A pipeline that breaks will tell you it broke. A starved one won't.
It reads an empty directory, writes nothing, marks itself complete, and the run history stays a clean column of green. Whatever depends on it keeps serving the last numbers it had. If nobody downstream happens to know what the number should be doing, this can go on for a while.
I bring it up because most pipeline monitoring answers one question, which is whether the job ran. That has a satisfying yes or no, and it's usually not the question that matters. What you want to know is what arrived, not what executed. Did anything land. Was it roughly the volume it normally is. Is the newest row newer than it was yesterday.
None of those look like failures to a scheduler.
Three checks
If a team has nothing at all, I'd put these in before anything else, and I wouldn't build a framework first.
The first is freshness. Every table anything depends on gets a maximum age, written down in units the business cares about. Six hours for some things, a day for others, longer at month end when a source is legitimately slow. The number matters less than having one somebody can argue with.
Then volume, meaning yesterday's row count against a trailing average, inside a band. Not an exact match, because real data moves and a tight band teaches everyone to ignore the alert within a fortnight. What you're hunting is the source that sent four rows when it normally sends four hundred thousand.
Schema last. Something that notices when a column arrives, disappears or changes type, before whatever reads it notices. People push back on this one because it fires when an upstream team adds a field they're proud of. Fire it anyway.
That's most of it. There's a great deal more you can do, and most teams aren't doing these three.
Half of this is already in system tables
If you're on Databricks, some of the work is done and nobody has gone looking for it.
Lineage settles the argument about which tables deserve watching, since it already knows what reads what. system.access.table_lineage carries source_table_full_name and target_table_full_name for every read and write, so ranking your tables by how many distinct things consume them comes down to a GROUP BY:
SELECT source_table_full_name, count(DISTINCT target_table_full_name) AS consumers
FROM system.access.table_lineage
WHERE source_table_full_name IS NOT NULL
AND event_time > current_date() - INTERVAL 30 DAYS
GROUP BY 1
ORDER BY consumers DESC
That list is your monitoring backlog, in priority order, and it took a minute. Audit logs then tell you when each of those was last written and by what. Join the two and your alert stops saying a table is stale and starts telling you that eleven things read it, two of them dashboards leadership opens on Monday morning.
Since the May 2025 schema change the lineage tables also carry statement_id, which joins to system.query.history. That's the difference between knowing a table changed and knowing which query changed it, under whose identity, and how long it ran.
Lakeflow expectations cover the row-level version, with constraints declared next to the transformation, dropping bad records or failing the run outright. Five expectations a team understands will beat a quality framework nobody maintains after the person who chose it moves on.
A couple of things I had backwards
For years I thought of this as a testing problem. Write assertions, run them after the load, alert on failures. Which is fine, and it completely misses the case above, because zero rows satisfy every constraint you can write about the shape of a row. The check that catches a starved pipeline is a check about nothing being there, and it took me an embarrassingly long time to see that nothing was a state worth alerting on.
I also used to route all of these the same way. A freshness alert that pages someone at two in the morning about a daily file that won't arrive until nine is an alert people learn to silence, and once they've learned that, they silence the ones that matter too. Most freshness failures are a working-hours problem.
The last one I only started doing recently. Keep the passes, not just the failures. A check that's been quietly succeeding for eight months is telling you something every single day, and when somebody eventually asks how long the number has been wrong, you want a table you can query instead of a guess based on when the alerts started.