How we handle GDPR data-subject requests on a deadline
Legal
The pipeline-health agent we run on Kortix — connected to the warehouse and GitHub. Every hour it checks table freshness, row-count anomalies, and schema drift, alerts Slack, and drafts a GitHub issue with the likely cause. It never touches the data.
A pipeline can fail quietly. A load job silently stops appending rows, a source schema changes upstream and half the fields come through null, a table that should refresh every hour hasn't moved since yesterday. None of that throws an error anyone sees — the dashboards built on top of it just start being wrong, and usually the first person to notice is whoever's report looks off in a stakeholder meeting.
We run a pipeline-monitor agent on Kortix that checks the warehouse itself, every hour: is every table as fresh as its SLA requires, does today's row count look like the rest of the trend, has the schema changed under anyone, did a load job fail. It only reads the warehouse. When something is wrong, it posts to the data team's Slack channel and drafts a GitHub issue with its best guess at the cause. This is how we watch our own warehouse.
Warehouse health doesn't announce itself. A load job can fail without an exception if it's built to skip a bad batch and move on. A table can go stale because an upstream cron got disabled, not because anything crashed. A schema can drift because a source API added a field, and the load still "succeeds" — it just drops or nulls what it doesn't recognize. Every one of these looks fine from the outside until a downstream query returns something wrong.
The common approaches don't hold up. Dashboards built on the data can't tell you the data itself is broken — they just render whatever's there, wrong or not. A "did the job run" check confirms the process exited zero, not that the table it wrote actually looks right. Someone eyeballing row counts in a spreadsheet catches it eventually, usually after a downstream report has already gone out wrong. And this is a different failure mode from an application throwing an error mid-request — the pipeline can succeed and the warehouse can still be unhealthy.
On Kortix, an hourly cron triggers an agent. It spawns a fresh session with read-only access to the warehouse and checks every monitored table against its freshness SLA, compares today's row count to the table's own trailing baseline, diffs the current schema against the last-seen shape, and looks for loads that failed or silently stalled. When it finds something out of bounds, it posts an alert to the data team's Slack channel with the affected table, the anomaly, and the most likely cause, and drafts a GitHub issue with the same diagnosis so the incident is tracked. It never writes to the warehouse — the alert and the draft issue are the only outputs.
Open your Kortix workspace, start a new session, paste this prompt and send it. The agent writes the agent file, the health check skill and the hourly trigger, and opens a change request for you to review. Prefer clicks? Choose Use this template instead.
Set up an hourly data pipeline monitor agent in this workspace.
Create the agent agents/data-pipeline-monitor.md. Its job is to watch the warehouse for stale tables, row-count anomalies, schema drift and failed loads. It may use the PostgreSQL and Slack connectors and a GH_TOKEN secret for GitHub.
Create the skill skills/pipeline-health-check/SKILL.md with these rules:
Add a cron trigger pipeline-health-sweep to kortix.yaml. It runs the data-pipeline-monitor agent every hour, on the hour, with the prompt "Load the pipeline-health-check skill and run one complete health sweep over the public schema."
Tell me which of these connectors are not connected yet.
Open a change request with all of it. Do not merge it.
Open Connectors in your workspace and connect your warehouse and Slack.
Credentials are brokered server-side and never enter the agent's machine. Add a GitHub token as the GH_TOKEN secret so the agent can draft
issues.
Checks freshness, row counts and schema drift, read-only.
Posts each anomaly with its evidence and likely cause.
Drafts an issue with the same diagnosis.
Merge the change request, then start a session with the data-pipeline-monitor agent and send this prompt. Check its alerts against what you know about your tables before you let the schedule take over.
Run one health sweep over the public schema.
The change request added this trigger to kortix.yaml. From now on the agent
runs every hour, on the hour, each run in a new session on its own cloud
computer.
To change the time, edit the cron line, or open Triggers in the workspace.
triggers:- slug: pipeline-health-sweep name: Hourly pipeline health sweep on: cron: "0 0 * * * *" run: agent: data-pipeline-monitor prompt: Load the pipeline-health-check skill and run one complete health sweep over the public schema.Every query the agent runs is a read. It never changes a row, a table or a pipeline, and it never restarts a load. Its only outputs are the Slack alert and the drafted GitHub issue. A person decides the fix.
A sweep that finds an anomaly posts one alert per table, and a clean sweep posts nothing. This is what a run looks like in the session:
Checked 32 tables. orders_daily is 31 hours old against a 24-hour SLA, and events has a new nullable column. Posted 2 alerts and drafted 2 issues. The other 30 tables are healthy.
Once it works, paste any of these into the same session.
Add to memory that the tables our revenue dashboard reads have a 2-hour freshness SLA, and check them first in every sweep.
Every weekday at 08:00, post one line per anomaly from the last 24 hours to #data-alerts, with whether its issue is still open.
For each anomaly, list the views and tables that read from the affected table, so the alert shows what else is wrong.
The agent reads across the entire warehouse, so its access is scoped and strictly one-directional:
Warehouse problems that used to surface as a wrong number in someone's dashboard now show up as an hourly health check with a specific table, a specific anomaly, and a likely cause — already in Slack and already drafted as an issue. The agent only reads and reports; the data team decides what to fix.