Oct 25, 2025
Observability for data pipelines: emitting OpenLineage from Dynatrace ingestion
How I designed a connector that ingests Dynatrace data, writes to ADLS, alerts on Teams, and publishes lineage events.
Most data teams talk about “visibility”, but can’t actually explain what ran, when, and why it failed.
In this post I walk through how I approached pipeline observability:
- snapshot vs incremental mode
- Teams alerting
- storing runtime state on disk / lake
- emitting OpenLineage events so downstream consumers know where the data came from.
This is based on real production work in an airline context.
Was this useful? Reach me on LinkedIn or Bluesky.