Data platform in operation

Peak memory and run time of the same run before and after reworking the processing.

Starting point

The platform from the first case study lives on data that arrives fresh every working day. What does not run overnight is missing in the morning. The data comes from four sources: the ERP system via a programming interface, a data warehouse, an analytics platform and the merchandise management system. Together that is around 155 million rows and ten years of history.

A data platform that manages this daily is not a collection of scripts. It is an operation with dependencies, run times, limits and failures. I am responsible for this operation alone, next to the application that depends on it. That sets the bar: every manual step I do not have to take at night counts twice.

Decision

First: orchestration by assets instead of by schedules. Every data set is an asset with known dependencies. If one is missing, I see immediately what depends on it and what has to be recomputed. A schedule does not know that.

Second: production readiness from incidents, not from guesswork. I did not build safeguards on suspicion. After every real failure I closed exactly the gap it had exposed. That sounds like negligence and is the opposite: every rule in the platform has an incident as its evidence.

Third: alerts with the whole error. A message saying “job failed” helps nobody. The alert carries the full traceback, so that I know at first glance where to start.

Implementation

The platform comprises 57 assets in 16 jobs. On a working day around 164 orchestrated steps run. They pull the raw data from the four sources, check it, prepare it and write the columnar files the application reads from.

The hardening consists of four building blocks. Concurrency limits, so that parallel pulls do not overload a source. Timeouts, so that a hanging pull does not block the whole night. Retry policies, so that a brief outage does not become a manual intervention. And alerts with traceback. Each of these blocks came out of a real incident.

The third site arrived with 6.85 million rows and 26 years of history. When connecting it, peak memory initially stood at 39.3 gigabytes and run time at 22.3 seconds. I reworked the processing. Afterwards: 9.7 gigabytes and 7.1 seconds.

Result and measurement

In operation: 57 assets, 16 jobs, around 164 steps per working day, around 155 million rows from four source systems, ten years of history. The third site adds 6.85 million rows with 26 years of history.

One number is deliberately missing here: the share of runs that complete without intervention. I will state it once I have measured it cleanly. Without that number, “hardened” remains a claim with evidence, not a measurement.

What I would do differently

Limits and timeouts from the start as a default that gets loosened, rather than a retrofit after the incident. That contradicts my second decision only in part. I would bring the basic safeguards forward. I would still derive the fine-tuning from real incidents, because only those show where it actually pinches.

A load test with real volumes before the third site. I would have seen the memory peak beforehand, not during the run.

A naming and layering convention for assets, fixed early. With 57 assets it pays off. Whoever introduces it later renames, and every rename is a risk in operation.

Counting the share of incident-free runs from day one. It is the one number by which an operation can be judged, and I do not have it today.

Technology

• Dagster
• Python
• DuckDB
• Parquet
• SAP R/3 (RFC)
• SQL Server
• Databricks
• Merchandise management system