Case studiesCase study Dongoing
Replacing an integration platform
Ongoing. Inventory captured automatically: 99 scheduled processes, 1,278 jobs, around 8,200 processing steps with dependency graph and risk classes. First process migrated and verified against the original: 501,546 rows identical.
- Role
- Responsible for inventory, target design and migration
- Period
- ongoing
- Setting
- Mid-sized meat industry group, three sites
- scheduled processes captured automatically
- 99
How measured
Read out of the integration platform by machine, not by interview.
- jobs in the inventory
- 1,278
How measured
Read out of the integration platform by machine, with dependency graph and risk class per process.
- processing steps, rounded
- 8,200
How measured
Read out of the integration platform by machine and rounded.
- rows in the first migrated process, identical to the original
- 501,546
How measured
Row-by-row comparison of the migrated process’s result with the result of the old platform: zero deviations.
Short version: the first paragraph of each section, plus key figures and chart.
Starting point
The group runs an integration platform that moves data between systems and executes processing on a schedule. It has grown over years. A complete overview of what runs in it and what it depends on did not exist.
My task is the replacement. The target is the same orchestration that runs the data platform from the third case study, on a current database platform. This case study is written in the present tense because the project is ongoing. I describe what exists and claim nothing that does not exist yet.
Decision
First: capture the inventory automatically, not by interview. I read the platform out by machine. That produces an inventory that is complete and repeatable, rather than a list from the memory of those involved.
Second: risk classes before sequence. Every process gets a risk class from its dependencies and its impact. The migration order follows the risk class, not whoever shouts loudest.
Third: every migrated process runs against the original. A process counts as replaced only once its result matches the result of the old platform row for row. Nothing is switched off without that proof.
Implementation
The inventory is captured: 99 scheduled processes, 1,278 jobs, around 8,200 processing steps. Plus the dependency graph, which shows which process requires which, and the risk classes per process.
The first process is migrated. I verified its result against the original: 501,546 rows, identical, zero deviations.
The remaining processes follow by risk class. Each passes the same check against the original before it replaces the old platform.
Result and measurement
| Status | Value |
|---|---|
| Scheduled processes captured | 99 |
| Jobs captured | 1,278 |
| Processing steps captured | around 8,200 |
| Processes migrated and verified | first process |
| Rows compared with the original | 501,546, zero deviations |
Nothing more exists yet. One migrated process out of 99 is a beginning, not a result. What I can prove is the method: the inventory is complete, the order is justified, and the verification against the original works.
What I would do differently
Build the comparison against the original as a fixed component that every process passes through automatically, before the first process is migrated. For the first process I set it up specifically. For the next ones it should be part of the path, not part of the work.
Not only capture the inventory but clarify with the business side from the start, process by process, whether it is still needed. A process nobody needs any more is not migrated. It is switched off. That is the cheapest migration.
Check the risk classes against the business side earlier. My classes come from dependencies and impact in the system. The business side knows which process someone misses at six in the morning. Both together give the right order.
Technology
- Python
- Dagster
- SQL Server 2022