Replacing an integration platform

As of October 2026. 96 of 99 processes are replaced, built or deliberately dropped. Switch-off follows only after sign-off.

Starting point

The group runs an integration platform that moves data between systems and executes processing on a schedule. It has grown over years. A complete overview of what runs in it and what it depends on did not exist.

My job is the replacement. The target is the same orchestration that runs in the data platform from project C, on a current database platform. This report is in the present tense because the project is ongoing. I describe what exists and claim nothing that does not exist yet.

Decision

First: capture the inventory automatically, not by interview. I read the platform out by machine. That produces an inventory that is complete and repeatable, instead of a list from the memory of those involved.

Second: risk classes before order. Every process gets a risk class from its dependencies and its impact. The migration order follows the risk class, not whoever asks loudest.

Third: every migrated process runs against the original. A process counts as replaced only once its result matches the result of the old platform. Without that proof, nothing is switched off.

Fourth: do not rebuild every process. What nobody needs any more is dropped. What only holds intermediate states for other processes goes away if the rebuild manages without it.

Implementation

The inventory is captured: 99 scheduled processes, 1,278 jobs, around 8,200 processing steps. Plus the dependency graph showing which process requires which, and the risk classes per process.

The first process was the test of the method. I checked its result against the original: 501,546 rows, identical, zero deviations.

Since then I have worked through all 99 processes. The new business layer bundles what used to be scattered. 23 of 25 target tables are done. 13 control and staging tables of the old platform go away without replacement. Tables that up to 8 processes used to write into now have a single building block. The processes that write back into other systems are rebuilt too. They are deliberately switched off until the cut-over is signed off. An error when writing back ends up in bookings, not in a report.

The rebuild revealed gaps the old platform had never closed. Delivery orders without a date were missing entirely. And one old process has been re-booking all goods receipts of one species on every run since mid-August, because a bracket is missing in its selection condition. I did not find that through a report but while rebuilding. I handed the case with evidence to IT management. My rebuild only sends what is still missing in the target system.

To make progress and errors visible, there is a control centre in the platform from project A. It shows jobs, runs, process chains and the data flow from source through job to target. Authorised users can start or rerun a run from there, and every intervention is logged. Management sees the state of the replacement in the same system as the business figures. The control centre and a service desk for IT shipped in 12 production releases over two days.

Result and measurement

96 of 99 processes are replaced, built or deliberately dropped. That does not yet mean the old platform is off. It is switched off only once the built processes are signed off and have run in parallel without deviation. That is the part coming now, and it does not depend on me alone.

What I would do differently

Build the comparison against the original as a fixed building block that every process passes automatically, before migrating the first process. For the first process I set it up specially. For the rest it became part of the path, but later than necessary.

Request access and sign-offs earlier. 51 processes are built and waiting. The technology was ready faster than the organisation behind it. I could have known that at the start and set it in motion in parallel.

Check the risk classes against the business side earlier. My classes come from dependencies and impact in the system. The business side knows which process someone misses at six in the morning. Both together give the right order.

Technology

• Python
• Dagster
• SQL Server 2022