01 / Faster payroll data
30 minutes to under 8
Processing the same data repeatedly made payroll ingestion slow. I changed the pipeline to process only what had changed.
CDC runtime fell from 30 minutes to under 8, with compute cost down approximately 67%.
How it works
Incremental merge-upserts replaced full-table reloads in SQL Server. Rerunnable ingestion and validation help keep the data correct.
