Resilient Data Pipeline

Page header background

For a source that changes without notice

View the source

Overview

.NET pipeline against a system it does not control, built to survive the redesign

You depend on a system you do not control and it changes without telling you. The question is what happens to your integration the morning after.

A .NET background worker drives a real browser, parses semi-structured pages defensively, normalises them into a typed model, writes idempotently to MongoDB and rolls the results up into aggregate figures.

Portfolio item image

Engineering

How it was built

Backend & services

  • .NET background worker driving a real browser session
  • Defensive parsing into a typed model
  • Newest-first selector chains with fallbacks to earlier markup generations
  • Sections parsed independently, so one failure does not cost the record
  • Idempotent MongoDB writes — an interrupted run resumes by being run again
  • Retry budget rather than unbounded retries against a failing source

Integrations

  • MongoDB aggregation pipelines for the rolled-up figures

Testing & verification

  • 41 tests

Applied

Skills on this project

.NETC#MongoDBETLData pipelinesResilience engineering

Next step

Ready to bring your project to life?

Schedule a free 30-minute strategy call with our team.

Book Your Free Consultation or email contact@disit.tech

No sales call. You will talk to the engineer who would do the work, and you will get a written summary either way.