CDC Event Streaming Across Disjoint Environments

Context

Role: Architect and full-stack development lead on a greenfield system. Environment: Disjoint deployment environments spanning on-premise and cloud, with no shared infrastructure between them.

Challenge

• Lead full-stack development of a greenfield system built around event-driven data integration.
• Propagate data changes across disjoint environments with no direct connectivity between data stores.
• Integrate complex graph relationships with change data capture (CDC) pipelines and fine-grained access controls.
• Evolve the system from a functional POC to a production-ready distributed system.

Architecture

• Event-driven services utilizing Golang, React.js, Neo4j, Postgres, and Kafka.
• Kafka-based CDC pipelines propagating changes between the relational system of record and the graph store.
• Event streams as the integration boundary between environments, decoupling producers and consumers across deployment targets.

Operational Model

• Work directly with the customer to define functional requirements.
• Lead full-stack development.

Outcomes

• Delivered a production-ready CDC streaming solution from POC through deployment across disjoint on-premise and cloud environments.