ENGINEERING LAB
01

Problem

M-Pesa transaction volumes for a small merchant were reconciled by hand at end of day, which meant discrepancies — failed callbacks, duplicate webhooks, timing mismatches — were only caught after the fact, sometimes days later.

02

Requirements

Reconcile transactions against the ledger continuously rather than in a daily batch. Flag anomalies for review within minutes, not days. Never lose a transaction record even if a downstream service is briefly unavailable.

03

Architecture

Webhook ingestion into an event processor, fanned out to a Kafka-backed fraud path and a direct-write validation path into PostgreSQL. See the full interactive diagram on the Architecture page.

04

Data model

Two core tables: transactions (system of record, keyed on transaction ID) and ledger_entries (external ledger snapshot). Reconciliation is a set difference between the two, not a mutation of either.

05

Implementation

Built incrementally: webhook ingestion and storage first, then the reconciliation job, then the Kafka fraud-scoring path once the core pipeline was stable. See selected snippets on the Code page.

06

Engineering decisions

Chose PostgreSQL over a NoSQL store for the ledger because reconciliation is fundamentally a relational join problem. Chose Kafka specifically for the fraud path (not the whole pipeline) because only that path needed replay and multiple independent consumers.

07

Bottlenecks

The reconciliation query slowed as the ledger grew, since it scanned all unsettled rows on every run. Resolved with a partial index — see the indexing snippet on the Code page — rather than denormalizing the schema.

08

Failure handling

Idempotency keys on transaction IDs make webhook retries safe. Validation failures route to a dead-letter topic instead of blocking the stream. The fraud scorer falls back to rule-based scoring if the statistical model is unavailable.

09

Security

Webhook signatures are verified with HMAC before any processing. Database roles are scoped per service with least privilege. No raw M-Pesa identifiers are logged outside the primary transaction table.

10

Performance

Rows written1,000,000
Tenants100
Write throughput~66K rows/sec
Concurrency8
ResultPASS

Load test conditions: 100 simulated tenants writing concurrently at concurrency level 8, sustained until 1,000,000 rows were written. Figures are from a load test run, not production traffic — see the Experiments page for method.

11

Results

Reconciliation moved from a manual, end-of-day process to a continuous automated check, with the pilot customer now seeing discrepancies flagged within the same processing cycle rather than the next business day.

12

Lessons learned

Introducing Kafka only where replay genuinely mattered kept the system easier to operate than putting every path through a message bus by default. A partial index solved the bottleneck more cheaply than a schema rewrite would have.