All work
Production

Kafka → AWS SQS/SNS Migration

Moving live payment and credit events to a new bus, with zero downtime and a rollback that is only a config change.

Role
SDE 1 – Backend (Java)
Company
Standard Chartered Research and Technology (SOLV)
Period
Jan 2024 – Present
Stack
Kafka · AWS SQS · AWS SNS · AWS Lambda · Ansible · Java
~800K
messages / month on the migrated reconciliation pipeline
99.94%
consumer success
10–20% → 100%
canary ramp
0
downtime

Production metrics · 30-day window · Sep 2026

Context

Our services published domain events (payment reconciliation, transaction conveyance, credit and charge calculations, order status) on an in-house Kafka-based event bus. The organization standardized on AWS SQS/SNS: SQS for point-to-point work queues, SNS for fan-out.

The problem

These are money-moving events. A lost or reordered message can mean an unreconciled payment or a wrong interest calculation. The migration had to happen live, stream by stream, with instant rollback and without changing consumer behavior.

Design

  • Per-event traffic split. Each event has a config key for the percentage of traffic sent to SQS. The producer rolls a random number per publish: below the threshold goes to SQS, otherwise to Kafka.
  • SQS-first with automatic fallback. If an SQS publish throws, the same event is published to Kafka in the catch block. No event is dropped because the new path failed.
  • Canary ramp. Start at 10–20%, watch consumer success, queue depth and fallback counts, then ramp to 100%. Remove the split and fallback in a separate cleanup PR.
  • Config-only rollback. The percentages live in Ansible-managed config, so rolling back is a config change and restart, not a code deploy.
  • Ordering where it matters. Events that must be processed in order use SQS FIFO queues with message-group ids, per credit line or per order, with deduplication ids. Other streams stay on standard queues for throughput.
  • Consumer compatibility. Consumers receive messages through a Lambda-to-HTTP bridge that calls their existing endpoints. The legacy idempotency, group and transaction-id headers are carried over, so consumers run unchanged.
  • Naming convention: <Env>_<Producer>_<Consumer>_<Event>_<UseCase>[.fifo].
Kafka → SQS canary
StageCanary
Sent to SQS
0
Sent to Kafka
0
Fallbacks (SQS → Kafka)
0
Dropped
0

0 events published · 100 events/s (sim) · every event lands in SQS or Kafka, so dropped stays at 0.

Drag the split. Failed SQS publishes fall back to Kafka, so nothing is dropped.
Kafka to SQS migration Before: a producer publishes to a Kafka topic read by consumers. After: the producer splits traffic by percentage to SQS / SNS, a Lambda bridge delivers it to consumer HTTP endpoints, and a dashed fallback path still lets the producer publish to Kafka. Before Producer Kafka topic Consumers After Producer SQS / SNS Lambda bridge Consumer HTTP endpoints split % Kafka topic fallback
Before and after: Kafka topic vs SNS/SQS fan-out

Key decisions & trade-offs

Results

  • The reconciliation pipeline (~800K messages/month) runs on SQS at 99.94% consumer success.
  • Payment and credit streams migrated with zero downtime and a config-only rollback path.

Tech used

  • Apache Kafka
  • AWS SQS (standard + FIFO)
  • AWS SNS
  • AWS Lambda
  • Ansible
  • Java
  • Datadog

Back to all work