Production
Kafka → AWS SQS/SNS Migration
Moving live payment and credit events to a new bus, with zero downtime and a rollback that is only a config change.
- Role
- SDE 1 – Backend (Java)
- Company
- Standard Chartered Research and Technology (SOLV)
- Period
- Jan 2024 – Present
- Stack
- Kafka · AWS SQS · AWS SNS · AWS Lambda · Ansible · Java
~800K
messages / month on the migrated reconciliation pipeline
99.94%
consumer success
10–20% → 100%
canary ramp
0
downtime
Production metrics · 30-day window · Sep 2026
Context
Our services published domain events (payment reconciliation, transaction conveyance, credit and charge calculations, order status) on an in-house Kafka-based event bus. The organization standardized on AWS SQS/SNS: SQS for point-to-point work queues, SNS for fan-out.
The problem
These are money-moving events. A lost or reordered message can mean an unreconciled payment or a wrong interest calculation. The migration had to happen live, stream by stream, with instant rollback and without changing consumer behavior.
Design
- Per-event traffic split. Each event has a config key for the percentage of traffic sent to SQS. The producer rolls a random number per publish: below the threshold goes to SQS, otherwise to Kafka.
- SQS-first with automatic fallback. If an SQS publish throws, the same event is published to Kafka in the catch block. No event is dropped because the new path failed.
- Canary ramp. Start at 10–20%, watch consumer success, queue depth and fallback counts, then ramp to 100%. Remove the split and fallback in a separate cleanup PR.
- Config-only rollback. The percentages live in Ansible-managed config, so rolling back is a config change and restart, not a code deploy.
- Ordering where it matters. Events that must be processed in order use SQS FIFO queues with message-group ids, per credit line or per order, with deduplication ids. Other streams stay on standard queues for throughput.
- Consumer compatibility. Consumers receive messages through a Lambda-to-HTTP bridge that calls their existing endpoints. The legacy idempotency, group and transaction-id headers are carried over, so consumers run unchanged.
- Naming convention:
<Env>_<Producer>_<Consumer>_<Event>_<UseCase>[.fifo].
Key decisions & trade-offs
Results
- The reconciliation pipeline (~800K messages/month) runs on SQS at 99.94% consumer success.
- Payment and credit streams migrated with zero downtime and a config-only rollback path.
Tech used
- Apache Kafka
- AWS SQS (standard + FIFO)
- AWS SNS
- AWS Lambda
- Ansible
- Java
- Datadog