Production
Reconciliation & Failover
Webhooks get lost and gateways go down. This is how payments still end up correct.
- Role
- SDE 1 – Backend (Java)
- Company
- Standard Chartered Research and Technology (SOLV)
- Period
- Jan 2024 – Present
- Stack
- Java · AWS SQS · MySQL · Datadog
~800K
reconciliation checks / month
99.94%
reconciliation success
±1 paisa
amount tolerance
15 min
late-callback P0 threshold
Production metrics · 30-day window · Sep 2026
Context
Even with good webhooks, some payments end up “stuck”: initiated or processing on our side, but resolved at the gateway. Refunds can sit in “refund initiated” too. And gateways have outages, sometimes exactly when delivery agents are collecting payments at a retailer’s door.
The problem
A stuck payment means either a customer who paid and sees “failed”, or an order waiting on money that already arrived. A gateway outage during delivery means agents cannot collect at all.
Design
- Reconciliation sweep. A scheduled job selects transactions in INITIATED, PROCESSING or REFUND_INITIATED status within a time window. It fans them out as individual messages on an SQS queue, and a consumer re-checks each with the gateway and completes, fails or updates it.
- Amount checks. The amount reported by the gateway must match ours to ±1 paisa (to absorb rounding), or the transaction is flagged.
- P0 alerting. Late authorizations (>15 minutes) and orders showing both a success and a refund page the on-call engineer.
- Circuit breaker with failover. A breaker wraps the primary gateway for delivery-time collections. When failures cross a threshold it opens and new collection requests fail over to Cashfree. After a cool-down it half-opens to probe recovery, then closes again.
Key decisions & trade-offs
Results
- ~800K reconciliation checks/month at 99.94% success.
- Delivery-time collections stay live through primary-gateway outages.
Tech used
- Java
- AWS SQS
- MySQL
- Razorpay
- Cashfree
- Datadog