All work
Production

Reconciliation & Failover

Webhooks get lost and gateways go down. This is how payments still end up correct.

Role
SDE 1 – Backend (Java)
Company
Standard Chartered Research and Technology (SOLV)
Period
Jan 2024 – Present
Stack
Java · AWS SQS · MySQL · Datadog
~800K
reconciliation checks / month
99.94%
reconciliation success
±1 paisa
amount tolerance
15 min
late-callback P0 threshold

Production metrics · 30-day window · Sep 2026

Context

Even with good webhooks, some payments end up “stuck”: initiated or processing on our side, but resolved at the gateway. Refunds can sit in “refund initiated” too. And gateways have outages, sometimes exactly when delivery agents are collecting payments at a retailer’s door.

The problem

A stuck payment means either a customer who paid and sees “failed”, or an order waiting on money that already arrived. A gateway outage during delivery means agents cannot collect at all.

Design

  • Reconciliation sweep. A scheduled job selects transactions in INITIATED, PROCESSING or REFUND_INITIATED status within a time window. It fans them out as individual messages on an SQS queue, and a consumer re-checks each with the gateway and completes, fails or updates it.
  • Amount checks. The amount reported by the gateway must match ours to ±1 paisa (to absorb rounding), or the transaction is flagged.
  • P0 alerting. Late authorizations (>15 minutes) and orders showing both a success and a refund page the on-call engineer.
  • Circuit breaker with failover. A breaker wraps the primary gateway for delivery-time collections. When failures cross a threshold it opens and new collection requests fail over to Cashfree. After a cool-down it half-opens to probe recovery, then closes again.
Circuit breaker · failover
BreakerCLOSEDt=0.0s
Total requests
0
Primary successes
0
Primary failures
0
Failovers to backup
0

Breaker CLOSED. Press "Send requests" to start.

  • Opens at ≥50% failures over the last 10 calls
  • Stays open 5s (sim time)
  • Half-open allows 3 trial calls

Illustrative parameters; production thresholds differ.

Closed → open → half-open → closed. While open, delivery collections route to the backup gateway.
Reconciliation pipeline A scheduler selects transactions in INITIATED, PROCESSING or REFUND_INITIATED within a time window and enqueues many small messages on an SQS queue; a consumer checks each against the gateway status API and either updates the transaction or raises a P0 alert. Scheduler Select in window INITIATED / PROCESSING / REFUND_INITIATED SQS queue many small messages Consumer Gateway status API Update transaction Alert (P0)
Reconciliation fan-out

Key decisions & trade-offs

Results

  • ~800K reconciliation checks/month at 99.94% success.
  • Delivery-time collections stay live through primary-gateway outages.

Tech used

  • Java
  • AWS SQS
  • MySQL
  • Razorpay
  • Cashfree
  • Datadog

Back to all work