[ALERT]14:14 · Support: 14 tickets in 12 minutes: 'I was credited twice for one transfer.' No errors on the dashboard.
14:02:11.004 inst-2 INFO webhook received ref=TRX_8841 acct=1042 amt=5000014:02:11.008 inst-3 INFO webhook received ref=TRX_8841 acct=1042 amt=5000014:02:11.012 inst-2 DEBUG dedupe check ref=TRX_8841 existing=014:02:11.013 inst-3 DEBUG dedupe check ref=TRX_8841 existing=014:02:11.019 inst-2 INFO ledger insert ref=TRX_884114:02:11.021 inst-3 INFO ledger insert ref=TRX_884114:02:11.030 inst-2 INFO balance update acct=1042 +5000014:02:11.031 inst-3 INFO balance update acct=1042 +50000--- metrics (last 12 min) ---duplicate_ledger_references: 37http_5xx: 0p95_latency_ms: 41support_tickets_tagged_double_credit: 14
Service Overview
- Service:
transfers-api - Stack: Node 20, Express, PostgreSQL 15
- Recent Change: PR #412 scaled service instances from 1 to 3.
- Deduplication Change: The deployment replaced an in-memory deduplication Set with a database check.
System Invariants & Behavior
- The payment provider can retry the same webhook.
- Each transfer reference must credit the associated account exactly once.
- Database Schema:
- -
ledger(id, reference, account_id, amount, created_at)— has no uniqueness constraint onreference. - -
accounts(id, balance)— stores current account balances. - The provider retries deliveries that are not successfully acknowledged with HTTP 200.
Investigation Progress
Step 1 of 5(Mitigate)
Immediate Containment
It's 14:14 and customers are being credited twice. What do you do in the next two minutes to stop the damage, before writing any code?
Be specific: what action, on what, and why it is safe.
Min 20 characters0 / 20 chars