Integrations are sold as connections and fail as assumptions. The connection is the easy part — a day of work, an API key, a webhook URL. What takes the rest of the project is everything that follows from a fact nobody puts in the proposal: the other system is not yours, and it will do things you did not plan for, at hours when nobody is awake.
It will send you the same payment notification three times. It will send one and then nothing. It will accept your request, time out before answering, and complete it anyway. It will change a field from a string to a number on a Tuesday. None of this is exotic; all of it is Tuesday.
The good news is that the defences are few, well understood, and cheap if you build them at the start. They are extremely expensive to retrofit onto a system that has been quietly double-counting for nine months.
The five failures, and what each one costs
1. The same message, twice
Every serious webhook provider guarantees at-least-once delivery, not exactly-once. That is not a defect, it is the only honest guarantee a network can make. It means duplicates are normal traffic.
If a duplicate payment notification credits a wallet twice, you have manufactured money. If a duplicate dispatch event posts an owner advance twice, you have paid somebody twice and will find out at settlement, from them, in a worse mood than yours.
The defence is idempotency, and it belongs in the database. Every inbound event is written to a raw-event table keyed on the sender’s own reference, with a unique index on that key, before any business logic runs. A duplicate hits the index and stops. Not an if in application code — a constraint, because two copies of the message can be in flight at the same instant, and an if will cheerfully let both through.
Get this right and a useful property falls out for free: you can replay a week of events and nothing changes. We rely on that in the wallet system we built for an EV rental operator — the Cashfree intake writes the raw event first and routes second, so a redelivery storm is a no-op rather than an incident.
2. The message that never arrives
Webhooks get lost. The provider had an outage, your endpoint returned a 500 during a deploy, a firewall rule changed. If your system’s only knowledge of the world arrives by webhook, a lost message means your record and reality have diverged permanently and silently.
The defence is reconciliation, and it is not optional. A scheduled job pulls the counterparty’s own list of transactions for the day and diffs it against yours. Anything on one side and not the other becomes a task somebody sees the next morning.
People skip this because it feels redundant when webhooks are working. It is not redundancy — it is the only mechanism that detects the failure of the primary mechanism. A system with webhooks and no reconciliation does not know when it is wrong.
3. The request that timed out but worked
You call the payout API. It takes thirty-one seconds. Your client gives up at thirty. Did the payout happen?
You cannot know from the timeout, and retrying blindly risks paying twice. The defence is an idempotency key on the way out — a key you generate and send with the request, so that a retry with the same key returns the original result instead of performing the action again. Every payment provider worth using supports this. Use it on the first day, not after the first double payout.
And do not mark money as sent when you sent the request. Mark it pending, and let the counterparty’s confirmation move it to sent. Money leaving the business should never be assumed to have left.
4. The schema that changed
A field that was "1200.00" becomes 1200. A status gains a new value you have never seen. An optional field stops being sent.
The defence is to store the raw payload forever, and to fail loudly on the unknown. Storing raw means that when you discover the change three weeks late, you can reprocess the affected window instead of asking the counterparty to resend history they may not have. Failing loudly means an unrecognised status routes to a human rather than falling through a default branch that silently treats it as success.
The worst outcome in integration work is not an error. It is an unrecognised value handled as if it were understood.
5. The failure nobody saw
A send fails, retries four times, exhausts its attempts, and writes a line to a log. The log is read by nobody. The customer was never told, and the first anyone hears is a complaint.
The defence is that failure produces a row, not a log line. Exhausted retries go to a dead-letter handler that files a task with the full payload attached, assigned to a queue a human actually works. The test of an integration is not whether it succeeds. It is whether you find out within a day when it does not.
- If the provider sends the same message twice tonight, what happens? Can you point at the constraint that stops it — not the code, the constraint?
- If a message is never delivered, how long until anyone knows?
- If an outbound call times out, do you retry, and is that retry safe?
- Can you reprocess last Tuesday from stored payloads without asking the counterparty for anything?
- When something fails permanently, whose queue does it land in?
Where the logic should live
One structural decision governs whether the above stays true as the system grows: the integration layer must not be where business rules live.
It is tempting, once you have a workflow tool or an integration service, to let it accumulate decisions — compute the balance here, apply the discount there. That works until two paths need the same rule, and now there are two implementations that will drift.
The durable arrangement is narrow: the integration layer moves messages, checks signatures, enforces idempotency, retries, and calls a function in the database that does the actual work inside a transaction. On the Go2Green wallet, no workflow computes a balance — each one calls a Postgres function that debits inside a transaction and returns the result. The workflows decide when something happens and who gets told. The database decides what is true.
That boundary is why the integration layer there can be a visual workflow tool at all without anybody losing sleep. The parts that must be exactly right are in one place, under test, behind a constraint — and the parts that change often are in the place that is easy to change.
Integrations do not fail at 2am because the network is unreliable, though it is. They fail at 2am because a system that assumed the happy path met the other 3%. The defences are all cheap. They are just only cheap before you need them.