A webhook is a message sent by one system to another as soon as something happens, such as an order being placed or a product being updated. Instead of you asking repeatedly whether anything has changed, the source system posts a small notification to an address you provide. It is fast and efficient, and it comes with conditions: you need a public endpoint that is always available, you must respond within a few seconds, and you have to assume messages can arrive twice, out of order, or not at all.

Polling is asking a system, on a schedule, what has changed since you last asked. A well built poll uses a change token or a modified since filter so the answer is usually a short list, and it stores a marker of the last successful position so it can restart cleanly after any interruption. It costs calls against your rate limit and introduces a delay equal to the interval, and in exchange it is predictable, restartable and easy to reason about when something goes wrong.

Which one should we use?

Per flow, not per project. Use events where a person or a customer is waiting for the result, such as stock availability, order capture or a payment confirmation. Use delta polling where a delay of minutes is harmless, which covers most master data, pricing and customer updates. Use scheduled batch for high volume, low urgency movements such as nightly financial summaries. Then put a reconciliation behind anything that would cause a real problem if a single change went missing.

Why do we get duplicate records?

Because delivery is usually promised at least once rather than exactly once. A network timeout on a request that actually succeeded, followed by a retry, sends the same message twice, and a slow acknowledgement causes the sender to retry as well. The fix is not to try to stop duplicates arriving, which is not achievable, but to make every write safe to repeat by using a stable reference from the source so a second attempt updates the existing record instead of creating a new one.

What happens if our endpoint is down for a while?

The vendor retries for a period, then gives up, and those events are lost. That is why a recovery path matters: a poll over the outage window, or a reconciliation that compares both systems and repairs the difference. It should be built and tested in advance, ideally as a routine operation rather than an emergency procedure. Integrations that rely on events alone with no backfill turn a short outage into a long and manual cleanup.

Is polling every minute a bad idea?

Not necessarily, but count the cost before deciding. One endpoint polled every minute is over a thousand requests a day, and ten endpoints is more than ten thousand, most returning nothing. That budget competes with the calls your real work needs, and the competition is fiercest during peak trading when both are highest. If the vendor supports change tokens the requests are cheap, if it does not they may be expensive, and either way the right interval comes from what the business actually needs rather than from a default.

Can we use both for the same data?

Yes, and for anything important you probably should. The common pattern is events for immediate updates, a low frequency catch up poll to collect anything that never arrived, and a scheduled reconciliation that compares counts or key values and reports the difference. The events give you speed, the poll gives you recovery, the reconciliation gives you evidence. Each layer is cheap on its own, and together they are what separates an integration you can trust from one you merely hope is working.

A plain English guide for people choosing between the two

Webhooks vs API Polling: Choosing How Data Moves

Two systems can stay in step in two ways. The source can announce each change as it happens, which is a webhook, or the destination can ask periodically what has changed, which is polling. Vendors present webhooks as the modern answer and polling as the old one. In production, the difference is not modern versus old. It is which failure you would rather design for.

Webhooks are fast and forgetful: they can arrive twice, out of order, or not at all, and you rarely find out which. Polling is dependable and costly: it burns your rate limit asking questions where the answer is usually nothing. Most integrations that survive a few years use both deliberately. This page explains how to decide and what each approach demands of you.

Book an Integration Call (03) 9003 0111

Realistic ROI

Seconds vs minutes

The real latency difference

Which only matters where somebody is waiting for the result of the change

At least once

Is what webhook delivery usually promises

Meaning duplicates are normal and every write has to be safe to repeat

Most polls

Return nothing at all

Which is fine with change tokens and wasteful when you re-read the whole dataset

Both, usually

The design that holds up over years

Events for speed, a scheduled sweep for certainty, reconciliation to prove it worked

Four Questions That Settle the Choice

Answer these per data flow, not per project. A single integration usually needs different answers for different data.

Is anyone waiting for this data

Stock availability during a sale, an order reaching the warehouse, a payment unlocking a booking: these are worth the complexity of event delivery because a customer or a staff member is actively waiting. A nightly general ledger summary, a price list refresh or a reporting extract is not. Applying real time delivery to data nobody is waiting for adds cost and fragility, and it is the most common overreach in integration design.

What happens if one change is missed

If a missed update means a stale field until the next sweep, events alone may be fine. If it means an order never reaches the warehouse or an invoice never posts, then no delivery mechanism should be trusted on its own. High consequence flows need a scheduled sweep or a reconciliation behind the events, because the failure mode of a webhook is silence, and silence looks identical to nothing having happened.

What the vendor limits allow

Polling costs calls, and calls are capped. Polling ten endpoints every minute across a working day is thousands of requests, most returning nothing, and that budget competes with the calls your actual work needs during peak trading. Where a vendor supports change tokens or a modified since filter, polling becomes cheap. Where it does not, frequent polling is often simply unaffordable and events become the only practical option.

Whether you can receive a call from the internet

A webhook requires a publicly reachable endpoint that is always available, verified as genuine, and hardened, because anyone who finds the address can post to it. For a cloud hosted integration layer that is straightforward. For an on premises system behind a corporate firewall it is a security conversation, and outbound polling is frequently the pragmatic answer regardless of what the vendor recommends.

The Six Mechanisms, and What Each Demands

Most real integrations combine three or four of these. The skill is knowing what each one obliges you to build.

Push, seconds

Webhooks

The source posts a small message when something happens. You must respond quickly, usually within a few seconds, or the vendor treats it as a failure and retries, so the correct pattern is to verify, queue and acknowledge immediately, then process from the queue. Delivery is typically at least once with no ordering guarantee, so duplicates and out of sequence arrivals are normal operating conditions rather than bugs.

Trust the source

Thin events, then fetch

Treat the webhook as a notification that something changed rather than as the data itself, then call the interface to read the current state of that record. This removes an entire class of problems: stale payloads, out of order updates and partial information. It costs one extra call per event and is almost always worth it, particularly for records that change several times in quick succession.

Pull, cheap

Delta polling

Ask for everything changed since a stored marker, usually a timestamp or a change token supplied by the vendor. Store the marker only after the batch is successfully processed, and allow a small overlap window because clocks and indexes are not perfectly aligned. Done well this is efficient, restartable and easy to reason about, which is why it remains the workhorse of reliable integration.

Certainty

Full sweep reconciliation

Periodically compare complete sets rather than changes: every active product, every open order, every customer modified this month. It is expensive, so it runs nightly or weekly and often only on identifiers and key fields. This is the layer that catches what both webhooks and delta polling silently missed, and it is the reason a mature integration can be trusted rather than merely believed.

Legacy friendly

Scheduled batch and files

A file dropped on a secure server at an agreed time, or a scheduled export from a system that offers nothing else. Unfashionable, entirely reliable, and often the only option for older Australian business systems. It demands its own discipline: an empty file is suspicious rather than normal, a missing file must raise an alarm, and every file needs a checked record count.

Safe to repeat

Idempotency and ordering

Whichever mechanism you use, the same message will eventually arrive twice, and two updates for the same record will occasionally arrive in the wrong order. Every write needs a stable reference so a repeat updates rather than duplicates, and every record needs a version or modified timestamp so an older update cannot overwrite a newer one. This is not optional detail. It is the difference between an integration and a source of duplicates.

Choosing the Mechanism, Flow by Flow

Task	Traditional	Designed Properly	Notes
Stock level changes	Hourly full export	Events plus a nightly sweep	Speed matters during trading, and the sweep catches whatever the events missed overnight.
New online order	Polled every 15 minutes	Event, then fetch the full order	High consequence, so it needs a scheduled catch up poll behind it as well.
Nightly financial summary	Real time posting per order	Scheduled batch after cut off	Nobody is waiting. Batch is cheaper, easier to correct and simpler to reconcile.
Price list refresh	Event per price change	Delta poll on a schedule	Bulk changes generate storms of events. Polling absorbs them without drama.
Customer record updated	Full sync of all customers	Delta poll with a change marker	Modified since queries are cheap and restartable after any interruption.
Legacy system with no interface	Staff export to a spreadsheet	Scheduled file with a record count	File based is legitimate design when the alternative is manual re-keying.
Vendor webhook goes quiet	Nobody notices for days	Silence alert plus catch up poll	The absence of events is the hardest failure to see, so monitor for it explicitly.
Bulk import by a supplier	Thousands of events at once	Queued, paced, processed in order	Backpressure protects downstream systems from a burst they cannot absorb.

Where Each Approach Bites

Trusting webhook delivery as if it were guaranteed

Endpoints go down for deployment, networks fail, vendors have incidents, and retry policies expire. When events stop arriving, everything looks calm: no errors, no queue, just quiet. Always pair events with either a periodic catch up poll or a reconciliation that compares record counts, and alert when a flow that normally receives events has received none for longer than usual for that time of day.

Processing inside the webhook response

Vendors expect a fast acknowledgement, often within a few seconds, and will retry if they do not get one, which means slow processing generates duplicate deliveries and can eventually see your endpoint disabled. Verify the signature, put the message on a queue, respond immediately, and do the real work separately. This one pattern prevents most webhook incidents in production.

Unverified endpoints

A webhook address is reachable by anyone who learns it. Without signature verification, an attacker can post fabricated orders, price changes or cancellations directly into your systems. Verify the signature on every request using the vendor’s secret, reject anything unsigned or stale, restrict by source address where the vendor publishes ranges, and treat the payload as untrusted input regardless. Where payloads carry personal information, remember the Privacy Act 1988 and the Australian Privacy Principles apply to what you log as well as what you store.

Polling that re-reads everything, every time

Requesting the full dataset on a schedule works during testing and collapses as data grows, taking your rate limit with it. Use change tokens or modified since filters, keep a marker of the last successful position, and only fall back to a full read as a deliberate reconciliation on a long interval. If a vendor offers no way to ask for changes, that constraint should shape the design rather than be worked around with brute force.

Out of order updates overwriting good data

An update sent at 10:00 can arrive after one sent at 10:01, particularly after a retry. Without a version number or a modified timestamp comparison, the older value wins and the corruption is silent. Compare before writing, discard stale updates, and log when you do so, because a pattern of stale arrivals usually means something upstream is retrying more than you realised.

No way to backfill after an outage

When a receiver has been down for six hours, you need to recover the changes that were missed, and events that have already expired their retries are gone. The recovery path is a poll over the affected window or a targeted reconciliation, and it should be built and tested before you need it. Discovering during an incident that there is no way to catch up is how a two hour outage becomes a fortnight of data cleanup.

How Yes AI Designs This

A mechanism chosen per flow, with reasons

We work through each data flow and recommend events, delta polling, batch or a combination, with the reasoning written down. Where the honest answer is that a nightly file will serve you better than a real time connection, we say so, because that is usually also the cheaper and more robust choice.

Receivers built defensively

Signature verification, immediate acknowledgement, queued processing, backpressure, idempotent writes and version checks are standard in what we build rather than refinements added after the first duplicate order. Payload logging is scoped so support has what it needs without hoarding personal information.

Hosted and monitored by us

Receivers and schedules run on a managed cloud automation layer we operate, with alerting on silence, on queue depth and on failed processing, plus a tested replay path for anything that did not make it through the first time.

Reconciliation behind the speed

For anything that matters, a scheduled comparison of counts and key values between the two systems, so you find out from a report rather than from a customer that something has been quietly missing since Tuesday.

How We Decide and Build

Five steps, applied per flow rather than once for the whole project.

List the flows and their urgency

What moves, in which direction, and whether anybody is waiting for it. Urgency is a business judgement, so it is agreed with you rather than assumed from the technology.

Check what each vendor actually offers

Available events, retry behaviour, signature scheme, change tokens or modified since filters, rate limits, bulk endpoints and file options. Documentation and reality sometimes differ, so we verify.

Choose the mechanism per flow

Events where speed matters, delta polling where it does not, batch where the system is old or the volume is high, and a reconciliation sweep behind anything with consequences.

Build defensively and test the nasty cases

Duplicate delivery, out of order updates, an endpoint outage with backfill, a burst of a thousand events, an empty file and an expired credential are all exercised before go live.

Monitor, including for silence

Alerting on failures, on queue depth and on the absence of expected traffic, with scheduled reconciliation reports so the health of the flow is visible rather than assumed.

FAQ

Get the Sync Design Right the First Time

Book a call. We map your flows, recommend the right mechanism for each, and give you a priced plan with the reasoning attached. The recommendations are yours either way.