For catalogues where a full sync takes hours

Large Catalogue Product Sync: When Your SKU Count Breaks the Usual Approach

Most product integrations are designed for a few thousand SKUs. They read everything, compare, and push changes, and at that size it works. Australian distributors of auto parts, hardware, electrical, office supplies, books, industrial consumables and wide homewares ranges routinely carry tens or hundreds of thousands of SKUs, and the same approach falls over.

A full sync that took ten minutes now takes most of a day, consumes the API allowance, and means a price change made at nine in the morning reaches the website by mid afternoon. This page explains how large catalogue syncs are designed differently: separate lanes for price and stock versus content, change detection instead of full reads, bulk interfaces, rate limit budgets, variant and image handling, and the monitoring needed to trust a catalogue nobody can check by eye.

Review Your Catalogue Sync (03) 9003 0111

The Arithmetic of a Large Catalogue

2 to 3 lanes

Price and stock, content, and media

Each with its own frequency and priority

1 to 5%

Of SKUs change on a typical day, roughly

Sending only those is the biggest single efficiency gain

Hours to minutes

Typical improvement in update latency

Depends on platform limits and how changes are detected

4 to 10 weeks

Typical build, as a guide

Initial load planning and data cleansing are usually the long poles

Why Large Catalogues Break Standard Syncs

Size changes the problem. These four constraints decide the design long before the first field is mapped.

Rate limits turn size into time

Every ecommerce platform and marketplace limits how fast you can write to it. At a given rate, the time to touch every product grows linearly with catalogue size. Sending 200,000 products one at a time through a standard API can take many hours, during which price and stock updates queue behind descriptions that did not change. Large syncs have to send less, send it in bulk where the platform allows, and prioritise what matters.

Most of the catalogue does not change

On a typical day only a small share of products change price, stock or content. A design that reads and compares everything to find those changes wastes most of its effort. Change detection at the source, by modified timestamps, change logs, event triggers or a stored fingerprint of each product, reduces the work to what actually moved.

Product structures do not translate one to one

An ERP may hold every size and colour as a separate SKU, while the store groups them as one product with options. Platforms impose limits on options and variants per product, and those limits have changed over time. A part number range with hundreds of variants, or a product with many option combinations, needs a deliberate grouping strategy before load.

Search and collections lag behind writes

Writing a product to the platform is only half the job. Search indexes, filters, collections and storefront caches update after the write, sometimes with delay. A large bulk update can leave filters inconsistent for a period. The sync design has to account for indexing time, schedule heavy loads outside peak trading, and verify the storefront rather than assume.

Six Techniques Behind Large Syncs That Work

None of these are exotic. Together they are the difference between a sync that runs all day and one that finishes in minutes.

Price and stock first

Separate lanes

Price and stock changes travel in a fast lane that runs frequently and touches only those fields. Content such as descriptions, attributes and categories travels in a slower lane. Images and media travel in their own lane because they are the heaviest. A wholesale price change never waits behind a batch of new product images. Lanes also fail independently, so a broken image URL from a supplier stalls only the media queue while prices and stock keep flowing.

Only what moved

Change detection

We detect changes at the source using modified dates, change log tables, event triggers or a fingerprint of each product compared with the last version sent. The sync then processes only changed products. A full reconciliation still runs periodically, for example weekly, to catch anything change detection missed. Fingerprints are calculated only over the fields the destination actually uses, so an internal ERP field changing does not trigger a pointless update to the store.

Fewer, larger operations

Bulk interfaces

Where the platform offers bulk operations, such as bulk mutation and bulk query jobs or file based imports, we use them for large changes and initial loads instead of thousands of individual calls. Smaller frequent updates still use standard calls. Choosing the right interface per job cuts sync time substantially.

Never throttled flat

Rate budget and queueing

Every lane gets a share of the platform rate allowance, with price and stock guaranteed headroom. Work is queued, paced to stay within limits, and retried with backoff when throttled. During peak periods the content lane can pause automatically so price and stock stay current.

Staged first load

Initial load plan

The first load of a large catalogue is a project of its own: cleanse and validate data, load in staged batches, verify counts and samples, then switch to change based sync. We plan it around trading hours and indexing time, and keep it repeatable so a rebuild is possible if ever needed.

Trust without checking by eye

Catalogue monitoring

Nobody can check 200,000 products by hand. Monitoring compares counts, prices and stock between source and destination on samples and in aggregate, reports products that failed validation, and alerts when queues grow or lanes fall behind. Problems surface as a report, not a customer complaint. A short daily summary tells the merchandising team how many products changed, how many failed validation and which suppliers sent the most errors, so data quality improves at the source over time.

Large Catalogue Scenarios

Task	Traditional	Designed for Scale	Notes
Price change across a supplier range	Full sync, live by afternoon	Fast lane, live within the hour	Only changed prices are sent, with price and stock prioritised over content.
Initial load of 150,000 SKUs	Single run that fails part way	Staged, verified, resumable load	Batches are validated, counted and sampled. A failure resumes from the last good batch.
Supplier feed with daily full file	Entire file re-sent to the store	File compared, only changes sent	Fingerprinting turns a full supplier file into a small set of actual changes.
Parts with hundreds of variants	Products split arbitrarily	Grouping strategy agreed before load	Group by what customers browse, within current platform variant and option limits.
Peak season with heavy order traffic	Sync competes with orders for API	Content lane paused, stock prioritised	Orders and stock updates keep their share of the rate budget during peak.
New images for thousands of products	Images block price updates	Separate media lane, off peak	Images are processed and uploaded in their own queue at quieter times.
Marketplace with stricter limits	Listings rejected or throttled	Channel specific queue and validation	Each channel gets its own pacing and validation rules before submission.
Discontinued lines across the range	Left live, oversold or hidden badly	End of life rules applied in bulk	Discontinued products are archived or redirected according to agreed rules.

Where Large Catalogue Syncs Go Wrong

One queue for everything

When price, stock, content and images share a queue, a large content change delays price and stock updates for hours. Customers see wrong prices and buy unavailable products. Separate lanes with priority for price and stock is the most important single design decision for large catalogues.

Full reads to detect changes

Reading every product from the source and destination to find differences consumes time and API allowance proportional to catalogue size. Use change detection at the source wherever possible, and keep full reconciliation to a scheduled, off peak job that catches anything missed.

Variant limits discovered during load

Platforms limit options and variants per product, and those limits change over time. Discovering partway through an initial load that some products exceed them means rework. Analyse the catalogue structure against current platform limits before loading and agree a grouping strategy for exceptions.

Bad data multiplied at scale

At large SKU counts, small data quality issues become thousands of broken products: missing weights that break shipping rates, inconsistent units, duplicate barcodes, invalid characters. Validate before sending, route failures to a report for correction at the source, and never let invalid data silently overwrite good data in the store.

Not planning for a rebuild

At some point a platform migration, a major restructure or corruption will require reloading the catalogue. If the initial load was a one off script nobody can rerun, that becomes a crisis. Keep the load process documented, repeatable and tested, so a rebuild is a planned job rather than an emergency.

Australian specifics left out of product data

GST treatment per product, Australian consumer warranty and safety information, dangerous goods flags for shipping, and units of measure that match Australian conventions must all be carried in the sync. In a large catalogue, missing attributes on even a small share of products creates many customer facing problems and compliance risk.

How Yes AI Helps With Large Catalogues

Honest platform advice

Some ecommerce platforms and connectors handle large catalogues well out of the box, especially with built in bulk import. If yours does and your needs are standard, we will say so. Custom sync design is for when standard approaches cannot keep up.

Structure before load

We analyse your catalogue structure, variant groupings, attributes and data quality against the destination platform before anything is loaded, so limits and rework are discovered on paper rather than mid load.

Built to scale and run for you

Syncs run on a managed cloud automation layer we operate, with lanes, queues, rate budgeting, retries and monitoring. You see what moved, what failed and why, without anyone watching a progress bar.

Repeatable loads

Initial loads are documented and repeatable, so a future migration, restructure or recovery uses the same tested process instead of starting again.

From All Day Syncs to Minutes

Five steps. Analysis and data cleansing usually take longer than the build itself.

Measure the catalogue and the sync

SKU counts, variant structures, attribute completeness, daily change rates, current sync duration and where time is being spent.

Design lanes and change detection

Which fields travel in which lane, how often, how changes are detected at the source, and how the rate budget is shared across lanes.

Cleanse and plan the initial load

Validation rules, data fixes at the source, grouping decisions and a staged load plan scheduled around trading and indexing time.

Build, load and verify

Built against the platform APIs and bulk interfaces, loaded in stages, verified by counts and samples, then switched to change based sync.

Monitor and tune

Lane latency, queue depth, failed validations and source versus destination checks reported continuously, with tuning before peak periods.

FAQ

How many products is a large catalogue?

There is no fixed line, but the standard approach usually starts to strain somewhere in the tens of thousands of SKUs, depending on platform limits, how often products change and how many channels you sync to. A practical test: if a full sync takes longer than the gap between the updates you need, such as hourly price changes, the catalogue is large for your setup.

How do you sync 100,000 products to Shopify or a similar platform?

Not one by one through standard calls. Use bulk operations or file based imports for the initial load and large changes, change detection so only modified products are sent day to day, separate lanes so price and stock are prioritised over content and images, and a rate budget that keeps each lane within platform limits. Check the current platform documentation for its limits and bulk interfaces.

What is delta sync?

Delta sync means sending only what changed since the last sync rather than the whole catalogue. Changes are detected at the source using modified timestamps, change logs, event triggers or a stored fingerprint of each product. It reduces sync time and API usage dramatically for large catalogues. A periodic full reconciliation still runs to catch anything delta detection missed.

Why do price changes take so long to reach our website?

Usually because price updates share a queue with everything else, so they wait behind content and image updates, or because the sync reads the entire catalogue to find changes. Separating price and stock into a fast lane, detecting changes at the source and sending only changed fields typically reduces latency from hours to minutes, subject to platform limits.

How do platform variant limits affect large catalogues?

Platforms limit how many options and variants a product can have, and these limits have changed over time. Catalogues with many sizes, colours or part variants may need grouping decisions: splitting a range into several products, combining options differently or using separate product pages. Analyse the catalogue against current limits before the initial load.

Should we use a PIM for a large catalogue?

Often it helps, particularly when content is enriched by several people or published to several channels. A PIM gives you one place to manage attributes, descriptions and media, while the ERP remains the owner of price and stock. It does not remove the need for a well designed sync to each channel. For a single channel with simple content, a PIM may be unnecessary.

How do we know the catalogue is accurate without checking it by hand?

Automated monitoring compares source and destination counts, prices and stock in aggregate and on samples, reports products that failed validation with the reason, and alerts when queues grow or lanes fall behind. With a large catalogue, this monitoring is how you know the sync is working. Silence from a sync is not evidence of accuracy.

Get a Large Catalogue Sync That Keeps Up

Book a call. We measure your catalogue and current sync, identify where the time goes, and give you a priced plan for a sync that stays current at your scale.

Book a Catalogue Sync Call (03) 9003 0111

All discussions held in confidence. Australian-based consultants.