2 min readby Romiel Inolino
n8n's batch processing guide: a workflow that handles 200 records can stop at 20,000. Chunk it, pace it and make every write safe to repeat
n8nworkflow automationdata syncCRMengineering
Most sync workflows don't fail on day one. They fail the day the client imports a big list.
What happened
n8n published a guide to batch processing at scale. It opens with a familiar problem: a workflow that breezes through 200 records might stop on 20,000, with memory limits, API quotas and no clear answer to what actually finished.
The patterns it recommends:
- Chunking: send records in groups, for example 100 or 500 at a time, so memory stays low and failures are easier to isolate.
- Rate limiting: pause between requests or batches and retry when an API asks you to slow down. In n8n that means the Loop Over Items and Wait nodes, or the HTTP Request node's built in Batching option.
- Per item error handling: separate failed items from successful ones so one bad record doesn't send a healthy batch back to the start.
- Idempotency: use upserts so a rerun updates an existing record instead of creating a duplicate, and save checkpoints as batches finish so a failed job can resume.
- Deduplication: n8n's Remove Duplicates node can drop items seen in previous executions, not just repeats within one run.
For larger self hosted loads, n8n points to sub workflows to manage memory and queue mode to spread executions across workers.
My take
This is the checklist I run on every CRM sync before it goes live:
- Find the unique key first. Email, external ID or deal number. Without it you can't upsert, and every retry becomes a duplicate contact. Duplicates in a CRM become wrong emails to real customers.
- Size batches to the slowest system. The limit is usually the CRM or email tool API, not n8n. Start small, watch a real run, then increase.
- Write a checkpoint the business can read. A "synced" flag or timestamp on the source row tells anyone what finished, without opening execution logs.
- Route failures somewhere visible. A Slack message or error sheet with the record ID beats a silent red execution nobody checks.
- Test with the big list before the client does. Run a few thousand records once in staging.
Batch jobs are not exciting, but they are where client trust is won or lost.
Sources
More posts
- Open OCR models now read 1,000 pages for under $0.10, and better serving saved more than switching modelsNov 20, 2026
- Gemini 4 Neon costs $3 in and $15 out per million tokens and tops Zapier's AutomationBench at 56.3%, though it uses more output tokens per taskNov 20, 2026
- Gemini API's Tool Retrieval loads tool schemas only when the model asks for them, so agents with hundreds of tools send fewer input tokensNov 20, 2026
