Batch APIs: The Easiest 50% Cost Cut You're Not Taking

Most teams route every LLM call through the synchronous chat completions endpoint, even when nothing on the other end is waiting for a reply within seconds. Nightly summarization of yesterday's support tickets. Re-tagging a product catalog. Running an eval suite against 4,000 test cases. Backfilling embeddings-adjacent metadata for a RAG index. None of this needs sub-second turnaround, yet it's billed at the same per-token rate as a live chat request.

Both OpenAI and Anthropic will process these jobs at a flat 50% discount if you're willing to wait up to 24 hours. That's the batch API. It's not a beta feature or a gimmick — it's a straightforward tradeoff of latency for cost, and for offline workloads it's close to a free lunch.

The problem it solves

Synchronous APIs are priced for the worst case: a human staring at a spinner. That pricing bakes in the cost of holding capacity ready for unpredictable, bursty, low-latency traffic. If your workload is a scheduled job that can tolerate hours of delay, you're paying for a guarantee you don't need.

The naive alternative — firing thousands of concurrent synchronous requests at 2 a.m. to "batch" them yourself — doesn't get you the discount, and it fights the provider's rate limits the whole way. You'll hit 429s, need custom backoff logic, and still pay full price.

Batch APIs solve this properly: you submit a file of requests, the provider schedules them into spare capacity, and you poll for a result file. No custom concurrency management, no rate-limit dance, half the cost.

How it actually works

The shape is nearly identical across providers:

1. Build a JSONL file where each line is one request, tagged with a custom_id you choose.
2. Upload the file.
3. Create a batch job pointing at that file.
4. Poll the job status until it's completed (typically minutes to a few hours, guaranteed within 24h).
5. Download the output file and join results back to your custom_ids.

OpenAI Batch API

import json
from openai import OpenAI

client = OpenAI()

# 1. Build the JSONL request file
requests = [
    {
        "custom_id": f"ticket-{ticket['id']}",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
            "model": "gpt-5-mini",
            "messages": [
                {"role": "system", "content": "Summarize this support ticket in one sentence."},
                {"role": "user", "content": ticket["body"]},
            ],
            "max_tokens": 100,
        },
    }
    for ticket in yesterdays_tickets
]

with open("batch_input.jsonl", "w") as f:
    for r in requests:
        f.write(json.dumps(r) + "\n")

# 2. Upload it
batch_file = client.files.create(file=open("batch_input.jsonl", "rb"), purpose="batch")

# 3. Create the batch job
batch = client.batches.create(
    input_file_id=batch_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
)

print(batch.id, batch.status)  # e.g. batch_abc123, validating

Polling and reading results:

import time

while True:
    batch = client.batches.retrieve(batch.id)
    if batch.status in ("completed", "failed", "expired", "cancelled"):
        break
    time.sleep(60)

if batch.status == "completed":
    output = client.files.content(batch.output_file_id).text
    results = {}
    for line in output.strip().split("\n"):
        row = json.loads(line)
        custom_id = row["custom_id"]
        if row.get("error"):
            results[custom_id] = {"error": row["error"]}
        else:
            results[custom_id] = row["response"]["body"]["choices"][0]["message"]["content"]

Anthropic Message Batches API

import anthropic

client = anthropic.Anthropic()

batch = client.messages.batches.create(
    requests=[
        {
            "custom_id": f"ticket-{ticket['id']}",
            "params": {
                "model": "claude-haiku-4-5",
                "max_tokens": 100,
                "messages": [
                    {"role": "user", "content": f"Summarize this ticket in one sentence:\n\n{ticket['body']}"}
                ],
            },
        }
        for ticket in yesterdays_tickets
    ]
)

print(batch.id, batch.processing_status)

Polling and streaming results (Anthropic streams results rather than handing you a single file):

import time

while True:
    batch = client.messages.batches.retrieve(batch.id)
    if batch.processing_status == "ended":
        break
    time.sleep(60)

results = {}
for entry in client.messages.batches.results(batch.id):
    if entry.result.type == "succeeded":
        results[entry.custom_id] = entry.result.message.content[0].text
    else:
        results[entry.custom_id] = {"error": entry.result.type}

The mental model is the same either way: write, submit, poll, join. The differences are in file format, result delivery (file vs. stream), and per-batch limits.

When batch is the right call — and when it isn't

Signal	Use Batch API	Use Synchronous API
A human is waiting for the response	No	Yes
Job runs on a schedule (nightly, hourly)	Yes	—
Latency tolerance measured in minutes-to-hours	Yes	No
Workload is a large, known set of independent requests	Yes	—
Requests depend on each other's output (agentic chains)	No	Yes
You need the result to unblock the next step in real time	No	Yes
Cost is the binding constraint, not speed	Yes	—
Volume is small (a handful of requests)	Marginal — overhead may not be worth it	Yes

Good batch candidates: dataset labeling, eval suite runs, log/ticket summarization, embeddings-adjacent enrichment, content moderation sweeps, re-scoring historical data after a prompt change, generating synthetic training data.

Bad batch candidates: chat UIs, anything in a request/response HTTP handler, multi-turn agent loops where step N's prompt depends on step N-1's output, real-time voice or streaming UX.

Building a batch job that survives production

The demo code above is the easy 80%. The parts that actually determine whether this works reliably in production:

1. Idempotent custom_ids. Derive the ID deterministically from your own primary key (f"ticket-{ticket_id}"), not a random UUID generated at submission time. If a batch job fails partway or you need to re-run it, deterministic IDs let you diff against what's already been processed instead of double-billing yourself.

2. Chunk large workloads. Both providers cap requests per batch (OpenAI: 50,000 requests or 200MB per file at time of writing; Anthropic: 100,000 requests or 256MB). For anything close to those limits, split into multiple batch files and track them as a set, not a single job.

3. Handle partial failures per-line, not per-batch. A batch can complete with some individual requests erroring out (bad input, content filter, model overload). Don't treat status: completed as "everything succeeded" — iterate the output file and check each row's error field before trusting the content.

4. Don't poll in a tight loop. Batches take minutes to hours. Poll every 60-300 seconds from a scheduled job (cron, Airflow, a queue worker with a delay), not a while True: sleep(1) inside a request handler. If your infra supports webhooks for job completion, prefer that over polling.

5. Set a real timeout and a dead-letter path. Batches can expire without completing (rare, but happens under provider-side load). Decide up front: does an expired batch retry automatically, alert a human, or fall back to synchronous calls for the unprocessed subset? Don't let a silently-expired batch just vanish from your pipeline.

6. Version your prompts alongside the batch file. If you re-run the same batch job weekly, store which prompt/model version produced each output file. When someone asks "why did last week's summaries look different," you want an answer that isn't "we changed the prompt on Tuesday but didn't write it down."

Common pitfalls

• Assuming batch guarantees ordering. Output rows are not guaranteed to come back in the same order as input. Always join on custom_id, never on line number.
• Forgetting rate limits still exist on the submission side. You can be rate-limited creating batches or uploading files even though the batch itself runs asynchronously.
• Using batch for latency-sensitive backfills "just this once." If a stakeholder wants results in 10 minutes, batch's up-to-24h SLA will burn you. Batch buys cost, not speed — never both.
• Not accounting for the discount in cost dashboards. If your cost monitoring assumes a flat per-token rate, batch traffic will look like an anomaly. Tag batch spend separately so cost reports don't get confusing.
• Re-submitting failed rows as a brand-new batch with new random IDs. This breaks your ability to reconcile what's been retried vs. what's still pending. Reuse the original custom_id.

Next steps

Start with your most expensive offline job — the nightly summarization run, the weekly re-scoring pipeline, the eval suite that runs in CI. Move it to batch first; it's the lowest-risk way to validate the plumbing (file format, polling, error handling) before you trust batch for anything that touches a deadline. Once that's solid, batch becomes the default for any workload where nobody's watching a spinner.

If you're building the muscle for this kind of production LLM engineering — cost modeling, pipeline design, the operational details that don't show up in API docs — that's exactly what we cover hands-on in Joinloop's AI engineering cohort.