Batch API Requests in Python: Throughput vs Rate Limit Trade-Off

You have 200,000 records to push to an API that allows 10 requests per second. One record per request, sequentially, that’s at best 5.5 hours. Throw 100 threads at it and you get a wall of 429s. The useful question isn’t “how fast can I send?” but “which knob am I actually limited by?”

There are three knobs:

• Rate (R): requests per second the API allows.
• Concurrency (C): requests in flight at the same time.
• Batch size (b): records per request, if the API has a batch endpoint.

Concurrency: size it with Little’s Law

Little’s Law: in-flight = throughput × latency. Applied to HTTP:

C_needed = R × L

If the limit is 10 req/s and each request takes 400 ms, you need 10 × 0.4 = 4 concurrent requests to saturate the limit. With 1 you’re capped at 2.5 req/s. With 50 you don’t go faster; you just reach the limit sooner and start collecting 429s.

So concurrency is not a speed knob past R × L. It’s a latency hiding knob. Above that point, the rate limit is the bottleneck, and only batch size moves record throughput.

Batch size: where the real gain is

If the endpoint accepts b records per call, record throughput becomes:

records/s = min(R, C / L(b)) × b

The catch is L(b): bigger payloads take longer to serialize, transfer and process. A simple illustrative model, L(b) = 200 ms + 4 ms × b with R = 10 and C = 8 (these are model numbers, not a benchmark):

Two things fall out of this:

1. Going from 1 to 100 is a 100× gain because you stay rate-bound.
2. Past ~200 the bottleneck flips to concurrency. You can push C up again (C ≥ R × L(b)), but gains flatten and each request now carries more risk.

What bigger batches cost you

Throughput isn’t free. Larger batches mean:

• Payload limits. Most APIs cap items per batch and body size. Read the docs before tuning.
• Partial failures. One bad record can fail the whole batch, or the API returns per-item results you must parse. Either way your retry unit is now b records.
• Retry cost. A 429 or 5xx on a 500-item batch re-sends 500 items.
• Quota accounting. Some providers count a batch as one request; others count each item against the quota. If it’s per item, batching saves round trips but not quota. Check before assuming.
• Memory and latency to first result. Relevant if a downstream step consumes results as a stream.

My default: pick the largest batch that keeps you rate-bound (not concurrency-bound) with a sane C, and stay well below the provider’s max.

Implementation: asyncio + httpx + token bucket

A semaphore limits concurrency. A token bucket limits rate. You need both: the semaphore alone doesn’t stop 8 fast requests from exceeding 10 req/s, and the bucket alone doesn’t stop slow requests from piling up connections.

# >GLB_ // rate-limited batch sender
import asyncio
import time
from collections.abc import Iterator, Sequence

import httpx


class TokenBucket:
    """Allow `rate` acquisitions per second, with bursts up to `capacity`."""

    def __init__(self, rate: float, capacity: int) -> None:
        self.rate = rate
        self.capacity = capacity
        self.tokens = float(capacity)
        self.updated = time.monotonic()
        self._lock = asyncio.Lock()

    async def acquire(self) -> None:
        while True:
            async with self._lock:
                now = time.monotonic()
                self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
                self.updated = now
                if self.tokens >= 1:
                    self.tokens -= 1
                    return
                wait = (1 - self.tokens) / self.rate
            await asyncio.sleep(wait)


def chunked(items: Sequence[dict], size: int) -> Iterator[Sequence[dict]]:
    for i in range(0, len(items), size):
        yield items[i : i + size]


async def send_batch(
    client: httpx.AsyncClient,
    bucket: TokenBucket,
    sem: asyncio.Semaphore,
    batch: Sequence[dict],
    max_retries: int = 5,
) -> dict:
    async with sem:
        for attempt in range(max_retries + 1):
            await bucket.acquire()
            resp = await client.post("/v1/records:batch", json={"records": list(batch)})
            if resp.status_code == 429 and attempt < max_retries:
                retry_after = float(resp.headers.get("Retry-After", 2 ** attempt))
                await asyncio.sleep(retry_after)
                continue
            resp.raise_for_status()
            return resp.json()
    raise RuntimeError("retries exhausted")


async def push_all(records: Sequence[dict], *, rate: float = 10, concurrency: int = 8, batch_size: int = 100):
    bucket = TokenBucket(rate=rate, capacity=int(rate))
    sem = asyncio.Semaphore(concurrency)
    limits = httpx.Limits(max_connections=concurrency, max_keepalive_connections=concurrency)

    async with httpx.AsyncClient(
        base_url="https://api.example.com",
        limits=limits,
        timeout=httpx.Timeout(30.0),
    ) as client:
        tasks = [send_batch(client, bucket, sem, b) for b in chunked(records, batch_size)]
        results = await asyncio.gather(*tasks, return_exceptions=True)

    failed = [r for r in results if isinstance(r, Exception)]
    print(f"batches: {len(results)}, failed: {len(failed)}")
    return results


if __name__ == "__main__":
    data = [{"id": i, "value": i * 2} for i in range(20_000)]
    asyncio.run(push_all(data))

Notes:

• Retry-After here is parsed as seconds only, for brevity. For the full version (HTTP-date support, jitter, idempotency rules) see retries in Python: from a basic retry loop to exponential backoff.
• Retries go back through bucket.acquire(), so retries count against your rate like any other request.
• httpx.Limits(max_connections=C) keeps the connection pool aligned with the semaphore; otherwise the pool’s default (100) is just unused capacity. (HTTPX resource limits)
• gather over pre-built coroutines is fine for tens of thousands of batches. For millions, feed an asyncio.Queue with a fixed pool of worker tasks so you don’t hold every coroutine in memory.
• Set the bucket a bit under the documented limit (e.g. 9 for a 10 req/s limit). Clock skew and the server’s window boundaries don’t line up with yours.
• If you’d rather not maintain the bucket, aiolimiter.AsyncLimiter(rate, 1) is a drop-in leaky bucket. (aiolimiter)

Measure, don’t guess

Before tuning, log per-request latency and status code, then compute:

• p50/p95 latency per batch size → gives you L(b)
• observed req/s vs configured rate → are you rate-bound or concurrency-bound?
• 429 ratio → anything above ~1% means the bucket is too generous

If p95 latency grows faster than linearly with b, the server is struggling with large payloads and you’ve passed the useful batch size.

Summary

• Concurrency beyond R × L buys nothing but 429s.
• Under a request-based rate limit, batch size is the main throughput lever.
• Bigger batches trade throughput for retry cost, partial-failure handling and payload limits.
• Use a semaphore and a token bucket; route retries through the bucket.
• Run slightly below the documented limit and measure p95 latency per batch size.

References

• Python — asyncio.Semaphore: https://docs.python.org/3/library/asyncio-sync.html#asyncio.Semaphore
• Python — asyncio.gather: https://docs.python.org/3/library/asyncio-task.html#asyncio.gather
• Python — asyncio.Queue: https://docs.python.org/3/library/asyncio-queue.html
• HTTPX — Async support: https://www.python-httpx.org/async/
• HTTPX — Resource limits: https://www.python-httpx.org/advanced/resource-limits/
• aiolimiter: https://aiolimiter.readthedocs.io/
• RFC 6585 §4 — 429 Too Many Requests: https://www.rfc-editor.org/rfc/rfc6585#section-4
• Little’s Law (Little, 1961, Operations Research): https://doi.org/10.1287/opre.9.3.383