Chunk an iterable into fixed-size batches

Python

itertools.batched on 3.12 and later; the islice version works everywhere and never materialises the whole input.

from itertools import batched, islice

for chunk in batched(rows, 500):
    insert_many(chunk)

# Before 3.12:
def chunks(it, size):
    it = iter(it)
    while batch := list(islice(it, size)):
        yield batch

Download

Python itertools batching chris

More in Python

Time a block of code Python A context manager that prints how long its body took, without scattering time.perf_counter calls through the function. Parse arguments without a dependency Python argparse covers subcommands, types and help text, which is more than most scripts ever need. Merge dictionaries Python The | operator returns a new dict; |= updates in place. Later keys win in both. Walk a directory tree lazily Python pathlib keeps the paths as objects rather than strings, so joining and suffix checks stay readable. Retry with exponential backoff and jitter Python Backs off on each failure and spreads retries out, so a recovering service is not hit by every client at once. A dataclass with validation Python __post_init__ runs after the generated constructor, which is where cheap invariants belong. Read a large file without loading it into memory Python Iterating a file object yields one line at a time, which is the difference between a constant footprint and an OOM.

Random picks

Undo the last commit but keep the changes Git Moves HEAD back one commit and leaves everything staged, which is what you usually want after committing too early. Make a shell script fail loudly Shell The four lines that turn a script from 'carries on after an error' into 'stops at the first problem'. Group rows into a JSON array Databases Returns one row per author with their posts nested, instead of a join that repeats the author on every row. Follow redirects and show only the headers Networking Traces a redirect chain to its destination without downloading the body. List branches merged into main Git Finds the branches that are safe to delete, and the ones that are not. Table sizes, largest first Databases Where the disk actually went, including indexes and TOAST. Time every phase of an HTTP request Networking Splits DNS, connect, TLS and first byte apart, which tells you which one is actually slow. Find what is taking up space in an image Containers Prints the size each layer adds, which usually points straight at the offending RUN line.