Read a large file without loading it into memory

Python

Iterating a file object yields one line at a time, which is the difference between a constant footprint and an OOM.

with open('access.log', encoding='utf-8') as fh:
    for line in fh:
        if 'ERROR' in line:
            print(line.rstrip())

Download

Python files memory io chris

More in Python

Parse arguments without a dependency Python argparse covers subcommands, types and help text, which is more than most scripts ever need. Retry with exponential backoff and jitter Python Backs off on each failure and spreads retries out, so a recovering service is not hit by every client at once. Time a block of code Python A context manager that prints how long its body took, without scattering time.perf_counter calls through the function. A dataclass with validation Python __post_init__ runs after the generated constructor, which is where cheap invariants belong. Chunk an iterable into fixed-size batches Python itertools.batched on 3.12 and later; the islice version works everywhere and never materialises the whole input. Walk a directory tree lazily Python pathlib keeps the paths as objects rather than strings, so joining and suffix checks stay readable. Merge dictionaries Python The | operator returns a new dict; |= updates in place. Later keys win in both.

Random picks

Find the commit that introduced a bug Git Binary search across history. Mark a known good and a known bad commit and git narrows it down for you. A starting Content-Security-Policy Security Blocks inline script and restricts everything to the origin. Tighten from here rather than loosening from nothing. Generate an SSH key and add it to the agent Security Ed25519 rather than RSA: shorter, faster, and no key-size decision to get wrong. Responsive images that do not blow the bandwidth budget Web & CSS srcset lets the browser pick; sizes tells it how wide the image will actually be rendered. Truncate text to a number of lines Web & CSS Ellipsis after two lines. The -webkit- prefixes are still required despite being supported everywhere. Read a file into a string in Go Systems os.ReadFile replaced ioutil.ReadFile in 1.16 and is the one-liner for small files. Context with a timeout in Go Systems Cancels the request and everything downstream of it after two seconds. The defer is not optional. Delete duplicate rows, keeping the oldest Databases Uses a window function to number the duplicates and removes everything but the first of each group.