The string that ate our memory

The bug was four characters long, and it had been sitting there quietly for a long time before anybody looked at it.

We had a FastAPI endpoint that streamed a model's response back to the browser token by token. Standard shape: an async generator, a chunk arrives, you forward it to the client, and — because you also need the finished text to store in the conversation history — you accumulate it as you go.

The accumulation looked like this.

async def stream(request):
    text = ""
    async for chunk in client.stream(prompt):
        text = text + chunk.delta + ""     # <- here
        yield chunk.delta
    await save(conversation_id, text)

Nothing about that line looks expensive. It is the most obvious way to build a string in any language, and in most languages it would be fine.

What actually happens

Python strings are immutable. text + chunk cannot modify text; it has to allocate a brand-new string large enough for both, copy the old contents in, copy the new contents after it, and hand back a new object. The old one is then garbage.

So for a response arriving in n chunks of average length L, you allocate strings of length L, 2L, 3L, and so on up to nL. Add those up and you have allocated

L × n(n+1) / 2 bytes of string, to produce one string of length L × n.

For a long answer — say sixty thousand chunks averaging twelve characters — that is roughly 21 GB of allocation traffic to produce a 720 KB string. Not held at once. Allocated, copied into, and thrown away, sixty thousand times, with each copy larger than the last.

Here is the part that makes it hard to spot. It is quadratic in work, but the live heap at any instant stays small, because each intermediate is freed the moment the next one is built. So a memory profiler that reports peak live allocation shows you almost nothing wrong. In the benchmark below, tracemalloc reports a 0.46 MB peak for the bad version and 0.40 MB for the good one — which tells you precisely nothing.

The clock tells you everything:

Python 3.11, 20,000 chunks

text = text + c + ""       306.2 ms      <- what we had
text += c                    7.1 ms      <- the version CPython can optimise
"".join(parts)               0.5 ms      <- the fix

Six hundred times slower than joining a list, for code that looks identical to a reviewer.

Why += isn't the answer either

You will find advice that says text += chunk is fine because CPython optimises it. That is half true and the half that isn't is where people get hurt.

CPython has a special case in the interpreter loop: when the target of a += on a str has a reference count of exactly one, it can resize the existing buffer in place instead of allocating a new one. That's the 7.1 ms row above — forty times better than the naive version, and still fourteen times worse than joining.

The optimisation evaporates the instant anything else holds a reference to that string. Which happens constantly, and always for a reason that looked sensible at the time:

• you also append it to a list for logging
• it is captured by a closure or a nested coroutine
• it's an attribute on self rather than a local
• someone writes text = text + a + b instead of text += a, which builds a temporary first and never takes the fast path at all

That last one was us. The + "" was left over from a formatting change nobody removed. It defeated the optimisation on its own.

So the performance of that line depends on an implementation detail that a refactor two files away can silently switch off. That is not a property you want load-bearing code to have.

The memory part, honestly

The reason I titled this after memory rather than speed is that memory is how it surfaced. The container's RSS climbed under load and did not come back down between bursts. It looked like a leak. It wasn't one — object counts were flat and gc.collect() freed nothing.

What it was: a high-water mark that ratchets. Under concurrency each in-flight request is running its own quadratic allocation loop, so the simultaneous transient footprint is many times what a single request suggests. And once the allocator has grown to serve that peak, it largely keeps it. Small objects come out of pools that are only released when every block in them is free, and larger blocks sit on free lists rather than being returned to the kernel. So RSS goes up on the worst minute of the day and stays there until the process restarts.

Which is why the obvious diagnostic sends you the wrong way. You look for something you forgot to free. There is nothing to free. You are looking for a leak and what you have is a peak.

The things that did point at it:

• RSS climbing while live object counts stayed flat — the signature of a high-water mark, not a leak
• a CPU profile with a large, unexplained share of time inside string concatenation
• the cost growing with response length, not with request rate

That last one is the tell. A leak scales with how many requests you serve. This scaled with how long each answer was.

The fix

Build a list, join once.

async def stream(request):
    parts = []
    async for chunk in client.stream(prompt):
        parts.append(chunk.delta)
        yield chunk.delta
    await save(conversation_id, "".join(parts))

list.append is amortised O(1) and stores a pointer, not a copy. "".join walks the list once to compute the total length, allocates exactly one buffer of that size, and copies each piece in once. Linear, one allocation, no dependence on refcounts.

For bytes, bytearray and .extend() do the same job. If you genuinely need a file-like interface, io.StringIO and .write() is the same idea wearing a different hat.

And the better question underneath all of it: do you need the accumulator at all? We did, for the history write. But we had a second endpoint accumulating a response it then threw away, purely because the pattern had been copied. The fastest string is the one you never build.

What I took from it

Two things, and neither is "avoid +=".

Reach for the collection, not the accumulator. parts.append(x) then "".join(parts) is not an optimisation you apply after profiling. It is the default way to build a string in a loop, it is shorter to read once you're used to it, and it removes an entire class of question about what the interpreter is doing.

Learn the difference between a leak and a peak before you go looking. They present identically on a memory graph and the investigations have nothing in common. A leak is something you kept. A peak is something you touched. If object counts are flat and RSS is not, stop hunting for a reference you forgot to drop and go and find out what your process is allocating and throwing away.

The four characters were + "". The lesson was the other bit.

If you want to reproduce the numbers, the benchmark is twenty lines: build a list of chunks, run each strategy under `time.perf_counter` and `tracemalloc`, and print. Do run it yourself — the ratio moves a lot with Python version and chunk size, and the point is the shape of the curve, not my figures.