July 11, 2026

The OOM wasn't where everyone thought it was

We blamed our ML model for a serverless out-of-memory crash. We were completely wrong — and a memory profiler proved it in five minutes.

Christoph Schütte

It started with a backfill.

At Solid we automatically sort every uploaded video into A-roll (a person talking to camera) and B-roll (everything else). It's the seam the whole product hangs off — A-roll becomes the spoken spine of an ad, B-roll becomes the cutaways. The classifier that does the sorting is an active-speaker model: it watches cropped faces and listens to the audio and decides, frame by frame, whether the person on screen is the one actually talking.

It runs on a 6 GB AWS SageMaker Serverless endpoint. Serverless because classification is spiky — nothing for minutes, then a batch of uploads all at once. Scale-to-zero, pay per call. Lovely.

Then we kicked off a backfill to classify ~8,000 older videos, and around file 7,800 it started falling over:

Inference failed due to insufficient memory on the Endpoint.
Please add more memory to the endpoint.

We spent the better part of a week convinced we knew why. We were wrong in an interesting way, so here's the whole thing — including the part where we shipped a fix for the wrong problem.

Lesson #1: An accelerating failure rate is a crash-loop, not a bad input

The failures didn't trickle in at a steady rate. They accelerated:

3 → 20 → 50 → 114 → 295 → 489 failures

That shape is a fingerprint. A flat error rate says "some of your inputs are bad." An accelerating one says crash-loop: a request OOMs the container, SageMaker's adaptive retries re-send it while the container is restarting cold, the extra load OOMs it again, and the spiral feeds itself. The retries we'd added for resilience were quietly pouring petrol on the fire.

There was a second clue, and we misread it completely: the failures correlated with 4K uploads. Big videos, big memory — of course. (Hold that thought.)

Remember: retries turn a transient OOM into a self-sustaining outage. An accelerating failure curve is telling you the system is fighting itself.

Lesson #2: The heavy, complicated component is not automatically the expensive one

Here's the trap we walked straight into.

The model is a TalkNet-style network running under ONNX Runtime. ONNX has a famous appetite: its CPU memory arena grabs a pool and holds on to it. A quick-and-dirty local repro showed a roughly flat ~1.3 GB per-call peak. That fit the story we already believed — the model is the big, scary, complicated part, so the model must be the memory hog — so we:

  1. Disabled the ONNX memory arena to stop it hoarding.
  2. Finished the backfill by running the model on a beefy local box instead of the endpoint.

It sort of worked. The arena trim shaved a bit. The local run limped over the line. And we walked away believing the fix was "buy a bigger endpoint."

Except live ingest of new uploads kept OOMing. And lowering client concurrency (40 → 8 → 3) did nothing. If concurrency were the whole story, three small concurrent calls could never breach 6 GB. The numbers refused to add up, which is usually the universe telling you your mental model is wrong.

Lesson #3: Measure the phase, not the call

Our whole first investigation measured the call as a single number. That's the mistake. One peak figure hides where the memory actually goes.

So we did the boring, decisive thing: checkpoint resident memory (RSS) at every phase, in a fresh process, on the exact videos that were failing. The result was not subtle. The peak lands the instant we finish decoding frames — and drops ~4× the moment we free them, before the model runs at all:

Window (portrait 360×640)framespeak (after decode)after del framesafter model
14.6 s362461 MB256 MB260 MB
17.9 s444512 MB254 MB258 MB
53.6 s13381107 MB268 MB267 MB
Resident memory across the four phases of one 53.6 s window — the decoded frame list is the entire spike; freeing it drops RSS ~4×, and the model barely registers.
Resident memory across the four phases of one 53.6 s window — the decoded frame list is the entire spike; freeing it drops RSS ~4×, and the model barely registers.

The model — the thing we'd spent a week blaming — adds ~30 MB. Its input tensor is about 4 MB. The idle container with both ONNX models loaded sits at ~210 MB. The model was never the problem. Not even close.

The actual culprit: to sample frames, we decoded the entire video window into a Python list of raw RGB frames and held it in memory through the whole face-detection pass. That list is:

≈ duration × 25 fps × height × width × 3 bytes

For a 53-second portrait window that's ~925 MB of decoded pixels just… sitting there. For a full 60-second square 640×640 preview it's ~2 GB (640 × 640 × 3 ≈ 1.23 MB per frame, times ~1,500 frames).

Remember: a peak-memory number tells you how much. Per-phase profiling tells you where — and "where" is the only thing you can actually fix.

Lesson #4: Distrust the correlation that flatters your first guess

Now the twist. The model never sees a 4K frame — we downscale every preview to ≤640 px before classification. So why did 4K uploads correlate with the OOMs?

Because 4K and social-format sources tend to export square or portrait previews (~2× the bytes per frame of a 16:9 letterbox) and tend to run longer. So they're precisely the uploads that hit a full 60-second window at the ~2 GB end of the scale. The resolution of the original was an innocent bystander. The shape and length of the preview was the real driver.

"4K → big memory" was true-sounding, mechanically plausible, and completely wrong about the causal chain. It cost us a week because it flattered the guess we'd already made.

And the real production trigger wasn't the backfill at all. Live ingest classifies one file's windows four at a time for throughput. So one long square upload fires:

4 concurrent windows × ~2 GB each = ~8 GB on a 6 GB container
Live ingest classifies one file four windows at a time, so a single long upload stacks four ~2 GB buffers onto a 6 GB endpoint.
Live ingest classifies one file four windows at a time, so a single long upload stacks four ~2 GB buffers onto a 6 GB endpoint.

A single upload could OOM the endpoint by itself, with zero competing traffic. There's your bug.

Lesson #5: Streaming beats scaling

Here's the part that stings a little: the frames were only ever consumed one at a time, and face detection keeps nothing but the tiny 112×112 crops (~12 KB each). We were materializing a 2 GB list to feed a consumer that wanted one frame at a time and threw ~99% of each one away.

So we turned frame extraction into a generator:

def iter_frames(path, start_sec, duration_sec):
    """Yield a video window's frames one at a time, at target fps.

    Face detection consumes frames one-by-one and keeps only tiny 112x112
    crops, so yielding — instead of materializing the whole window — keeps
    the decoded frames from piling up. That list is the dominant per-call
    allocation (~1GB to ~2GB for a full 60s window), and on the 6GB endpoint
    it is the difference between a flat ~330MB per invocation and an OOM.
    """
    cap = cv2.VideoCapture(str(path))
    for i in range(total_source_frames):
        ret, frame = cap.read()
        if not ret:
            break
        if i >= next_sample:
            resized = _resize_frame(frame, hparams.face.max_detect_dim)
            yield cv2.cvtColor(resized, cv2.COLOR_BGR2RGB)
            next_sample += frame_interval

Detection pulls the iterator lazily. Only one decoded frame is ever resident, plus the little list of crops. The full window never exists in memory at once. Same numbers, measured on the same failing videos:

Windowbefore (list)after (streaming)
14.6 s461 MB~358 MB
17.9 s512 MB~360 MB
53.6 s1107 MB~376 MB

Per-call peak is now flat at ~330–380 MB regardless of window length or preview shape. A 60-second square window that used to hit ~2 GB lands in the same band as a 15-second clip. That's ~15+ concurrent calls on the same 6 GB endpoint. No resize, no bigger instance, no leaving Serverless, no added cost.

And — the part we care about most — the output is bit-identical. Detection does the same work on the same frames in the same order; we verified max|Δ| = 0.0 across the probability timelines. We didn't trade accuracy for memory. We just stopped hoarding.

Remember: before you scale the box up, check whether you're holding something you could stream. Streaming is free; a bigger endpoint is a monthly bill.

The job was never the model

The most expensive line in this whole system wasn't the neural network. It was a list().

We anchored on the model because it's the most complicated thing in the pipeline, and complicated feels expensive. But complexity and cost aren't the same axis. The model was a rounding error; the humble frame buffer was 98% of the peak. One for loop and a yield later, the endpoint has been flat ever since.

If there's one thing to steal from this: the next time something is OOMing, don't argue about the fix until a profiler has told you where the bytes actually are. We'd have saved a week.

Latest posts
How Fast-Growing Brands Source Video ContentInsights · November 19, 2026Supplement Ads 2026: Rules, Angles, AI WorkflowInsights · November 19, 2026How to Build a Creative Engine With a Lean Marketing TeamInsights · November 18, 2026AI Tools for Marketers in 2026: The Stack by JobInsights · November 18, 2026Pet Food Advertising: How Brands Scale Video Ads With AIInsights · November 18, 2026