We wrap captions with the TeX algorithm — but the penalty is silence

Auto-generated captions tend to share a tell. They break in the wrong place:

I saved twelve euros with Too
Good To Go last night

Nobody talks like that. The brand name is split, the line ends on a preposition, and you have to reassemble the sentence as you read it. The captioning tool did what it was told: fill the line until the next word won't fit, then wrap. It optimized for the width of the box.

But the box isn't the point. A caption is read in a second or two, one or two lines at a time, while a person is also listening. Where you break the line is a reading decision — and the wrapper ignores the signal that would make it a good one.

This post is about that signal, and the algorithm we use to act on it.

Greedy wrapping optimizes the wrong thing

Start with plain line wrapping, no timing yet. The naïve approach is greedy: pack each line as full as it goes, wrap, repeat. It's what most tools do, and it produces the familiar lopsided caption: a full first line and a near-empty one below.

Take "Welcome to our brand new feature update." at a typical vertical-video width. Greedy fills the first line with five words and drops two:

Greedy:            slack² = 51,104
  Welcome to our brand new
  feature update.

Balanced:          slack² = 37,064
  Welcome to our brand
  new feature update.

The fix is 45 years old and has a paper. In 1981, Donald Knuth and Michael Plass published "Breaking Paragraphs into Lines" — the line breaker they had just built into TeX. The key move: instead of deciding one line at a time, it "considers the paragraph as a whole," assigns each candidate line a badness that grows steeply with its leftover space, and lets dynamic programming pick the set of breaks with the least total badness — no backtracking, no orphaned tail.

The backstory is a long detour. In 1977 Knuth saw the phototypeset galleys for a new edition of The Art of Computer Programming, didn't want to put his name on them, and paused the books to build his own typesetting system. The detour became TeX and took the better part of a decade; the line breaker inside it, worked out with Plass — then his PhD student at Stanford — is still what LaTeX uses today, and still the version most people port.

We run a reduced version of that: Knuth–Plass without the hyphenation and justification machinery, because captions are short and ragged-right. Badness is slack² — the squaring does the same job as the paper's steeper curve: two half-empty lines score better than one full line and one near-empty one, so the text spreads out evenly. For the sentence above the DP prefers 4+3 over greedy's 5+2 because 37,064 beats 51,104.

Remember: balancing line width only makes captions look tidy. It says nothing about whether the line breaks where the sentence does.

The best place to break a line is where the voice stops

We have an optimizer that minimizes a cost, and so far the only cost is slack — how full each line is. But slack is just geometry. What we want is to break where a human would: at the end of a thought, in the space between phrases.

The transcript already carries a good signal for where those are: the gaps between the words. Every word arrives with a start and end time, so the silence between any two words is free to read off. Speakers tend to pause at the boundaries a reader wants to see — the ends of clauses, the beat before a punchline.

So we fold timing into the cost function, using the model from the 1981 paper. It has three primitives: "boxes" (the unbreakable things, with widths), "glue" (the stretchable space between them), and "penalties" (a signed charge on each place you might break — positive discourages, negative invites). Our boxes are the measured word widths, our glue is the space width, and our penalties are these — a bonus or a charge for what kind of boundary it is:

// Negative invites a break, positive forbids one. The absolute scale is
// arbitrary; only the ordering matters. gapCost is unbounded — it scales with
// the length of the silence — so a long enough pause outweighs everything else.
const LINE_COST = {
  gapThreshold: 0.3, // ignore pauses shorter than 300ms
  gapCost, // − per second of silence past the threshold
  tupleCost, // + never split a brand name (more below)
  numberUnitCost, // + never split "500 ml" (more below)
  sentenceEndCost, // − ". ! ?" — good place to break
  clauseBreakCost, // − ", ; :" — decent place to break
  beforePunctuationCost, // + don't strand a lone "—" at the start of a line
};

Negative pulls the break toward a boundary; positive pushes it away. A pause is scored per second past a 300ms threshold, so a two-second gap pulls much harder than a half-second one — hard enough to overpower almost any amount of slack. The absolute scale doesn't matter; what the layout depends on is how these compare to each other.

In 45 years the algebra hasn't changed. What changed is where the numbers come from: in TeX the penalties come from the markup. In ours they come from the recording.

The result: take "Wait a sec… [two second pause] …then I'll begin." It fits comfortably on one line, so width wrapping would leave it there. But two seconds of silence in the middle outweighs any amount of slack the split costs, so the layout breaks it in two and puts the break in the silence, where the speaker put it:

Width says:                     Timing says:
  Wait a sec then I'll begin      Wait a sec
                                  then I'll begin

The caption now follows the voice rather than the viewport. That's the whole idea, and it comes out of the optimizer without a special case — we didn't write a rule that says "break at pauses," we told the cost function that silence is cheap to break on and let the DP find the rest.

Remember: the signal for where a human would break a line is already in your data — the pauses in the audio. You don't need a model to guess it; you need a cost function that will spend it.

Some things must never break, in any language

The flip side of "silence is cheap to break on" is that some boundaries are expensive. You've seen this one too:

It's only
500 ml

…except split as 500 / ml. A number and its unit are one atom; so is a currency symbol and its amount; so is a temperature. Every one of those boundaries gets the heaviest fixed charge in the table — the same order of magnitude as a long pause pulls the other way — which in practice means "never break here unless there is genuinely no alternative."

TeX has this concept too, with one difference: there, any penalty of 10,000 or more is treated as infinite — a hard never. We kept the magnitude and dropped the infinity. Ours is a very large finite number, and that difference matters, as the next section shows.

The catch is that "a number and its unit" has to work in the languages our customers actually caption in, which is most of them. So the dictionary is deliberately broad:

// currency, either side of the number: $5 and 5 €, 12 £, ¥300, ₹99…
const CURRENCY_SYMBOLS = new Set([
  '$', '€', '£', '¥', '₹', '₽', '₩', '¢', '₪', '₫', '₴', '₦', '฿', // …24 total
]);

// SI + data + imperial suffixes: 500 ml, 25 mg, 4 km, 8 GB, 5 lbs…
const UNIT_SUFFIXES = new Set([
  'mm', 'cm', 'km', 'µm', 'mg', 'kg', 'µg', 'ml', 'cl', 'dl', 'hl', 'ms',
  'hz', 'khz', 'mhz', 'ghz', 'kw', 'kwh', 'kb', 'mb', 'gb', 'tb', 'gib',
  'mbps', 'mph', 'km/h', 'm/s', 'rpm', 'dpi', 'px', 'ft', 'yd', 'oz',
  'lb', 'lbs', 'gal', // …and more
]);

Plus a percent sign, per-mille, and a temperature regex so 37 °C stays whole. Numbers are matched with grouped thousands and either-side decimals, so 1,000.00 and 1 000,00 both read as one token. It's unglamorous code, and you only notice it when it fails — a caption that reads € on one line and the price on the next is a bad look in a paid ad.

Make your constraints costs, not rules

One design decision here came out of a real bug.

Ticket SOL-4264, from a customer's footage: the brand "Too Good To Go" was getting split across lines. Easy enough — add it to a keep-together dictionary and charge any break inside the phrase exactly like a number-unit split.

Except a keep-together dictionary collides with everything else the optimizer knows. Three cases came up:

1. It fits. "Try Too Good To Go tonight" — the brand sits on one line and the penalty is never charged.
2. It doesn't fit, but the page can hold it. On a narrow phone render "Too Good To Go" has to wrap to two lines — and if there's a 1.5-second pause in the middle of the brand, the pause wants to break the caption onto a new page right there, cutting the brand in half across a screen transition. The keep-together cost is stronger than that pause, so both halves stay on the same page. Good.
3. The "brand" is a coincidence. Same setup, but now there's a fifteen-second gap between "Good" and "To." Two words fifteen seconds apart are almost certainly not a brand name — they're a transcription artifact, or someone said "good" and then, much later, "to." Here we want the split.

Case 3 is why the penalties are finite. If "keep this brand together" were a hard constraint — a rule, TeX's infinity — case 3 would hold a phantom brand together across a fifteen-second gap. Because it's a cost instead, a pause of that length outvotes it by a wide margin. Both extremes work without a special case: real brands stay together, and misclassified ones come apart once the evidence for a break is strong enough.

Those three cases aren't hypotheticals. They're the regression suite, verbatim — same brand, same 1.5-second pause, same fifteen-second gap. The ticket closed; the transcript stayed behind as tests.

Remember: prefer costs to rules. A rule can't tell a brand name from a coincidence. A cost can be outvoted when the data disagrees with it — which is exactly what you want the first time your dictionary is wrong.

One dynamic program, run twice

It takes less machinery than it sounds like. Lines and pages are the same problem at two scales — partition a sequence into contiguous groups at minimum total cost — so there's exactly one solver, and both layers call it:

// Least-cost partition of n items into contiguous groups (Knuth–Plass, no hyphenation).
// minimise  Σ groupCost(group) + Σ breakCost(internal boundary)
function partitionLeastCost(n, feasible, groupCost, breakCost) {
  const dp = new Float64Array(n + 1).fill(Infinity);
  const parent = new Int32Array(n + 1).fill(-1);
  dp[0] = 0;
  for (let end = 1; end <= n; end++) {
    const closing = end < n ? breakCost(end) : 0;
    for (let start = end - 1; start >= 0; start--) {
      if (!feasible(start, end)) continue;
      const cost = dp[start] + groupCost(start, end) + closing;
      if (cost < dp[end]) {
        dp[end] = cost;
        parent[end] = start;
      }
    }
  }
  // …walk `parent` back to recover the group starts.
}

Breaking words into lines calls it with groupCost = slack², breakCost = the boundary penalties above, and feasibility = "the line isn't wider than the caption box" — with one escape hatch: a single word wider than the box is still feasible, because no other layout would fit it either. Grouping lines into pages calls the same function, with the same boundary classifier, in a different currency. Line penalties have to outweigh slack², which is counted in pixels squared, so they run large. Page penalties have nothing to outweigh — groupCost is zero — so the whole scale collapses:

// Same shape, its own scale — page costs never compete with slack².
const PAGE_COST = {
  base, // + every page break costs a little…
  gapThreshold: 1.0,
  gapCost, // − …but a pause past a second more than pays for it
  tupleCost, // + and a brand still refuses to straddle a page
  numberUnitCost, // +
};

A pause a little past the one-second threshold makes a page break free; anything longer makes it profitable, so pagination drifts toward the pauses on its own. Same twenty lines of DP, two different jobs, by swapping the three cost functions.

One small touch in the solver itself: on a cost tie, it starts the final group as late as possible, so when something has to be short, it's the last line. A short last line reads as an ending; a short middle line reads as a mistake.

One last detail. After pages are chosen, each page's on-screen time is stretched to reach into the silence before the next one — capped at 300ms — so a caption doesn't disappear the instant the last word ends.

And when you export, none of this runs again. The optimizer's verdict — pages, lines, words — is baked into the composition document itself, and the headless C++ renderer that produces the final video just draws those pages. There is no second layout engine to keep in sync, no server-side re-wrap to drift from the preview. The caption you approved is the caption you ship — the same data, not a re-computation of it.

Captions aren't laid out. They're scored.

The width of the caption box, the thing a naïve wrapper optimizes, turns out to be one of the less important constraints. Once you frame layout as minimize a cost rather than fill a line, the things you care about — break where the sentence breaks, never split a price, keep the brand whole unless it isn't one, linger into the pause — each become another term in the cost function. The 1981 algorithm does the search; the judgment is in what you make cheap and what you make expensive.

If there's one thing to steal from this: the next time you're tempted to write layout rules, write a cost function instead. Rules fight each other and you arbitrate the collisions by hand. Costs add up, and the optimizer arbitrates them for you, for the price of picking the right numbers.

The best caption break is the one you can hear, and it's already in the audio.

Solid is an AI video editor. If turning fussy craft problems into clean optimization problems sounds like your idea of fun, we're hiring.