Most fine-tuning guides answer “how many examples” and skip “how long should each one be.” That second question is the one that quietly decides whether your fine-tune helps at inference or fights it. Example length isn’t a property you inherit from your data — it’s a design choice, and the default (whatever length your dumped transcripts happen to be) is usually wrong in one of two expensive directions.
The framing I keep coming back to: the context window is a cache, not a memory. Fine-tuning changes what the weights know; it does not change the fact that at inference the model reasons over whatever you put in the window right now. Size your training examples to the window you’ll actually serve, or you’re training for a world you won’t deploy into.
The two failure directions
Too short is the sneakier one. Say your real requests arrive with 6–8K tokens of retrieved context, but your training examples are tidy 800-token snippets because that’s what your labeling tool exported. You’ve now fine-tuned a model whose learned prior is “the answer is near the top of a short prompt.” At inference you hand it 8K tokens and the relevant fact sits at position 5,000, and the model underweights it — not because the base model can’t attend that far, but because your fine-tune taught a length distribution that never occurs in production. You optimized the model onto a distribution you will never sample from.
Too long is the one that shows up on the invoice. Attention is quadratic in sequence length, so a training set of 32K-token examples doesn’t cost 4× a set of 8K-token examples — it costs closer to 16× per step in the attention term, plus the memory that forces you into smaller batches or gradient checkpointing, which slows you down again. Worse, long examples tempt you into teaching the model to memorize reference material that belongs in retrieval. You pay quadratic training cost to bake facts into weights that a RAG lookup would have served fresh, and now those facts are frozen at training time and go stale.
Match the training distribution to the serving distribution
The rule is boring and load-bearing: the length distribution of your training examples should match the length distribution of your production requests. Not the max, the distribution. If prod requests are lognormal with a median of 4K and a p95 of 12K, your training data should look like that too — a spread, not a single padded length.
Measure it before you build the set:
import numpy as np
# token counts of real production prompts (sample from logs)
lengths = np.array([count_tokens(p) for p in sampled_prod_prompts])
for q in (50, 90, 95, 99):
print(f"p{q}: {np.percentile(lengths, q):6.0f} tokens")
print(f"max sequence to train on: ~p99 = {np.percentile(lengths, 99):.0f}")
Set your training max_seq_len at roughly the p99 of production, not the max. The single 60K-token outlier request shouldn’t force every batch to reserve 60K of sequence budget; truncate or drop the long tail and handle it separately. And critically: don’t pad-and-collapse your examples to one length. Bucket by length so a batch of short examples trains cheaply and only the genuinely long batches pay the quadratic cost. Length bucketing is the single highest-leverage efficiency lever in fine-tuning and it’s routinely skipped.
Where the labels sit changes the sizing
There’s a second-order effect people miss. If your examples are long and the label (the tokens you compute loss on) is short and at the end — a classic “long context in, short answer out” shape — then most of the sequence is loss-masked context the model reads but isn’t scored on. That’s fine functionally, but it means your effective training signal per token is low: you’re paying to process 12K tokens to get gradient from 200.
Two consequences. First, you may need more examples than a short-answer intuition suggests, because each one carries little supervised signal relative to its cost. Second, this is often the signal that you should be retrieving that context at inference rather than teaching the model to condition on a specific long document — if the long part is reference material rather than the reasoning you want to instill, it belongs in the cache, not the memory.
Don’t fine-tune the summarizer’s mistakes in
If your production pipeline compacts context — summarizing earlier turns to fit the window — then your serving distribution includes compacted, lossy context. Your training examples had better include it too. Fine-tuning exclusively on full, un-compacted transcripts and then serving compacted ones at inference is another train/serve mismatch: you taught the model to rely on detail that your own pipeline strips before the model ever sees it in production. If you compact at inference, compact (a sample of) your training examples the same way, so the model learns to reason over the degraded input it will actually get.
What I’d actually do
- Sample real prod prompts and plot the length distribution first. Everything downstream keys off p50/p95/p99. Guessing here is guessing at the whole design.
- Set `max_seq_len ≈ p99 of production, and length-bucket batches.** Don’t let the tail dictate the batch, and don’t pay quadratic cost on short examples by padding them long.
- Match the shape, not just the cap. A spread of lengths that mirrors production beats one padded length, even if the padded length is “safe.”
- Ask whether the long part is reasoning or reference. Reasoning you want in the weights; reference you want in retrieval. Fine-tuning reference material is paying quadratic cost to freeze facts that go stale.
- If you compact at inference, compact your training data too. Train on the distribution you serve, degradations included.
Example length is a lever, and it’s one of the few in fine-tuning where the wrong default costs you on both axes at once — quality and dollars. Measure the serving distribution, then build training examples that look like it. The model can only learn the world you show it, and the window is that world.
Quadratic-in-sequence-length is the standard dense-attention cost model; architectures with sparse or linear attention change the constant but not the direction of the argument. Percentile targets and bucket boundaries are workload-specific — the method (match training length distribution to serving length distribution) is what transfers.