Benchmark Believers Welcome. Benchmark Skeptics, You Were Right.

29 Jul, 2026

The pitch

The idea was simple: take Qwen3-8B, fine-tune it on math problems where every single training example is generated and graded by symbolic computation — sympy, not an LLM — so the training data literally cannot hallucinate a wrong answer. No scraped datasets, no LLM-generated "synthetic" data quietly poisoning itself with confidently wrong reasoning. If sympy can't solve a problem and independently re-verify the solution, it doesn't make it into the dataset.

Full code and data pipeline: github.com/Dev-X25874/math-finetune-pipeline

That part worked. It's the rest of the story — the part where a legitimate-looking 85% accuracy turned out to be mostly fake, and a "let's just crush GSM8K" side quest took five training runs to actually work — that's worth writing down.

Round one: the pipeline, and a very good-looking number

The setup: six domains of math (arithmetic, linear equations, quadratic equations, derivatives, integrals, systems of equations), each backed by a sympy-templated generator and an independent verifier — a second function that re-parses the problem from scratch and re-derives the answer using a separate code path, so a bug in the generator can't just agree with itself.

First training run, first eval: baseline Qwen3-8B scored 14.9%. After LoRA fine-tuning: 85.1%. A clean 5.7x improvement. Genuinely exciting number.

It was also wrong.

The self-audit that ruined my day

Before trusting that number, I ran a check that should be standard practice and often isn't: I regenerated the training set and the eval set using their actual seeds, and checked for literal string overlap between them.

The result: up to 99.3% of the eval problems in the quadratic_equation domain were byte-identical to problems already seen during training. Not similar — identical. The cause was almost embarrassingly simple: my coefficient ranges were too narrow (a, b, c drawn from a space of about 867 possible distinct quadratics), and I'd asked the generator for 5000 training examples from that space. There literally weren't enough unique problems to go around, so a "held-out" eval set built from a different random seed wasn't held out at all.

The tell, in retrospect, was sitting right there in the per-domain breakdown the whole time. The one domain with genuinely zero overlap — linear_system — was also the worst-scoring domain, both before and after the leak was found. Every other domain's inflated score should have been suspicious precisely because it was so much better than the one domain that couldn't have been contaminated.

Fix: widen the parameter ranges (quadratic's problem space went from 867 to over 32,000), and add a hard --exclude-file dedup check that filters any eval prompt already present in the training set — verified directly, not assumed, before training even starts.

Round two: the honest number

Same pipeline, clean data, verified zero overlap (0/1800, checked by direct comparison, not by faith in wider ranges). Retrained.

Lower than the fake 85.1%, but real. A legitimate 6.5x lift, with the domains that got smaller improvements (quadratic_equation, linear_system) being honestly harder rather than artificially inflated. This felt like the actual result — solid, defensible, done.

It was not done.

The GSM8K gut-check

Out of curiosity — and because a benchmark you built yourself is a much easier bar to clear than a standard, external one — I ran the fine-tuned model against GSM8K, the widely-used grade-school math word problem benchmark. Different problem style entirely: natural language story problems ("Janet's ducks lay 16 eggs a day...") instead of rigid symbolic templates.

Base model: 83.0%. My fine-tuned checkpoint: 76.0%.

Fine-tuning made the model worse at general math reasoning.

This is not actually shocking, once you think about the training data. Every one of my 30,000 symbolic examples ends in a terse two-to-four-line answer: compute, state, done. The base model's natural style, by contrast, is long and exploratory — it reasons out loud, checks itself, sometimes rambles for 300 words before landing on an answer. Thirty thousand examples of "answer fast, minimal reasoning" is enough to meaningfully reshape a model's default response style, not just its knowledge. GSM8K word problems benefit from that longer, more careful reasoning; my training data actively discouraged it.

Not a bug. A real trade-off, and a useful one to understand: narrow fine-tuning can specialize a model in a way that costs it something elsewhere, even within the same broad subject.

The "let's fix it" saga (which took four more tries)

Attempt 1 (v3): Mix in 12,000 new synthetic word-problem examples, generated the same way as the symbolic data — sympy for ground truth, independently verified. Result: 59.0%. Worse. My new word-problem templates were also terse (two lines, "Final answer: X") — I'd added more training data reinforcing the exact behavior that was already hurting the model. Diagnosed correctly, fixed wrong.

Attempt 2 (v4): Stop writing my own word problems. Use GSM8K's actual official training split instead — 7,473 problems, completely separate from the 1,319-problem test set (verified: zero overlap, by direct comparison, exactly like the earlier audit). This is not training on the test set; it's the standard, intended way to use a benchmark with a train/test split. Mixed with the existing 30k symbolic examples. Result: 70.0%. Better than v3, still below base. The symbolic data — still 4x the volume of the GSM8K data — was dominating the mix and dragging the model's style back toward terseness.

Attempt 3 (v5): Drop the symbolic data entirely. Train purely on the 7,473 real GSM8K examples, three epochs. Result: 75.8% (confirmed on a larger 500-example sample, not just noise from a small one). Close to base, still short.

The bug that was actually costing the most: somewhere in this process I noticed something dumb — my eval script appended an instruction to every prompt ("Solve step by step, then end with: Final answer: ") that the training prompts never included. The model was being evaluated on a slightly different prompt shape than the one it trained on. A real train/inference mismatch, hiding in plain sight.

Attempt 4 (v6): Same GSM8K-only training, but with training prompts now formatted identically to the eval prompts. Result:

A real win — beats base by 4.4 points, using only real, non-contaminated training data, no synthetic tricks. The fix wasn't "train harder," it was "stop feeding the model inconsistent formats and see what it can actually do."

The number I haven't beaten yet

One more honest thing worth saying, because it would be easy to stop the story here on a high note. Digging into how Qwen's own team (and independent researchers) report GSM8K performance for this model, the picture gets more humbling: when Qwen3-8B is allowed to reason at length — 2,000+ tokens of thinking per problem — independent evaluations put it at 95–96% on GSM8K. My eval setup capped generation at 1,500 tokens. It's entirely possible a meaningful chunk of my "wrong" answers, on both the base and fine-tuned models, are actually answers that got cut off mid-reasoning, not answers that were genuinely incorrect.

I haven't checked this yet. It's the obvious next thing to check, and I'm noting it here specifically because it would have been easy to declare victory at 83.8% and never look for the ceiling above it.

What actually happened here, in order

1. Built a synthetic data pipeline with real safeguards (independent verification, not self-consistency).
2. Got an exciting number (85.1%) that was mostly a data leakage artifact.
3. Caught it myself, by actually checking instead of trusting the pipeline design.
4. Got an honest, much less exciting, more real number (69.7%).
5. Checked the result against a standard external benchmark and discovered a real regression (83.0% → 76.0%) that the internal benchmark had no way of revealing.
6. Diagnosed the cause correctly (style/format overfitting) but fixed it wrong twice before fixing it right.
7. Found a legitimate win (83.8%) — while now knowing there's likely another 10+ points sitting behind a token-budget ceiling I haven't tested yet.

None of the individual mistakes here are exotic. Data leakage from a too-small parameter space, a narrow training distribution reshaping model behavior in ways the target metric doesn't capture, a silent prompt-format mismatch between train and eval — these are common, well-documented failure modes in ML work generally. What made the difference wasn't avoiding them; it was checking for them directly, with code, rather than trusting that the pipeline was fine because the headline number looked good.

The uncomfortable, useful takeaway: the biggest jumps in this whole project — finding the leak, finding the regression, finding the prompt mismatch — all came from actively trying to prove the current result was wrong, not from trying to make it look better.