Soup du Jour.

We're cooking, and today we made model soup. We trained three versions of a model, and averaged its weights into one. It bumped it's accuracy from 77.8% to 92.2%. This is our recipe.

01 / The recipe

First, what is model soup?

Everyone’s me-ma had a soup. The recipe was nothing: chicken, carrots, salt, a bay leaf she fished out later. What made it hers was knowing how much of each to throw in. Post-training a model is the same job, except me-ma doesn’t have to deal with PyTorch.

Model soup is what you get when you train the same model a few times and blend the weights.

No new training run, no new data. Just take the versions you already made, blend them, and taste. That’s it. It’s an old ML trick from the paper “Model Soups” by Wortsman et al. (ICML 2022).

Naturally you’re asking: why doesn’t this just produce garbage? Because every version started from the same pretrained base. Fine-tuning only nudges the weights, and all the nudges land in the same valley of the loss landscape. Average three points in one valley and you’re still in the valley, closer to the middle, which is where the good ingredients are. Our ingredients: three runs of one recipe, identical except for the random seed.

One bad apple can ruin the bunch

We fine-tuned Qwen3-1.7B to find the right passage in a contract and answer fixed-choice questions about it. Before training, we set five checks the model had to pass. No moving the goalposts after kickoff. The first run passed four.

On one check, it scored 0.612. The bar was 0.80. It missed by 0.19, which means into the garbage it goes. So we ran the identical recipe twice more and changed nothing but the seed. The next one came back at 0.878, and the next one after that came back at 0.952. Same recipe, same data, same GPU. Winner winner chicken dinner.

02 / A better model

The soup became our new model.

Equal parts of each seed, including the one we threw in the garbage. A third, a third, a third.

The soup scored 0.878 on fiduciary duty and 84.5% on material-adverse-effect questions. It passed all five checks, so it became the new model. It also came out better calibrated than any of its ingredients: expected calibration error of 0.0087, against 0.014 for the run we threw in the garbage.

Did it keep the good stuff?

On held out tests, the soup put a correct passage in first place for 92.2% of 320 contract–clause pairs. The previous model hit 77.8%. False flags per contract fell from 49.8 to 12.2.

03 / Check the temperature

We over-seasoned it.

Before evaluating the soup, we got fancy. We refit its calibration temperatures on 2,000 training examples. The model had already seen those, so they looked easy. Calibrate a model on easy questions and it gets cocky on everything else. Expected calibration error came in at 0.0296. Our limit was 0.03. It passed by 0.0004, which is the eval version of making your flight because the flight was also late.

Then we threw out the refit and used the soup’s own blended temperatures, computed from the same saved predictions. ECE: 0.0087. Three times better, for doing less.

Calibrating on training rows is asking the cook whether dinner was good. Recalibrate a soup on held-out data, or leave it alone.