When Your Metric Improves and Your Product Gets Worse

The training run itself was the part that went to plan. I fine-tuned Llama 3.2 3B with QLoRA — 4-bit NF4 quantization, LoRA rank 16 — on an 8 GB RTX 3070 laptop GPU, using 10,752 emotional-support conversations. Validation perplexity fell from 32.6 for the base model to 9.79 for the fine-tune: a drop of roughly two thirds, and comfortably past the success criterion I had written down before touching any data.

By my own definition of success, the project had worked. The model sounded warmer, stayed in role, and predicted held-out supportive dialogue far better than the base model. If perplexity had been the only thing I measured, this post would be a victory lap — and I would have shipped a worse product.

Alongside perplexity I ran an 18-prompt behavioural suite split into crisis, control, and boundary prompts — the questions where the right answer is not a warmer answer. The crisis results were the ones that mattered: the fine-tuned model's crisis pass rate fell to 25%, against 62.5% for the base model I had started from. My training run had made the model measurably less safe than doing nothing at all.

The mechanism, once I saw it, was obvious. The empathy data taught the model warmth, and warmth is exactly the wrong register for a crisis message. Where the base model would break role and point to help, the fine-tune had learned to stay soft, reflective, and conversational — because that is what ten thousand supportive conversations rewarded. The regression wasn't a bug in the pipeline. It was the pipeline doing precisely what I asked.

The first repair was the one every ML practitioner reaches for: more training. I built a targeted safety-patch set and ran a second pass, and it worked as far as training can — the crisis pass rate recovered to 75%, beating the base model's 62.5%. That is a genuinely better model, and it is also not an acceptable answer. When the failure mode is a person in crisis getting a soft deflection instead of a crisis line, a number that starts with 7 is not a number to celebrate.

So the second repair moved the requirement out of the model entirely. Haven's server now has a crisis layer that detects crisis messages and guarantees — regex-verified, on every response — that real resources appear: 988, text 741741, 911. The model still generates the reply, but the property I actually care about no longer depends on the model. No future fine-tune, no new dataset, no clever prompt can trade it away, because the training loop cannot touch it.

The general lesson took me longer to phrase than to learn: iteration produces expected behaviour, not guaranteed behaviour. Training pushes a distribution around; it does not sign a contract. That is fine for the requirements you are optimizing — tone, fluency, helpfulness — and disqualifying for the requirements you are promising.

So the discipline I took from Haven is to name, before training, which requirements are optimization targets and which are constraints — and then to put the constraints somewhere no future training run can trade them away. Targets live in the loss function and the eval dashboard. Constraints live in code, at a layer the model cannot reach. If a requirement would be a headline when it fails, it should not be a metric; it should be architecture.

The same week handed me a smaller version of the same lesson from Haven's other model. My DistilRoBERTa emotion classifier, scoring 28 labels per message, looked respectable at a flat 0.30 decision threshold: micro-F1 of 0.596. But macro-F1 — where every class counts equally — was 0.395, below my TF-IDF baseline's 0.411, and several rare classes, including grief and nervousness, sat at an F1 of exactly zero. For a mental-wellness app, grief is not a class you are allowed to miss.

Per-class threshold tuning fixed it honestly: micro-F1 eased to 0.563 while macro-F1 rose to 0.433, beating the baseline on both. I traded a little average performance for the classes that matter. It is the same failure shape as the perplexity story, one level down — averages hide exactly the failures that matter most, and any single headline number is an average of something.