My diabetic retinopathy classifier reports 95.2% accuracy on 231 held-out test images. The natural reading is that this is the best model from training, the one with the highest validation accuracy; the notebook even prints “Best Validation Accuracy” just before testing. While writing up the project, I reread the training loop to explain exactly how that model was chosen, and found that it wasn’t the model the code claimed to restore.

The number did not change. What changed is which model it describes.

The setup

The model is an ImageNet-pretrained ResNet-18 with its backbone frozen and a small new head (512 → 256 → 2) trained for 20 epochs. A common pattern is to track validation accuracy each epoch, remember the weights from the best epoch, and restore them before the final test. Here is that part of the training function, copied from the notebook:

if phase == 'valid' and epoch_acc > best_acc:
    best_acc = epoch_acc
    best_model_wts = model.state_dict()

print(f"\nBest Validation Accuracy: {best_acc:.4f}")
model.load_state_dict(best_model_wts)
return model

It reads correctly. It even prints a reassuring line at the end.

What actually happens

model.state_dict() does not return a copy of the weights. It returns a dictionary of tensors that share memory with the model’s parameters and buffers. When the optimiser later updates the parameters in place, the tensors in best_model_wts change with them, because they are the same memory.

Two things keep changing after the best epoch:

  1. The head’s weights, updated by every optimizer.step().
  2. BatchNorm running statistics in the backbone. The backbone’s weights are frozen, but calling model.train() still updates BatchNorm’s running mean and variance on every training batch, and those buffers are part of the state dict too.

So by the end of training, best_model_wts holds the final weights and statistics. model.load_state_dict(best_model_wts) copies the model’s current state onto itself and changes nothing.

The evidence in the log

The notebook’s saved training log makes the gap visible:

Epoch Validation accuracy Role
17 96.99% Highest validation accuracy, printed as “Best Validation Accuracy: 0.9699”
20 93.79% Final epoch: the weights actually tested

Replay training · validation accuracy per epoch

93%94%95%96%97%15101520live weightsdeepcopy
best_model_wts = model.state_dict()

What the notebook kept: a reference to the live weights. It changes every epoch.

Holds epoch 20 · validation 93.79%

best_model_wts = copy.deepcopy(model.state_dict())

The fix: a frozen copy, taken only when validation improves.

Would hold epoch 17 · validation 96.99%

Outcome · after epoch 20

model.load_state_dict(best_model_wts) restored the epoch-20 weights (validation 93.79%). Those were tested: 95.2% (220 of 231), confusion matrix 103 · 10 · 1 · 117.

The epoch-17 checkpoint (96.99% validation) was never evaluated on the test set, so its test accuracy is unknown.

Drag (or use the arrow keys) to replay training. Larger dots mark epochs where validation accuracy improved.Source: notebook training log and test output

The printed message describes a model that, by the time it was printed, no longer existed anywhere in memory. The test evaluation that follows ran on the epoch-20 weights, and the notebook then saved those same weights with torch.save. The Flask app loads a file with the same name (retinopathy_model.pth), so, assuming it is that file, the app serves the final-epoch model too.

What the 95.2% means now

The result is still a legitimate measurement: the final-epoch model scored 95.2% (220 of 231) on a test set it never saw during training, with 91.2% sensitivity and 99.2% specificity. What I can no longer say is that it is the best-validation model. The epoch-17 model was never evaluated on the test set, so I make no claim about how it would score. It could be better, worse or the same; validation accuracy on its own does not tell us.

Why this matters more than one line of code

  • Provenance is part of the result. A metric is only meaningful together with the exact artefact it was measured on. Here the metric was right and the description of the artefact was wrong.
  • Model selection is part of the method. “Best validation epoch” is a methodological choice. If it silently does not happen, the method on paper and the method in code diverge.
  • The failure is silent. No error, no warning, a confident log line. The only way to catch it is to verify the path from training to the evaluated weights.
  • It affects deployment too. The saved file, and therefore the app, inherits whatever the training function returns.

The fix, and how to make it hard to repeat

The minimal fix is to copy:

import copy

if phase == 'valid' and epoch_acc > best_acc:
    best_acc = epoch_acc
    best_epoch = epoch + 1
    best_model_wts = copy.deepcopy(model.state_dict())

Better still is to write the checkpoint to disk at the moment it is best, with the metadata needed to identify it later:

torch.save(
    {"epoch": best_epoch, "val_acc": float(best_acc), "state_dict": model.state_dict()},
    "checkpoints/best.pt",
)

and to make the training run say which weights it is returning, for example by logging best_epoch next to the test result. A one-line check after restoring, asserting that the restored head weights differ from the final ones whenever the best epoch is not the last, would have caught this immediately. Framework utilities that handle this for you (such as checkpoint callbacks that restore the best weights) exist for exactly this reason.

What I took from it

I now treat “which weights produced this number?” as a question every reported metric must answer, and I verify the answer in code rather than trusting a variable name. The next step for this project is simple: apply the fix, re-run, and report the best-validation model and the final model side by side. Until then, the number stays labelled for exactly what it is.

The full project, including the confusion matrix and training curves, is in the diabetic retinopathy case study.