I have spent five years building production systems, where the rules are clear: everything is logged, everything is version controlled, and if something cannot be reproduced, it is treated as broken. When I started training models in PyTorch, I noticed how easy it is to drop all of those habits. You run a script, watch numbers scroll by, get a final accuracy, and a week later you cannot say which settings produced it.
So when I built my CIFAR-10 training harness (github.com/JonMirzoDev/vision-baselines), I decided to treat it like a small production service from the first commit. This post is about what that means in practice, and about the moment it paid off.
Seed everything, and mean it
PyTorch training has at least three sources of randomness: Python’s random, NumPy, and torch itself. Weight initialization, data shuffling, and augmentation all draw from them. Controlling one and not the others gives you runs that are almost reproducible, which is worse than not reproducible at all, because it invites you to trust comparisons that are actually noise.
def set_seed(seed: int) -> None:
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
Four lines. The seed is an ordinary CLI argument with a default (--seed 42), so every run is seeded whether or not I think about it.
Metrics are artifacts, not console output
The training loop prints progress, but printing is not logging. Every run writes a metrics.csv with one row per epoch: train loss, test accuracy, learning rate. The file lands in a per-run directory next to the best checkpoint.
epoch,train_loss,test_accuracy,lr
1,2.0050,0.2974,0.09973
2,1.8376,0.3379,0.09891
3,1.7197,0.4084,0.09755
This is the same instinct as shipping metrics from a service instead of grepping its stdout. A CSV can be diffed, plotted, and committed. A scrollback buffer cannot. When my LeNet run finished at 80.25% test accuracy after 30 epochs, that number was not a claim from my memory. It was row 30 of a file in the repository.
The interruption that proved the point
Here is the part that made me trust the setup. My first LeNet run got killed at epoch 21, because training on a laptop competes with the laptop’s actual job. Two weeks later I restarted the run from scratch.
First epoch of the original run:
epoch 1/30 loss 2.0050 test acc 0.2974
First epoch of the rerun, weeks later, same machine:
epoch 1/30 loss 2.0050 test acc 0.2974
Identical to four decimal places. That is what seeding buys you: an interrupted experiment is an inconvenience, not a loss. In software terms, my training run was idempotent. I did not have to wonder whether the rerun was comparable to the one that died. It was the same run.
Honest results tables
The repository README has a results table. While a run has not happened yet, the cell says “run pending”. It would take thirty seconds to type a plausible number instead, and plenty of tutorial repositories effectively do that by copying numbers from papers. I think the discipline matters more in ML than in normal software, because in ML the result is the product. A results table you cannot trace to a run is a bug report waiting to happen.
This is also the habit I would want reviewing someone else’s work: show me the metrics file, the seed, and the command. My whole harness is:
python train.py --model lenet --epochs 30
python train.py --model resnet18 --epochs 60
Everything else (SGD with momentum, cosine learning-rate schedule, standard CIFAR augmentation) is visible in one file of about 130 lines.
What transfers from software engineering
None of this is novel ML. It is ordinary engineering discipline applied to experiments: deterministic behavior, artifacts over console output, honest state reporting, small reproducible units of work. Those habits are cheap to keep and expensive to retrofit.
The machine learning itself, I am still learning, and the repository says so openly. But I have noticed that the parts of research practice that fail most often in the wild (results that cannot be reproduced, evaluation that leaks, numbers with no provenance) are exactly the parts that engineering already knows how to solve. That overlap is a big part of why I am making this move.