SEED(3407) IS ALL YOU NEED torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision David Picard [email protected] LIGM, Ecole de

SEED(3407) IS ALL YOU NEED
torch.manual_seed(3407) is all you need: On the influence of
random seeds in deep learning architectures for computer
vision
David Picard [email protected]
LIGM, Ecole des Ponts, 77455 Marnes vallée, France
я
i=)
я Abstract
In this paper I investigate the effect of random seed selection on the accuracy when using
popular deep learning architectures for computer vision. I scan a large amount of seeds (up
to 10*) on CIFAR 10 and also scan fewer seeds on Imagenet using pre-trained models to.
investigate large scale datasets. The conclusions are that even if the variance is not very
large, it is surprisingly easy to find an outlier that performs much better or much worse
than the average.
‘Keywords: ‘Deep Vision; Randomness
9
7)
1. Introduction
In this report, I want to test the influence of the random generator seed on the results of
ubiquitous deep learning models used in computer vision. I ask the following question:
1. What is the distribution of scores with respect to the choice of seed?
2. Are there black swans, seeds that produce radically different results?
5 3. Does pretraining on larger datasets mitigate variability induced the choice of seed?
I feel this questions are important to ask and test because with the rise of large training sets
and large deep learning models, it has become common practice in computer vision (and
2 to other domains relying heavily on machine learning) to report only a single run of the
experiment, without testing for its statistical validity. This trend started when computing
я power was limited, and it is perfectly understandable that the result of a single experiment
is better than no result at all. To the best of my understanding, this seems to happen all
the time in physics first results get quickly announced, and are later confirmed or denied.
‘The deep learning community has what may be a faulty approach in that it never confirms
results running as many reproductions as required. Just as the noise in the measurement
of a physical experiment, we know there are random factors in a deep learning experiment,
like the split between training and testing data, the random initialization, the stochasticity
of the optimization process, etc. It has been shown in Mehrer (2020) that these
affect the obtained networks such a way that they differ measurably. I strongly believe
it is important to have a sense of the influence of these factors on the results in terms of
evaluation metrics. This is essentially what was done Bouthillier (2021) on different
tasks, including computer vision, but limited to smaller datasets and only 200 runs. Instead
1