I have been troubled by the randomness of ImageNetVID, and finally found the reason. In some versions of python, e.g. python 3.5, the cache files in JSON format will be unorderly loaded. This won't happen when we use python 2.7 or python 3.6. This greatly prevented me from reproducing my experiments, since the random order of the training data between runs will lead to different gradients in early epochs when training SiamFC. I suggest we may cache the dataset in a more stable way, e.g. using numpy or cpickle, and using OrderedDict or else.

Details:

import json
seq_dict = json.load(open('imagenet_vid_train/seq_dict.json', 'r'))
seq_names = [n for n in seq_dict]
print(seq_names[0])

gives ILSVRC2015_train_00000000.0 twice when using python 3.6 (completely in order),
gives ILSVRC2015_train_00646001.0 twice when using python 2.7 (not in order but repeatable),
but gives ILSVRC2015_train_00053009.1 and ILSVRC2015_train_00047000.2 when using python 3.5.