Download data files at runtime or in an installation script

[rouson]

I just uploaded mnist.tar.gz to GitHub file storage in way that is persistent without being part of the git repository. I recommend against storing binary files in a repository. Over time, they increase download times and because git can't do useful diffs on binary files, every change to the file means a completely new version must be stored. What's worse, git rm doesn't take any of the committed versions of the file out of the commit history so the only way to fix the problem is to rewrite history, which essentially means that everyone who has a local copy of the repository will need to do a fresh clone and if having everyone do a fresh clone is not practical, then it's probably necessary to set up a new repository.

I recommend either

1. Use Fortran's intrinsic execute_commnand_line() subroutine to download and uncompress the file from the above location at runtime if it's missing or
2. Provide a short script that users can run to handle basic set up tasks (including downloading data files) and then build and test the repository's software.

On the OpenCoarrays project, I found that our downloads went up by a factor of 2-3 soon after I wrote an installation script. More recently, I settled on a much simpler approach to writing an installer for Caffeine that is an order of magnitude smaller than the OpenCoarrays installer, more robust, and much more maintainable. I'll be glad to adapt Caffeine's install.sh script to neural-fortran if you like. When someone can get a package built and tested by typing nothing more than ./install.sh, it saves a lot of time over reading build/test instructions, no matter how simple those instructions are.

[milancurcic]

I agree and sounds good. I didn't know that GitHub offered file storage, thanks for uploading it.

I'm fine with either of the two approaches. Option 1 seems more attractive, because option 2 would be doing much of what fpm and CMake are already doing. So install.sh would be downloading the data automatically, but for fpm we'd still need separate instructions for manual download of the data. Alternative would be to have a script to download the data.

Option 1 would also have a nice UX that Keras has with its toy datasets. It works just like you wrote--the first time you invoke the function to load the dataset, it's downloaded to a conventional directory, and loaded from disk on any subsequent call. Even if you decide to add the script, we'll be looking at this option long-term.

[milancurcic]

Another good reason to do this with the dataset is that with the addition of CNNs and hopefully other architectures (RNNs, etc.), I'd want to add several other toy datasets.

[milancurcic]

Another improvement that can be made with the MNIST dataset is to store it as int8s instead of real32.

[rouson]

Sounds good. I'll work on Option 1.

[rouson]

The file storage is not something GitHub advertises as a standalone feature. It's more of repurposing of an existing feature. Zaak Beekman (@zbeekman) gave me the idea: dragging a file into an issue comment causes GitHub to upload the file and generate a URL for it. The URL starts with the repository's URL so I suspect the file lives as long as the repository does. At least I've never seen one disappear. ☺️

[milancurcic]

That's a cool trick. Let's use it, and if it ever shows to not persist, we can set it up permanently on a CDN.

[rouson]

Fixed by PR #55