This repository houses the configuration files and scripts for building a TU Delft Conda-based standalone offline installer that includes annually requested software packages. This is suitable for installing on TU Delft's lab computers including those used for digital examinations.

Current:

• Jason K. Moore, ME, j.k.moore@tudelft.nl
• Jan-Maarten Brockhoff, ICT, J.J.M.Brockhoff@tudelft.nl
• Kevin Geboers, Workplace Support Education, k.f.geboers@tudelft.nl

Past:

• Bart Gerritsen, EEMCS, b.h.m.gerritsen@tudelft.nl

Managing a university-wide software distribution takes a large effort. At the minimum, we ask that you contribute by opening issues in this repository to report bugs and software/feature requests. At the maximum, help to collaboratively improve the distribution through merge requests and testing the build and installation. We will do our best to address issues and merge requests in a timely manner and will also provide commit rights to those that would like to help maintain this repository. Reach out if that interests you.

Version numbering schema: v<year>.<two digit integer>.

• Make open source scientific software available in TU Delft computer labs
• Make open source scientific software available for use during TU Delft computer exams
• Software should be at the latest versions (latest within ~18 months) and mutually compatible
• Installation must primarily support Windows (exam computers), but should also install on Linux and Mac OSX
• Installation must be scriptable on Windows
• Software installation must occur on an air-gapped computer (no internet)

We create a Windows executable installer for a collection of Conda packages using Conda Constructor and script its installation for batch deployment on Windows. We install a set of software packages from Conda Forge and make them available via the Conda package manager.

The goal of agreeing on a joint installation of a conda-based distribution is multifold:

1. to avoid a cluttering of (slightly) different Python environments across the infrastructure within the TU Delft
2. to avoid intricate and disturbing differences between Python environments used during instruction and exams
3. to mitigate the burden of maintaining the configurations involved

This installation is revised and updated in a yearly cycle. ICT-WPS ("Werkplekbeheer") receives an installation instruction, drafted on behalf of all faculties by the team that runs this repository, taking care of the actual implementation of a prepared package. This package may then be deployed through https://software.tudelft.nl and is at the basis of the implementation of the DigitalExam environment.

The (annual) process flow is roughly as follows:

If you need a conda package that is not available in Conda Forge installation, you can request inclusion of the package. The first step is to check if your package is available in Conda Forge. Search this list to check (note that package names may differ from PyPi, CRAN, etc.). If your package is not present in Conda Forge, then you should initiate a new package build following the instructions to add a recipe.

Once the package is available on Conda Forge for Windows 64 bit systems, open a merge request on this repository adding your package name to the construct.yaml file. Test building the installer locally on Windows and report your success (or failures) in the pull request.

My software is not available on the exam computers. How do I make it available?

The first step is to ensure that your desired package can be installed from Conda Forge. Search https://conda-forge.org/packages to see if your package is already available and if not you will need to add a recipe following the instructions here: https://conda-forge.org/docs/maintainer/adding_pkgs. Once the package is available on Conda Forge, open a merge request on this repository adding your package to the construct.yaml file. Make sure to submit a merge request here months before the freeze deadline for the new academic year.

The package I need is missing from Conda Forge and adding it is beyond my skills.

Open an issue here and we will coach and help you get the package built on Conda Forge.

Is it possible to have different conda environments other than the base environment?

We can add new environments using Conda Constructor and make them selectable by end users, but the current limitation is the 2GB maximum for our exe installer on Windows. It may be difficult to keep the installer under 2GB with even one additional environment.

Why can't my package be installed with PyPi/Cargo/NPM/CRAN?

Software building and installing ranges from simple to extremely complex, even for a single package. This is further complicated by trying to have a mutually compatible set of hundreds or thousands of packages installed together. We manage a monolithic installation of packages that come from a variety of languages (Python, C++, C, R, Julia, CUDA, etc.). Firstly, many of the packages cannot be installed from a single languages' repository, e.g. PyPi only hosts Python packages. Secondly, it is quite difficult, if not impossible, to create a compatible set of packages if relying on PyPi (see https://pypackaging-native.github.io/ for detailed explanations). A tenable solution that does not require excessive amounts of time and complexity for this team is to rely on Conda Forge for solving these problems. The binaries available on Conda Forge are guaranteed to provide a compatible set by the nature of its design. Even though it may seem simple to install your package on your computer, it may not be so in a offline installer. Every special case we add, costs us more time. So we default to Conda Forge packages with rare exceptions that depend on available volunteer time. This provides a minimal headache way to deliver a working software environments to our students.

Why can't students pip install or conda install packages while on the lab computers?

The computers are disconnected to the internet during most exams so these commands will not work in general. It may be possible to host our own copies of Conda Forge or PyPi behind our firewall in the future, but this is a large undertaking. We also do not give students write access to the locations that either of these tools install packages to, so the default install will not work. You can provide packages to the students via the shared drive and they can install them in their personal drive as a workaround.

Why are there no version specifications in the construct.yaml file? How do we know what version will be installed?

We install the latest Conda Forge version of all packages up to the July 1 version freeze date. After the July 1 freeze date the versions are added to the construct.yaml file. Build the installer to see what versions are installed or check the latest Github Actions output (see below).

Will you install may favorite IDE (e.g. PyCharm, VS Code, etc.)?

We can install any IDE that is installable from a Conda Forge package. For example, we install and provide Spyder and Jupyter Lab as part of this software distribution. If your desired IDE is not installable from Conda Forge, you should contact ICT with your request as it is outside the scope of this effort.

This is a list of people who have requested packages in the past or expressed interest in doing so.

You will need conda installed on Windows. You can use conda from Anaconda, Miniconda, Miniforge, or any other conda-based installation. For example, you can install the latest of the mentioned three installers from:

• Anaconda: https://www.anaconda.com/download/
• Miniconda: https://docs.anaconda.com/free/miniconda/miniconda-install/
• Miniforge: https://conda-forge.org/miniforge/

For example, the latest Miniconda Windows installer is at:

https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe

Install Git, either:

• Follow general installation instructions for your Operating System. For example, Github provides this guidance: https://github.com/git-guides/install-git
• or, install Git with conda (open the Anaconda/Conda prompt in Windows or a Terminal in OSX or Linux) and type:

conda install -c conda-forge git

If you want to build a specific version of the installer then download the release files from:

https://gitlab.ewi.tudelft.nl/bhmgerritsen/anaconda-dee-config/-/releases

If you want to work with the development version, then use one of the following methods to obtain the files:

If you have SSH set up for the EWI Gitlab use:

git clone git@gitlab.ewi.tudelft.nl:bhmgerritsen/anaconda-dee-config.git
cd anaconda-dee-config

else use if you do not have SSH set up:

git clone https://gitlab.ewi.tudelft.nl/bhmgerritsen/anaconda-dee-config.git
cd anaconda-dee-config

If you don't want to use Git you can download a zip file of the latest version of this repository from:

https://gitlab.ewi.tudelft.nl/bhmgerritsen/anaconda-dee-config/-/archive/master/anaconda-dee-config-master.zip

This makes sure conda is up-to-date and is using the recently added libmamba solver. The libmamba solver is faster, uses less memory, gives better error messages, and can solve harder sets of constraints.

conda update -n base conda

You will build the installer with constructor, which is the same tool used to build the official Anaconda installer. Create an environment with only constructor and activate it.

conda create -c conda-forge -n constructor "constructor>=3.12.2"
conda activate constructor

constructor will read the contents of the construct.yaml file, solve for the compatible set of packages and build a .exe installer that contains them.

On Windows, to build the installer, run:

set "CONDA_CHANNEL_PRIORITY=strict"
constructor .

or on Mac or Linux, run:

CONDA_CHANNEL_PRIORITY=strict constructor .

in the directory with the construct.yaml file. This can take 10-30 minutes depending on your computer's speed and download speeds. After it finishes there will be an executable file, for example tudelft-conda-<version>-Windows-x86_64.exe, in the directory. This installer can be executed on any computer to install the DEE.

Once you have built the installer exe, sh, pkg, you can use it to install the TU Delft Anaconda on other computers.

For example on Windows, download tudelft-conda-<version>-Windows-x86_64.exe to the Windows computer you want to install it on.

We build Windows, Linux, and Mac installers in this Github CI action:

https://github.com/moorepants/anaconda-dee-config/actions

The files can be downloaded up to 90 days after the build. Click the workflow run (likely the latest one) and then download from the "Artifacts" section. You can likely only see these if you are logged into Github.

Double click on the tudelft-conda installer (exe, pkg) or execute the shell script and follow the prompts. By default this will install the TU Delft Conda distribution in C:\Program Files\tudelft-conda on Windows and /home/user/tudelft-conda on Linux and Mac.

On Windows, you will get a warning about long path names. Either select an install location with a shorter base path, e.g. C:\, or install as an administrator to bypass the path length restrictions.

The full list of packages and their versions can be created with:

conda list --verbose --show-channel-urls -n base > base-pkgs.lst

After executing the installer you can run these basic tests to check if things are working properly:

conda run -n base python tests/test_imports.py
conda run -n base python tests/test_slycot.py
conda run -n base python tests/test_biogeme.py
conda run -n pytorch python tests/test_pytorch.py

You can also ask WPS to provide access to a TU Delft computer for testing. Two options are:

• assign a test machine and test using RemoteDesktop (Citrix)
• assign access rights to a beta tester