Improve interactive debugging

[casparvl]

Problem
We often have issues reproducing build failures interactively. Currently, we have about three options:

1. Run with EESSI-extend. This is the easiest, but also the most 'different' from the bot's build environment. It uses EESSI for dependencies, but it doesn't run in a build container.
2. Run the bot/inspect.sh from the job dir of the failed build job. This script was designed to mimic bot/build.sh and drop you into an interactive shell in the failed build environment. However, it is very much outdated, and keeping it in sync currently requires a lot of code duplication from bot/build.sh. An example of how out-of-sync it is: it does not have any GPU support (no NVIDIA GPU support either).
3. Try to set all relevant environment variables, extracted from the build log. Then, start a build container manually with the command reported in the build log. Then, try to build to reproduce the failure. This is potentially the most 'similar' in terms of environment to the build job, but very error prone, as it's easy to miss environment variables, or steps that are executed as part of bot/build.sh. It is also very time-consuming.

Proposed solution
The solution presented here is to tightly integrate an option for interactive debugging into the bot/build.sh, that will drop you into an interactive debugging shell if EESSI_INTERACTIVE_BUILD is set. To get an environment that is as similar as possible, one should call the job script that the bot submits interactively. I.e. if a bot is configured to use the default, that would be EESSI_INTERACTIVE_BUILD=<mode> eessi-bot-software-layer/scripts/bot-build.slurm.

mode supports three options:

• shell - start a plain shell in the build container instead of running the build
• before - run the build, but start a shell (inside the compat layer, EasyBuild loaded) right before EasyBuild is called
• onfail - run the build, and start a shell when EasyBuild fails (or the build script exits non-zero)

The idea is that this is ran from the job directory left behind by the bot, with the failing build. By default, it will resume from the latest tarball of the build step (this is selectable). If accelerators are used, it will run for the last accelerator in the list by default (this is also selectable).

Limitations

• If you're running this from a different user than the bot user, some things may fail on file permission - though typically this just means that things in the shared_fs_directory can not be updated (what's there can be used as long as your user has read permissions to that dir).
• Currently, the bot/build.sh from software-layer hard-clones and links the software-layer-scripts/bot/build.sh. That results in an exit, because the script runs with set -e. My current hack is just to replace the bot/build.sh in the job dir with a one-liner: software-layer-scripts/bot/build.sh. We should fix this more properly, and maybe make the whole clone-business conditional on EESSI_INTERACTIVE_BUILD not being set.