Pin Python build dependencies to defined versions

[chrissimpkins]

Establish builds that can be built in a reproducible fashion at a specific commit at any point in time.

Suggestion by @anthrotype to create requirements.txt file:

#398 (comment)

Modified following discussions to use pipenv with Pipfile + Pipfile.lock Python build dependency pinning.

TODO:

• create pipenv venv with new Pipfile and Pipfile.lock files to define pinned Python build dependency versions 8490024
• modify build scripting to use the pipenv venv to build font files under Python 3 with pinned build dependency versions as defined in the Pipfile.lock file
• modify ttf build scripts
• modify woff build scripts

[anthrotype]

If you don’t mind the noise in your inbox pyup.io runs a bit that will automatically send PRs to update a requirements.txt files in your repo whenever any package mentioned there has a new version published in PyPI.

[anthrotype]

(a bot, not a bit)

[chrissimpkins]

Good service. It serves the changelogs as part of the PR too when they are available through its automated changelog parsing method. Good idea. Thanks Cosimo!

[anthrotype]

Yeah, it’s good. You may want to configure it’s frequency to say once a week or so because it gets pretty quickly annoying. Check the configuration file in fontmake repo for example

[chrissimpkins]

Convert to the following approach in build scripting to install all Python dependencies:

pip install -r /path/to/requirements.txt

and eliminate all pip install [project] commands in the scripts. This means that we should be documenting the best approach for builds to be within a virtualenv.

[chrissimpkins]

@anthrotype I could use your feedback about how to accomplish this on my development machine. Do you suggest that I create a virtualenv, activate & install requirements.txt dependencies in the venv, build from this, delete the venv after build, and create new venv for every new build? I've used venv for one off isolation of dependencies but have never kept one around for ongoing development as other things change on the machine. Homebrew versions of the Python interpreter will be rolling on my development system and this is desired for other reasons on that machine. Possible to swap out the Python 3 interpreter in an existing venv so that the libraries and executables do not need to be reinstalled each time this changes? Is the Python interpreter completely isolated in the venv or does it break if the venv is defined with Homebrew installed Python where cleanups remove previous system installed versions of Pythons?

[anthrotype]

Possible to swap out the Python 3 interpreter in an existing venv so that the libraries and executables do not need to be reinstalled each time this changes?

no, each venv is connected to one specific python interpreter, cannot be shared. You need to create two different ones.

Is the Python interpreter completely isolated in the venv and does it break if the venv is defined with Homebrew installed Python where cleanups remove previous versions of the installed Pythons?

the venv will break if you update the host python, yeah. Venvs are meant to be throw-away and easy to delete and recreate, the only things that defines then is the requirements.txt with all the dependencies pinned to a specific version.

[chrissimpkins]

Probably best to script the venv creation then. OK, that is very helpful. Thanks Cosimo.

[anthrotype]

there's a new kid in town, called pipenv, which has recently also become officially blessed by the Python Packaging Authority (PyPA), though I haven't checked it out yet. It seems more geared towards "applications" than "libraries" (which is mostly what I deal with). Perhaps it may work in your case though. Try it out.
There's also a tutorial of PyPUG
https://packaging.python.org/tutorials/managing-dependencies/?highlight=pipenv

[chrissimpkins]

Interesting. It looks like the Python scripts are run like this for the local install of Python isolated through pipenv:

pipenv run python main.py

or maybe this to use Python3?:

pipenv run python3 main.py

I wonder how you use it to run isolated executables installed with that approach. Will take a deeper dive into the docs for both that and virtualenv to see what approach will work the best. Thanks again Cosimo.

[chrissimpkins]

By the way, this was a very good suggestion for the project. Thanks for the push to get this moving. It needs to happen. We are going to pin all build dependency versions across the Python projects and non-Python projects in the tooling as well as eliminate pathways to build from system installed versions of build tools when detected as present on the build machine. Work is underway.