Python tools to help with the development & deployment of company-private Python packages.
It was developped to use Sonatype Nexus as a private Pypi mirror (until it supports this natively), but should be adaptable to any repository supporting artifact upload with HTTP.
It is composed of 2 distincts tools :
nexus_uploader
: a simple module to be used as asetup_requires
entry insetup.py
, in order to easily upload Python packages onto a Nexus.pyRequirements2nexus
: a CLI tool to convert standard Pythonrequirements.txt
intonexus-requirements.txt
files, made of URLs pointing to installable packages thatpyRequirements2nexus
mirrored on your Nexus.
Contents
- fully compatible Python 2 & 3
- easy integration in
setup.py
- HTTP BasicAuth to connect to Sonatype Nexus
- full dependency resolution of Python packages, working with both Pypi public ones & private ones on a Nexus
- to install packages, the end machine only require a connexion to the Nexus host, not to the Internet
- support
-e
editable packages inrequirements.txt
- support package URL fallbacks as comments in
requirements.txt
- can select only Python 2 packages with --allowed-pkg-classifiers py2.py3-none-any,py2-none-any
- a list of packages already included in the base environment can be
specified (e.g. if you are using Anaconda), so that they will be
excluded from the final
nexus-requirements.txt
Limitations
- only support
==
version locking inrequirements.txt
(not>=
). This is intentional, to ensure package versions do not change unexpectedly.
Here is a handy recipe you can put in your setup.py
:
setup( ... setup_requires=['nexus_uploader'] )
Usage:
$ python setup.py sdist nexus_upload --repository http://nexus/content/repositories/Snapshots/com/vsct/my_project --username $REPOSITORY_USER --password $REPOSITORY_PASSWORD
Contrary to requirements.txt
files, there is no way to include URLs
to dependencies in setup.py
install_requires
(nor
setup_requires
). To work around this limitation,
pyrequirements2nexus
takes advantage of the deprecated
"dependency_links" mechanism. Using them, it is able to discover the
full chain of Python packages dependencies, both from Pypi AND from
your local Nexus.
To easily generate the "dependency_links.txt" in your Nexus-hosted
private package out of your requirements.txt
, use the following
recipe in your its setup.py
:
with open(os.path.join(os.path.dirname(os.path.realpath(__file__)), 'requirements.txt')) as requirements_file: requirements = requirements_file.readlines() dependency_links = [req for req in requirements if req.startswith('http')] install_requires = [req for req in requirements if not req.startswith('http')] setup( ... install_requires=install_requires, dependency_links=dependency_links, )
Because "dependency_links" are not supported since pip
1.6,
they will NOT be installed by pip install
normal recursive
dependency-retrieval algorithm. You should always use the flat
nexus-requirements.txt
with pip install
.
pip install nexus_uploader pyRequirements2nexus --help
Also take a look at jenkins-install-python-requirements.sh
for an
example of how we use it on our Jenkins.
pip install --user --no-index --no-deps --no-cache-dir --upgrade --requirement nexus-requirements.txt
http://nexus/content/repositories/repo_id/my/project/group/mypkgname/1.0/mypkgname-1.0-py2.py3-none-any.tar.gz nose==1.3.7 # -> transformed into an URL like this: http://nexus/content/repositories/repo_id/my/project/group/... -e ../my/relative/path # http://nexus/content/repositories/...fallback_url...
pre-commit hooks installation:
pip install -r dev-requirements pre-commit install
Unit tests:
py.test tests/
Smoke tests using Pypi:
ipython3 --pdb tests/smoke_test_extract_classifier_and_extension.py 200
pip install - Download error on https://pypi.python.org / Couldn't find index page for
The stack trace :
Collecting http://nexus/content/repositories/pip/com/vsct/pip/jsonschema/2.5.1/jsonschema-2.5.1-py2.py3-none-any.tar.gz (from -r scripts/requirements.pip (line 12)) Downloading http://nexus/content/repositories/pip/com/vsct/pip/jsonschema/2.5.1/jsonschema-2.5.1-py2.py3-none-any.tar.gz (50kB) Complete output from command python setup.py egg_info: Download error on https://pypi.python.org/simple/vcversioner/: [Errno -2] Name or service not known -- Some packages may not be found! Couldn't find index page for 'vcversioner' (maybe misspelled?) Download error on https://pypi.python.org/simple/: [Errno -2] Name or service not known -- Some packages may not be found! No local packages or download links found for vcversioner
Explanation : python-jsonschema/jsonschema#276
Solution :
$ cat <<EOF > ~/.pydistutils.cfg [easy_install] allow_hosts = nexus find_links = http://nexus/content/repositories/pip/com/vsct/pip/vcversioner/2.14.0.0/ EOF
grep -aF 'extract_dist ' Anaconda3-2.4.1-Linux-x86_64.sh \ | perl -p -e 's/extract_dist (.+?[0-9])[^.]*$/\1\n/;' -e 's/^(.+)-(.+)$/\1 == \2/;' \ | grep -vE '^(_cache|_license|anaconda|python) ' > anaconda3-2.4.1_included_packages.txt
pip install --user repositorytools export REPOSITORY_USER=... export REPOSITORY_PASSWORD= artifact delete http://nexus/content/repositories/pip/com/vsct/pip/ultrajson/1.35/ultrajson-1.35-macosx-10.6-intel.tar.gz
- detect if package releases on Pypi require gcc compilation (are they
using setuptools/distutils
Extension
insetup.py
?) - classifier-based selection of Python packages
- add support for md5 & sha1 upload/checks