Releasing a GitHub-Hosted, GitFlow-Release-Managed, Wheel-Packaged Python Package to PyPI Using Twine
This week I released a package to the Python Package Index (PyPI) for the first time. While I've developed a number of packages over the past decade and configured a goodly number of them to be installed via pip by adding a setup script and properly modularizing them, this was the first time I felt I had a package that added significant value to the community to move it from the GitHub development environment and release it on PyPI.
It was not a completely straightforward practice. Most tutorials and directions I found covered eggs and easy_install rather than pip and wheel and used sdist rather than twine. I found a few good resources explaining the necessary files and procedures, however:
The Python Project Howto, while aging, does a good job of covering general best practices before deciding to release a module.
How to submit a package to PyPI provided a good basic overview of what to include in the
setup.py script and how to link to documentation when it is in a markdown-formatted readme file. This also provided all the directions for creating PyPI Live and PyPI Test accounts and a
.pypirc configuration file.
The Python Packaging and Users Guide seems to be the actual, most-up-to-date documentation on the process. It covers the additional parameters of the setup file including the more complicated variables of
classifiers and possible values therein. It also includes instructions for uploading files using the better practice of using twine instead of sdist as well as the wheel binary package distribution format intended to replace eggs.
Creating a Development Environment for a Python Package and Uploading it to PyPI
Assuming you've created a
.pypirc configuration file and registered for PyPI Live and PyPI Test, the directions for using twine rather than sdist are pretty much a drop-in-place replacement.
The directions that follow assume that you have a project you've built and have been hosting on GitHub, of which you have maybe been installing using the
-e flag in pip, and are ready to release it to PyPi.
Starting from the top, we will create a new virtual environment, clone in the repo, run tests and lint the project, register the package to PyPi Test and PyPi Live and then upload.
Clone the Repo
$ git clone firstname.lastname@example.org:myuser/myproject.git
This creates a directory tree as follows:
├── myproject ├── LICENSE.txt ├── README.md ├── myproject │ ├── __init__.py ├── setup.cfg ├── setup.py └── tests ├── __init__.py └── test_myproject.py
Create a Virtual Environment
(I'm using virtualenvwrapper):
$ cd myproject $ mkvirtualenv -a . myVirtualEnvName
Install the Package for Development
$ pip install -e .
Install Wheel, Twine and PyLint
$ pip install wheel twine pylint
Run the Tests
$ python ./tests/test_myproject.py $ pylint ./myproject
Setup Git Flow
I'm using Nvie's Git Flow to manage releases to GitHub. The first time the project is cloned and setup, GitFlow needs to be initialized.
By default my project clones the
develop branch which is my default working branch. Git Flow uses
master to track the latest release. This branch is also on GitHub so it needs to be checked out.
$ git checkout master
master branch exists alongside
develop, Git Flow can be initialized with the defaults.
$ git flow init -d
Create the Distribution
This needs to be done each time the version (or subpoint version) gets bumped.
Tag the version for release.
$ git flow release start versionNumberAndSubPoint
setup.py and bump the
download_url versions (my
download_url points to a specific tarball of a tag on GitHub).
$ git flow release finish versionNumberAndSubPoint
Push everything back to GitHub.
$ git push --all; git push --tags
Create the dist.
$ python setup.py sdist
Create the Wheel
Depending on whether the package is universal to Python 2 and 3 or not changes how this is run. Instructions can be found in the Python Packaging User Guide. To create a Universal package that works with both Python 2 and 3:
$ python setup.py bdist_wheel --universal
Register the package to PyPI Test
This only needs to be done the first time.
$ python setup.py register -r pypitest
Upload to PyPI Test
$ twine upload dist/* -r pypitest
Register the package to PyPI Live
This only needs to be done the first time.
$ python setup.py register -r pypi
Upload to PyPI Live
$ twine upload dist/*
- August 20, 2015 -- Revised virtualenv instructions to utilize virtualenvwrapper. Added instructions for using wheel. Added GitFlow instructions.
- February 23, 2016 -- Noted that Universal wheels support both Python 2 and Python 3; consolidated installation of Python packages necessary for linting and pushing to PyPi