Releasing a GitHub-Hosted, GitFlow-Release-Managed, Wheel-Packaged Python Package to PyPI Using Twine

This week I released a package to the Python Package Index (PyPI) for the first time. While I've developed a number of packages over the past decade and configured a goodly number of them to be installed via pip by adding a setup script and properly modularizing them, this was the first time I felt I had a package that added significant value to the community to move it from the GitHub development environment and release it on PyPI.

It was not a completely straightforward practice. Most tutorials and directions I found covered eggs and easy_install rather than pip and wheel and used sdist rather than twine. I found a few good resources explaining the necessary files and procedures, however:

The Python Project Howto, while aging, does a good job of covering general best practices before deciding to release a module.

How to submit a package to PyPI provided a good basic overview of what to include in the setup.py script and how to link to documentation when it is in a markdown-formatted readme file. This also provided all the directions for creating PyPI Live and PyPI Test accounts and a .pypirc configuration file.

The Python Packaging and Users Guide seems to be the actual, most-up-to-date documentation on the process. It covers the additional parameters of the setup file including the more complicated variables of classifiers and possible values therein. It also includes instructions for uploading files using the better practice of using twine instead of sdist as well as the wheel binary package distribution format intended to replace eggs.

Creating a Development Environment for a Python Package and Uploading it to PyPI

Assuming you've created a .pypirc configuration file and registered for PyPI Live and PyPI Test, the directions for using twine rather than sdist are pretty much a drop-in-place replacement.

The directions that follow assume that you have a project you've built and have been hosting on GitHub, of which you have maybe been installing using the -e flag in pip, and are ready to release it to PyPi.

Starting from the top, we will create a new virtual environment, clone in the repo, run tests and lint the project, register the package to PyPi Test and PyPi Live and then upload.

Clone the Repo

$ git clone git@github.com:myuser/myproject.git

This creates a directory tree as follows:

├── myproject
    ├── LICENSE.txt
    ├── README.md
    ├── myproject
    │   ├── __init__.py
    ├── setup.cfg
    ├── setup.py
    └── tests
        ├── __init__.py
        └── test_myproject.py

Create a Virtual Environment

(I'm using virtualenvwrapper):

$ cd myproject
$ mkvirtualenv -a . myVirtualEnvName

Install the Package for Development

$ pip install -e .

Install Wheel, Twine and PyLint

$ pip install wheel twine pylint

Run the Tests

$ python ./tests/test_myproject.py
$ pylint ./myproject 

Setup Git Flow

I'm using Nvie's Git Flow to manage releases to GitHub. The first time the project is cloned and setup, GitFlow needs to be initialized.

By default my project clones the develop branch which is my default working branch. Git Flow uses master to track the latest release. This branch is also on GitHub so it needs to be checked out.

$ git checkout master

Once the master branch exists alongside develop, Git Flow can be initialized with the defaults.

$ git flow init -d

Create the Distribution

This needs to be done each time the version (or subpoint version) gets bumped.

Tag the version for release.

$ git flow release start versionNumberAndSubPoint

Edit setup.py and bump the version and download_url versions (my download_url points to a specific tarball of a tag on GitHub).

$ git flow release finish versionNumberAndSubPoint

Push everything back to GitHub.

$ git push --all; git push --tags

Create the dist.

$ python setup.py sdist

Create the Wheel

Depending on whether the package is universal to Python 2 and 3 or not changes how this is run. Instructions can be found in the Python Packaging User Guide. To create a Universal package that works with both Python 2 and 3:

$ python setup.py bdist_wheel --universal

Register the package to PyPI Test

This only needs to be done the first time.

$ python setup.py register -r pypitest

Upload to PyPI Test

$ twine upload dist/* -r pypitest

Register the package to PyPI Live

This only needs to be done the first time.

$ python setup.py register -r pypi

Upload to PyPI Live

$ twine upload dist/*

Revisions

  1. August 20, 2015 -- Revised virtualenv instructions to utilize virtualenvwrapper. Added instructions for using wheel. Added GitFlow instructions.
  2. February 23, 2016 -- Noted that Universal wheels support both Python 2 and Python 3; consolidated installation of Python packages necessary for linting and pushing to PyPi