How to publish your package to PyPI, 2018

Page loading, please, little wait
Version 0.1.0
Posted —
Last modified —
  1. Overview
  2. Audience
  3. Relevance
  4. Motivation
  5. Demonstration
  6. Features
  7. Limitations
  8. Installation
  9. Configuration
    1. Files
    2. __init__.py
    3. License
    4. README
      1. README.md
      2. README.rst
    5. setup.cfg
      1. long_description
      2. classifiers
      3. zip_safe
      4. packages
      5. console_scripts
      6. bdist_wheel
    6. Dependencies
      1. Difference between requirements.txt and install_requires
        1. install_requires
        2. requirements.txt
      2. setup.py
      3. requirements.txt
      4. MANIFEST.in
    7. pyroma
  10. TestPyPI
    1. What is TestPyPI
    2. Registration
    3. Twine settings
    4. Build package
    5. Publish to TestPyPI
    6. Visit TestPyPI
    7. virtualenv
      1. Why virtual environment?
      2. Using virtualenv and virtualenvwrapper
      3. Checking
  11. PyPI
  12. Updating
  13. Automatic updating, release and changelog
    1. Demonstration
    2. Features
    3. Limitations
    4. Installation and setting-up
      1. .release-it.json
    5. Usage
  14. Testing environment

1. Overview

Tutorial, how you can add your Python package to PyPI — Python Package Index.

If you want, that users install your Python package via command:

pip install <your_package>

Read this article.

Info

In this article I wrote the most common method at February 2018. But you can consider another tools as flit and poet.

2. Audience

Python developers, that:

  1. publish package to PyPI first time,
  2. already published packages to PyPI, but want to do it more qualitatively.

3. Relevance

This article is relevant for February 2018. In the future, the data in this article may be obsolete.

4. Motivation

One more article? Why?

I read some articles about PyPI publishing → I think, that articles, which I read, have disadvantages:

  1. outdated;
  2. not show real examples and demonstrations;
  3. not all-in-one; I was forced to read other resources;
  4. They do not describe all the difficulties the developer has encountered, when try to publish package on PyPI.

I don't want, that another beginner Python developers take a lot of time for PyPI publishing as me. And I try to write an article, that:

  1. non-outdated at February 2018;
  2. show examples and demos;
  3. all-in-one; I try to add in my article all information, that I need for first PyPI publishing;
  4. described all problems, to solve that personally I spent my time.

5. Demonstration

For examples in this article, I select my Erichek package.

https://pypi.python.org/pypi/erichek page at February 2018:

Erichek on pypi.python.org

https://pypi.org/project/erichek Warehouse page at February 2018:

Erichek on pypi.org

Erichek on pypi.org 2

You can install Erichek, use command:

pip install erichek

If you want to have similar behavior, read on.

If I have explained something incomprehensibly, see files of Erichek GitHub repository.

6. Features

  1. This article — cross platform solution.

7. Limitations

  1. You must be able to write working Python packages.
  2. Erichek and another my packages only for Python 3. I don't want support Python 2. If you want support Python 2, possibly, you will need some other action.
  3. If you can have a configuration different, that Erichek, possibly, you will need in another actions.

8. Installation

Please, install via pip (pip install <package>):

  1. twine,
  2. wheel,
  3. virtualenv,
  4. virtualenvwrapper for Linux/macOS, virtualenvwrapper-win for Windows,
  5. pyroma,
  6. update your setuptools, to be sure, that pyroma works correctpip install -U setuptools.

9. Configuration

9.1. Files

Simplify configuration. Real project configuration see in Erichek GitHub repository.

│   .release-it.json
│   CHANGELOG.md
│   LICENSE.md
│   MANIFEST.in
│   README.rst
│   package.json
│   requirements.txt
│   setup.cfg
│   setup.py
│
├───erichek
│       __init__.py
│       __main__.py

In Software Engineering site recommends use uppercase for README and some another text files.

Question

I ask a question, need I use uppercase for file extension or no? README.rst or README.RST? But I get 7 minuses and my question will delete.

I don't find standard. Personally I use lowercase for file extensions, but you can use uppercase.

9.2. __init__.py

See answer, why __init__.py need. This file may be blank or contains content.

9.3. License

Add text of license for your package to LICENSE.md file.

If your editor is Sublime Text 3, you can use License Snippets package.

9.4. README

9.4.1. README.md

If you preferred write README in Markdown, you can have problems.

At February 2018 PyPI doesn't support Readme.md without dependencies. Possibly, it seems like there is no easy way to use a markdown README for PyPI. Solutions involve requiring pandoc locally, which is a heavy dependency.

9.4.2. README.rst

I recommend write README in reStructuredText — you need use file README.rst. I add to my README.rst next information:

  1. Short package description,
  2. Badges,
  3. Link to long package description on my site.

I prefer to add long description to personal site, not to README, because I have much more options in the design of the description.

9.5. setup.cfg

Info

You can add your parameters to setup.py, not to setup.cfg, see discussion. I believe, that adding to setup.cfg more convenient.

See example setup.cfg. Copy content of this file to your setup.cfg and change example values of parameters to your real values.

In the subsections below I describe the places that caused me difficulties.

9.5.1. long_description

my value:

long_description = file: Readme.rst

Set your README file as value. Letters must be in same register as in file. For example, if your file is Readme.rst, you need to set file: README.rst, not file: Readme.rst or file: README.RST.

9.5.2. classifiers

You can use only classifiers from this list. If no, you can't publish your package to PyPI.

Possibly, UNIX users can select classifiers via pypi-classifiers GUI, but I can't set this program for Windows 10.

9.5.3. zip_safe

my value:

zip_safe = False

If you build your package use Wheels, you don't need this parameter, but pyroma will show lower value. So I recommend add zip_safe = False or zip_safe = True (unimportant) to your setup.cfg file.

9.5.4. packages

my value:

packages = find:

If no packages = find:, users doesn't download folder with your Python module.

9.5.5. console_scripts

What is it. If Erichek user want run Erichek, he/she wrote in console:

python "path/to/__main__.py"

But since console_scripts exist in setup.cfg, Erichek user can use simply command, that get same behavior:

erichek

my value:

console_scripts =
        erichek = erichek.__main__:main

Parameter and values:

9.5.6. bdist_wheel

my value:

[bdist_wheel]
python-tag = py3

See what is Wheels and why .whl preferred than .egg.

Erichek — Python 3 package, so I have python-tag = py3. If your package support Python 2 and 3 both, you need use next code:

[bdist_wheel]
universal = 1

9.6. Dependencies

In this section I tell, what you need, that dependencies of your package from file requirements.txt automatically install for user. You don't need to write same dependencies in requirements.txt and setup.cfg/setup.py.

Caution

Some developers criticize this method. If you need different dependencies for requirements.txt and install_requires, please, add list of your dependencies in requirements.txt and setup.cfg both.

9.6.1. Difference between requirements.txt and install_requires

Simplified definition.

9.6.1.1. install_requires

install_requires parameter in setup.cfg or setup.py show, which packages will install additionally, if user install your package via pip. For example, user install clize package:

pip install clize

Part of clize setup.py:

install_requires=[
	'six',
	'sigtools >= 2.0',
	'attrs >= 17.4.0',
	'od',
	'docutils',
],

In clize installation process six, sigtools, docutils, od and attrs packages automatically install for user, if this packages no already installed.

D:\>pip install clize
Collecting clize
	Using cached clize-4.0.3-py2.py3-none-any.whl
Requirement already satisfied: six in c:\python36\lib\site-packages (from clize)
Requirement already satisfied: sigtools>=2.0 in c:\python36\lib\site-packages (from clize)
Requirement already satisfied: docutils in c:\python36\lib\site-packages (from clize)
Collecting od (from clize)
	Using cached od-1.0-py3-none-any.whl
Requirement already satisfied: attrs>=17.4.0 in c:\python36\lib\site-packages (from clize)
Installing collected packages: od, clize
Successfully installed clize-4.0.3 od-1.0

9.6.1.2. requirements.txt

For example, I am Python package developer and have local dependencies for my package.

  1. If I want to update local dependencies, I update versions in requirements.txt → I run in console:

    pip install -r requirements.txt -t . --upgrade
    

    my local dependencies are updated.

  2. Services as Dependabot or Pyup check dependencies from requirements.txt and update numbers of outdated versions.

9.6.2. setup.py

In this and 2 next sections I wrote, how you can get install_requires behavior, use only requirements.txt file. You don't need additional add your dependencies to install_requires.

Add this code to your setup.py:

from setuptools import setup

from pip.req import parse_requirements

# parse_requirements() returns generator of pip.req.InstallRequirement objects
install_reqs = parse_requirements('requirements.txt', session='hack')

# reqs is a list of requirements
# e.g. ['django==1.5.1', 'mezzanine==1.4.6']
reqs = [str(ir.req) for ir in install_reqs]

setup(
	install_requires = reqs
)

Note

Author of answer by link above doesn't add a line in answer:

from setuptools import setup

But you need to add it.

9.6.3. requirements.txt

Add list of your dependencies to requirements.txt.

9.6.4. MANIFEST.in

Add a line to MANIFEST.in file:

include requirements.txt

If no, users can't install your package.

9.7. pyroma

pyroma — PyPI configuration validator.

If you set all your configuration files, run command in root directory of your Python module:

pyroma .

You need get 10/10 final rating, example:

$ pyroma .
------------------------------
Checking .
Registered VCS backend: git
Registered VCS backend: hg
Registered VCS backend: svn
Registered VCS backend: bzr
Found erichek
------------------------------
Final rating: 10/10
Your cheese is so fresh most people think it's a cream: Mascarpone
------------------------------

If no 10/10, please, see pyroma messages and try fix your package.

10. TestPyPI

10.1. What is TestPyPI

If you publish your package first time or not sure that everything is doing right, I recommend at first publish package to TestPyPI. TestPyPI — is a service, where you can test uploading, downloading and display your package on PyPI site before you share package to PyPI.

10.2. Registration

Please, register on TestPyPI and PyPI sites.

It would be nice, if you will use same username and password for both sites, that Twine works correct.

10.3. Twine settings

Add environment variables TWINE_USERNAME and TWINE_PASSWORD with values — your PyPI and TestPyPI username and password.

See, how you can add environment variables for:

Caution

Also you can add your username and password to .pypirc file, but it not recommended.

10.4. Build package

Print in your terminal:

python setup.py bdist_wheel

Info

If by some reasons you don't want use Wheels, you can build your package, use command python setup.py sdist

10.5. Publish to TestPyPI

Enter in your terminal:

twine upload --repository-url https://test.pypi.org/legacy/ dist/*

If no errors in your configuration, package must successful upload to TestPyPI.

Bug

Known bug at February 2018 — in PyPI and TestPyPI you never can't overwrite specific version of your package, even if you delete your package. I.e., for example, you publish to PyPI or TestPyPI version 4.14.7 of your package mypackage → you delete mypackage from PyPI and/or TestPyPI → you can't upload 4.14.7 version of mypackage again.

10.6. Visit TestPyPI

After uploading visit 2 pages:

Examples for Erichek at February 2018:

Erichek TestPyPI

Erichek TestPyPI 2

Erichek TestPyPI Python

If the result suits you, in next step install package, that you upload, use virtual environment.

Note

On https://testpypi.python.org can't show last version of your package. That fix it, login at https://testpypi.python.org → visit https://testpypi.python.org/pypi?%3Aaction=pkg_edit&name=<your_package> (https://testpypi.python.org/pypi?%3Aaction=pkg_edit&name=erichek for Erichek) → set Hide? No for latest version.

If you have this problem for https://pypi.python.org, make same actions.

10.7. virtualenv

I recommend install your TestPyPI package, use virtual environment.

10.7.1. Why virtual environment?

Citation from Pro Python Best Practices book:

Virtualenv is like building a moat around the house. It prevents a fire from spreading — in both directions. Likewise, a virtual environment prevents that Python projects interfere with each other.

virtualenv

Reasons of use virtual environment for TestPyPI installation:

  1. You can have bugs in your package. Bugs may have a negative impact of your environment. If you will use virtual environment, these bugs does not affect your global Python environment.
  2. You can have some globally dependencies in your environment, but users of your package may haven't them. If you use virtual environment, you see, how your package will install and works without, possibly, pre-installed global dependencies on your machine.

10.7.2. Using virtualenv and virtualenvwrapper

Caution

If you on Windows, please, use standard Windows console for correct virtualenvwrapper-win works or use plugin for your preferred terminal. You can't use command lines command from this section in PowerShell or Far Manager.

Open terminal in any directory. Run these commands:

C:\Users\SashaChernykh>mkvirtualenv erichekenv
Using base prefix 'c:\\python36'
New python executable in C:\Users\SashaChernykh\Envs\erichekenv\Scripts\python.exe
Installing setuptools, pip, wheel...done.

(erichekenv) C:\Users\SashaChernykh>toggleglobalsitepackages

	Disabled global site-packages
(erichekenv) C:\Users\SashaChernykh>pip install --extra-index-url https://test.pypi.org/simple/ erichek

Where:

Note

Use --extra-index-url, not --index-url command line argument for correct dependencies installation.

Caution

Can take ~10 minutes, before you can install last version of your package. For example, you upload 4.14.7 version your package mypackage to TestPyPI → you print in a terminal pip install --extra-index-url https://test.pypi.org/simple/ mypackage==4.14.7 → you can get an error:

Could not find a version that satisfies the requirement mypackage==4.14.7 (from versions: 4.14.6)
No matching distribution found for mypackage==4.14.7

You may need wait ~10 minutes, that 4.14.7 version index on TestPyPI. You can have a similar problem for PyPI.

10.7.3. Checking

If you can't have bugs in installation process, check, that you package works correct. For example, Erichek check, contains errors in .txt files in a folder or no. I run erichek console command.

If error in one of text file:

C:\ErichekExamples>erichek
[2018-02-08 08:49:43.058353] NOTICE: eric_encoding logbook: All files in Windows-1251 encoding
[2018-02-08 08:49:43.060355] NOTICE: eric_body logbook: All files contains <body>
[2018-02-08 08:49:43.062358] NOTICE: eric_asterisks logbook: All needest lines contains asterisks
[2018-02-08 08:49:43.063357] ERROR: eric_head logbook: Air_crashes.txt not contains «Постоянный адрес пакета:»
[2018-02-08 08:49:43.064359] ERROR: eric_head logbook: One or more packages not contains one or more head data. Please, add correct head data to your package.
[2018-02-08 08:49:43.064359] ERROR: summary logbook: You have errors in your packages. Please, fix them.

If all .txt files correct:

C:\ErichekExamples>erichek
[2018-02-08 08:51:23.840987] NOTICE: eric_encoding logbook: All files in Windows-1251 encoding
[2018-02-08 08:51:23.841997] NOTICE: eric_body logbook: All files contains <body>
[2018-02-08 08:51:23.843991] NOTICE: eric_asterisks logbook: All needest lines contains asterisks
[2018-02-08 08:51:23.844992] NOTICE: eric_head logbook: All files contains correct head data
[2018-02-08 08:51:23.844992] NOTICE: summary logbook: Congratulations! You haven't errors in your packages!

Erichek works as expected.

After succsessful checking you can remove your virtual environment (erichekenv in example):

(erichekenv) C:\Users\SashaChernykh>rmvirtualenv erichekenv

	Deleted C:\Users\SashaChernykh\Envs\erichekenv

C:\Users\SashaChernykh>

11. PyPI

If all actions from TestPyPI section success for you, you can publish your package to PyPI and then install it.

  1. change version in your setup.cfg file (it must be semver-compatible);
  2. in root folder of your package open terminal and print:

    python setup.py bdist_wheel && twine upload dist/*
    
  3. visit https://pypi.python.org/pypi/<your_package> and https://pypi.org/project/<your_package>, as you visit TestPyPI and check, is everything okay;

  4. install your package, use pip:

    pip install <your_package>
    
  5. check, that your package works correct as in previous section.

If all okay, congratulations! You successful publish your package to PyPI!

Congratulations

12. Updating

That update your package in PyPI:

  1. Make new changes,
  2. Make same actions as in section above.

13. Automatic updating, release and changelog

Note

In this article I wrote not detailed. If you want to read details, how it worked, see my another article. I recommend read it before making actions from this section.

Caution

I recommend in first do actions from this section for test project, not real. Argumentation:

  • You or I can make a typo(s);
  • Differences in my and your environment;
  • Different versions of tools from this article can do another behavior, example

13.1. Demonstration

I enter command to the terminal:

release-it -n -V

I get behavior:

  1. All commits description add to CHANGELOG.md:

    Changelog

  2. New release publish to GitHub:

    New release

  3. Version updates in setup.cfg:

    Version in setup.cfg

  4. New version of Erichek publish to PyPI.

If you want to have same behavior, read on.

13.2. Features

See these features.

13.3. Limitations

See these limitations.

13.4. Installation and setting-up

  1. you need to install all from this section except tee, cat, mv and js-beautify;
  2. please, follow these, these and these instructions;
  3. create a file package.json in root folder of your repository.

13.4.1. .release-it.json

Create a file .release-it.json in root folder of your repository with content:

{
	"buildCommand": "changelog -u https:\/\/${repo.host}\/${repo.repository} -f CHANGELOG.md && sed -i 's\/^version = .*$\/version = ${version}\/g' setup.cfg && python setup.py bdist_wheel && twine upload dist\/*",
	"changelogCommand": "changelog -f -",
	"github": {
		"release": true,
		"tokenRef": "GITHUB_TOKEN"
	},
	"npm": {
		"publish": false
	},
	"safeBump": false
}

Where:

Note

In .release-it.json of real Erichek repository I add also next text:

&& sed -i 's\/^VERSION = \".*\"$\/VERSION = \"${version}\"\/g' \"erichek\/eric_config.py\"

It change a version in eric_config.py file, that command erichek --version works correct.

Version in eric_config.py

If you don't want to have this behavior, please, do not add this text to your .release-it.json.

See here, if you want to know details about another parts of .release-it.json.

13.5. Usage

See usage instructions.

14. Testing environment