Publishing Python packages to PyPI

After many years of shipping my code via git, finally I dedicated some time to learn Python packaging.

You can publish to the Python Package Index (PyPI) repository using couple of steps.

Before you start, you may want to check if your preffered project name avaliable. If someone already publish anything with that name in PyPI, you’ll need to find an alternative.

I wanted to publish a program that will allow end-to-end analysis of tRNA isodecoders for data from Nano-tRNAseq protocol. Luckily my preffered, tRNAdecoder name was available.

Program structure

tRNAdecoder analysis will be split in several steps executed as sparate subcommads. One of them, tRNAdecoder align, will align the reads tothe genome.

First, let’s create a folder for then project

mkdir -p ~/src/tRNAdecoder
cd ~/src/tRNAdecoder

Here, it’s recommended to init git repository (details beyond the scope of this post).

So, we’ll need to create several files as follows:

1. src/tRNAdecoder/src/align.py is an actual subprogram performing alignment. Below code will allow running it using both, tRNAdecoder align and src/tRNAdecorer/src/align.py. The latter will be useful for development and debugging.

import argparse, sys

def align_reads(outfn, infn, fasta):
    print("it works!")

def main(o=None):
    if not o:
        parser = argparser(True)
        o = parser.parse_args()
        if o.verbose: sys.stderr.write("Options: %s\n" % str(o))

    align_reads(o.output, o.input, o.fasta)

def argparser(add_help=False):
    parser = argparse.ArgumentParser(formatter_class=argparse.RawTextHelpFormatter, add_help=add_help)
    parser.add_argument("-v", "--verbose", action="store_true", help="verbose")
    parser.add_argument("-f", "--fasta", required=True, help="reference FASTA file")
    parser.add_argument("-i", "--input", nargs="+", help="input FastQ/BAM file(s)")
    #...
    return parser

if __name__ == '__main__': main()

2. src/tRNAdecoder/src/__init__.py enables tRNAdecoder package with set of subcommands.

from argparse import ArgumentDefaultsHelpFormatter, ArgumentParser
from tRNAdecoder import align

modules = [
    align,
]

__version__ = '1.1.2'

def main():
    parser = ArgumentParser(formatter_class=ArgumentDefaultsHelpFormatter)

    parser.add_argument('-v', '--version', action='version', version=f'{__name__} {__version__}')

    subparsers = parser.add_subparsers(
        title='subcommands', description='valid commands',
        help='additional help', dest='command'
    )
    subparsers.required = True

    for m in modules:
        p = subparsers.add_parser(m.__name__.split('.')[-1], parents=[m.argparser()])
        p.set_defaults(func=m.main)

    args = parser.parse_args()
    args.func(args)

3. src/tRNAdecoder/src/__main__.py define tRNAdecoder:__main__ for step 4.

import sys
from tRNAdecoder import main
main()
sys.exit(0)

4. pyproject.toml defines Python package using hatch. This config will read version of software from package __init__.py.

[build-system]
requires = ["hatchling >= 1.26"]
build-backend = "hatchling.build"

[tool.hatch.version]
path = "src/tRNAdecoder/__init__.py"

[project]
name = "tRNAdecoder"
dynamic = ["version"]
dependencies = [
  "mappy==2.30",
  "numpy",
  "pysam",
]
requires-python = ">=3.9"
authors = [
  { name="Name Surname", email="somone@crg.es" },
]
description = "Isodecoder level analysis for Nano-tRNAseq runs"
readme = "README.md"
license = "MIT"
license-files = ["LICEN[CS]E*"]
keywords = ["keyword1", ]
classifiers = [
    "Programming Language :: Python :: 3",
    "Operating System :: OS Independent",
]

[project.urls]
Homepage = "https://github.com/..."
Issues = "https://github.com/.../issues"

[project.scripts]
tRNAdecoder = "tRNAdecoder:__main__"

Finally, don’t forget to add LICENSE and README.md.

Adding new subcommand, ie tRNAdecoder count, will require:

• creating new src/tRNAdecoder/src/count.py program like in step 1.

• and adding count to modules in step 2.

Neat, right?

Build

Now it’s time to build your package and install it from local .whl file.

# install dependencies
python3 -m pip install --upgrade pip build twine

# build first removing previous builds
rm dist/*
python3 -m build

# try installing and runnig it
pip install --force-reinstall --no-deps dist/*.whl

tRNAdecode align -i in -o out -f ref.fa

Publish to PyPI

Register PyPI account

If you haven’t done it yet, register an account at - test PyPI (sort of playground) - and PyPI (the real software index).

Enable automatic login to PyPI by - adding API token either all or selected projects - and creating ~/.pypirc file with username and created token

[pypi]
  username = username
  password = pypi-...

[testpypi]
  username = username
  password = pypi-...

Definitely read more about pypirc and API tokens.

Upload package

Finally, we can upload the package to PyPI. Uploading it first to test PyPI is recommended.

# upload to PyPI
python3 -m twine upload dist/* #--repository testpypi
# install from PyPI
python3 -m pip install tRNAdecoder #--index-url https://test.pypi.org/simple/
# try running it
tRNAdecoder align -h

You can also see it in the PyPI repository.

Note, PyPI doesn’t allow re-submitting the same version of the software. Therefore before re-uploading update version you’ll need to change version of your software.

Publish to bioconda

Once your package is in PyPI, you can rather simply publish it also to bioconda.

One of the benefit of publishing to bioconda is automatic container creation.

First, you’ll need to creat a fork of bioconda/bioconda-recipes and clone your fork locally.

# install grayskull - the only tool you need if starting from PyPI
pip install grayskull

cd ~/src/bioconda-recipes/recipes
git checkout master
git pull upstream master
git push origin master

git checkout -b add-tRNAdecoder
cd recipies
grayskull pypi tRNAdecoder

Note, you’ll need to add some additional info in your meta.yaml file, such as:

• recipe-maintainers (your github username without @)

You can have a look at my PR and meta.yaml .

Commit and push your changes:

# bioconda recipies/packages cannot use capital letters
mv tRNAdecoder trnadecoder
git add trnadecoder/meta.yaml && git commit -m add-tRNAdecoder
git push --set-upstream origin add-tRNAdecoder

Finally, you need to add your pull-request.

Once all test on your PR are passed (it took several tweaks on my first try, so be patient…), don’t forget to issue the @BiocondaBot please add label command, so your PR is tagged for merging (it’ll need to be reviewed.

Once it’s merged and built on bioconda ends, you can install your package using

conda install trnadecoder

It’ll appear also in biocontainers, so you could use it easily via apptainer/singularity or docker.

You can delete branch in your fork

git branch -D add-tRNAdecoder && git push origin -d add-tRNAdecoder

Definitely read more about guidelines, workflow and adding software to bioconda.

Automate deployment

TDB…

You can find more information about packaging Python projects here.

About giving up