Remediating poor PyPi performance with DevPi
Performance of the primary PyPi service has been so bad lately that it’s become very disruptive. Tasks that used to take a few seconds will now churn along for 15-20 minutes or longer before completing, which is incredibly frustrating.
I first went looking to see if there was a PyPi mirror infrastructure, like we see with CPAN for Perl or CTAN for Tex (and similarly for most Linux distributions). There is apparently no such beast,
I didn’t really want to set up a PyPi mirror locally, since the number of packages I actually use is small vs. the number of packages available. I figured there must be some sort of caching proxy available that would act as a shim between me and PyPi, fetching packages from PyPi and caching them if they weren’t already available locally.
I was previously aware of Artifactory, which I suspected (and confirmed) was capable of this, but while looking around I came across DevPi, which unlike Artifactory is written exclusively for managing Python packages. DevPi itself is hosted on PyPi, and the documentation made things look easy to configure.
After reading through their Quickstart: running a pypi mirror on your laptop documentation, I built a containerized service that would be easy for me to run on my desktop, laptop, work computer, etc. You can find the complete configuration at https://github.com/oddbit-dot-com/docker-devpi-server.
I started with the following
Dockerfile (note I’m using
podman rather than Docker as my container runtime, but the
resulting image will work fine for either environment):
FROM python:3.9 RUN pip install devpi-server devpi-web WORKDIR /root VOLUME /root/.devpi COPY docker-entrypoint.sh /docker-entrypoint.sh ENTRYPOINT ["sh", "/docker-entrypoint.sh"] CMD ["devpi-server", "--host", "0.0.0.0"]
This installs both
devpi-server, which provides the basic caching
pip install, as well as
devpi-web, which provides support for
To ensure that things are initialized correctly when the container
start up, I’ve set the
ENYTRYPOINT to the following script:
#!/bin/sh if ! [ -f /root/.devpi/server ]; then devpi-init fi exec "$@"
This will run
devpi-init if the target directory hasn’t already been
The repository includes a GitHub workflow that builds a new image on each commit
and pushes the result to the
oddbit/devpi-server repository on
Once the image was available on Docker Hub, I created the following systemd unit to run the service locally:
[Service] Restart=on-failure ExecStartPre=/usr/bin/rm -f %t/%n-pid ExecStart=/usr/bin/podman run --replace \ --conmon-pidfile %t/%n-pid --cgroups=no-conmon \ --name %n -d -p 127.0.0.1:3141:3141 \ -v devpi:/root/.devpi oddbit/devpi-server ExecStopPost=/usr/bin/rm -f %t/%n-pid PIDFile=%t/%n-pid Type=forking [Install] WantedBy=multi-user.target default.target
There are a couple items of note in this unitfile:
The service is exposed only on
-p 127.0.0.1:3141:3141. I don’t want this service exposed on externally visible addresses since I haven’t bothered setting up any sort of authentication.
The service mounts a named volume for use by
-v devpi:/root/.devpicommand line option.
This unit file gets installed into
systemctl --user enable --now devpi.service both enables the service to start at boot
and actually starts it up immediately.
With the service running, the last thing to do is configure
utilize it. The following configuration, placed in
~/.config/pip/pip.conf, does the trick:
[install] index-url = http://localhost:3141/root/pypi/+simple/ [search] index = http://localhost:3141/root/pypi/
pip install and
pip search hit the local cache instead of
the upstream PyPi server, and things are generally much, much faster.
For Poetry Users⌗
Poetry respects the
pip configuration and will Just Work.
For Pipenv Users⌗
Pipenv does not respect the pip configuration [1,
2], so you will
need to set the
PIPENV_PYPI_MIRROR environment variable. E.g: