Navigation Menu

Skip to content

Instantly share code, notes, and snippets.

@Jaza

Jaza/Private-pypi-howto

Last active July 2, 2023 16:24
Show Gist options
  • Save Jaza/fcea493dd0ba6ebf09d3 to your computer and use it in GitHub Desktop.
Save Jaza/fcea493dd0ba6ebf09d3 to your computer and use it in GitHub Desktop.
Download ZIP
Guide for how to create a (minimal) private PyPI repo, just using Apache with directory autoindex, and pip with an extra index URL.
Raw
private_pypi_howto.md

How to set up and use a private PyPI repo

Accompanying blog post:

Splitting a Python codebase into dependencies for fun and profit

Based on:

Create a local PyPi repository using only mod_rewrite

See also:

Local PyPI Options

Setting up a private, team-wide PyPI repository

For a more advanced private PyPI, see:

devpi: PyPI server and packaging/testing/release tool

Create root directory for private PyPI

ssh dude@myserver.com

(On remote server)

mkdir /path/to/pypi.myserver.com
mkdir /path/to/pypi.myserver.com/simple

Create self-signed SSL certificate

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
-keyout /etc/apache2/ssl/pypi.myserver.com.key \
-out /etc/apache2/ssl/pypi.myserver.com.pem

Set up Apache vhost

sudo htpasswd -c /etc/apache2/passwords_pypi pypi

sudo vi /etc/apache2/sites-available/pypi.myserver.com

(Add these lines)

<VirtualHost *:80>
    ServerName pypi.myserver.com

    RewriteEngine On
    RewriteCond %{HTTPS} off
    RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
</VirtualHost>

<VirtualHost *:443>
    ServerName pypi.myserver.com
    DocumentRoot /data/www/pypi.myserver.com

    SSLEngine On
    SSLCertificateFile /etc/apache2/ssl/pypi.myserver.com.pem
    SSLCertificateKeyFile /etc/apache2/ssl/pypi.myserver.com.key

    <Directory /data/www/pypi.myserver.com/>
        AllowOverride None
        Options +Indexes
        IndexOptions SuppressColumnSorting
        IndexIgnore ..
        Order deny,allow
        Allow from all

        AuthType Basic
        AuthName "My Server"
        AuthBasicProvider file
        AuthUserFile /etc/apache2/passwords_pypi
        Require valid-user
    </Directory>

    LogLevel warn
    ErrorLog /var/log/apache2/pypi-error.log
    CustomLog /var/log/apache2/pypi-access.log combined
</VirtualHost>

cd /etc/apache2/sites-enabled
sudo ln -s ../sites-available/pypi.myserver.com pypi.myserver.com
sudo apache2ctl graceful

Create directory for new library in private PyPI

mkdir /path/to/pypi.myserver.com/simple/foobar-utils

exit

Update library's code

(On local machine)

cd /path/to/foobar-utils

vi foobar_utils.py

(Add these lines)

__version__ = '0.1.0'

foobar = 'Hey foo, I am a bar!'
vi setup.py

(Add these lines)

import os

import setuptools

module_path = os.path.join(os.path.dirname(__file__), 'foobar_utils.py')
version_line = [line for line in open(module_path)
                if line.startswith('__version__')][0]

__version__ = version_line.split('__version__ = ')[-1][1:][:-2]

setuptools.setup(
    name="foobar-utils",
    version=__version__,
    url="https://git.myserver.com/foobar-utils/",

    author="Mister foo",
    author_email="mister@foo.com",

    description="Utils for handling Foo and Bar.",
    long_description=open('README.rst').read(),

    py_modules=['foobar_utils'],
    zip_safe=False,
    platforms='any',

    install_requires=[],

    classifiers=[
        'Development Status :: 2 - Pre-Alpha',
        'Environment :: Web Environment',
        'Intended Audience :: Developers',
        'Operating System :: OS Independent',
        'Programming Language :: Python',
        'Programming Language :: Python :: 2',
        'Programming Language :: Python :: 2.7',
        'Programming Language :: Python :: 3',
        'Programming Language :: Python :: 3.3',
    ],
)
vi README.rst

(Add these lines)

foobar-utils
============

Utils for handling Foo and Bar.

Upload new version of library code to private PyPI

python setup.py bdist_wheel --universal
scp dist/foobar_utils-0.1.0-py2.py3-none-any.whl \
dude@myserver.com:/path/to/pypi.myserver.com/simple/foobar-utils/

Configure pip to use private PyPI

vi ~/.pip/pip.conf

(Add these lines)

[global]
; Extra index to private pypi dependencies
extra-index-url = https://pypi:pyp1@pypi.myserver.com/simple/
trusted-host = pypi.myserver.com

Use private library in a project's requirements.txt

cd /path/to/projectfoo
virtualenv .
source bin/activate

vi requirements.txt

(Add these lines)

foobar-utils==0.1.0

pip install -r requirements.txt
@brettswift
Copy link

brettswift commented Aug 17, 2017 •
edited

any idea how to use this without the user dot files? That kinda sucks when you want to have other people set up on your team right away. Obviously these packages will be sourced from the same spot. I'm new to python - just wondering if it can go into a config file in the project, just no luck finding this on google yet.

Also supplying them with environment variables, so we can do continuous delivery of our modules from a CI server, would be good info :)

@brettswift
Copy link

https://pypi.python.org/pypi/twine

looks like this is a preferred way of deploying

@alexcthomas
Copy link

Could you post the equivalent nginx config?

@militiaonly
Copy link

@brettswift Actually there are parameter to tell pip to use a different server and without TLS. example:
pip install requests --index-url=http://mirrors.aliyun.com/pypi/simple/ --trusted-host=mirrors.aliyun.com

@JulienPalard
Copy link

Nginx config I used today:

server {
    listen 443 ssl;
    server_name  {{ pypi_domain }};
    include snippets/letsencrypt-{{ pypi_domain }}.conf;
    location / {
        root /data/pypi/;
        autoindex on;
        try_files $uri $uri/ =404;
        client_body_temp_path /data/pypi.tmp/;
        dav_methods PUT;
        create_full_put_path on;
        dav_access group:rw all:r;
        limit_except GET {
                        allow {{ ci_public_ip }};
                        deny  all;
        }
    }
    index index.html;
}

We're uploading from our CI with a simple curl -XPUT https://pypi.redacted/simple/demo-0.0.2.tar.gz --data-binary @dist/demo-0.0.2.tar.gz from a whitelisted IP and it's done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment