We have 100+ private packages and so far we've been using s3pypi to set up a private pypi in an s3 bucket. Our private packages have dependencies on each other (and on public packages), and it is (of course) important that our GitLab pipelines find the latest functional version of packages it relies on. I.e. we're not interested in the latest checked in code. We create new wheels only after tests and qa has run against a push to master (which is a long-winded way of explaining that -e <vcs> requirements will not work).
Our setup works really well until someone creates a new public package on the official pypi that shadows one of our package names. We can force our private package to be chosen by increasing the version number so it is higher than the new package on pypi.org - or by renaming our package to something that haven't yet been taken on pypi.org.
This is obviously a hacky and fragile solution, but apparently the functionality is this way by-design.
After the initial bucket setup s3pypi has required no maintenance or administration. The above ticket suggests using devpi but that seems like a very heavy solution that requires administration/monitoring/etc.
GitLab's pypi solution seems to be at individual package level (meaning we'd have to list up to 100+ urls - one for each package). This doesn't seem practical, but maybe I'm not understanding something (I can see the package registry menu under our group as well, but the docs point to the "package-pypi" docs).
We can't be the first small company that has faced this issue..? Is there a better way than to register dummy versions of all our packages on pypi.org (with version=0.0.1, so the s3pypi version will be preferred)?
It might not be the solution for you, but I tell what we do.
company.product.tool).requirements.txt file including our PyPI URL. We run everything in container(s) and we install all public dependencies in them when we are building the images.Your company could redirect all requests to pypi to a service you control first (perhaps just at your build servers' hosts file(s))
This would potentially allow you to
We use VCS for this. I see you've explicitly ruled that out, but have you considered using branches to mark your latest stable builds in VCS?
If you aren't interested in the latest version of master or the dev branch, but you are running test/QA against commits, then I would configure your test/QA suite to merge into a branch named something like "stable" or "pypi-stable" and then your requirements files look like this:
pip install git+https://gitlab.com/yourorg/yourpackage.git@pypi-stable
The same configuration will work for setup.py requirements blocks (which allows for chained internal dependencies).
Am I missing something?
You could perhaps get the behavior you are looking for from a requirements.txt and two pip calls:
cat requirements.txt | xargs -n 1 pip install -i <your-s3pipy>
pip install -r requirements.txt
The first one tries to install what it can from your local repository and ignores a package if it fails. The second call tries to install everything that failed before from pipy.
This works because --upgrade-strategy only-if-needed is the default (as of pip 10.X I believe, don't quote me on that). If you are using an old pip you may have to specify this manually.
A limitation of this approach is if you expect/request a local package, but it doesn't exist and a package with the same name exists on pipy. In this case, you will get that package instead. Not sure if that is a concern.