Are you writing a simple library? Create a setup.py. Copy paste an existing setup.py and modify it to suit your purposes. Now you have a working, pip installable python package.
Want to publish to PyPI? Use twine. It's standard and it's simple.
You don't need complicated tooling for simple projects.
1) It’s batteries included. The standard library is a core feature, and it is extensive in order to reduce the number 3rd party library dependencies.
2) “There should be one— and preferably only one —obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.” The Pythonic way is to use the standard library, setup.py, pypi, and so on. Python is not Rust, or Node, or Clojure, or anything else. Do it Pythonicly and good things will happen.
In the history of Python, it’s only because of the Zen’s guiding principles and the “come join us” attitude of the community that Python accidentally became “enterprise grade”. It succeeded in spite of the lack of Cargo or NPM, or (shudder) Mavin[, or CI/CD, or 3D game engines, or easy binary package productization, or everything else Python is obviously bad at]; almost precisely because it’s not those things. Python encourages me to get my task done, and all it asks is that I leave the flame wars at the door and embrace it for what it is.
It saddens me to see so many people jump down zomglings throat for making the daring suggestion that Pythonistas use the language as designed. Because, he is absolutely right about this: “Complex is better than complicated.”
Like it or not Python is now “enterprise grade”. The sooner it either grows up and figures out packages, ci/cd, binaries etc, or it looses its reputation at a great beginner and general programming language to something like Julia or Crystal, the better.
In the python ecosystem, poorly defined packages is a wide problem. You never know what you get in the next version upgrade.
So my suggestion: burn all the "simple" python packaging tools in a big fire and move eveything to Poetry today. It will be painful at start but will increase the quality of the ecosystem by a huge margin if just 50% of the projects do this.
Rust is incredibly lucky to have been created when it was because it benefits immensely from these things (I think it's the best new-to-middle-aged production-usable systems language out there today).
I agree with the idea but languages like Python and their ecosystems are really hard to move (remember python 2->3? is that even over?) -- it's a herculean and often impossible task.
If poetry create consume non-poetry packages and create packages which other package management systems can consume? If so, then:
1. Projects can move more effectively independently.
2. Projects can re-package as a single task rather than a massive rewrite effort.
It's the best one I've tried so far.
* virtual environments (granted, venv is now part of the stdlib, but it's still an "extra tool")
* publishing to PyPI or another index (twine)
* dependency management (both for development and actual dependencies)
* unit tests (pytests)
* static typing (mypy)
* linting (flake8, pylint)
* styling (black)
Finally, tools like Poetry  offer a whole set of functionalities in one place (dependency management, virtual environments, publishing), which means that they need fewer "extra tools" than just setuptools.
poetry build -f wheel
Sure, experts know how to do all these things because they spent many days learning them, but I'd rather outsource to a tool.
Version pinning can be done in setup.py using the same syntax you would see in a requirements.txt file. You should be very conservative when pinning versions in a library, though.
You can lean on your ci tool (eg. Github actions) to handle testing, hash checking, credential management, etc. But I recommend all of this start as a bunch of locally runnable scripts.
I typically bump version directly in a version file and move on with my life.
This stuff usually builds up iteratively and at least for me has never been the starting point. Starting point should be a library worth sharing. It is not the end of the world of you release the first few versions manually.
Even if that set of tools is kinda crappy (Glances at go), it's just so nice to not have all the bikeshedding and just get on with it.
E.g. more flexible than Cargo in that you could have a large codebase with two different versions of a dependency isolated to their compile units allowing you to gradually adopt a breaking change dependency in a large code base. I was kinda bowled over with that feature (the isolation part is key).
For python I’m finding poetry much more ergonomic than pipenv, it’s not just the speed difference it’s the convenience of one tool which aligns with what you’re saying although the existence of poetry doesn’t delete historical aberrations in Python’s long history.
What we saw with npm was the entire community iterating towards a feature set and everyone reaping the benefits automatically with npm updates. package-lock.json is a good example of this.
Leiningen is the weird one where people did actually settle fairly well around an unofficial solution in the absence of an official one. I think the norm with languages that forego official tooling is closer to what we've seen in Python.
If you're curious, the email threads about Conda and defining wheels are interesting.
But with or without a BDFL, one packaging tool to rule them all is a pretty tall order. The needs of a package like scipy, which incorporates C & Fortran code, are pretty different from something like requests. And different communities using Python have their own entrenched tools and techniques. It takes more than someone saying "Foo is officially blessed" to shift that, even if everyone respects the person saying that.
Also, yes, Guido and many other early contributers have been been active for 30ish years.
And NumPy/Scipy/Pandas exist (and are successful) because...? /s
The js tooling is particularly immature, e.g. fake packages along with typo squatting is rife. Compare with boring old maven where that’s not been an issue in >10 years at this point.
Personally I much prefer pip over npm, it feels much more polished.
python is a mature, old, software system. three times older than go or rust. way older than zig. these modern languages have learned from the field as a whole and implemented tools that people are taking for granted nowadays.
as with any mature software system, people & companies have established their preferred ways of doing things. it's going to be hard to have the language council dictate ways of doing things.
i would recommend going simple and using the standard set of modules until poetry or whatever gets into it.
Strongly disagree. There are so many footguns with low-level tools like pip that I can't recommend it to anybody but an expert (but an expert doesn't need my recommendation anyway).
It's the succeeding sections (A, B, C, D, E) that get more advanced, but they're all optional. You should definitely do A, but the rest I'd say it's a lot more opinionated and definitely not needed.
The problem with this approach is that it doesn't handle transitive dependencies well. Say you depend on version 1.4.6 of a particular library. And then that library depends on version >= 2 of some other library. When you install your package, you know that you'll get version 1.4.6 of the first library but have no idea what version you'll get of the second library. You can of course pin all the transitive dependencies - except that clutters up the setup.py and is a massive pain to keep up to date as you bump individual dependency versions.
the version selected during a build is the one with the most minimal version that satisfies all other constraints. this means if you have libA that needs dep>=1.1 and libB that needs dep>=1.3, you get dep=1.3 even if dep1.9 is out. your build never changes because of a new version release, as long as they release with proper semantic versioning. if you later include libC that needs dep>=1.8, you'll get that version. but because you changed your immediate dependencies, not due to a surprise down the dependency line.
Imagine that you depend on library A of a particular version which itself depends on library B. With minimal version selection, as long you don't bump you dependency on library A (or some other library that depends on library B), you'll continue to get that same version of library B. But, then library B releases a critical security fix. With minimal version selection, there isn't a great way to pick up that fix. You can _hope_ that library A releases a new version that requires the fix - but that may or may not happen and could take a while. Or, you could add an explicit dependency on the new version of library B - which is unfortunate, since, your main package doesn't depend on library B directly.
Lock files solve this problem. You can depend on whatever version of library A that you need and lock the transitive dependencies. And once library B releases its fix, you can update your lock file without having up bump the version of library A.
Tools like poetry providing the features to automate this workflow.
No; you should be very conservative when pinning versions in an application, not in a library. Check this article for the explanation: https://caremad.io/posts/2013/07/setup-vs-requirement/
Why spend time tweaking the setup when you can just get it right in the first place with less work?
I've been meaning to try it but haven't had the time to migrate an existing project to poetry.
poetry new foo
There is still room for improvement, e.g. I had some trouble with their install script at times (python vs python3)
If you're an expert, you can do it with setup.py or setup.cfg, but I don't think it's normally worth the trouble.
You seem to be confusing packages with "apps". It's very important to understand the clear distinction between these.
I'm not confusing libraries with applications. Pinning dependency versions enables a repeatable test environment.
It's by far the simplest option I've found.
If your project is a library, just use setup.py and express your deps abstractly (min, max or version range). Don't pin to specific versions at all, if you can help it.
The author is correct that you want a tool such as flit or poetry which will work with pyproject.toml. Setting up a basic package will be no harder than using setuptools, and it is much more future-proof. You won't have to copy-paste some other crufty setup config either.
It is fair that you don't need all the external tools in this tutorial. In particular, using make is very silly since you can configure different linting and testing workflows directly in pyproject.toml, rather than pull in a whole other system which only works decently on *nix. Poetry also removes the need for tox.
It does not correctly verify hashes (if at all) . You can't add packages without updating all your dependencies. Monorepos are not supported. PEP-508-compliant Git dependencies cause the tool to crash with TypeError .
I think Poetry is the right direction, I use it for everything, but it's not the silver bullet you're painting it to be (yet). It's definitely not on par with Cargo, or maybe even npm.
I have to say that it probably isn't a fare comparison because python is much older than nodejs. Python package management might very well have been state of the art in 1995.
This approach ends up with multiple, potentially-incompatible versions of the same package in a project. True that's less of a problem in JS, since it's interpreted (deferring imports to runtime) and un(i)typed (no need to check if interfaces match up). Yet even that has lead to replacements/complements like yarn.
Yes. There's hardly even a standard directory structure, let alone a standard way to convert source code to published code. Every slightly non-trivial repo basically has an ad hoc build system of its own. Ever tried to fix a bug in a package, and realized that using git://github.com/user/repo#branch doesn't work, because npm only downloads the source code, which bears no resemblance to the built products? I fixed two bugs in two third party packages within the past week, had to deal with this twice. Ran into the Node 12+ and Gulp 3.x incompatibility issue twice in the past month (with totally modern, actively developed packages), too.
npm has more sophisticated dependency resolution and locking than pip, sure. Python packaging is more consistent in basically every other regard.
If data generated by one version of a library is being consumed by another version, that's a bug in the code that moves data between them.
I no longer start Python projects without Poetry. It's really good.
EDIT: Also, it's being integrated to most PaaS as well. I deploy to Render.com with Poetry now.
No it doesn't. Neither setuptools nor pip are part of the standard library. Yes, they are installed by default in many cases, but they are still "extra tools".
If you've solved 1-99 and download size is an issue, you're in heaven already.
Poetry doesn't fragment the ecosystem. Unlike setuptools it uses pyproject.toml, which can be read by other tools, and is the correct way of storing package configuration.
A package built using Poetry is installable without Poetry in the exact same way as one built using setuptools.
See details upthread https://news.ycombinator.com/item?id=26739234
> Other unethical or unprofessional conduct
from https://jazzband.co/about/conduct ? Or something else?
I just published a repo today using Poetry and it didn’t take me more than 5 minutes. Poetry build && poetry publish
This is one of the most ridiculous issues I've read. If the rest of the "drama" is like that, then eh.
I did like the emojis in the logs though.
Poetry by contrast works pretty much like any modern dependency system you'd be familiar with from another language like cargo, npm, or hex.
With a title like this, I'd be expecting to see an article describing the latest tools and recommendations from the PyPA, which are here:
(In short, it's setup.cfg + pyproject.toml, `python3 -m build` to build, twine to upload.)
Though `python -m build` only works _if_ you use something like flit or setup.py in the backend to do build the package and hence why you can set flit as a build-backend.
So yes, flit is one of the latest tool, and yes it is one of the things that push for the ability to use pyproject.toml+python3 -m build, you just seem to miss some subtleties of the toolchain.
Flit is more aimed at being the simplest possible thing to put a package on PyPI, if that's all you want to do. It expects you to list dependencies in pyproject.toml manually.
The nice part about Pip-tools versus Flit or Poetry or Pipenv is that Pip-tools lets you keep using Setuptools if you want to, or if you're unable to switch to one of the others for some reason (and valid reasons do exist).
- Write a simple setup.py file.
- Generate a source or binary release by e.g. running "python setup.py sdist"
- You're done!
Adding a setup.py file is already enough to make your library pip-installable, so you could argue that this is a package already. The files generated by "setup.py" can also be pip-imported, so they are also packages. Now you might want to upload your package to a repository, for which there are different tools available. The simplest one being twine. Again, you just install it and run "twine upload -r dist/*" and your packages get uploaded to PyPi (it will ask for a username and password). So why complicate things?
(flit install does wrap pip, however)
If you've never run into setuptools compatibility problems, you've either been much luckier than me, or you haven't done much with Python packages.
Vanilla ubuntu used to come with a version of setuptools which didn't work for installing many recent packages.
pip install git+https://myg.it/repo.git
pip install https://github.com/django/django/archive/master.zip
pip install https://github.com/django/django/archive/stable/1.7.x.zip
More controversially: With a few (<10) lines of code in your setup file, you can make an install-able module out of jupyter notebooks.
And what if some of the dependencies are incompatible with the versions already on your system?
Even as a long time Python user, the packaging ecosystem feels fragmented and error-prone at best. Honestly, it sours the experience of writing Python code knowing you might eventually need to make it work on another computer.
What I actually like is using the system package manager to install stuff. pacman -s or apt-get install
letting multiple packaging systems muck with your system is a recipe for hurt somewhere down the line.
It's great for beginners and experts. As a long time python veteran it completely changed how I work with python for the better. It is lengthy with lots of optional steps. Just skip the ones you don't find relevant.
I do feel like tox gets a bad rap despite having a complete feature set. I think a part of it is that the documentation is complete but not organized in the "Tox user"'s perspective, so for someone who shows up on a project using it, it's hard to figure out the quickstart( though the "general tips and tricks" page gets somewhere )
Anyways yeah, would not recommend Make over just leaning into tox more here.
EDIT: also, this article reminded me of how much I really dislike Github Action's configuration syntax. Just balls of mud on top of the Docker "ball of mud" strategy. I will re-iterate my belief that a CI system where the configuration system isn't declarative but just like..... procedural Lua will be a billion dollar business. CI is about running commands one after another! If you want declarative DAGs use Bazel
The recipes for each target describe not only how a project intends to run each tool, but which tools it intends to run. Instead of having to know that this project runs tests under tox, while that one runs only under pytest, we can run ‘make test’ in each, and count on the recipe doing the Right Thing.
That consistency across projects makes it much easier for someone to get started on a new project (or to remember how the pieces fit together on your own project from a few months ago)
I generally agree with your sentiment, though. I’m usually limiting myself to Python stuff so don’t have much exposure to make, it’s always felt like a less powerful task runner than other stuff
When phony targets are all written out in the beginning of the Makefile, it's easy to forget to alter this list when a phony target is added or removed later.
This rarely causes an error, but still I've seen many Makefiles with outdated .PHONY lists.
It sets up GitHub actions for publishing the package to PyPI when you create a release on GitHub - which I find way to be a really productive way of working.
Here's an example of how I like to do my "bash scripts that sorta work like make": https://github.com/francislavoie/laravel-websockets-example/... basically each function is a "command", so I do like "./utils start" or whatever.
If you really want make, you can also just call out to your bash scripts from make:
If you want to see a powerful and actually simple tool that does the job, take a look at DJB redo:
Bash, Python, or even Typescript are much easier, safer, and more widely standardized environments to maintain and grow your scripts once you get past a few lines.
There are a bunch of lightweight alternatives to Make out there (I hear Ninja is pretty good). My personal preference is Nix these days (although that's quite heavyweight).
Python is fantastic at gluing specialized tools/libraries, but a lot of these require non-python dependencies (most are written in more performant languages). IMO, this is a big differences when comparing with Cargo for Rust because most of the dependencies in Rust are written in Rust.
The state of packaging in Python is kinda meh, the official documentation here  suggests to create 3 additional files in order to create a package:
- setup.py # optional, needed to make editable pip installs work
- meta.yaml # to build you conda package
 -- https://packaging.python.org/tutorials/packaging-projects/
The worst part of Python.
I have no problems using setuptools directly, as outlined in https://packaging.python.org/guides/distributing-packages-us...
`gem` from Ruby does that (well, it doesn't 2 AFAIK but at least it does 1)
1) venv at the same level:
create new project on GH
git clone https://github.com/user/myproject
python -m venv myproject
2) clone first, venv in subdirectory:
python -m venv venv
3) something completely different? if so, what?
Flit is not an unreasonable pick, but it's not a silver bullet.
> dynamic metadata [(setup.py)] should be used only as an escape hatch when absolutely necessary.
I've gotten to the point where a number of smaller tools I've put together can now be used in larger projects. I learned the hard way that just copying files around makes it hard to know which version is in that project, and upgrading, especially once there is more than one file, becomes a lot tougher. I learned the harder way that trying to link the same file into multiple projects is a great way to really screw things up.
About 4 or 5 months ago I made a real effort to try to learn how to use virtual environments (pipenv) and packaging to make it so that if I update one of the smaller tools, I don't clobber all the downstream projects that rely on it. I wanted to make it so that when I update something to add features or change things, I can go back and fix older projects it's used in at my leisure. I haven't even begun to touch on unit testing, and I have no clue what linting is. Things are kind of working so far, but it feels very hacky and fragile, and I know it can (and SHOULD) be better.
All of this stuff around packaging and being able to install those packages very daunting, and trying to stumble on the right tutorials is extremely frustrating. The vast majority of them assume I want to share my stuff with the world on PyPI, or that I have servers available to me to create private PyPI indexes, but I don't. Yet I still want my packages to "resolve their own dependencies" when I install or upgrade them.
And when it comes to learning things like testing, the few tutorials I've looked at either use different tools to do it, or their examples are so oversimplified that when I look at my own code, I don't know where to begin.
I say all of this because looking at this tutorial, it's more of the same. I want to make my code better. I want to make it easier to use those smaller projects in larger projects. But then it says things like "Every solid open-source project runs cloud tests after each commit, so we will too," but it doesn't do anything to explain what that is or why it should be done, besides "everyone does it, so you should, too."
I think what makes it even harder is that when something like this gets shared, there are so many conflicting opinions. Some people say to just use setuptools, other say that setuptools is on its way out and to use pyproject.toml (or some other tool) instead. It's all just so... hard!
I'm sorry. This is coming off a bit ranty, and that's not what I intended. I'm just feeling frustrated and I'm not sure of a better way to express that I need help with finding help. There are even a lot of things that I'm sure I need help with, but I just don't know what they are. It makes it really hard to verbalize what I need to another person, let alone to get the right words into a search engine to take me there.