I guess that because Python is extremely dynamic, we may never have a full everything-works compiler, but I’m excited about the possibility of this becoming some kind of intermediate step where different parts of the program get compiled when possible.
Edit: apparently GraalPython is a thing.
D compiles quickly and has much nicer syntax than C or C++.
: https://forum.dlang.org/post/[email protected]
"D, the Best Programming Language, for Former Python Developers: (Learn D Programming for Python Developers)"
It generally feels pretty pythonic (At least as much as anything typed can), and it certainly scratches the "compiles to efficient machine code with no dependencies" itch.
Who would have guessed....
typed_python is new to me. I'll check it out. I'm too am keeping an eye on this space. I think that compiling or transpiling python may be the solution to both the major problems I have with python: performance and distribution. Exciting times.
I compared it to cython, numba and Julia for DSO, which I wrote about here: https://jochenschroeder.com/blog/articles/DSP_with_Python2/
There isn't any Python dynamic feature that those languages lack, yet they solved out the problem a couple of decades ago, and SELF JIT is the genesis of HotSpot.
What Python lacks is JIT love and being too much attached to CPython.
It is quite promising though, if it becomes more robust and compatible. I also believe they have still only scratched the surface of possible optimizations.
As for the performance gain - 13.5 s with Python, 9 s compiled. It was a naive implementation of AoC 2020/23, so a lot of array cutting, concatenation etc. So this isn't really math, rather lot of RAM I/O
The reason goto is considered bad is that it can make code hard to follow for humans. Since this is an intermediate step in compilation, that's not an issue here.
What is the difference between cython and mypyc? I think they should answer the question why anyone would want this over cython on the readme.
Mypy aims to be compatible with the standard Python type annotations and still be able to optimize them. So in theory, you don’t need to modify your existing type-annotated program. In practice I believe there are limitations currently.
I'm not sure that the Numpy/Pandas hegemony over Python scientific computing will last. Eventually the ecosystem might move toward Arrow or something else. In this case it's probably not such a big deal because Arrow's mainstream debut will probably predate any serious adoption of Cython, but if it didn't then the latter would effectively preclude the former--Arrow becomes infeasible because everyone is using Cython/Numpy and Cython/Arrow performance is too poor to make the move, and since no one is making the move it's not worth investing in an Arrow special case in Cython and now no one gets the benefits that Arrow confers over Numpy/Pandas.
[^1]: Yes, Pypy exists and its maintainers have done yeoman's work in striving for compatibility with the ecosystem, and still (last I checked) you couldn't do such exotic things as "talking to a Postgres database via a production-ready (read: 'maintained, performant, secure, tested, stable, etc') package".
Arrow is a low-level implementation detail, like BLAS. "Using" Arrow in data science in Python would mean implementing an Arrow-backed Pandas (or Pandas-like) DataFrame.
Your rank-and-file data scientist doesn't even know that Arrow exists, let alone that you can theoretically implement arrays, matrices, and data frames backed by it.
If you want to break the hegemony of Numpy, you will have to reimplement Numpy using CFFI instead of the CPython C API. There is no other way, unless you get everyone to switch to Julia.
C = A + B
Python with Numpy perfectly service just that need. We all have our grief with the status quo, but Python needs data processing acceleration from somewhere. In my view, Python needs to implement a JIT to alleviate 95% of the need for Numpy.
> In my view, Python needs to implement a JIT to alleviate 95% of the need for Numpy.
Numpy is just statically typed arrays. This seems like best case for AOT compilers, no? I'm all for JIT as well, but I don't have faith in the Python community to get us there.
I say this as a Computer Scientist at NASA that tends to re-write the scientific code in straight C. But for many workloads, a JIT would make my team more productive, basically for free as a user.
The problem is not that Python lacks JITs, rather the community culture of rewriting code in C instead of contributing to JIT efforts.
Personally I just use a JVM/.NET based language, and if I need I can use the same C, C++ and Fortran libraries that Python uses anyway.
The hope is to create a new C API which doesn't expose CPython interpreter details, is easily exposed by interpreters other than CPython, and then port C-based APIs to it. Sadly it seems they aren't making much progress in 2020/2021. And I don't think it will eliminate Cython/Numpy overhead entirely, so Cython adding Numpy-specific features will still improve performance.
Also Pypy now has a compatibility shim for CPython extension modules. But last time I checked, it was slower than CPython for running one of my Numpy-based programs (corrscope), due to interfacing overhead.
Even without types cython provides a neat way to embed your code and the interpreter into a native executable and has applications for distributing python programs on systems that are tricky for python like Android and WASM.
This seems like an important difference to me. Your regular type annotations can be used.
Obviously organizations that don’t manage human lives or large amounts of money can use ‘riskier’ tools without as much worry. This isn’t an argument against cython generally. But I worked at a hospital and wrote a lot of Python, and would not have been able to get the security team to support cython on their SELinux servers without a really good argument. Cython is just an unnecessary liability when your job manages identifiers and medical details on servers accessible on a fairly wide (albeit private) network.
Note that GraalPython has the C structs memory layout too.
The biggest issue right now seems to be miscompiles and the resulting errors being a bit inscrutable. It leaves you in the “am I wrong or is the system what’s wrong?” stuff a bit still.
But overall I think the techniques are really sound and I believe this is the most promising way forward for perf in Python.
- "batteries included" including a massive set of libraries (any one of which won't be supported by the compiler)
- dynamism which makes it easy to wrangle syntax to your needs (including the creation of domain-specific languages), but which destroys the performance improvement of compilation, even if the compiler can handle all the crazy introspection.
One consequence is that it never makes sense to use static typing or compilation as application-wide absolutes for any language or paradigm.
You should virtually never be writing whole applications in Rust, C, C++, Java, Haskell, etc. It is a huge sign of bad premature optimization and dogmatism. Compiling subsets in these languages and then exposing them through extension modules in other languages that don’t force those constant trade offs is almost always superior, and it’s very telling about poor engineering culture when this results in debates or vitriolic dismissiveness from people with prior dogmatic commitments to static typing or AOT compilation.
But you seem to think everyone agrees that dynamic languages are more productive and that using (say) haskell is a trade-off for performance. For people used to (good) static type systems — for me that'd be OCaml — this is just not the case. Types do not impede, they help. I guess it might be a question of taste or habit, but don't make it a universal truth and accuse others of being biased when they disagree.
I say that as someone who spent most of a decade doing Haskell and Scala professionally in large companies that built lots of developer tooling and workflows for them.
The most critical aspect of business software is to be able to drop into any particular local section of the code and make significant changes according to shifting business constraints. Anything that enforces a program structure that makes this harder to do, or requires a sequence of significant refactors around things like type class design / OOP interfaces and so forth, is strictly a loss for the business, not a win, even considering correctness, safety, a developer efficiency as critical success measures.
It’s often much worse than “a loss for the business” too, given that 99% of the time, those type class designs and OOP interfaces or nested inheritance models were premature abstractions and all the extensibility or well factored SOLID code (or equivalent ideas in FP) winds up being sheer debt that fails to be extensible in the ways that reality turned out to require but which nobody foresaw.
In a world with excellent foresight and ability to hit pause to refactor architectures, then baking in domain modeling constraints through type system designs would be great. Unfortunately that doesn’t map to the real world at all.
You say this, but then never address this claim in the text below. Could you expand on it?
You also do a lot of conflating typeclasses and OOP concepts like interfaces and inheritance, while those OOP concepts don't have much of anything to do with static typing and exist in object-oriented dynamic languages as well.
The "types prevent you from easily changing things to meet business needs" argument is one I've heard a lot but I'm not familiar with any concrete scenarios where that would be the case. Do you have any examples you can share from your time working with Haskell or Scala?
I believe I did answer this in my original comment, so I will just refer you back to that.
I disagree that there was any conflating going on. Type system designs enforced with static typing are a hallmark aspect of most of these design patterns around things like type classes, interfaces and inheritance. Of course similar things can exist in dynamically typed languages but they are not the same. For example “interfaces” in Python are just duck typing conventions (apart from built-in CPython data model properties). That duck typing interface is not at all similar to interfaces as a type system design pattern in a statically types language. Any similarity is purely semantic.
As for examples, one example that I worked on heavily involved a Scala system for managing DAG dependencies in task execution. The system was set up using phantom typing and a bunch of sealed case classes such that for any logical type of Task that could exist in a DAG, the task had an “Active” and “Passive” variant, where the “Active” variant could only be obtained through a monadic validator processing a “Passive” variant.
The goal was to use the type system itself to encode the concept of “this task has passed through validation and it’s allowed to be processed.”
Because this was designed at the type system level, it created huge problems and never added any real value in the sense of making it “logically impossible” to create invalid Tasks. Number one, it led to huge, painful boiler plate to create the case classes for every type of Task and specialize the type class with a “validator” function. Number two, we eventually realized there were many different aspects to “validation” that did not map well to the concept of “passing through a validator.” For example, some tasks depended on data that didn’t exist in the required location at a certain time, and hence needed retry logic to validate. Some situations involved re-running an already complete task (usually for resource usage observability reasons, or because an external data dependency changed). At any rate, baking validation status into a static type via the phantom type design was nothing but a headache. For all the beautiful code supposedly protecting us from processing invalid jobs, all that we got was difficult constant refactors.
Eventually we abandoned it and just used Luigi instead, and wrote all DAG management code in Python. It was the best decision we made. We lost zero safety and our defect rate did not get worse. Testing caught all the same bugs that compilation would have caught in Scala, and more, with less total code. And because the nature of the tasks in Luigi was just “whatever arbitrary Python you want” it was super easy to write effective validators, accommodate new use cases on the fly, and keep the code clean without dogmatic adherence to a precommitted type system design. Luigi happening to use some lightweight inheritance patterns was forgivable, given the dynamic typing flexibility.
If I have a pure python, fully type hinted library I'm consuming, hats off to them, and they choose to use this, awesome.
Which is why this compiles specified modules, which can freely call noncompiled modules, not “complete Python programs”.
It makes sense for distribution of apps to end users, which is a particular pain point with Python.
After evaluating a number of different solutions, I ended up being quite happy with pex: https://github.com/pantsbuild/pex
It basically bundles up the wheels for whatever your workspace needs, and then ships them in an archive with a bootstrap script that can recreate that environment on your target. But critically, it natively supports the idea of targeting multiple OS and Python versions, you just explicitly tell it which ones to include, eg:
--platform=manylinux2014_x86_64-cp-38-cp38 # 16.04
--platform=manylinux2014_x86_64-cp-36-cp36m # 18.04
--platform=manylinux2014_x86_64-cp-35-cp35m # 20.04
And you can see the tags in use for any package on PyPI which ships compiled parts, eg: https://pypi.org/project/numpy/#files
I don't know that this would be suitable for something like a game, but in my case for a small utility supporting a commercial product, it was perfect.
Why wouldn’t you want to use pip?
In my particular case the "application" was in fact interactive bootstrap/install scripts for a large, proprietary blob which wouldn't have been suitable for publishing on PyPI, anyway. Setting up a separate, possibly authenticated PyPI instance, and then training end users how to use it, vs just shipping everything together in a single package? Total non-starter.
Definitely a complicating factor was that all of this was meant to be usable by non-nerds and in an environment with limited or possibly no internet access. It wouldn't have been acceptable to download your installer package at the hotel, and then get to site and invoke it only to discover that you were on the hook for a few GBs of transfer from docker.io.
Is there any way to say "no, a really want a __dict__ class here, please"?
Write it in a module you aren't compiling, and import it, since this supports compiled modules using noncompiled ones.
"The aim of Pyccel is to provide a simple way to generate automatically, parallel low level code. The main uses would be:
Convert a Python code (or project) into a Fortran or C code.
Accelerate Python functions by converting them to Fortran or C functions.
Pyccel can be viewed as:
a compiler for a Domain Specific Language with Python syntax"
When I say build static like a Go binary, I mean that the binary contains everything, and is not allowed to dynamically load anything at all. Also, preferrably it doesn't need a C standard library, and does system calls manually.
Just code? Or are you talking about data (e.g. HTML files) too? A lot of python's ecosystem seems pretty built around the assumption that non-code assets won't be bundled into single files, so any magic single-file Python compiler would probably have to include the ability to distribute those assets separately with the executable.
> preferrably it doesn't need a C standard library
I don't believe that golang even meets this standard--not with default options, at least. Also, why? How often have dynamically linked libc version differences really been a challenge for you in production?
> does system calls manually
This is generally considered to be a bad idea. Golang has moved away from this on several platforms, and people are making noise about doing it on Linux, too. A minority of compiled languages opt to go to this route. Why do you want this?
The big draw with mypyc has got to be direct integration with other source code in C.
Can anyone answer if it’s possible to replace PyPy’s VM backend with LLVM for AOT compilation? I wonder if that will results in any performance improvements.
Nowadays, CUDA hardware uses C++11 memory semantics, which are based on Java's memory model.
This is one of the reasons why Krhonos insistence in keeping OpenCL a "C99 dialect on GPUs" has not gained the love of most researchers, and now is too late to win them back, despite SPIR.
Would love to see some benchmarks on this.
The motivating use case is mypy, so I guess if someone wants to hand code mypy in native C we can assess this. But not doing that is as much, I would expect, of the motivation as speeding up mypy is.
What does it do?
My wager is that it does not. It may if you have math intensive code, but if you have an algorithm that touches lots of python built in datatypes, access to those types will be the bottleneck.