It always bugs me to think about the Python community not rallying around PyPy. Faster Python is right there and has been there for years. Sure there can be C-extension issues but projects like HPy showed you can remedy that. Why doesn’t everyone just use and support PyPy?
"C-extension issues" are not something you can just gloss over. They're a huge chunk of python usage. Users won't rally around that. And Hpy is dead. If CPython embraced PuPy, worked with it to smooth those issues then maybe things would be different.
I do think Go and Java are strong examples of this. Those languages have native code support, but it’s clunky to use and, more importantly, very rare.
If you removed JNA/JNI/CGo, plenty of people would complain…but the vast majority of uses of those languages would still work, unaltered, because most common tasks on the JVM and Go runtime don’t require external native deps.
Well, see popularity of CGO among Go devs, or JNI among Java devs, or the ongoing work to make all the learnings from System C# (from Midori) available in regular C#, while C++/CLI is slowly left in maintenance mode (last updated for partial C++20 support, and not part of FOSS .NET)
All of those are for non native code to call code in those languages. Nobody is saying "all Go code must never invoke native code". That would be wild and to my knowledge, there's no major language that forces such a constraint.
I can't tell if you're being willfully ignorant or not. The comment I was replying to was proposing banning C extensions: invoking native binaries that are not written in the host language and compiled in (at build or runtime). What you're describing has nothing to do with that.
Why would a Go binary (to keep the examples short), compiled to native code, with a compiler toolchain that includes support for Assembly if necessary, and a syscall package as well, have to depend on C extensions?
PyPy makes tradeoffs that might not be acceptable. For anything that isn't a long running server, PyPy can be much, much slower since the JIT doesn't have a chance to warm up. PyPy also implements things differently: repeated string concatenation in a loop is quadratic instead of linear like in CPython. There's also memory tradeoffs, depending on what you're doing.
The list of differences is quite long, and even though most of them are edge cases, it's not hard to get bitten by one on a project of any substantial size:
It's not that people are writing weird code, it's that you have a package that does something clever but safe, and then another package using it that calls it in a way that's unexpected but not documented as unsafe, and layers and layers of that compound until you have things like exceptions raised because non-string keys are getting set on type objects.
At the end of the day, Python isn't a terribly well-specified language. CPython behavior is functionally the specification. "Make it match CPython but faster" isn't really an option because it kills your ability to do a lot of the things that make PyPy fast. And if that was all it took, we wouldn't still be arguing over a JIT inside CPython. Pyston was an honest attempt at doing that, but it has been dead for years partially because it hit many of these walls.
> but it has been dead for years partially because it hit many of these walls
Maybe this is one of the reasons. But I've also heard that Dropbox (which was making Pyston [0]), that used to heavily rely on Python, just decided that gradual migration to Go [1] is a more feasible approach than creating a faster Python implementation.
It always bugs me to think about the Python community not rallying around PyPy. Faster Python is right there and has been there for years. Sure there can be C-extension issues but projects like HPy showed you can remedy that. Why doesn’t everyone just use and support PyPy?
"C-extension issues" are not something you can just gloss over. They're a huge chunk of python usage. Users won't rally around that. And Hpy is dead. If CPython embraced PuPy, worked with it to smooth those issues then maybe things would be different.
To the point that in Python culture those C extensions are seen as Python libraries, not bindings.
Maybe PyPy needs a Steve Jobs-like figure to write a "no C extension" manifesto, declaring that all Python apps must be 100% Python.
Is there a single example of a language+runtime where this approach has ever been successful
I do think Go and Java are strong examples of this. Those languages have native code support, but it’s clunky to use and, more importantly, very rare.
If you removed JNA/JNI/CGo, plenty of people would complain…but the vast majority of uses of those languages would still work, unaltered, because most common tasks on the JVM and Go runtime don’t require external native deps.
Well, see popularity of CGO among Go devs, or JNI among Java devs, or the ongoing work to make all the learnings from System C# (from Midori) available in regular C#, while C++/CLI is slowly left in maintenance mode (last updated for partial C++20 support, and not part of FOSS .NET)
All of those are for non native code to call code in those languages. Nobody is saying "all Go code must never invoke native code". That would be wild and to my knowledge, there's no major language that forces such a constraint.
All the languages I mentioned compile to native code.
I can't tell if you're being willfully ignorant or not. The comment I was replying to was proposing banning C extensions: invoking native binaries that are not written in the host language and compiled in (at build or runtime). What you're describing has nothing to do with that.
Why would a Go binary (to keep the examples short), compiled to native code, with a compiler toolchain that includes support for Assembly if necessary, and a syscall package as well, have to depend on C extensions?
CGO only exists as a matter of convenience.
PyPy makes tradeoffs that might not be acceptable. For anything that isn't a long running server, PyPy can be much, much slower since the JIT doesn't have a chance to warm up. PyPy also implements things differently: repeated string concatenation in a loop is quadratic instead of linear like in CPython. There's also memory tradeoffs, depending on what you're doing.
The list of differences is quite long, and even though most of them are edge cases, it's not hard to get bitten by one on a project of any substantial size:
https://doc.pypy.org/cpython_differences.html
It's not that people are writing weird code, it's that you have a package that does something clever but safe, and then another package using it that calls it in a way that's unexpected but not documented as unsafe, and layers and layers of that compound until you have things like exceptions raised because non-string keys are getting set on type objects.
At the end of the day, Python isn't a terribly well-specified language. CPython behavior is functionally the specification. "Make it match CPython but faster" isn't really an option because it kills your ability to do a lot of the things that make PyPy fast. And if that was all it took, we wouldn't still be arguing over a JIT inside CPython. Pyston was an honest attempt at doing that, but it has been dead for years partially because it hit many of these walls.
> Pyston was an honest attempt at doing that
I'm still kinda sad that Pyston didn't happen
> but it has been dead for years partially because it hit many of these walls
Maybe this is one of the reasons. But I've also heard that Dropbox (which was making Pyston [0]), that used to heavily rely on Python, just decided that gradual migration to Go [1] is a more feasible approach than creating a faster Python implementation.
[0] https://dropbox.tech/infrastructure/introducing-pyston-an-up...
[1] "About a year ago, we decided to migrate our performance-critical backends from Python to Go to leverage better concurrency support and faster execution speed." - https://dropbox.tech/infrastructure/open-sourcing-our-go-lib...
A lot of Python code is just a relatively thin layer of "glue" between C libraries.
An ML pipeline will probably rely on numpy to do all the heavy lifting.
A web backend will probably use something like psycopg, which is a wrapper around libpq (official postgres client lib, written in C).
etc
like bun?