Love this. My https://llm.datasette.io/ CLI tool supports plugins, and people were complaining about really slow start times even for commands like "llm --help" - it turned out there were popular plugins that did things like import pytorch at the base level, so the entire startup was blocked on heavy imports.
Note that this is global to the entire process, so for example if you make an import of Numpy lazy this way, then so are the imports of all the sub-modules. Meaning that large parts of Numpy might not be imported at all if they aren't needed, but pauses for importing individual modules might be distributed unpredictably across the runtime.
Edit: from further experimentation, it appears that if the source does something like `import foo.bar.baz` then `foo` and `foo.bar` will still be eagerly loaded, and only `foo.bar.baz` itself is deferred. This might be part of what the PEP meant by "mostly". But it might also be possible to improve my implementation to fix that.
Parse the command line and do things like "--help" without doing the imports.
Only do imports when you know you need them -- or as an easy approximation, only if the easy command line options have been handled and there's still something to do.
Lazy imports have been proposed before, and were rejected most recently back in 2022: https://discuss.python.org/t/pep-690-lazy-imports-again/1966.... If I recall correctly, lazy imports are a feature supported in Cinder, Meta's version of CPython, and the PEP was driven by folks that worked on Cinder. Last time, a lot of the discussion centered around questions like: Should this be opt-in or opt-out? At what level? Should it be a build-flag for CPython itself? Etc. The linked post suggests that the Steering Council ultimately rejected it because of the complexity it would introduce to have two divergent "modes" of importing.
I hope this proposal succeeds. I would love to use this feature.
I also hope this proposal succeeds, but I'm not optimistic. This will break tons of code and introduce a slew of footguns. Import statements fundamentally have side effects, and when and how these side effects are applied will cause mysterious breakages that will keep people up for many nights.
This is not fearmongering. There is a reason why the only flavor of Python with lazy imports comes from Meta, which is one of the most well-resourced companies in the world.
Too many people in this thread hold the view of "importing {pandas, numpy, my weird module that is more tangled than an eight-player game of Twister} takes too long and I will gladly support anything that makes them faster". I would be willing to bet a large sum of money that most people who hold this opinion are unable to describe how Python's import system works, let alone describe how to implement lazy imports.
PEP 690 describes a number of drawbacks. For example, lazy imports break code that uses decorators to add functions to a central registry. This behavior is crucial for Dash, a popular library for building frontends that has been around for more than a decade. At import-time, Dash uses decorators to bind a JavaScript-based interface to callbacks written in Python. If these imports were made lazy, Dash would break. Frontends used by thousands, if not millions of people, would immediately become unresponsive.
You may cry, "But lazy imports are opt-in! Developers can choose to opt-out of lazy imports if it doesn't work for them." What if these imports were transitive? What if our frontend needed to be completely initialized before starting a critical process, else it would cause a production outage? What if you were a maintainer of a library that was used by millions of people? How could you be sure that adding lazy imports wouldn't break any code downstream? Many people made this argument for type hints, which is sensible because type hints have no effect on runtime behavior*. This is not true for lazy imports; import statements exist in essentially every nontrivial Python program, and changing them to be lazy will fundamentally alter runtime behavior.
This is before we even get to the rest of the issues the PEP describes, which are even weirder and crazier than this. This is a far more difficult undertaking than many people realize.
---
* You can make a program change its behavior based on type annotations, but you'd need to explicitly call into typing APIs to do this. Discussion about this is beyond the scope of this post.
Some of these worries make sense, but wouldn’t it be relatively trivial to pass a flag to the interpreter or something similar in order to force all imports to evaluate, as in the current behavior? But to be a bit cheeky if some of these issues cause serious production outages for you it might be time to consider moving on from a scripting language altogether.
The issue is that some imports can be made lazy and some cannot. A binaristic all-or-nothing approach does not address the issue. (I also think that there is zero basis to claim that adding such a flag is trivial, since there’s no reference implementation of this flavor of lazy imports.)
What if we have a program where one feature works only when lazy imports are enabled and one feature only when lazy imports are disabled?
This is not a contrived concern. Let’s say I’m a maintainer of an open-source library and I choose to use lazy imports in my library. Because I’m volunteering my time, I don’t test whether my code works with eager imports.
Now, let’s say someone comes and builds an application on top of this library. It doesn’t work with lazy imports for some unknown reason. If they reach for a “force all imports” flag, their application might break in another mysterious way because the code they depend on is not built to work with eager imports. And even if my dependency doesn’t break, what about all the other packages the application may depend on?
The only solution here would be for the maintainer to ensure that their code works with both lazy and eager imports. However, this imposes a high maintenance cost and is part of the reason why PEP 690 was rejected. (And if your proposed solution was “don’t use libraries made by random strangers on the Internet”, boy do I have news for you...)
My point is that many things _will_ break if migrated to lazy imports. Whether they should have been written in Python in the first place is a separate question that isn’t relevant to this discussion.
It's been explained many times before why this is not possible: the library doesn't actually have a version number. The distribution of source code on PyPI has a version number, but the name of this is not connected to the name of any module or package you import in the source code. The distribution can validly define zero or more modules (packages are a subset of modules, represented using the same type in the Python type system).
You got three other responses before me all pointing at uv. They are all wrong, because uv did not introduce this functionality to the Python ecosystem. It is a standard defined by https://peps.python.org/pep-0723/, implemented by multiple other tools, notably pipx.
> It's been explained many times before why this is not possible: the library doesn't actually have a version number. The distribution of source code on PyPI has a version number, but the name of this is not connected to the name of any module or package you import in the source code.
You're making the common mistake of conflating how things currently work with how things could work if the responsible group agrees to change how things work. Something being the way it is right now is not the same as something else being "not possible".
No, changing this breaks the world. A huge fraction of PyPI becomes completely invalid overnight, and the rest fails the expected version checks. Not to mention that the language is fundamentally designed around the expectation that modules are singleton. I've written about this at length before but I can't easily find it right now (I have way too many bookmarks and not nearly enough idea how to organize them).
Yes, you absolutely can create a language that has syntax otherwise identical to Python (or at least damn close) which implements a feature like this. No, you cannot just replace Python with it. If the Python ecosystem just accepted that clearly better things were clearly better, and started using them promptly, we wouldn't have https://pypi.org/project/six/ making it onto https://pypistats.org/top (see also https://sethmlarson.dev/winning-a-bet-about-six-the-python-2...).
The hard part is making the change. Adding an escape hatch so older code still works is easy in comparison.
Nobody is claiming this is a trivial problem to solve but its also not an impossible problem. Other languages have managed to figure out how to achieve this and still maintain backwards compatibility.
Note that you will be expected to have familiarized yourself generally with previous failed proposals of this sort, and proactively considered all the reasonably obvious corner cases.
In spite of the 'You're welcome to bring' this does not actually sound like an encouragement but more of a veiled statement that some non-technical reason will be found to shoot down the proposal if it were to be made so you might as well not bother.
No, the point is that most people in this thread do not appreciate the complexity of implementing lazy imports. If you disagree, your energy is better spent talking to a CPython core developer about implementation details of making baseless assertions from an ivory tower.
There are many people here who think enabling lazy imports is as simple as flipping a light switch. They have no idea what they're talking about.
> It's been explained many times before why this is not possible: the library doesn't actually have a version number.
That sounds like it is absolutely fixable to me, but more of a matter of not having the will to fix it based on some kind of traditionalism. I've used python, a lot. But it is stuff like this that is just maddeningly broken for no good reason at all that has turned me away from it. So as long as I have any alternative I will avoid python because I've seen way too many accidents on account of stuff like this and many lost nights of debugging only to find out that an easily avoidable issue became - once again - the source of much headscratching.
You could absolutely have this be part of the language in any regard. The question then becomes how does one implement it in a reasonable way. I think every package should have a __version__ property you should be able to call, then you could have versioned imports.
In fact there's already many packages already defining __version__ at a package level.
Edit: What they are solving with UV is at the moment of standing up an environment, but you're more concerned about code-level protection, where are they're more concerned about environment setup protection for versioning.
If vereioned imports were added to the language versioned library support obviously would have to become part of the language as well.
However it isn't trivial. First problem coming to my mind:
module a importing first somelib>=1.2.0 and then b and b then requiring somelib>1.2.1 and both being available, will it be the same or will I have a mess from combining?
I know I'm missing something but wouldn't it be possible to just throw an import error when that happens? Would it even break anything? If I try:
import numpy==2.1
And let's say numpy didn't expose a version number in a standard (which could be agreed upon in a PEP) field, then it would just throw an import exception. It wouldn't break any old code. And only packages with that explicit field would support the pinned version import.
And it wouldn't involve trying to extract and parse versions from older packages with some super spotty heuristics.
But it would make new code impossible to use with older versions of python, and older packages, but that's already the case.
> It's been explained many times before why this is not possible: the library doesn't actually have a version number.
Not possible? Come on.
Almost everyone already uses one of a small handfull of conventional ways to specify it, eg `__version__` attribute. It's long overdue that this be standardized so library versions can reliably be introspected at runtime.
Allowing multiple versions to be installed side-by-side and imported explicitly would be a massive improvement.
I believe the charitable interpretation is that it is not possible without breaking an enormous amount of legacy code. Which does feel close enough to “not possible”.
Some situations could be improved by allowing multiple library versions, but this would introduce new headaches elsewhere. I certainly do not want my program to have N copies of numpy, PyTorch, etc because some intermediate library claims to have just-so dependency tree.
I should be able to do "python foo.py" and everything should just work. foo.py should define what it wants and python should fetch it and provide it to foo. I should be able to do "pyc foo.py; ./foo" and everything should just work, dependencies balled up and statically included like Rust or Go. Even NodeJS can turn an entire project into one file to execute. That's what a modern language should look and work like.
The moment I see "--this --that" just to run the default version of something you've lost me. This is 2025.
I don't hate it but I don't love it. It sounds like everyone will start writing `lazy` before essentially every single import, with rare exceptions where eager importing is actually needed. That makes Python code visually noisier. And with no plan to ever change the default, the noise will stay forever.
I would have preferred a system where modules opt in to being lazy-loaded, with no extra syntax on the import side. That would simplify things since only large libraries would have to care about laziness. To be fair, in such a design, the interpreter would have to eagerly look up imports on the filesystem to decide whether they should be lazy-loaded. And there are probably other downsides I'm not thinking of.
If everyone starts favoring lazy imports with not much fuss then it means that lazy should have been the default behavior and eager is the keyword we're missing. This isn't the first time Python revisits this paradigm. Many constructs that used to eagerly produce lists in v2 were turned into generators in v3 with next to no problems.
The PEP includes the ability to enable (or disable) lazy imports globally via a command-line flag or environment variable, in addition to the import syntax.
I don't think this makes sense to be on the module side, the caller is the one with the information as to whether the module can or needs to be lazily loaded. There's nothing really for the module being imported to decide, every module can be lazily loaded. Even if it has side effects the caller may want to defer those as well.
I think side-effects are exactly the problem, you can't have the runtime default to lazy-loading all modules without breaking code that e.g. relies on side effects running before thread creation or forking.
We heard that about types, the walrus, asyncio, dataclasses and so much more. But it didn't happen, if people don't need something (and many don't know it exists or what it does), it's unlikely they use it.
In fact, half of the community basically uses only a modernized set of python 2.4 features and that's one of the beauties of the language. You don't need a lot to be productive, and if you want more, you can optionally reach for it.
It has worked very well for the last 2 decades and it will likely work again.
People said the same about Perl and its “there’s more than one way to do things” ethos, which gained much criticism.
Same is true for C++.
In this specific case, I think a lazy load directive isn’t a bad addition. But one does need to be careful about adding new language features just because you have an active community.
Perhaps people won't use it. But I for one want it to be used, and I will certainly be using it in my own code if the PEP is accepted. Startup time is very important to me, more important than the cost of making the code noisier. I just wish I didn't have to make that tradeoff in the first place.
If anyone's interested I've implemented a fairly user friendly lazy import mechanism in the form of context managers (auto_proxy_import/init) at https://pypi.org/project/lazyimp/ that I use fairly heavily. Syntactically it's just wrapping otherwise unmodified import statements in a with block, so tools 'just work' and it can be easily disabled or told to import eagerly for debugging. It's powered primarily by swapping out the frame's f_builtins in a cext (as it needs more power than importlib hooks provide), but has a lame attempt at a threadsafe pure python version, and a super dumb global hook version.
I was skeptical and cautious with it at first but I've since moved large chunks of my codebase to it - it's caused surprisingly few problems (honestly none besides forgetting to handle some import-time registration in some modules) and the speed boost is addictive.
One thing the PEP doesn't really talk about, and that I find very annoying is that many python linters will complain if you don't put all of your imports at the top of the file, so you get lint warnings if you do the most obvious way to implement lazy imports.
And that is actually a problem for more than just performance. In some cases, importing at the top might actually just fail. For example if you need a platform specific library, but only if it is running on that platform.
I think they're understating the thread safety risks here. The import is going to wind up happening at a random nondeterministic time, in who knows what thread holding who knows what locks (aside from the importer lock).
Previously, if you had some thread hazardous code at module import time, it was highly likely to only run during the single threaded process startup phase, so it was likely harmless. Lazy loading is going to unearth these errors in the most inconvenient way (as Heisenbugs)
(Function level import can trigger this as well, but the top of a function is at least a slightly more deterministic place for imports to happen, and an explicit line of syntax triggering the import/bug)
> The standard library provides the LazyLoader class to solve some of these inefficiency problems. It permits imports at the module level to work mostly like inline imports do.
The use of these sorts of Python import internals is highly non-obvious. The Stack Overflow Q&A I found about it (https://stackoverflow.com/questions/42703908/) doesn't result in an especially nice-looking UX.
So here's a proof of concept in existing Python for getting all imports to be lazy automatically, with no special syntax for the caller:
import sys
import threading # needed for python 3.13, at least at the REPL, because reasons
from importlib.util import LazyLoader # this has to be eagerly imported!
class LazyPathFinder(sys.meta_path[-1]): # <class '_frozen_importlib_external.PathFinder'>
@classmethod
def find_spec(cls, fullname, path=None, target=None):
base = super().find_spec(fullname, path, target)
base.loader = LazyLoader(base.loader)
return base
sys.meta_path[-1] = LazyPathFinder
We've replaced the "meta path finder" (which implements the logic "when the module isn't in sys.modules, look on sys.path for source code and/or bytecode, including bytecode in __pycache__ subfolders, and create a 'spec' for it") with our own wrapper. The "loader" attached to the resulting spec is replaced with an importlib.util.LazyLoader instance, which wraps the base PathFinder's provided loader. When an import statement actually imports the module, the name will actually get bound to a <class 'importlib.util._LazyModule'> instance, rather than an ordinary module. Attempting to access any attribute of this instance will trigger the normal module loading procedure — which even replaces the global name.
Now we can do:
import this # nothing shows up
print(type(this)) # <class 'importlib.util._LazyModule'>
rot13 = this.s # the module is loaded, printing the Zen
print(type(this)) # <class 'module'>
That said, I don't know what the PEP means by "mostly" here.
Yeah, I think this is one of the cleanest PEPs to come around in quite a while, at least from the userspace perspective. Interested to see what happens after the traditional syntax bikeshedding ritual has been completed.
Agree, they really did their homework, listed edge cases, made practical compromises, chose not to overdo it, reworked it again and again quite a bit and compared it to real life experience.
It's really beautiful work, especially since touching the back bone (the import system) of a language as popular as Python with such a diverse community is super dangerous surgery.
Hopefully they learned lessons from why PEP-690 was rejected. I've spent quite a while trying to build this stuff for our codebase and it's never worked well enough to use.
Lazy imports are a great way to create runtime errors far into the operation of a long lived service. Yes, it gives the superficial benefit of 'fast startup', but that upside is negated by the downside of not being sure that once something runs it will run to completion due to a failed import much further down the line. It also allows for some interesting edge cases with the items that are going to be imported no longer being what is on the tin at the time the program is started.
That's fine, because this is still a genuine problem in need of a solution. It's not just about startup time for the sake of it (not that this is even a superficial concern - python startup time with large dependencies quickly gets awful). Large projects can have hefty dependencies that not every user will use. And bundling it all for everyone can sometimes be intractable. The work arounds people use already have the sort of problems you're talking about, on top of being diabolical and hacky. Not having to duplicate and hide imports in functions alone would be a big improvement. It's not like it isn't being proposed as an optional language feature.
An automated test mitigates the risk you describe, and is well worth the tradeoff for fast startup.
I don't consider startup time "superficial" at all; I work in a Django monolith where this problem resulted in each and every management command, test invokation, and container reload incurring a 10-15sec penalty because of just a handful of heavy-to-import libraries used by certain tasks/views. Deferring these made a massive difference.
We tend to prefer explicit top-level imports specifically because they reveal dependency problems as soon as the program starts, rather than potentially hours or days later when a specific code path is executed.
As a counterpoint, having all the imports automatically deferred would instantly dramatically speed up pip for short tasks.
$ time pip install --disable-pip-version-check
ERROR: You must give at least one requirement to install (see "pip help install")
real 0m0.399s
user 0m0.360s
sys 0m0.041s
Almost all of this time is spent importing (and later unloading) ultimately useless vendored code. From my testing (hacking the wrapper script to output some diagnostics), literally about 500 modules get imported in total (on top of the baseline for a default Python process), including almost a hundred modules related to Requests and its dependencies, even though no web request was necessary for this command.
I think this makes a ton of sense in the very specific narrow use case of python CLI tools. For a web app or other long-lived process, startup time is typically not of extreme concern, and having more simplicity and legibility to the import process seems better.
That's not to say this PEP should not be accepted. One could always apply a no-lazy-imports style rule or disable it via global lazy import control.
Right, Not sure why but a lot of code which claude generates also comes with local dependencies vs globally declared import statements. Don't promote that pattern because
- It reduces visibility into a module’s dependencies.
- It increases the risk of introducing circular dependencies later on.
I haven't fully digested the PEP but perhaps there would be a command-line flag or external tool for dependency validation, a bit like how there are external tools for type annotations?
My current team at my current company (see bio if you're really interested), though I should say I'm not authorized to speak on behalf of my employer, so I should really say something more like "I".
I'm a fan because it's something you can explicitly turn on and off. For my Docker based app, I really want to verify the completeness of imports. Preferably, at build and test time. In fact, most of the time I will likely disable lazy loading outright. But, I would really appreciate a faster loading CLI tool.
However, there is a pattern in python to raise an error if, say, pandas doesn't have an excel library installed, which is fine. In the future, will maintainers opt to include a bunch of unused libraries since they won't negatively impact startup time? (Think pandas including 3-4 excel parsers by default, since it will only be loaded when called). It's a much better UX, but, now if you opt out of lazy loading, your code will take longer to load than without it.
I don't like the idea of introducing a new keyword. We need a backward compatible solution. I feel like Python needs some kind of universal annotation syntax such as in go (comments) or in Rust (macros). New keyword means all parsers, lsps, editors should be updated.
I’m pretty sure there will be new keywords in Python in the future that only solve one thing.
Right now all the imports are getting resolved at runtime example in a code like below
from file1 import function1
When you write this, the entire file1 module is executed right away, which may trigger side effects.
If lazy imports suddenly defer execution, those side effects won’t run until much later (or not at all, if the code path isn’t hit). That shift in timing could easily break existing code that depends on import-time behavior.
To avoid using lazy, this there is also a proposal of adding the modules you want to load lazily to a global `__lazy_modules__` variable.
I was talking about syntax. I'm pretty sure there will be new features that will require a new keyword or syntax given the speed of Python growth. It can be universal, for example same as decorator, but it can be applied anywhere.
from lazy import make_lazy
from package import module @make_lazy @local @nogil
Let's say this syntax gets introduced in Python 3.16. The @nogil feature can be introduced in 3.17. If such code is running in Python 3.16, the @nogil marker will be ignored.
The problem with new keywords is that you have to stick to the newest Python version every time a new keyword is added. Older Python versions will give a syntax error. It's a big problem for libraries. You need to wait for 3-5 years before adding it to a library. There are a lot of people who still use Python 3.8 from 2019.
I think they mean backwards-compatible syntax-wise, rather than actually allowing this feature to be used on existing code. If I’m understanding correctly they would prefer for the Python grammar to stay the same (hence the comment about updating parsers and IDEs).
But I don’t think I really agree, the extensible annotation syntaxes they mention always feel clunky and awkward to me. For a first-party language feature (especially used as often as this will be), I think dedicated syntax seems right.
Async or not in modules is a huge pain in the ass for web-dev, but thankfully a lot of Python still isn't web dev. As a Data Scientist you can live your life peacefully without ever worrying about this. But I can see why web devs like it, even though personally I really don't want to see anymore Javascript sneaking inty my Python. Especially when there is already support for IO bound concurrency elsewhere in the language. If I want to do JS syntax, I'll fucking use JS. And I really don't want to see Python go the C++ route where it just wants to be everything and do everything so that you end up with so many possible approaches to the same problem that two devs can't read each others code anymore.
I'm relatively new to Python - how does one do concurrent IO without async/await?
My main complaint, though, about Python async is - because it is opt-in I never know if I forgot a sync IO call somewhere that will block my worker. JS makes everything async by default and there is effectively no chance of blocking.
Async was a soft keyword for many, many years in order to maintain compat. And before that, asyncio used yield to be even more compatible.
They took a decade to solidify that. At some point, you have to balance evolution and stability. For a language as popular as Python, you can not break the world every week, but you can't stay put for 5 years.
They thought about backward compatibility, and offer an alternative syntax that use no keyword for library that want to activate it yet stay compact with old version. It's already in the spec.
This is the wrong syntax, comparable to how "u" strings were the wrong syntax and "b" strings are the right syntax.
They make that, what should be the default, a special case. Soon, every new code will use "lazy". The long term effect of such changes is a verbose language syntax.
They should have had a period where one, if they want lazy imports, has to do "from __future__ import lazy_import". After that period, lazy imports become the default. For old-style immediate imports, introduce a syntax: "import now foo" and "from foo import now bar".
All which authors of old code would have to do is run a provided fix script in the root directory of their code.
This would be a huge deal for Python startup time *if* it was applied to all the standard library packages recursively. Right now importing asyncio brings in half the standard library through transitive imports.
This is needed, but I don't like new keywords. What I would love, for many reasons, is if we could decorate statements. Then things like:
import expensive_module
could be:
@lazy
import expensive_module
or you could do:
@retry(3)
x = failure_prone_call(y)
lazy is needed, but maybe there is a more basic change that could give more power with more organic syntax, and not create a new keyword that is special purpose (and extending an already special purpose keyword)
As a side note, It would be great to have a 'preview' for HN comments. I updated the above because I forgot to add line breaks. Sigh. I bet 'preview' would stop a lot of not well thought out comments too.
I like the approach of ES6 where you pull in bindings that are generally lazily resolved. That is IMO the approach that should be the general strategy for Python.
The import/require schism was necessary precisely of all the issues with module side effects the python community is rediscovering now. I suppose they won’t get around a similar solution eventually; but judging from years of discussions in the JS world, this is going to be dragging on for a while.
Remember mercurial? Me neither. But what I remeber is this article I've read about all the hacks they had to do to achieve reasonable startup time for CLI in python. And the no #1 cause was loading the whole world you don't ever need. As I recall they somehow monkeypatched the interpreter to ignore imports and just remember their existence until they were actually needed, at which point the import happened. So all the dead paths were just skipped.
Ugh...I like the idea, but I wish lazy imports were the default. Python allows side effects in the top level though so that would be a breaking change.
Soooo instead now we're going to be in a situation where you're going to be writing "lazy import ..." 99% of the time: unless you're a barbarian, you basically never have side effects at the top level.
We need some kind of unexec revival in Linux so we don't have to resort to crap like this. Maybe CRIU based. Worst case, some Python specific hack. But Python's import system is wacky enough without lazy loading. This sounds like combining the worst of multiple worlds.
Kinda related, I wish there was an easy way to exclude dependencies at pip-install time and mock them at runtime so an import doesn't cause an exception. Basically a way for me to approximate "extras" when the author isn't motivated to do it for me, even though it'd be super brittle.
Recursively importing other modules is itself a side effect.
In fact, all the code you see in the module is "side effects", in a sense. A `class` body, for example, has to actually run at import time, creating the class object and attaching it as an attribute of the module object. Similarly for functions. Even a simple assignment of a constant actually has to run at module import. And all of these things add up.
Further, if there isn't already cached bytecode available for the module, by default it will be written to disk as part of the import process. That's inarguably a side effect.
Side-effect means you're changing state outside the scope the code is running.
Sure thing you can declare globals variable and run anything on a module file global scope (outside funcs and class body), but even that 'global' scope is just an illusion, and everything declared there, as yourself said, is scoped to the module's namespace
(and you can't leak the 'globals' when importing the module unless you explicity do so 'from foo import *'. Think of python's import as eval but safer because it doesn't leaks the results from the module execution)
So for a module to have side-effect (for me) it would either:
- Change/Create attributes from other modules
- Call some other function that does side-effect (reflection builtins? IO stuff)
All my code which uses import probing would fail, such as fallbacks:
try:
import module
except ImportError:
import slow_module as module
Conditional support testing would also break, like having tests which only run if module2 is available:
try:
import module2
except ImportError:
def if_has_module2(f):
return unittest.skip("module2 not available")(f)
else:
def if_has_module2(f):
return f
@if_has_module2
class TestModule2Bindings(....
The proto-PEP also gives an example of using
with suppress_warnings():
import module3
where some global configuration changes only during import.
In general, "import this" and "import antigravity" - anything with import side-effects - would stop working.
Oh, and as the proto-PEP points out, changes to sys.path and others can cause problems because of the delay between time of lazy import and time of resolution.
Then there's code where you do a long computation then make use of a package which might not be present.
>>> import nonexistent_module
Traceback (most recent call last):
File "<python-input-2>", line 1, in <module>
import nonexistent_module
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "<frozen importlib._bootstrap>", line 1322, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1262, in _find_spec
File "<python-input-0>", line 8, in find_spec
base.loader = LazyLoader(base.loader)
^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'loader'
The implementation should probably convert that exception back to ImportError for you, but the point is that the absence of an implementation can still be detected eagerly while the actual loading occurs lazily.
Ahh, so you do the find first, and keep that around before loading.
I have bad memories of using a network filesystem where my Python app's startup time was 5 or more seconds because of all the small file lookups for the import were really slow.
I fixed it by importing modules in functions, only when needed, so the time went down to less than a second. (It was even better using a zipimport, but for other reasons we didn't use that option.)
If I understand things correctly, your code would have the same several-second delay as it tries to resolve everything?
Yes, if checking for a file is slow, then checking for a file is slow. If you need to know up front whether the module exists, then you can't get around using the "figure out whether the module exists" machinery up front. And if the definition of "a module exists" includes cases where the module is represented by a file whose existence you have to check for, then there's no getting around that, either.
(Trying to do "fallback" logic with lazily-loaded modules is also susceptible to race conditions, of course. What if someone defines the module before you try to use it?)
You have to opt in with the lazy import keyword no matter what. This pep also prevents lazy import in try catch. I think your concern matters if ‘module’ itself has a lazy import that you want to check exists. With this you now need to be more rigorous in check those sub dependencies.
This can already happen with non top level imports so it is not a necessarily a new issue, but could become more prevalent if there is an overall uptake in this feature for optional dependencies.
Very unfortunate how from json lazy import dumps would result in backward compatibility issues. It reads much better and makes it easier to search for lazy imports, especially if in the future something else becomes optionally lazy as well.
Looks good to me. I use a tab-completion trick where the tab-completer tool calls the script I'm about to invoke with special arguments, and the script reflects on itself and responds with possible completions. But because of slow imports, it often takes a while for the completion to respond.
I could, and sometimes do, go through all the imports to figure out which ones are taking a long time to load, but it's a chore.
Really excited about this - we've recently been struggling with making imports lazy without completely messing up the code in DeepInverse https://deepinv.github.io/deepinv/
> The dominant convention in Python code is to place all imports at the beginning of the file. This avoids repetition, makes import dependencies clear and minimizes runtime overhead.
> A somewhat common way to delay imports is to move the imports into functions, but this practice requires more work [and] obfuscates the full set of dependencies for a module.
The first part is just saying the traditions exist because the traditions have always existed. Traditions are allowed to change!
The second part is basically saying if you do your own logic-based lazy imports (inline imports in functions) then you’re going against the traditions. Again, traditions are allowed to change!
The point about the import graph being obfuscated would ring more true if Python didn’t already provide lightning fast static analysis tools like ast. If you care about import graphs at the module level then you’re probably already automating everything with ast anyway, at which point you just walk the whole tree looking for imports rather than the top level.
So, really, the whole argument for a new lazy keyword (instead of inlining giant imports where they are needed) is because people like to see import pytorch at the top
of the file, and baulk at seeing it — and will refuse to even look
for it! — anywhere else? Hmmm.
What does seem like a pain in the ass is having to do this kind of repetitive crap (which they mention in their intro):
>The second part is basically saying if you do your own logic-based lazy imports (inline imports in functions) then you’re going against the traditions. Again, traditions are allowed to change!
No, they are saying the tradition is there for a reason. Imports at the beginning of the file makes reasoning about the dependencies of a module much easier and faster. I've had to deal with both and I sure as hell know which I'd prefer. Lazy imports by functions is a sometimes necessary evil and it would be very nice if it became unnecessary.
I love the feature but I really dislike using the word lazy as a new language keyword. It just feels off somehow. I think maybe defer might be a better word. It is at least keeps the grammar right because it would be lazily.
lazily import package.foo
vs
defer import package.foo
Also the grammar is super weird for from imports.
lazy from package import foo
vs.
from package defer import foo.
Even with eager importing there is only a "circular import problem" if you try to import names `from` the modules — as I pointed out a few days ago (https://news.ycombinator.com/item?id=45400783).
This is what I thought of too. I really only know python, do other languages not have that issue? In python it does not seem like a "problem" to me - whenever I have seen circular import issues it is because the code is organized poorly. I worry that this feature will lead to devs "fixing" circular import issues by using lazy imports.
Sometimes it's hard to avoid cyclic imports, without blaming the design. Like if a Parent has a Child, and the Child needs to know of the parent. Only way to solve that in python is to put everything in the same file, which also feels like bad deisgn.
I would say in that case, the Parent and Child shouldn't need to know about each other - some kind of handler in a different file should.
Although I guess that doesn't work in all cases, like defining foreign key relationships when using an orm (like sqlalchemy) for example. But in the orm case, the way to get around that is... lazy resolution :^)
Interesting. I find that I always have this problem in any non-trivial python project, and don't find this to be due to poorly organized code. I have only seen this requirement in Python.
IME circular import errors aren't due to poor organization; they're due to an arbitrary restriction Python has.
I don’t want lazy imports. That’s just makes performance shitty later and harder to debug. It’s a hacky workaround.
What I want is for imports to not suck and be slow. I’ve had projects where it was faster to compile and run C++ than launch and start a Python CLI. It’s so bad.
That's a hack that forces you to duplicate and hide imports. The tradition is to specify imports at the top because it's that much better readability wise.
I know/heard there are "some" (which I haven't seen by the way) libraries that depend on import side effects, but the advantage is much bigger.
First of all, the circular import problem will go away, especially on type hints. Although there was a PEP or recent addition to make the annotation not not cause such issue.
Second and most important of all, is the launch time of Python applications. A CLI that uses many different parts of the app has to wait for all the imports to be done.
The second point becomes a lot painful when you have a large application, like a Django project where the auto reload becomes several seconds. Not only auto reload crawls, the testing cycle is slow as well. Every time you want to run test command, it has to wait several seconds. Painful.
So far the solution has been to do the lazy import by importing inside the methods where it's required. That is something, I never got to like to be honest.
Maybe it will be fixed in Python 4, where the JIT uses the type hints as well /s
Top-level code should not be able to fail except in incredibly deterministic ways that are tested during development. Failing fast is not as good as not failing at all. Lazy imports mean the power to avoid importing things that don't need to be imported at all on this run. Good design also cares about performance to some extent. On my machine, asking pip to do literally nothing takes several times as long as creating a new virtual environment --without-pip .
Python really seems like a bad fit for that. So your imports succeed, what now? Do they have all the functions or fields your program needs? Those are still resolved at the last possible moment. If you want to be sure your program actually runs you will have to write and run tests with and without lazy imports.
Relative imports have been supported for approximately forever (https://stackoverflow.com/questions/72852). If you mean "by explicitly specifying a path string" (as opposed to a symbolic name), that has also been supported for approximately forever (https://stackoverflow.com/questions/67631). Today, the `importlib` standard library exposes all the steps of the import process — including figuring out where the source code is, checking the `sys.modules` cache etc. — and lets you hook into everything (in particular, you can bypass the entire "finder" process and turn a file path into a dummy "spec" which is fed to a "loader").
The flexibility of this system also entails that you can in effect define a completely new programming language, describe the process of creating Python bytecode from your custom source, and have clients transparently `import` source in the other language as a result. Or you can define an import process that grabs code from the Internet (not that it would be a good idea...).
If you mean "by explicitly specifying a relative path, and having it be interpreted according to the path of the current module's source code", well first you have to consider that the current module isn't required to have source code. But if it does, then generally it will have been loaded with the default mechanism, which means the module object will have a `__file__` attribute with an absolute path, and you just set your path relative to that.
> Relative imports work by relative package path, which is not at all the same.
Exactly. This gives you the flexibility to distribute a complex package across multiple locations.
> Often when you run Python you don't even have a package path.
Any time you successfully import foo.bar, you necessarily have imported foo (because bar is an attribute of that object!), and therefore bar can `from . import` its siblings.
> Using `importlib` is a horrible hack that breaks basically all tooling. You very very obviously are not supposed to do that.
It is exactly as obvious (and true) that you are not "supposed to", in the exact same sense, directly specify where on disk the source code file you want to import is. After all, this constrains the import process to use a source code file. (Similarly if you specify a .pyc directly.) Your relative path doesn't necessarily make any sense after you have packaged and distributed your code and someone else has installed it. It definitely doesn't make any sense if you pack all your modules into a zipapp.
You… can? I mean in the strictest sense you're technically not importing by file path but if you make your folder a module by slapping an __init__.py in there then your relative imports will follow the directory tree. I think as of Python 3.3 the init file is optional so it will do it by default but I can't remember if there are still some cases where it's required. The only thing you can't do is go "up" to a higher directory than the root module.
Also if that doesn't strike your fancy all of the importlib machinery is at your disposal and it's really not very much work to write an import_path() function. It's one of the patterns plug-in systems use and so is stable and expected to be used by end users. No arcane magic required.
> if you make your folder a module by slapping an __init__.py in there then your relative imports will follow the directory tree.
`__init__.py` has nothing to do with making this work. It is neither necessary (as of 3.3, yes, you got it right: see https://peps.python.org/pep-0420/) nor sufficient (careless use of sys.path and absolute imports could make it so that the current folder hasn't been imported yet, so you can't even go "up into" it). The folder will already be represented by a module object.
What `__init__.py` does is:
1. Prevents relative imports from also potentially checking in other paths.
2. Provides a space for code that runs before sub-modules, for example to set useful package attributes such as `__all__` (which controls star-imports).
Love this. My https://llm.datasette.io/ CLI tool supports plugins, and people were complaining about really slow start times even for commands like "llm --help" - it turned out there were popular plugins that did things like import pytorch at the base level, so the entire startup was blocked on heavy imports.
I ended up adding a note to the plugin author docs suggesting lazy loading inside of functions - https://llm.datasette.io/en/stable/plugins/advanced-model-pl... - but having a core Python language feature for this would be really nice.
You can implement this from your tool today: https://news.ycombinator.com/item?id=45467489
Note that this is global to the entire process, so for example if you make an import of Numpy lazy this way, then so are the imports of all the sub-modules. Meaning that large parts of Numpy might not be imported at all if they aren't needed, but pauses for importing individual modules might be distributed unpredictably across the runtime.
Edit: from further experimentation, it appears that if the source does something like `import foo.bar.baz` then `foo` and `foo.bar` will still be eagerly loaded, and only `foo.bar.baz` itself is deferred. This might be part of what the PEP meant by "mostly". But it might also be possible to improve my implementation to fix that.
Parse the command line and do things like "--help" without doing the imports.
Only do imports when you know you need them -- or as an easy approximation, only if the easy command line options have been handled and there's still something to do.
In the llm project, plugins can modify the command line arguments, so it's not that simple.
Yea, that's the core problem here: plugins can add new CLI subcommands, which means they all need to be loaded on startup.
https://llm.datasette.io/en/stable/plugins/plugin-hooks.html...
Could you cache the help doc after first full loaded run and only regenerate when new plugins are added / updated?
Well yes, or you can just use the `lazy` keyword, when it makes it into core.
You can “just” use a feature which does not exist yet? How is that something you “just” do?
You always find a way to spam links to your site. It’s honestly impressive.
Lazy imports have been proposed before, and were rejected most recently back in 2022: https://discuss.python.org/t/pep-690-lazy-imports-again/1966.... If I recall correctly, lazy imports are a feature supported in Cinder, Meta's version of CPython, and the PEP was driven by folks that worked on Cinder. Last time, a lot of the discussion centered around questions like: Should this be opt-in or opt-out? At what level? Should it be a build-flag for CPython itself? Etc. The linked post suggests that the Steering Council ultimately rejected it because of the complexity it would introduce to have two divergent "modes" of importing.
I hope this proposal succeeds. I would love to use this feature.
I also hope this proposal succeeds, but I'm not optimistic. This will break tons of code and introduce a slew of footguns. Import statements fundamentally have side effects, and when and how these side effects are applied will cause mysterious breakages that will keep people up for many nights.
This is not fearmongering. There is a reason why the only flavor of Python with lazy imports comes from Meta, which is one of the most well-resourced companies in the world.
Too many people in this thread hold the view of "importing {pandas, numpy, my weird module that is more tangled than an eight-player game of Twister} takes too long and I will gladly support anything that makes them faster". I would be willing to bet a large sum of money that most people who hold this opinion are unable to describe how Python's import system works, let alone describe how to implement lazy imports.
PEP 690 describes a number of drawbacks. For example, lazy imports break code that uses decorators to add functions to a central registry. This behavior is crucial for Dash, a popular library for building frontends that has been around for more than a decade. At import-time, Dash uses decorators to bind a JavaScript-based interface to callbacks written in Python. If these imports were made lazy, Dash would break. Frontends used by thousands, if not millions of people, would immediately become unresponsive.
You may cry, "But lazy imports are opt-in! Developers can choose to opt-out of lazy imports if it doesn't work for them." What if these imports were transitive? What if our frontend needed to be completely initialized before starting a critical process, else it would cause a production outage? What if you were a maintainer of a library that was used by millions of people? How could you be sure that adding lazy imports wouldn't break any code downstream? Many people made this argument for type hints, which is sensible because type hints have no effect on runtime behavior*. This is not true for lazy imports; import statements exist in essentially every nontrivial Python program, and changing them to be lazy will fundamentally alter runtime behavior.
This is before we even get to the rest of the issues the PEP describes, which are even weirder and crazier than this. This is a far more difficult undertaking than many people realize.
---
* You can make a program change its behavior based on type annotations, but you'd need to explicitly call into typing APIs to do this. Discussion about this is beyond the scope of this post.
Some of these worries make sense, but wouldn’t it be relatively trivial to pass a flag to the interpreter or something similar in order to force all imports to evaluate, as in the current behavior? But to be a bit cheeky if some of these issues cause serious production outages for you it might be time to consider moving on from a scripting language altogether.
The issue is that some imports can be made lazy and some cannot. A binaristic all-or-nothing approach does not address the issue. (I also think that there is zero basis to claim that adding such a flag is trivial, since there’s no reference implementation of this flavor of lazy imports.)
What if we have a program where one feature works only when lazy imports are enabled and one feature only when lazy imports are disabled?
This is not a contrived concern. Let’s say I’m a maintainer of an open-source library and I choose to use lazy imports in my library. Because I’m volunteering my time, I don’t test whether my code works with eager imports.
Now, let’s say someone comes and builds an application on top of this library. It doesn’t work with lazy imports for some unknown reason. If they reach for a “force all imports” flag, their application might break in another mysterious way because the code they depend on is not built to work with eager imports. And even if my dependency doesn’t break, what about all the other packages the application may depend on?
The only solution here would be for the maintainer to ensure that their code works with both lazy and eager imports. However, this imposes a high maintenance cost and is part of the reason why PEP 690 was rejected. (And if your proposed solution was “don’t use libraries made by random strangers on the Internet”, boy do I have news for you...)
My point is that many things _will_ break if migrated to lazy imports. Whether they should have been written in Python in the first place is a separate question that isn’t relevant to this discussion.
Especially since it is opt in, with various level of granularity, and a global off switch. Very well constructed spec given the constraints.
Oof. I wish they could support version imports
and support having multiple simultaneous versions of any Python library installed. End this conda/virtualenv/docker/bazel/[pick your poison] messIt's been explained many times before why this is not possible: the library doesn't actually have a version number. The distribution of source code on PyPI has a version number, but the name of this is not connected to the name of any module or package you import in the source code. The distribution can validly define zero or more modules (packages are a subset of modules, represented using the same type in the Python type system).
You got three other responses before me all pointing at uv. They are all wrong, because uv did not introduce this functionality to the Python ecosystem. It is a standard defined by https://peps.python.org/pep-0723/, implemented by multiple other tools, notably pipx.
> It's been explained many times before why this is not possible: the library doesn't actually have a version number. The distribution of source code on PyPI has a version number, but the name of this is not connected to the name of any module or package you import in the source code.
You're making the common mistake of conflating how things currently work with how things could work if the responsible group agrees to change how things work. Something being the way it is right now is not the same as something else being "not possible".
No, changing this breaks the world. A huge fraction of PyPI becomes completely invalid overnight, and the rest fails the expected version checks. Not to mention that the language is fundamentally designed around the expectation that modules are singleton. I've written about this at length before but I can't easily find it right now (I have way too many bookmarks and not nearly enough idea how to organize them).
Yes, you absolutely can create a language that has syntax otherwise identical to Python (or at least damn close) which implements a feature like this. No, you cannot just replace Python with it. If the Python ecosystem just accepted that clearly better things were clearly better, and started using them promptly, we wouldn't have https://pypi.org/project/six/ making it onto https://pypistats.org/top (see also https://sethmlarson.dev/winning-a-bet-about-six-the-python-2...).
The hard part is making the change. Adding an escape hatch so older code still works is easy in comparison.
Nobody is claiming this is a trivial problem to solve but its also not an impossible problem. Other languages have managed to figure out how to achieve this and still maintain backwards compatibility.
You're welcome to bring a concrete proposal to, e.g., https://discuss.python.org/c/ideas/6 , or ask around the core devs to find a PEP sponsor.
Note that you will be expected to have familiarized yourself generally with previous failed proposals of this sort, and proactively considered all the reasonably obvious corner cases.
Tbh you’re just reinforcing mine and others point there that the issue isn’t a technical one.
"Talk is cheap. Show me the code."
In spite of the 'You're welcome to bring' this does not actually sound like an encouragement but more of a veiled statement that some non-technical reason will be found to shoot down the proposal if it were to be made so you might as well not bother.
No, the point is that most people in this thread do not appreciate the complexity of implementing lazy imports. If you disagree, your energy is better spent talking to a CPython core developer about implementation details of making baseless assertions from an ivory tower.
There are many people here who think enabling lazy imports is as simple as flipping a light switch. They have no idea what they're talking about.
> It's been explained many times before why this is not possible: the library doesn't actually have a version number.
That sounds like it is absolutely fixable to me, but more of a matter of not having the will to fix it based on some kind of traditionalism. I've used python, a lot. But it is stuff like this that is just maddeningly broken for no good reason at all that has turned me away from it. So as long as I have any alternative I will avoid python because I've seen way too many accidents on account of stuff like this and many lost nights of debugging only to find out that an easily avoidable issue became - once again - the source of much headscratching.
You could absolutely have this be part of the language in any regard. The question then becomes how does one implement it in a reasonable way. I think every package should have a __version__ property you should be able to call, then you could have versioned imports.
In fact there's already many packages already defining __version__ at a package level.
https://packaging.python.org/en/latest/discussions/versionin...
Edit: What they are solving with UV is at the moment of standing up an environment, but you're more concerned about code-level protection, where are they're more concerned about environment setup protection for versioning.
If vereioned imports were added to the language versioned library support obviously would have to become part of the language as well.
However it isn't trivial. First problem coming to my mind:
module a importing first somelib>=1.2.0 and then b and b then requiring somelib>1.2.1 and both being available, will it be the same or will I have a mess from combining?
I know I'm missing something but wouldn't it be possible to just throw an import error when that happens? Would it even break anything? If I try:
import numpy==2.1
And let's say numpy didn't expose a version number in a standard (which could be agreed upon in a PEP) field, then it would just throw an import exception. It wouldn't break any old code. And only packages with that explicit field would support the pinned version import.
And it wouldn't involve trying to extract and parse versions from older packages with some super spotty heuristics.
But it would make new code impossible to use with older versions of python, and older packages, but that's already the case.
Maybe the issue is with module name spacing?
> It's been explained many times before why this is not possible: the library doesn't actually have a version number.
Not possible? Come on.
Almost everyone already uses one of a small handfull of conventional ways to specify it, eg `__version__` attribute. It's long overdue that this be standardized so library versions can reliably be introspected at runtime.
Allowing multiple versions to be installed side-by-side and imported explicitly would be a massive improvement.
I believe the charitable interpretation is that it is not possible without breaking an enormous amount of legacy code. Which does feel close enough to “not possible”.
Some situations could be improved by allowing multiple library versions, but this would introduce new headaches elsewhere. I certainly do not want my program to have N copies of numpy, PyTorch, etc because some intermediate library claims to have just-so dependency tree.
You could do that with uv.
How would version imports be handled across the codebase? Also, what do you gain with those over PEP 723 – Inline script metadata? https://packaging.python.org/en/latest/specifications/inline...
The mess has ended thanks to uv.
Oof. This feature request has nothing to do with lazy imports. It’s also solved far more cleanly with inline script metadata.
Really what is the headache with virtual environments? They’ve been solved. Use UV or python’s built in venv creator and you’re good to go.
uv venv —seed —python=3.12 && source .venv/bin/activate && pip3 install requests && …
That's messy.
I should be able to do "python foo.py" and everything should just work. foo.py should define what it wants and python should fetch it and provide it to foo. I should be able to do "pyc foo.py; ./foo" and everything should just work, dependencies balled up and statically included like Rust or Go. Even NodeJS can turn an entire project into one file to execute. That's what a modern language should look and work like.
The moment I see "--this --that" just to run the default version of something you've lost me. This is 2025.
uv is good.
I don't hate it but I don't love it. It sounds like everyone will start writing `lazy` before essentially every single import, with rare exceptions where eager importing is actually needed. That makes Python code visually noisier. And with no plan to ever change the default, the noise will stay forever.
I would have preferred a system where modules opt in to being lazy-loaded, with no extra syntax on the import side. That would simplify things since only large libraries would have to care about laziness. To be fair, in such a design, the interpreter would have to eagerly look up imports on the filesystem to decide whether they should be lazy-loaded. And there are probably other downsides I'm not thinking of.
If everyone starts favoring lazy imports with not much fuss then it means that lazy should have been the default behavior and eager is the keyword we're missing. This isn't the first time Python revisits this paradigm. Many constructs that used to eagerly produce lists in v2 were turned into generators in v3 with next to no problems.
Yes, there were totally no problems in the transition from 2 to 3 :)
I would gladly take a command line flag that I can pass to python that makes all module loading lazy.
Unless you are writing scripts or very simple stuff running side effects when modules are loaded should be avoided at all cost anyway.
The PEP includes the ability to enable (or disable) lazy imports globally via a command-line flag or environment variable, in addition to the import syntax.
That's already part of the PIP. There is a flag to enable lazy imports for all possible imports.
I don't think this makes sense to be on the module side, the caller is the one with the information as to whether the module can or needs to be lazily loaded. There's nothing really for the module being imported to decide, every module can be lazily loaded. Even if it has side effects the caller may want to defer those as well.
I think side-effects are exactly the problem, you can't have the runtime default to lazy-loading all modules without breaking code that e.g. relies on side effects running before thread creation or forking.
> Even if it has side effects the caller may want to defer those as well.
But that's rare, and could be handled with existing workarounds.
Normally, a module needs to be eagerly imported if and only if it has side effects.
We heard that about types, the walrus, asyncio, dataclasses and so much more. But it didn't happen, if people don't need something (and many don't know it exists or what it does), it's unlikely they use it.
In fact, half of the community basically uses only a modernized set of python 2.4 features and that's one of the beauties of the language. You don't need a lot to be productive, and if you want more, you can optionally reach for it.
It has worked very well for the last 2 decades and it will likely work again.
People said the same about Perl and its “there’s more than one way to do things” ethos, which gained much criticism.
Same is true for C++.
In this specific case, I think a lazy load directive isn’t a bad addition. But one does need to be careful about adding new language features just because you have an active community.
Perhaps people won't use it. But I for one want it to be used, and I will certainly be using it in my own code if the PEP is accepted. Startup time is very important to me, more important than the cost of making the code noisier. I just wish I didn't have to make that tradeoff in the first place.
Wish pyproject.toml was enhanced to specify lazy loading via regexs.
If anyone's interested I've implemented a fairly user friendly lazy import mechanism in the form of context managers (auto_proxy_import/init) at https://pypi.org/project/lazyimp/ that I use fairly heavily. Syntactically it's just wrapping otherwise unmodified import statements in a with block, so tools 'just work' and it can be easily disabled or told to import eagerly for debugging. It's powered primarily by swapping out the frame's f_builtins in a cext (as it needs more power than importlib hooks provide), but has a lame attempt at a threadsafe pure python version, and a super dumb global hook version.
I was skeptical and cautious with it at first but I've since moved large chunks of my codebase to it - it's caused surprisingly few problems (honestly none besides forgetting to handle some import-time registration in some modules) and the speed boost is addictive.
looks very interesting! i might use this for some of my projects as well
One thing the PEP doesn't really talk about, and that I find very annoying is that many python linters will complain if you don't put all of your imports at the top of the file, so you get lint warnings if you do the most obvious way to implement lazy imports.
And that is actually a problem for more than just performance. In some cases, importing at the top might actually just fail. For example if you need a platform specific library, but only if it is running on that platform.
I don't think there is any solution for that but "fix your broken linter".
It isn't just one though. Every linter I've used has warned about that.
Probably because PEP 8 says
> Imports are always put at the top of the file, just after any module comments and docstrings, and before module globals and constants
Ruff doesn't do this, and in fact even lets you specify modules that _must_ not be imported at the top level (banned-module-level-imports = [...])
I banished the worst/heaviest libraries to this list at my workplace and it's been really helpful at keeping startup times from regressing.
It is annoying, but most linters will accept a `#noqa E402` comment to ignore it
I think they're understating the thread safety risks here. The import is going to wind up happening at a random nondeterministic time, in who knows what thread holding who knows what locks (aside from the importer lock).
Previously, if you had some thread hazardous code at module import time, it was highly likely to only run during the single threaded process startup phase, so it was likely harmless. Lazy loading is going to unearth these errors in the most inconvenient way (as Heisenbugs)
(Function level import can trigger this as well, but the top of a function is at least a slightly more deterministic place for imports to happen, and an explicit line of syntax triggering the import/bug)
> The standard library provides the LazyLoader class to solve some of these inefficiency problems. It permits imports at the module level to work mostly like inline imports do.
The use of these sorts of Python import internals is highly non-obvious. The Stack Overflow Q&A I found about it (https://stackoverflow.com/questions/42703908/) doesn't result in an especially nice-looking UX.
So here's a proof of concept in existing Python for getting all imports to be lazy automatically, with no special syntax for the caller:
We've replaced the "meta path finder" (which implements the logic "when the module isn't in sys.modules, look on sys.path for source code and/or bytecode, including bytecode in __pycache__ subfolders, and create a 'spec' for it") with our own wrapper. The "loader" attached to the resulting spec is replaced with an importlib.util.LazyLoader instance, which wraps the base PathFinder's provided loader. When an import statement actually imports the module, the name will actually get bound to a <class 'importlib.util._LazyModule'> instance, rather than an ordinary module. Attempting to access any attribute of this instance will trigger the normal module loading procedure — which even replaces the global name.Now we can do:
That said, I don't know what the PEP means by "mostly" here.Feels like a good feature, with a simple explanation, real world use cases, and a scoped solution (global only, pretty simple keyword). I like it!
Yeah, I think this is one of the cleanest PEPs to come around in quite a while, at least from the userspace perspective. Interested to see what happens after the traditional syntax bikeshedding ritual has been completed.
Agree, they really did their homework, listed edge cases, made practical compromises, chose not to overdo it, reworked it again and again quite a bit and compared it to real life experience.
It's really beautiful work, especially since touching the back bone (the import system) of a language as popular as Python with such a diverse community is super dangerous surgery.
I'm impressed.
Hopefully they learned lessons from why PEP-690 was rejected. I've spent quite a while trying to build this stuff for our codebase and it's never worked well enough to use.
Lazy imports are a great way to create runtime errors far into the operation of a long lived service. Yes, it gives the superficial benefit of 'fast startup', but that upside is negated by the downside of not being sure that once something runs it will run to completion due to a failed import much further down the line. It also allows for some interesting edge cases with the items that are going to be imported no longer being what is on the tin at the time the program is started.
That's fine, because this is still a genuine problem in need of a solution. It's not just about startup time for the sake of it (not that this is even a superficial concern - python startup time with large dependencies quickly gets awful). Large projects can have hefty dependencies that not every user will use. And bundling it all for everyone can sometimes be intractable. The work arounds people use already have the sort of problems you're talking about, on top of being diabolical and hacky. Not having to duplicate and hide imports in functions alone would be a big improvement. It's not like it isn't being proposed as an optional language feature.
An automated test mitigates the risk you describe, and is well worth the tradeoff for fast startup.
I don't consider startup time "superficial" at all; I work in a Django monolith where this problem resulted in each and every management command, test invokation, and container reload incurring a 10-15sec penalty because of just a handful of heavy-to-import libraries used by certain tasks/views. Deferring these made a massive difference.
We tend to prefer explicit top-level imports specifically because they reveal dependency problems as soon as the program starts, rather than potentially hours or days later when a specific code path is executed.
As a counterpoint, having all the imports automatically deferred would instantly dramatically speed up pip for short tasks.
Almost all of this time is spent importing (and later unloading) ultimately useless vendored code. From my testing (hacking the wrapper script to output some diagnostics), literally about 500 modules get imported in total (on top of the baseline for a default Python process), including almost a hundred modules related to Requests and its dependencies, even though no web request was necessary for this command.I think this makes a ton of sense in the very specific narrow use case of python CLI tools. For a web app or other long-lived process, startup time is typically not of extreme concern, and having more simplicity and legibility to the import process seems better.
That's not to say this PEP should not be accepted. One could always apply a no-lazy-imports style rule or disable it via global lazy import control.
https://peps.python.org/pep-0810/#global-lazy-imports-contro...
It saddens me to think that the use case of "python CLI tools" is thought of as anything like "very specific" or "narrow".
These tools already have a million ways to avoid this slowdown. Is it so much to ask their authors to care?
Right, Not sure why but a lot of code which claude generates also comes with local dependencies vs globally declared import statements. Don't promote that pattern because
- It reduces visibility into a module’s dependencies.
- It increases the risk of introducing circular dependencies later on.
I haven't fully digested the PEP but perhaps there would be a command-line flag or external tool for dependency validation, a bit like how there are external tools for type annotations?
Who is "we"?
My current team at my current company (see bio if you're really interested), though I should say I'm not authorized to speak on behalf of my employer, so I should really say something more like "I".
This sounds like something that should be covered by tests?
I'm a fan because it's something you can explicitly turn on and off. For my Docker based app, I really want to verify the completeness of imports. Preferably, at build and test time. In fact, most of the time I will likely disable lazy loading outright. But, I would really appreciate a faster loading CLI tool.
However, there is a pattern in python to raise an error if, say, pandas doesn't have an excel library installed, which is fine. In the future, will maintainers opt to include a bunch of unused libraries since they won't negatively impact startup time? (Think pandas including 3-4 excel parsers by default, since it will only be loaded when called). It's a much better UX, but, now if you opt out of lazy loading, your code will take longer to load than without it.
I don't like the idea of introducing a new keyword. We need a backward compatible solution. I feel like Python needs some kind of universal annotation syntax such as in go (comments) or in Rust (macros). New keyword means all parsers, lsps, editors should be updated.
I’m pretty sure there will be new keywords in Python in the future that only solve one thing.
Not sure if this can be made backward compatible.
Right now all the imports are getting resolved at runtime example in a code like below
When you write this, the entire file1 module is executed right away, which may trigger side effects.If lazy imports suddenly defer execution, those side effects won’t run until much later (or not at all, if the code path isn’t hit). That shift in timing could easily break existing code that depends on import-time behavior.
To avoid using lazy, this there is also a proposal of adding the modules you want to load lazily to a global `__lazy_modules__` variable.
I was talking about syntax. I'm pretty sure there will be new features that will require a new keyword or syntax given the speed of Python growth. It can be universal, for example same as decorator, but it can be applied anywhere.
Let's say this syntax gets introduced in Python 3.16. The @nogil feature can be introduced in 3.17. If such code is running in Python 3.16, the @nogil marker will be ignored.The problem with new keywords is that you have to stick to the newest Python version every time a new keyword is added. Older Python versions will give a syntax error. It's a big problem for libraries. You need to wait for 3-5 years before adding it to a library. There are a lot of people who still use Python 3.8 from 2019.
Understood make sense
I think they mean backwards-compatible syntax-wise, rather than actually allowing this feature to be used on existing code. If I’m understanding correctly they would prefer for the Python grammar to stay the same (hence the comment about updating parsers and IDEs).
But I don’t think I really agree, the extensible annotation syntaxes they mention always feel clunky and awkward to me. For a first-party language feature (especially used as often as this will be), I think dedicated syntax seems right.
Me neither.
Introducing new keyword has become a recent thing in Python.
Seems Python has a deep scare since Python2 to Python3 time and is scared to do anything that causes such drama again.
For me, the worst of all is "async". If 2to3 didn't cause much division, the async definitely divided Python libraries in 2. Sync and Async.
Maybe if they want backward compatible solution, this can be done by some compile or runtime flag like they did with free threading no-gil.
Async or not in modules is a huge pain in the ass for web-dev, but thankfully a lot of Python still isn't web dev. As a Data Scientist you can live your life peacefully without ever worrying about this. But I can see why web devs like it, even though personally I really don't want to see anymore Javascript sneaking inty my Python. Especially when there is already support for IO bound concurrency elsewhere in the language. If I want to do JS syntax, I'll fucking use JS. And I really don't want to see Python go the C++ route where it just wants to be everything and do everything so that you end up with so many possible approaches to the same problem that two devs can't read each others code anymore.
I'm relatively new to Python - how does one do concurrent IO without async/await?
My main complaint, though, about Python async is - because it is opt-in I never know if I forgot a sync IO call somewhere that will block my worker. JS makes everything async by default and there is effectively no chance of blocking.
The same way you always did it: sync io in threads. Caveats are similar to async/await, but stack traces don’t suck.
take your pick: threads, fork, non-explicit coroutines (gevent), io multiplexing (select/poll/epoll/kqueue), virtual threads/goroutines
Async was a soft keyword for many, many years in order to maintain compat. And before that, asyncio used yield to be even more compatible.
They took a decade to solidify that. At some point, you have to balance evolution and stability. For a language as popular as Python, you can not break the world every week, but you can't stay put for 5 years.
Yeah I like the feature but hate the keyword. Dunder lazy imports sounds good enough imho.
They thought about backward compatibility, and offer an alternative syntax that use no keyword for library that want to activate it yet stay compact with old version. It's already in the spec.
This is the wrong syntax, comparable to how "u" strings were the wrong syntax and "b" strings are the right syntax.
They make that, what should be the default, a special case. Soon, every new code will use "lazy". The long term effect of such changes is a verbose language syntax.
They should have had a period where one, if they want lazy imports, has to do "from __future__ import lazy_import". After that period, lazy imports become the default. For old-style immediate imports, introduce a syntax: "import now foo" and "from foo import now bar".
All which authors of old code would have to do is run a provided fix script in the root directory of their code.
In short, grandiose change, wrong syntax.
This would be a huge deal for Python startup time *if* it was applied to all the standard library packages recursively. Right now importing asyncio brings in half the standard library through transitive imports.
This is needed, but I don't like new keywords. What I would love, for many reasons, is if we could decorate statements. Then things like:
import expensive_module
could be:
@lazy
import expensive_module
or you could do:
@retry(3)
x = failure_prone_call(y)
lazy is needed, but maybe there is a more basic change that could give more power with more organic syntax, and not create a new keyword that is special purpose (and extending an already special purpose keyword)
As a side note, It would be great to have a 'preview' for HN comments. I updated the above because I forgot to add line breaks. Sigh. I bet 'preview' would stop a lot of not well thought out comments too.
For what it's worth, you can already do: x = retry(3)(failure_prone_call)(y)
And, for that matter, expensive_module = lazy(importlib.import_module)('expensive_module') .
Would this statement decorator then manipulate the AST of the following statement or how would that work?
I like the approach of ES6 where you pull in bindings that are generally lazily resolved. That is IMO the approach that should be the general strategy for Python.
The import/require schism was necessary precisely of all the issues with module side effects the python community is rediscovering now. I suppose they won’t get around a similar solution eventually; but judging from years of discussions in the JS world, this is going to be dragging on for a while.
Remember mercurial? Me neither. But what I remeber is this article I've read about all the hacks they had to do to achieve reasonable startup time for CLI in python. And the no #1 cause was loading the whole world you don't ever need. As I recall they somehow monkeypatched the interpreter to ignore imports and just remember their existence until they were actually needed, at which point the import happened. So all the dead paths were just skipped.
I recall chg was a must.
Ugh...I like the idea, but I wish lazy imports were the default. Python allows side effects in the top level though so that would be a breaking change.
Soooo instead now we're going to be in a situation where you're going to be writing "lazy import ..." 99% of the time: unless you're a barbarian, you basically never have side effects at the top level.
We need some kind of unexec revival in Linux so we don't have to resort to crap like this. Maybe CRIU based. Worst case, some Python specific hack. But Python's import system is wacky enough without lazy loading. This sounds like combining the worst of multiple worlds.
Kinda related, I wish there was an easy way to exclude dependencies at pip-install time and mock them at runtime so an import doesn't cause an exception. Basically a way for me to approximate "extras" when the author isn't motivated to do it for me, even though it'd be super brittle.
I wonder how much things would break if all imports were lazy by default.
Maybe nothing would break ?
edit: ok well "xxx in sys.modules" would indeed be a problem
Yeah unfortunately in real world Python code people put side effects in their modules all the time.
In Python? I almost never see that. And I’ve certainly never signed off on a PR that did that.
Recursively importing other modules is itself a side effect.
In fact, all the code you see in the module is "side effects", in a sense. A `class` body, for example, has to actually run at import time, creating the class object and attaching it as an attribute of the module object. Similarly for functions. Even a simple assignment of a constant actually has to run at module import. And all of these things add up.
Further, if there isn't already cached bytecode available for the module, by default it will be written to disk as part of the import process. That's inarguably a side effect.
Side-effect means you're changing state outside the scope the code is running.
Sure thing you can declare globals variable and run anything on a module file global scope (outside funcs and class body), but even that 'global' scope is just an illusion, and everything declared there, as yourself said, is scoped to the module's namespace
(and you can't leak the 'globals' when importing the module unless you explicity do so 'from foo import *'. Think of python's import as eval but safer because it doesn't leaks the results from the module execution)
So for a module to have side-effect (for me) it would either:
- Change/Create attributes from other modules
- Call some other function that does side-effect (reflection builtins? IO stuff)
Maybe your code is awesome, but you're almost certainly using a library which does it.
This. The most egregious example I've ever seen is the inflect library, takes almost 2 seconds(!) to import: https://github.com/jaraco/inflect/issues/212
There will be chaos, all code bases usually have tons of side effects.
All my code which uses import probing would fail, such as fallbacks:
Conditional support testing would also break, like having tests which only run if module2 is available: The proto-PEP also gives an example of using where some global configuration changes only during import.In general, "import this" and "import antigravity" - anything with import side-effects - would stop working.
Oh, and as the proto-PEP points out, changes to sys.path and others can cause problems because of the delay between time of lazy import and time of resolution.
Then there's code where you do a long computation then make use of a package which might not be present.
All of these would be replaced with "import module; module.__name__" or something to force the import, or by an explicit use to __import__.With the LazyLoader technique I described at https://news.ycombinator.com/item?id=45467489 , there is no problem:
The implementation should probably convert that exception back to ImportError for you, but the point is that the absence of an implementation can still be detected eagerly while the actual loading occurs lazily.Ahh, so you do the find first, and keep that around before loading.
I have bad memories of using a network filesystem where my Python app's startup time was 5 or more seconds because of all the small file lookups for the import were really slow.
I fixed it by importing modules in functions, only when needed, so the time went down to less than a second. (It was even better using a zipimport, but for other reasons we didn't use that option.)
If I understand things correctly, your code would have the same several-second delay as it tries to resolve everything?
Yes, if checking for a file is slow, then checking for a file is slow. If you need to know up front whether the module exists, then you can't get around using the "figure out whether the module exists" machinery up front. And if the definition of "a module exists" includes cases where the module is represented by a file whose existence you have to check for, then there's no getting around that, either.
(Trying to do "fallback" logic with lazily-loaded modules is also susceptible to race conditions, of course. What if someone defines the module before you try to use it?)
You have to opt in with the lazy import keyword no matter what. This pep also prevents lazy import in try catch. I think your concern matters if ‘module’ itself has a lazy import that you want to check exists. With this you now need to be more rigorous in check those sub dependencies.
This can already happen with non top level imports so it is not a necessarily a new issue, but could become more prevalent if there is an overall uptake in this feature for optional dependencies.
My post was a response to the sedatk conjectural "I wonder how much things would break if all imports were lazy by default."
I have zero concerns about this PEP and look forward to its implementation.
Very unfortunate how from json lazy import dumps would result in backward compatibility issues. It reads much better and makes it easier to search for lazy imports, especially if in the future something else becomes optionally lazy as well.
Looks good to me. I use a tab-completion trick where the tab-completer tool calls the script I'm about to invoke with special arguments, and the script reflects on itself and responds with possible completions. But because of slow imports, it often takes a while for the completion to respond.
I could, and sometimes do, go through all the imports to figure out which ones are taking a long time to load, but it's a chore.
Really excited about this - we've recently been struggling with making imports lazy without completely messing up the code in DeepInverse https://deepinv.github.io/deepinv/
That seems really great, .moving imports in-line for CLI tools startup and test discovery has always been a pain.
I don’t really agree with the premise:
> The dominant convention in Python code is to place all imports at the beginning of the file. This avoids repetition, makes import dependencies clear and minimizes runtime overhead.
> A somewhat common way to delay imports is to move the imports into functions, but this practice requires more work [and] obfuscates the full set of dependencies for a module.
The first part is just saying the traditions exist because the traditions have always existed. Traditions are allowed to change!
The second part is basically saying if you do your own logic-based lazy imports (inline imports in functions) then you’re going against the traditions. Again, traditions are allowed to change!
The point about the import graph being obfuscated would ring more true if Python didn’t already provide lightning fast static analysis tools like ast. If you care about import graphs at the module level then you’re probably already automating everything with ast anyway, at which point you just walk the whole tree looking for imports rather than the top level.
So, really, the whole argument for a new lazy keyword (instead of inlining giant imports where they are needed) is because people like to see import pytorch at the top of the file, and baulk at seeing it — and will refuse to even look for it! — anywhere else? Hmmm.
What does seem like a pain in the ass is having to do this kind of repetitive crap (which they mention in their intro):
But perhaps the solution is a pattern where you put all your stuff like that in your own module and it’s that module which is lazy loaded instead?>The second part is basically saying if you do your own logic-based lazy imports (inline imports in functions) then you’re going against the traditions. Again, traditions are allowed to change!
No, they are saying the tradition is there for a reason. Imports at the beginning of the file makes reasoning about the dependencies of a module much easier and faster. I've had to deal with both and I sure as hell know which I'd prefer. Lazy imports by functions is a sometimes necessary evil and it would be very nice if it became unnecessary.
For posterity, as the submitted link seems temporary: https://github.com/python/peps/pull/4622
The proper permanent link is https://peps.python.org/pep-0810/ .
I love the feature but I really dislike using the word lazy as a new language keyword. It just feels off somehow. I think maybe defer might be a better word. It is at least keeps the grammar right because it would be lazily.
Also the grammar is super weird for from imports.second this.
Just might as well add `defer` keyword like Golang.
Crossing my fingers that it goes through this time. Been in the top 3 of my Python wishlist for nearly a decade.
Does this fix the circular imports problem that comes up if you don't structural your programs in a hierarchical way?
Even with eager importing there is only a "circular import problem" if you try to import names `from` the modules — as I pointed out a few days ago (https://news.ycombinator.com/item?id=45400783).
It should, when you do lazy imports on your own it fixes the problem.
This is what I thought of too. I really only know python, do other languages not have that issue? In python it does not seem like a "problem" to me - whenever I have seen circular import issues it is because the code is organized poorly. I worry that this feature will lead to devs "fixing" circular import issues by using lazy imports.
Sometimes it's hard to avoid cyclic imports, without blaming the design. Like if a Parent has a Child, and the Child needs to know of the parent. Only way to solve that in python is to put everything in the same file, which also feels like bad deisgn.
I would say in that case, the Parent and Child shouldn't need to know about each other - some kind of handler in a different file should.
Although I guess that doesn't work in all cases, like defining foreign key relationships when using an orm (like sqlalchemy) for example. But in the orm case, the way to get around that is... lazy resolution :^)
Interesting. I find that I always have this problem in any non-trivial python project, and don't find this to be due to poorly organized code. I have only seen this requirement in Python.
IME circular import errors aren't due to poor organization; they're due to an arbitrary restriction Python has.
I don’t want lazy imports. That’s just makes performance shitty later and harder to debug. It’s a hacky workaround.
What I want is for imports to not suck and be slow. I’ve had projects where it was faster to compile and run C++ than launch and start a Python CLI. It’s so bad.
imports run arbitrary Python code, making them faster is the same as making every other language feature faster
Make Python more like Haskell, LFG!
what is the point of this? you can just import inside function definitions:
thus numpy.linalg is only imported the first time you call the antislash function. Much cleaner than a global import.Ignore wrong traditions. Put all imports in the innermost scopes of your code!
That's a hack that forces you to duplicate and hide imports. The tradition is to specify imports at the top because it's that much better readability wise.
Won’t this have a very noticeable performance hit on the first request? Thinking web frameworks like Flask and Django.
why does this have to be a syntax feature and not a lazy code loader at the intepreter level?
YES!!!
I wish all imports were lazy by default.
I know/heard there are "some" (which I haven't seen by the way) libraries that depend on import side effects, but the advantage is much bigger.
First of all, the circular import problem will go away, especially on type hints. Although there was a PEP or recent addition to make the annotation not not cause such issue.
Second and most important of all, is the launch time of Python applications. A CLI that uses many different parts of the app has to wait for all the imports to be done.
The second point becomes a lot painful when you have a large application, like a Django project where the auto reload becomes several seconds. Not only auto reload crawls, the testing cycle is slow as well. Every time you want to run test command, it has to wait several seconds. Painful.
So far the solution has been to do the lazy import by importing inside the methods where it's required. That is something, I never got to like to be honest.
Maybe it will be fixed in Python 4, where the JIT uses the type hints as well /s
Lazy imports mean late errors. Fail fast is a good design principle.
Top-level code should not be able to fail except in incredibly deterministic ways that are tested during development. Failing fast is not as good as not failing at all. Lazy imports mean the power to avoid importing things that don't need to be imported at all on this run. Good design also cares about performance to some extent. On my machine, asking pip to do literally nothing takes several times as long as creating a new virtual environment --without-pip .
Python really seems like a bad fit for that. So your imports succeed, what now? Do they have all the functions or fields your program needs? Those are still resolved at the last possible moment. If you want to be sure your program actually runs you will have to write and run tests with and without lazy imports.
Wake me up when we can import a module by relative file path.
Relative imports have been supported for approximately forever (https://stackoverflow.com/questions/72852). If you mean "by explicitly specifying a path string" (as opposed to a symbolic name), that has also been supported for approximately forever (https://stackoverflow.com/questions/67631). Today, the `importlib` standard library exposes all the steps of the import process — including figuring out where the source code is, checking the `sys.modules` cache etc. — and lets you hook into everything (in particular, you can bypass the entire "finder" process and turn a file path into a dummy "spec" which is fed to a "loader").
The flexibility of this system also entails that you can in effect define a completely new programming language, describe the process of creating Python bytecode from your custom source, and have clients transparently `import` source in the other language as a result. Or you can define an import process that grabs code from the Internet (not that it would be a good idea...).
If you mean "by explicitly specifying a relative path, and having it be interpreted according to the path of the current module's source code", well first you have to consider that the current module isn't required to have source code. But if it does, then generally it will have been loaded with the default mechanism, which means the module object will have a `__file__` attribute with an absolute path, and you just set your path relative to that.
Nope. Relative imports work by relative package path, which is not at all the same. Often when you run Python you don't even have a package path.
Using `importlib` is a horrible hack that breaks basically all tooling. You very very obviously are not supposed to do that.
> Relative imports work by relative package path, which is not at all the same.
Exactly. This gives you the flexibility to distribute a complex package across multiple locations.
> Often when you run Python you don't even have a package path.
Any time you successfully import foo.bar, you necessarily have imported foo (because bar is an attribute of that object!), and therefore bar can `from . import` its siblings.
> Using `importlib` is a horrible hack that breaks basically all tooling. You very very obviously are not supposed to do that.
It is exactly as obvious (and true) that you are not "supposed to", in the exact same sense, directly specify where on disk the source code file you want to import is. After all, this constrains the import process to use a source code file. (Similarly if you specify a .pyc directly.) Your relative path doesn't necessarily make any sense after you have packaged and distributed your code and someone else has installed it. It definitely doesn't make any sense if you pack all your modules into a zipapp.
> Using `importlib` is a horrible hack that breaks basically all tooling. You very very obviously are not supposed to do that.
This is an assertion that has absolutely no reasoning behind it. I'm not saying I disagree; I'm just saying there is a time and a place for importlib.
You… can? I mean in the strictest sense you're technically not importing by file path but if you make your folder a module by slapping an __init__.py in there then your relative imports will follow the directory tree. I think as of Python 3.3 the init file is optional so it will do it by default but I can't remember if there are still some cases where it's required. The only thing you can't do is go "up" to a higher directory than the root module.
Also if that doesn't strike your fancy all of the importlib machinery is at your disposal and it's really not very much work to write an import_path() function. It's one of the patterns plug-in systems use and so is stable and expected to be used by end users. No arcane magic required.
> if you make your folder a module by slapping an __init__.py in there then your relative imports will follow the directory tree.
`__init__.py` has nothing to do with making this work. It is neither necessary (as of 3.3, yes, you got it right: see https://peps.python.org/pep-0420/) nor sufficient (careless use of sys.path and absolute imports could make it so that the current folder hasn't been imported yet, so you can't even go "up into" it). The folder will already be represented by a module object.
What `__init__.py` does is:
1. Prevents relative imports from also potentially checking in other paths.
2. Provides a space for code that runs before sub-modules, for example to set useful package attributes such as `__all__` (which controls star-imports).
> if you make your folder a module by slapping an __init__.py in there then your relative imports will follow the directory tree
haha no
> all of the importlib machinery is at your disposal
This breaks all tooling. Awful option.