This article is written with Python 3.7 in mind. If you’re using an older version, keep in mind that some things discussed in this article might work differently, especially if you’re using Python2.
If you’ve been using Python for more than just simple proof-of-concept apps, you have probably encountered issues due to circular dependencies at some point. While some other languages, like Java, will allow you to get away with this, with Python it’s not always that simple.
Before we start, I will note that most resources will tell you these kinds of issues hint at a bigger, design-level problem. While I mostly agree with that statement, this will not be the focus of this article. Instead, I will attempt to show you how and why import-related exceptions occur and what you can do in situations where making significant design changes, such as splitting modules into more independent ones, is not an option.
How do errors caused by circular imports manifest?
In simplest terms, a circular import occurs when module A tries to import and use an object from module B, while module B tries to import and use an object from module A.
You can see it on on an example with simple modules a.py and b.py:
# a.py snippet
print('First line of a.py')
from package.package_b.b import fun_b
print('Imported fun_b inside a.py')
def fun_a():
print('Executing fun_a')
fun_b()
return 'a'
print('Last line of a.py')
# b.py snippet
print('First line of b.py')
from package.package_a.a import fun_a
print('Imported fun_a inside b.py')
def fun_b():
print('Executing fun_b')
return 'b'
fun_a()
print('Last line of b.py')
We’ll run the code from run.py, which just imports a.py.
# run.py snippet import package.package_a.a
This is the output we get:
# output of run.py /home/petar/Projects/python_circular_import_article/venv/bin/python /home/petar/Projects/python_circular_import_article/package/run.py First line of a.py First line of b.py Traceback (most recent call last): File "/home/petar/Projects/python_circular_import_article/package/run.py", line 1, in <module> import package.package_a.a File "/home/petar/Projects/python_circular_import_article/package/package_a/a.py", line 3, in <module> from package.package_b.b import fun_b File "/home/petar/Projects/python_circular_import_article/package/package_b/b.py", line 3, in <module> from package.package_a.a import fun_a ImportError: cannot import name 'fun_a' from 'package.package_a.a' (/home/petar/Projects/python_circular_import_article/package/package_a/a.py) Process finished with exit code 1
As you can see, we get an exception as soon as b.py tries to import a.py. Nothing but the first two printouts from each file get displayed.
Notice how we’re using the ‘from import ‘ syntax. With this format, Python expects that function to exist within that module and tries to access it in that exact moment. Whenever the module is imported, Python checks if the sys.modules dictionary contains that import. If not, it tries to import that module and add it to the dictionary. The way it does this is it goes over the module line by line.
Let’s see what happens in detail:
- run.py imports module a –> module a does not exist in sys.modules, interpreter starts going over the module a lines
- module a gets to the import fun_b line –> module b does not exist in sys.modules, interpereter starts going over the module b lines
- module b tries to import fun_a from module a –> module a exists in sys.modules, so there’s no need to import it again line by line.
This is why ‘First line of a.py’ only gets printed once. Since it’s already imported, Python tries to access fun_a. However, the interpreter did not finish going over every line of module a, it stopped at line 2 (before fun_a could be interpreted). Finally, we get an ImportError.
The main thing to take away from this is that sys.modules can contain a module without that module being fully imported. So how do we work around this?
Absolute module imports
Absolute imports have the import package_a.a format, or alternatively, from package_a import a. This way, we’re importing the module and not the functions.
The main difference is that with relative imports, the interpreter will attempt to immediately access the function and potentially raise an exception. This is not the case with absolute imports, because the function is only needed at the exact line it’s called.
This can help us if the only place we use our functions is inside other functions or classes, because code inside of them doesn’t get evaluated on import time.
Let’s try our code with absolute imports:
# a.py snippet
print('First line of a.py')
from package.package_b import b # alternative way of writing import package.package_b.b
print('Imported module b inside a.py')
def fun_a():
print('Executing fun_a')
b.fun_b()
return 'a'
print('Last line of a.py')
# b.py snippet
print('First line of b.py')
from package.package_a import a # alternative way of writing import package.package_a.a
print('Imported module a inside b.py')
def fun_b():
print('Executing fun_b')
return 'b'
print('Executing fun_a from b.py')
a.fun_a()
print('Last line of b.py')
…And again, we get an error:
# output of run.py /home/petar/Projects/python_circular_import_article/venv/bin/python /home/petar/Projects/python_circular_import_article/package/run.py Traceback (most recent call last): File "/home/petar/Projects/python_circular_import_article/package/run.py", line 1, in <module> import package.package_a.a File "/home/petar/Projects/python_circular_import_article/package/package_a/a.py", line 3, in <module> from package.package_b import b # alternative way of writing import package.package_b.b File "/home/petar/Projects/python_circular_import_article/package/package_b/b.py", line 14, in <module> a.fun_a() AttributeError: module 'package.package_a.a' has no attribute 'fun_a' First line of a.py First line of b.py Imported module a inside b.py Executing fun_a from b.py Process finished with exit code 1
Let’s see what happened:
- module a starts importing module b
- module b starts importing module a and does so successfully, importing of module b continues
- we get to line 14 of module b where we try to call a.fun_a(). Since module a is not fully imported, we get an error.
This time, we get a different exception – AttributeError. This happens because fun_a() gets called during import time and the module ‘a’ obviously doesn’t have it since it didn’t finish importing. The issue is that we’re calling fun_a() at the top module level and the import mechanism evaluates this line while trying to import.
Before going further, there’s one more interesting interaction I would like to show you. Instead of running run.py, let’s call a.py directly:
# output of a.py /home/petar/Projects/python_circular_import_article/venv/bin/python /home/petar/Projects/python_circular_import_article/package/package_a/a.py First line of a.py First line of b.py First line of a.py Imported module b inside a.py Last line of a.py Imported module a inside b.py Executing fun_a from b.py Executing fun_a Executing fun_b Last line of b.py Imported module b inside a.py Last line of a.py Process finished with exit code 0
It works?! Let’s see how exactly this happens.
This is related to a specific behavior that’s worth keeping in mind. The first module that gets ran, i.e. the entry point of the program, will immediately be loaded into sys.modules, but it will be called ‘main‘ instead of its original name. So, this is the execution flow in the case of running a.py first:
- interpreter starts evaluating a.py, it’s loaded into sys.modules by the name main
- it gets to the import b step, b is not in sys.modules so it has to import it
- importing of b begins, it gets to the import a step
- this is the strange part – a is not in sys.modules yet, because it was loaded under the name main. This means a has to be imported as well!
- importing of a starts (from the beginning – it doesn’t continue because main is considered a different module) and is successful
- importing of b continues after the import a line, it too is successful
- execution of a continues and the program successfully exits
With that out of the way, let’s get back to our issue of making run.py run correctly. The root cause of our problem is calling fun_a() at the top level of module b. In this situation, the only ‘quick’ fix we can do is move the call to a function so that it doesn’t get evaluated at import time:
# b.py
print('First line of b.py')
from package.package_a import a # alternative way of writing import package.package_a.a
print('Imported module a inside b.py')
def fun_b():
print('Executing fun_b')
return 'b'
print('Executing fun_a from b.py')
a.fun_a()
print('Last line of b.py')
# output of run.py /home/petar/Projects/python_circular_import_article/venv/bin/python /home/petar/Projects/python_circular_import_article/package/run.py First line of a.py First line of b.py Imported module a inside b.py Last line of b.py Imported module b inside a.py Last line of a.py Process finished with exit code 0
Quick note: in this situation, you could actually use the from package_a.a import fun_a syntax, but you would have to put that import inside the function, right before usage. I would advise against relying on this too much: it goes against PEP8 and your IDE/linter will complain. In addition, any potential errors will be not be detected at ‘import time’ and you will encounter exceptions later on when the function is invoked. I think we can agree that it’s much safer to have your program crash right at startup during imports instead of randomly crashing after a while.
You might think ‘Whatever, I never call functions at the top level anyway!’. Sure, but there is one more very common case during which this can happen.
I’m talking about extending a class that’s inside of a module with circular dependency.
# new file with just a class import snippet import package.package_class class B(package_class.class_a):
This line of code is at the top level of a module and the interpreter will try to evaluate it at import time. At that point, if the relationship between modules is circular just like in the previous example, you have no viable option to avoid this. Your only choice is to either give up on trying to extend the class or refactor your code to avoid the circular import altogether.
Takeaways
We’ve seen a general high-level overview of the import mechanism. So how should you import in real projects?
First of all, I suggest using explicit absolute imports whenever you can (e.g. from package_a.a import fun_a), or rather, when there are no circular dependencies preventing you from doing that. They are explicit, result in the most concise code and if there is a problem you will know right away because the error will happen as soon as you try to import.
If you do have circular dependencies, you should try your best to avoid that situation altogether by changing your project structure, e.g. splitting modules into multiple smaller ones. In case this is not an option for you, you can try using deferred explicit imports (import right before usage, for example before a function call) or you can use absolute module imports (e.g. import package_a.a).
Just be aware that these solutions are not optimal and if you find that the amount of coupling/tangling in your project is increasing, you should really try to resolve the underlying issue.
