Project structure and dependency managment#
In this chapter we will focus on the structure of the project as well as the tools to manage project specifications and dependencies.
Not so long ago, Python packages were managed with files such as setup.py, setup.cfg, or requirements.txt. Nowadays their usage is discouraged as new better solutions have been introduced (see for instance PEP621).
All projects use modern packaging (pyproject.toml, no more setup.py !!!)
In addition to the packaging specification format, the Python community has developped very useful tools to ease the work of developpers. In the Aramis team we have made the choice to rely on a tools called Poetry.
Dependencies are managed with `Poetry` (the file `poetry.lock` fixes all the dependencies, no more `requirements.txt` !!!)
Install Poetry#
It is recommended to install Poetry with a tool called pipx such that Poetry will run in an isolated environment.
Let’s start by installing pipx then:
$ brew install pipx
$ pipx ensurepath
You should now be able to install Poetry with pipx:
$ pipx install poetry
If everything worked, you should have poetry installed in ~/.local/bin/poetry
$ which poetry
/Users/nicolas.gensollen/.local/bin/poetry
It’s also interesting to check our version of Poetry:
$ poetry --version
Poetry (version 2.1.3)
If all these commands are running correctly, then you can move on to the next section.
Exercice#
We are going to continue or small project that we started in Chapter 1.
Define the specifications of our package using Poetry#
Let’s use Poetry to manage our project’s configuration and dependencies. We can generate our package specifications with the poetry init command. After hitting Enter, Poetry will ask you a few questions in order to write an appropriate pyproject.toml file:
$ poetry init
This command will guide you through creating your pyproject.toml config.
Package name [calculator-lib]: calculator
Version [0.1.0]:
Description []: Python library to compute things.
Author [NicolasGensollen <nicolas.gensollen@gmail.com>, n to skip]:
License []: MIT
Compatible Python versions [>=3.12]:
Would you like to define your main dependencies interactively? (yes/no) [yes] no
Would you like to define your development dependencies interactively? (yes/no) [yes] no
Generated file
[project]
name = "calculator"
version = "0.1.0"
description = "Python library to compute things."
authors = [
{name = "NicolasGensollen",email = "nicolas.gensollen@gmail.com"}
]
license = {text = "MIT"}
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
]
[build-system]
requires = ["poetry-core>=2.0.0,<3.0.0"]
build-backend = "poetry.core.masonry.api"
Do you confirm generation? (yes/no) [yes] yes
Once finished, you can inspect your project and verify that you have indeed one more file: pyproject.toml which describes your project and how to build it:
$ tree
.
├── LICENSE
├── README.md
└── pyproject.toml
1 directory, 3 files
Note that the poetry init command didn’t do anything else, and we could totally have written the pyproject.toml file ourselves if we wanted to. Poetry just made our life easier by only requesting specific information.
Where to put the code: the flat layout and the src layout#
It’s now time to decide where we will put the meat of our package: the code itself !
There are two main responses to this question: the flat layout and the src layout. The flat layout consists in creating a folder with the package name and put the code files within this folder. People have been using the flat layout for decades and it is probably still the most frequent solution you’d come accross. However, it was pointed out that, because of the way Python discovers packages, doing this can lead to very nasty bugs.
The solution is very simple and consists in putting the code in the subfolder src/name_of_the_package. We therefore encourage you to use an src layout to organize our code every time this is possible.
You can read more about this subject on this page.
If the project is a library, use an `src` layout.
Let’s initialize this:
$ mkdir -p src/calculator
$ touch src/calculator/__init__.py
$ tree
.
├── LICENSE
├── README.md
├── pyproject.toml
└── src
└── calculator
└── __init__.py
3 directories, 4 files
We still need to tell Poetry that we are using such a layout. Modify the pyproject.toml file and add this:
...
readme = "README.md"
packages = [
{ include = "calculator", from = "src" },
]
requires-python = ">=3.12"
...
Where to install the package: a virtual environment#
At this point, we have almost all we need to install our package, but we didn’t think about where we would want to install it…
We could install our package without worrying about that, but that would install it (as well as all its dependencies) into our computer native environment. Doing that is a VERY BAD IDEA as you will very quickly pollute your system with thousands of dependencies with potentially conflicting versions.
The solution to this problem is very simple:
Always use dedicated virtual environments.
One project = One virtual environment.
When the project ends, delete the environment.
There are multiple tools to create and manage virtual environments but, at Aramis, we tend to use conda for that.
Note that conda is much more than a simple virtual environments manager, but this is the functionality that we are going to use in this tutorial.
In our simple package example, as we don’t really have dependencies, we could let the users worry about virtual environments, but let’s follow good practices and facilitate our life by creating a new file named environment.yml with the following content:
name: calculator
channels:
- conda-forge
dependencies:
- python=3.12
As mentionned above, we only specify the Python version and the environment’s name as we don’t have any dependencies yet. But if we had dependencies not installable with pip, this is where we could add them (assuming they were packaged with conda of course…).
We can use this file to create our environment:
$ conda env create -f environment.yml
$ conda activate calculator
At this point, you should have the following structure:
$ tree
.
├── LICENSE
├── README.md
├── environment.yml
├── pyproject.toml
└── src
└── calculator
└── __init__.py
3 directories, 5 files
Let’s modify the file src/calculator/__init__.py and add this line:
PI = 3.1415
such that we have at least one object in our package.
We can now install our package, in our virtual environment, using Poetry:
$ poetry install
Updating dependencies
Resolving dependencies... (0.1s)
Writing lock file
Installing the current project: calculator (0.1.0)
This did a few things for us: it resolved our dependencies (more on that in the next section) and created a new file poetry.lock out of this resolution. This file should have a content similar to this:
# This file is automatically @generated by Poetry 2.1.3 and should not be changed by hand.
package = []
[metadata]
lock-version = "2.1"
python-versions = ">=3.12"
content-hash = "75265641fd1a3f2a4d608312a3879427b7141ac2a51d0873da5711cbc8ead28e"
We will come back to this file in the next section to understand why it is so important when working on software collaboratively.
Poetry also installed our dependencies (there is None at the moment, but if we had some they would have been installed), and it finished by installing our package, all of that in our virtual environment.
To finish this part, we can test that everything is working as expected. Open a Python interpreter and test that we can import and use our package:
$ python
Python 3.12.11 | packaged by conda-forge | (main, Jun 4 2025, 14:38:53) [Clang 18.1.8 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from calculator import PI
>>> PI
3.1415
Dependencies and the poetry.lock file#
In order to better understand why a developement team woul benefit from the poetry.lock file we saw in the prevous section, let’s add a dependency to our package. We will add Numpy as a dependency which is installable with pip. We don’t really know what version of Numpy we want, but we know that we want something more recent that v2.0.0 because we don’t want to deal with the deprecated things from the v1. To do that, we can put a constraint on the version of Numpy that we want:
$ poetry add "numpy>=2.0.0"
Updating dependencies
Resolving dependencies... (0.1s)
Package operations: 1 install, 0 updates, 0 removals
- Installing numpy (2.3.0)
Writing lock file
Again, Poetry does a few things under the hood. It first added the dependency with our constraint to our package specifications. You can verify that by looking at the pyproject.toml file:
$ git diff pyproject.toml
diff --git a/pyproject.toml b/pyproject.toml
index 804db9b..17ad9b5 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -12,6 +12,7 @@ packages = [
]
requires-python = ">=3.12"
dependencies = [
+ "numpy (>=2.0.0)"
]
We now have one dependency listed in our specifications. But note that there is only a version constraint here, we are nowhere telling our users which version of Numpy to install. v2.0.0, v2.1.0, and v2.2.1 would all be valid versions of Numpy. Moreover, if Numpy v46.12.2 existed, it would also satisfy this constraint.
Hopefully, Poetry did more than that: it resolved the dependencies, meaning that it computed, from the constraints that we imposed, the exact versions of all the dependencies that should be installed. Since this process is sometimes called locking the dependencies, Poetry write the results of its calculations into a file called poetry.lock.
Note that Poetry cannot solve impossible constraints. Imagine we had our numpy constraint (>=2.0.0), but another dependency on a package X which itself depends on Numpy, but with a constraint <2.0.0. In this situation, there is no version of Numpy that would satisfy our specifications, and Poetry will fail to lock and install our package. Here the problem seems obvious, but as your project grows and depends on a lot of packages, which in turn depend on other packages, it can become extremely difficult to understand what’s going wrong when Poetry fails to lock.
Let’s take a look at the poetry.lock file:
# This file is automatically @generated by Poetry 2.1.3 and should not be changed by hand.
[[package]]
name = "numpy"
version = "2.3.0"
description = "Fundamental package for array computing in Python"
optional = false
python-versions = ">=3.11"
groups = ["main"]
files = [
{file = "numpy-2.3.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:c3c9fdde0fa18afa1099d6257eb82890ea4f3102847e692193b54e00312a9ae9"},
{file = "numpy-2.3.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:46d16f72c2192da7b83984aa5455baee640e33a9f1e61e656f29adf55e406c2b"},
{file = "numpy-2.3.0-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:a0be278be9307c4ab06b788f2a077f05e180aea817b3e41cebbd5aaf7bd85ed3"},
...
{file = "numpy-2.3.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:e017a8a251ff4d18d71f139e28bdc7c31edba7a507f72b1414ed902cbe48c74d"},
{file = "numpy-2.3.0.tar.gz", hash = "sha256:581f87f9e9e9db2cba2141400e160e9dd644ee248788d6f90636eeb8fd9260a6"},
]
[metadata]
lock-version = "2.1"
python-versions = ">=3.12"
content-hash = "e959f25fb0916f5459e0d6efdf2d97b969a8c756ecdceb6883588e37c9c05822"
There are a lot of small details that we won’t cover here, but the important point is that poetry has locked Numpy to be v2.3.0, which is the most recent version of Numpy that satisfies our constraints.
Finally, Poetry did one more thing for us, it installed the locked version of Numpy in our virtual environment. This can be verified by opening a Python interpreter, importing Numpy, and checking the version manually:
$ python
Python 3.12.11 | packaged by conda-forge | (main, Jun 4 2025, 14:38:53) [Clang 18.1.8 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.__version__
'2.3.0'
Summing up, there is an important distinction to understand between the pyproject.toml file which is a mandatory package specification file and the poetry.lock file which is a tool-specific optional file. The former describes our package and, as part of this description, gives the list of dependencies with the version constraints, if any. Poetry takes this information and tries to solve the dependency version constraints we wrote. If it is successful, it writes them in the poetry.lock file. All users installing the package in development mode will then rely on this file and install the exact same versions of all dependencies.
Note that prior techniques such as the requirements.txt files were only describing the constraints, such that two users could end up installing different versions of the dependencies for the same requirements.txt file. This can naturally lead to very nasty bugs as developers of the project could experience different behaviors. By relying on Poetry and its lock files, we are making sure to avoid those bugs in our development teams!
Do not forget to commit !#
We have one more thing to do in this chapter: commit our work to version control. If you followed the tutorial you should be on the main branch. Since we are setting up the project, let’s not worry about feature branches for now and directly push to this branch:
$ git add .
$ git commit -m "Initialize project with Poetry"
$ git push origin main