Learnings

Python in practice, part 1 of 9

Python Project Setup

So far every program has been one file that you run with python3 file.py. Real work is a folder of files, some packages written by other people, and a way to run it all on another machine. This page grows one small project, a package called bank, from a single file into a project you can install and run as a command. The terminal output shown was captured from real runs with Python 3.9 on macOS. Long folder paths are shortened to ....

Environments and pip

Group 1 of 4 · a private Python for each project

Before writing any code, give the project its own place to install packages. Everything in this group is typed in a terminal.

The problem with one Python

Your machine has one main Python install. If every project installs its packages there, they all share one set. A package here means code written by someone else that you download, such as requests or tabulate.

Sharing breaks in three ways:

  • Version clashes. Project A needs version 1 of a package and project B needs version 2. Only one version can be installed at a time.
  • No record. After a year nobody knows which of the 80 installed packages this project really needs, so nobody else can run it.
  • Risk to the system. On macOS and Linux the system Python is used by the operating system itself. Changing its packages can break other tools.

The fix is a virtual environment: a private folder of packages that belongs to one project.

Creating a virtual environment

The venv module comes with Python. It creates a folder that holds a link to the Python interpreter and an empty place for packages. The usual name for that folder is .venv.

bin holds python, pip and the activate scripts. lib is where installed packages will go. On Windows the folder is called Scripts instead of bin.

Rule: never commit .venv to git. It is large, it only works on the machine that made it, and it can be rebuilt in a minute from a list of package names. Add .venv/ to .gitignore.

Activating and leaving it

Activating an environment changes your current terminal so that python and pip mean the ones inside .venv. It lasts until you close that terminal or type deactivate.

SystemActivateCheck which Python runs
macOS, Linuxsource .venv/bin/activatewhich python
Windows, Command Prompt.venv\Scripts\activatewhere python
Windows, PowerShell.venv\Scripts\Activate.ps1Get-Command python

There are three ways to tell that an environment is active. The prompt starts with (.venv). which python prints a path inside .venv. And Python itself can tell you:

Concept: sys.prefix is the folder of the Python that is running. sys.base_prefix is the folder of the real install it was made from. They differ only inside a virtual environment.

You do not have to activate at all. Calling .venv/bin/python directly uses the environment too. Activation is only a shortcut.

Installing packages with pip

pip downloads packages from the Python Package Index (PyPI) and puts them into the active environment. The bank will print tables, so install tabulate, a small package that formats rows as a text table.

==0.9.0 pins the version: you get exactly that one. Without it pip installs the newest release, which can differ from one month to the next.

CommandWhat it does
pip install tabulateInstall the newest version
pip install tabulate==0.9.0Install exactly 0.9.0
pip install "tabulate>=0.9"Install 0.9 or newer. The quotes stop the shell from reading > as a redirect
pip install --upgrade tabulateMove to the newest version
pip uninstall tabulateRemove it
pip show tabulateVersion, summary and install location of one package
pip listEverything installed in this environment

This is the one Python block on this page that needs pip install first. All the others use the standard library only.

Tip: python -m pip install ... does the same as pip install ..., but it is certain to install into the Python you just named. Use it whenever you are not sure which pip your terminal picked.

requirements.txt

Since .venv is not committed, the project needs a text file that lists its packages. pip freeze prints every installed package with its exact version, in the format pip can read back.

Anyone who clones the project then rebuilds the same environment with three commands:

Note: pip freeze also lists the packages your packages depend on. Installing requests, for example, adds four more lines. That is useful, because it pins everything, but it hides which packages you chose yourself. The pyproject.toml chapter shows where to record only your own choices.

Modules and packages

Group 2 of 4 · splitting code into files and folders

This group is about import: what it loads, where Python looks, and why the same file behaves differently when it is run and when it is imported.

A module is one file

A module is a .py file. Its name is the file name without .py. Here is the whole bank as one module.

A second file in the same folder can use it. import bank runs bank.py once from top to bottom and gives you a module object called bank. Everything defined in the file is reached with a dot.

Concept: a module is imported only once per program. A second import bank anywhere else reuses the object that already exists, so the file does not run again.

Ways to import

There are four forms. They all load the same module. They differ only in which names appear in your file. These examples need bank.py from the previous chapter in the same folder.

FormUse it when
import bankThe default. Every use says where the name comes from
from bank import AccountYou use a few names many times
import bank as bThe name is long, or there is a well known short form such as import numpy as np
from bank import *Avoid in files. See below

Why not from x import *

The star copies every public name of the module into your file. You cannot see which names arrived, and they silently replace names you already had. The os module has its own function called open:

After the star import, open("notes.txt") calls os.open, which takes different arguments and returns a number instead of a file object. Nothing on the import line warned you.

The __name__ check

Every module has a variable called __name__, set by Python before the file runs. Its value depends on how the file was started:

  • The file you run gets __name__ == "__main__".
  • A file that is imported gets its module name, for example "bank".

That lets one file do two jobs: be imported for its functions, and do something useful when run directly. Add a demo function and four lines to the bottom of bank.py. The first print is only there so you can watch the value.

Run it directly and the condition is true, so the demo runs:

Import it from another file and the condition is false, so the demo is skipped:

Why it matters: without the if, the demo would run every time anyone imported bank, including your tests. Keep the top level of a module to definitions, and put anything that acts (printing, reading input, starting a server) under the check.

A package is a folder

When one file grows too long, turn it into a package: a folder of modules. The folder name becomes the import name. A file called __init__.py inside it marks the folder as a package and runs when the package is first imported.

accounts.py is the Account class from before. It runs on its own, although it prints nothing.

The next two files import from other files of the package, so they cannot be run alone. They are marked as not runnable here. Both were tested as part of the finished project.

The two imports in __init__.py lift the main names to the top of the package. Users can write from bank import Account and do not need to know which file holds the class. __all__ is the list of names the package offers to the outside.

Note: __init__.py may be completely empty. Python 3 can even import a folder without one, as a "namespace package", but that is a special tool for splitting one package across several folders. For a normal package always add the file.

Absolute and relative imports

Inside a package there are two ways to import a neighbour.

KindWritten asMeaning
Absolutefrom bank.accounts import AccountStart from the top package name
Relativefrom .accounts import AccountOne dot: start from the folder this file is in
Relativefrom ..utils import helperTwo dots: start from the folder above

Both work. Absolute imports are easier to read and to search for, and they are what PEP 8, the Python style guide, recommends. Relative imports are shorter and keep working if the package is renamed. Pick one style for a project. The rest of this page uses absolute imports.

Relative imports have one trap. They only work when the file is loaded as part of its package. Run the file directly and Python does not know which package the dot refers to:

Rule: do not run files that live inside a package by their path. Import them, or start the package with python -m, which a later chapter covers.

How Python finds modules

For import bank, Python walks through a list of folders called sys.path and takes the first bank.py file or bank folder it finds. The list is built when Python starts, in this order:

OrderFolderHolds
1The folder of the script you ran. With python -m or python -c, the current folder insteadYour own files
2Folders listed in the PYTHONPATH environment variable, if it is setExtra folders you asked for
3The standard library foldersjson, os, random and the rest
4site-packages of the active environmentEverything installed with pip

Every module has a __file__ attribute. When an import gives you something unexpected, print it to see which file Python really loaded.

ModuleNotFoundError

If no folder in the list has the module, you get the error most people meet in their first week:

Python is not saying the module does not exist on your disk. It is saying the module is not in any folder on sys.path for the Python that is running. There are three usual causes:

  • Not installed here. You ran pip install with one Python and the script with another, or you forgot to activate the environment.
  • Wrong starting folder. Rule 1 uses the folder of the script, not the folder you are standing in.
  • A typo, or an install name that differs from the import name. For example you pip install pyyaml but import yaml.

The second cause is the confusing one. Take a project with the bank package and a script in a sub-folder:

You are standing in project, and bank is right there. But the first entry of sys.path is project/scripts, and there is no bank in that folder.

Fix: do not edit sys.path in your code. Either run the script as a module from the project folder with python3 -m scripts.report, or install your project into the environment so that bank can be imported from anywhere. The editable install chapter shows how.

Running with python -m

python -m name means: find the module called name through sys.path, the same way import would, and run it as the main program. You have used it already: python3 -m venv and python -m pip run modules named venv and pip.

python scripts/report.pypython -m scripts.report
You giveA file path, with slashes and .pyA module name, with dots and no .py
First entry of sys.pathThe folder of the fileThe current folder
Relative importsFailWork

From the project folder of the last chapter, this form finds bank, because the current folder is on the path:

Making a package runnable

python -m bank with a package name runs the file __main__.py inside that package. Without that file you get this:

The file is usually three lines that hand over to a main function. The command-line chapter writes that function. This block is not runnable alone, because it needs the package.

Concept: sys.exit(number) ends the program and reports the number to the terminal. 0 means success. Anything else means failure. Other programs and scripts read that number to decide what to do next.

A real project

Group 3 of 4 · a layout others can install and run

Now put the pieces together: where the files go, one configuration file that describes the project, and a command you can type in the terminal.

Project layout

This is the layout used for the rest of the page. It is called the src layout, because the package sits inside a folder named src.

The other common choice is the flat layout: the bank folder sits directly in bank-project, with no src.

src layoutFlat layout
Import without installingFails from the project folder. You must install the project firstWorks from the project folder, by accident of rule 1
Tests run againstThe installed package, as your users will get itWhatever folder happens to be on the path
Good forAnything you will share or deploySmall scripts and quick experiments

Tests live in their own tests folder, outside the package, so they are not shipped to users. Python Testing fills that folder.

Concept: __pycache__ folders hold compiled copies of your modules that Python makes to start faster. They are rebuilt automatically and should never be committed.

pyproject.toml

pyproject.toml is the standard file that describes a Python project. It is written in TOML, a simple format of [sections] and key = value lines. pip reads it, and so do most Python tools.

PartMeaning
[build-system]Which tool turns this folder into an installable package. setuptools is the long-standing default. pip downloads it by itself when needed
name, versionWhat pip calls the project and which release this is
requires-pythonpip refuses to install on an older Python
dependenciesPackages the code needs to run. pip installs them together with your project
[project.optional-dependencies]Named extra groups. dev holds tools only developers need, such as the test runner
[project.scripts]Terminal commands to create. Explained in the command-line chapter

With the src layout, setuptools finds the bank package inside src without being told.

Ranges or pins: in pyproject.toml write the loosest range that works, such as tabulate>=0.9. That says what the code needs. Exact pins of every package belong in requirements.txt or a lock file, which says what was installed on one tested setup. Many projects keep both.

Editable install

pip install . installs the project in the current folder like any other package, by copying it into site-packages. While you are developing that is annoying, because every edit would need a new install. An editable install, pip install -e ., does not copy. It records the location of your src folder, so the environment always sees your latest code.

Before it is installed, the package cannot be found from the project folder. This is the src layout doing its job:

Installing from pyproject.toml in editable mode needs a reasonably new pip. The pip that Python 3.9 puts into a fresh environment is too old, and says so:

Upgrade pip inside the environment, then install. The output is shortened to its first and last lines.

pip installed tabulate as well, because it is listed under dependencies. Now bank can be imported from any folder while this environment is active, and it points at your source files:

To install the developer tools too, name the extra group in square brackets. The quotes are needed in some shells.

Note: you install in editable mode once. Edits to .py files take effect on the next run. Run the install again only after changing pyproject.toml, for example after adding a dependency or a command.

A command-line entry point

The last piece is a command. cli.py reads the words typed after bank with argparse from the standard library and prints a table. The balances are sample data built in memory on every run, so a transfer is not remembered by the next command. Saving data comes in Python Databases.

This file imports tabulate and the package itself, so it is marked as not runnable alone. It was run inside the finished project to produce the output below.

With __main__.py from the python -m chapter in place, the package runs as a module:

The line bank = "bank.cli:main" under [project.scripts] goes one step further. It reads as: create a command named bank that imports bank.cli and calls main. During install, pip writes a small launcher file into .venv/bin, so the command exists whenever the environment is active.

Leave out the command word and argparse prints a usage line and exits with code 2:

Design: main(argv=None) takes the arguments as a parameter and returns the exit code instead of calling sys.exit itself. A test can then call main(["balances"]) directly, without starting a new process.

Tools, mistakes and reference

Group 4 of 4 · what else exists and what goes wrong

Newer tools

venv and pip come with Python and work everywhere, which is why this page uses them. You will meet other tools in real projects. They were not run for this page, so here they are by name only.

ToolIn one line
uvA very fast replacement for pip and venv that can also install Python versions and keep a lock file
PoetryManages dependencies, a lock file, the environment and publishing through its own commands, configured in pyproject.toml
pipxInstalls command-line programs written in Python, each in its own hidden environment, so they are available everywhere

Concept: a lock file records the exact version of every package that was installed, including the packages your packages depend on. It does the job of a frozen requirements.txt, written and updated by the tool.

All of them build on the ideas above: an isolated environment per project, and dependencies written in a file. Learn those with venv and pip first and the other tools will make sense.

Common mistakes

Naming your file like a standard library module

The folder of your script comes first on sys.path. If you save a file as random.py, then import random finds your file, not the standard one. Here the file even imports itself:

The block is marked as not runnable because it only fails when the file is really named random.py. The same happens with json.py, email.py, test.py, calendar.py and any other standard name. The cure is to rename your file and delete the __pycache__ folder next to it.

Installing into the wrong Python

pip says "Successfully installed", and the script still fails with ModuleNotFoundError. Usually pip and python belong to two different installs. Ask both where they live:

Both paths should point into the same .venv. Installing with python -m pip install ... makes a mismatch impossible, because pip then runs under the Python you named.

Forgetting to activate

A new terminal window starts without any environment. If you install a package there, it goes into the global Python, which is the problem this page began with. Check for (.venv) at the start of the prompt before you type pip install. Editors such as VS Code and PyCharm have their own interpreter setting, and it must point at .venv as well.

Quick reference

GoalCommand or file
Create an environmentpython3 -m venv .venv
Activate itsource .venv/bin/activate, on Windows .venv\Scripts\activate
Leave itdeactivate
Install a package, pinnedpip install tabulate==0.9.0
Record what is installedpip freeze > requirements.txt
Install from the recordpip install -r requirements.txt
Describe the projectpyproject.toml
Install your own project for developmentpip install -e . or pip install -e ".[dev]"
Run a module or packagepython -m bank, which needs __main__.py in the package
Create a terminal command[project.scripts] in pyproject.toml
Code that runs only when the file is run directlyif __name__ == "__main__":
See where Python looks for modulespython -c "import sys; print(sys.path)"
See which file an import loadedprint(module.__file__)

The project now has a tests folder with nothing explained yet. Next in the practice path: Python Testing.