Python in practice, part 5 of 9
Python Standard Library Tour
The standard library is the set of modules that is installed together with Python. You only have to import them. This page is a guided tour of the ones worth knowing, each with a small example from a bank and a line that says when to reach for it. Nothing on this page needs pip install, and every example runs on Python 3.9 or later.
Text
Part 1 of 7 · finding, checking and shaping text
Programs spend a lot of time reading text that people wrote: account numbers, bank codes, SMS alerts. String methods such as split() and startswith() handle the simple cases. When the text follows a pattern, the re module is the tool.
re: finding a pattern
A regular expression (regex for short) is a small pattern that describes what a piece of text should look like. re.search(pattern, text) looks for the first place in the text where the pattern fits. It returns a match object, or None when nothing fits.
The simplest pattern is plain text. It matches itself.
Most characters match themselves, but some have a special meaning. A dot means "any one character". To match a real dot, put a backslash before it: \.
Why the r before the quotes
Patterns use many backslashes, and Python strings use backslashes too ("\n" is a new line). The r prefix makes a raw string, in which a backslash is just a backslash. Always write patterns as raw strings.
Reach for it when: you need to check or pull out text by its shape. If in, split() or startswith() can do the job, use them. They are easier to read.
re: character classes and quantifiers
A character class matches one character out of a set. Square brackets list the set: [A-Z] is any capital letter, [0-9] any digit. Common sets have short names: \d is a digit, \w a letter, digit or underscore, \s a space, tab or new line.
A quantifier says how many times the thing before it may repeat: {4} exactly four, {9,18} nine to eighteen, + one or more, * zero or more, ? zero or one.
Put the pieces together to check a whole value. An Indian bank branch code (IFSC) is four capital letters, then a zero, then six letters or digits. re.fullmatch succeeds only if the whole string fits the pattern.
The same idea checks an account number: digits only, between 9 and 18 of them.
Concept: Inside square brackets, ^ at the start means "not". [^0-9] is any character that is not a digit. Outside brackets, ^ means the start of the text and $ the end.
re: groups and named groups
Round brackets make a group: a part of the match that you can ask for separately. Groups are numbered from 1, left to right. group(0), or just group(), is the whole match.
Numbers are easy to mix up. A named group, written (?P<name>...), lets you ask by name, and groupdict() returns them all as a dictionary.
Two more things appear in that pattern. The bar | means "or". And (?:...) is a group that only holds things together and is not captured, so (?:from|to) does not get a number.
Tip: Python joins string literals that sit next to each other, which is how the long pattern above is split over two lines.
re: the functions
The pattern language is the same everywhere. What differs is where the function looks and what it gives back.
| Function | Where it looks | Returns |
|---|---|---|
re.search(p, text) | Anywhere in the text | The first match, or None |
re.match(p, text) | Only at the start | A match, or None |
re.fullmatch(p, text) | The whole text must fit | A match, or None |
re.findall(p, text) | Everywhere | A list of strings (or of tuples, if the pattern has several groups) |
re.finditer(p, text) | Everywhere | Match objects, one at a time |
re.sub(p, new, text) | Everywhere | A new string with each match replaced |
re.split(p, text) | Everywhere | A list of the pieces between matches |
match found nothing for \d+ because the text does not start with a digit. fullmatch found nothing because the text holds more than digits.
findall collects every match. With one group in the pattern it returns just that group, which is ideal for pulling the amounts out of an alert.
sub replaces. The replacement can use \1, \2 to put groups back, or it can be a function that receives each match.
compile
re.compile(pattern) turns a pattern into an object with the same functions as methods. Give it a name once and reuse it. The code reads better, and the pattern is prepared only one time.
re: greedy, lazy and flags
Quantifiers are greedy: they take as much text as they can. That often swallows more than you meant. Adding ? after a quantifier makes it lazy: it takes as little as it can.
The greedy .* ran from the first quote to the very last one. The lazy .*? stopped at the nearest closing quote. The third form says exactly what is wanted, "anything that is not a quote", and is the clearest.
Flags
A flag is an extra argument that changes how the pattern is read.
| Flag | Short | Effect |
|---|---|---|
re.IGNORECASE | re.I | Capital and small letters count as the same |
re.MULTILINE | re.M | ^ and $ match at the start and end of every line, not only of the whole text |
re.DOTALL | re.S | The dot also matches a new line |
re.VERBOSE | re.X | Spaces and # comments inside the pattern are ignored, so it can be laid out and explained |
Combine flags with a bar: re.IGNORECASE | re.MULTILINE.
re: cheat sheet
| Pattern | Matches | Example |
|---|---|---|
abc | Exactly that text | debited |
. | Any one character except a new line | A.c fits A/c |
\d \w \s | A digit, a word character, a space character | \d\d fits 26 |
\D \W \S | The opposite of each | \D fits x |
[abc] [A-Z] | One character from the set or range | [A-Z]{4} fits HDFC |
[^abc] | One character not in the set | [^"]* |
* + ? | Zero or more, one or more, zero or one of the thing before | \d+ fits 4821 |
{4} {9,18} {2,} | Exactly 4, from 9 to 18, 2 or more | \d{9,18} |
*? +? | The lazy forms: as few as possible | ".*?" |
^ $ | Start and end of the text (or of a line with re.M) | ^DEBIT |
\b | A word boundary: the edge between a word and what is next to it | \bRs\b |
a|b | Either one | debited|credited |
(...) | A group that is captured | (\d{4}) |
(?P<name>...) | A captured group with a name | (?P<last4>\d{4}) |
(?:...) | A group that is not captured | (?:from|to) |
\. \( \+ | The real character, with its special meaning switched off | Rs\. |
Watch out: A regex can check the shape of a value, not whether it is real. HDFC0999999 has the right shape for a branch code and may still not exist. Keep patterns short. A pattern nobody can read is a bug waiting to happen.
string and textwrap
The string module holds ready-made sets of characters and a simple template class. Template fills $name placeholders. It is safer than f-strings when the template text comes from a file or a user, because it cannot run code.
textwrap fits long text into a fixed width, which is useful for letters, emails and terminal output.
Reach for it when: you need the alphabet or digits as a string (string), a template that other people may edit (string.Template), or text wrapped to a width (textwrap). textwrap.dedent() removes the common indentation from a triple-quoted string.
Files and system
Part 2 of 7 · paths, folders and the world around the program
These modules let a program find its files, tidy up after itself, and learn how it was started. Reading and writing files with open() is covered in Python Errors and Files.
- pathlib: paths as objects
- os and shutil: rename, copy, move, delete
- tempfile: files that clean themselves up
- sys: arguments, exit codes and the three streams
- Environment variables: settings from outside the code
pathlib
A path is the address of a file or folder. pathlib.Path turns that address into an object. The / operator joins parts, and it uses the right separator on every operating system.
A Path can also do the work: make folders, write and read a whole file in one call, and check what is there.
glob() finds files by a wildcard pattern. * stands for any run of characters, and ** for any depth of folders. The order is not guaranteed, so sort the result.
Reach for it when: you touch a file path at all. Prefer it to joining strings with "/" and to the older os.path functions. Useful extras: Path.cwd() is the current folder, Path.home() the user's home folder, and path.resolve() the full absolute path.
os and shutil
os talks to the operating system: list a folder, rename or delete one file. shutil ("shell utilities") does the bigger jobs: copy files, move them, delete or archive a whole folder tree.
| Job | Call |
|---|---|
| Current folder | os.getcwd() |
| Walk a folder tree | os.walk("bank"), or Path("bank").rglob("*") |
| Copy a file | shutil.copy(src, dst). copy2 also keeps the file times |
| Copy a folder tree | shutil.copytree(src, dst) |
| Make a zip of a folder | shutil.make_archive("backup", "zip", "bank") |
| Find a program | shutil.which("git") gives its path, or None |
| Free disk space | shutil.disk_usage(".") |
Watch out: shutil.rmtree() and os.remove() delete for good. Nothing goes to a recycle bin. Print the path and check it before you run such a line on real data.
tempfile
A temporary file is one you need only for a moment: a download being checked, a report being built, a file for a test. tempfile creates it in a safe place with a name that cannot clash, and deletes it for you when the with block ends.
The folder name is different on every run, so it is not printed here. There is a single-file form too.
Reach for it when: a file is needed for a short time and must not be left behind, above all in tests. Use tempfile.NamedTemporaryFile() when another program has to open the file by its name.
sys
sys holds information about the running Python and the way the program was started.
sys.argvis the list of words typed on the command line.sys.argv[0]is the script name. Runningpython3 pay.py Rishikesh 500gives['pay.py', 'Rishikesh', '500']. Every item is a string.sys.stdoutandsys.stderrare two separate output streams. Results go to the first and problems to the second, so that a user can save the results to a file and still see the errors.sys.executableis the full path of the Python that is running. It is used later on this page to start another Python.
Exit codes
Every program hands a number back when it ends, called the exit code. Zero means success. Anything else means failure. sys.exit(n) ends the program with that code. It works by raising SystemExit, which the block below catches so that it can keep running.
Reading standard input
sys.stdin is the input stream. Another program can be connected to it with a pipe (|), so your script can be one step in a chain. This script is not runnable here because it waits for input.
Reach for it when: you need the raw command line, an exit code, or the input and output streams. For anything more than one or two arguments, use argparse, shown below.
Environment variables
An environment variable is a named setting that lives outside the program and is handed to it when it starts. It is the usual place for things that differ between your laptop and the server: database addresses, modes, secret keys. os.environ is a dictionary of them.
In a terminal you set one for a single run by writing it before the command:
Reach for it when: a value must not be written in the code, either because it is a secret or because it changes from machine to machine. Read all settings in one place at the start of the program, and fail early with a clear message if a required one is missing.
Command-line programs
Part 3 of 7 · scripts that take arguments
A command-line program is one you run from a terminal with words after its name, such as bank deposit Rishikesh 500. argparse reads those words, checks them, converts them and writes the help text for you.
argparse: positional and optional arguments
You describe the arguments, and parse_args() returns an object with one attribute for each. There are two kinds:
- A positional argument is known by its place. It has no dashes and is required.
- An optional argument starts with
--and may be left out. Its value comes after it.
Called with no argument, parse_args() reads the real command line from sys.argv. The examples below pass a list instead, so that each block runs as it stands. The list holds exactly what you would type after the script name.
Two things to notice. Every value arrives as a string, and an optional argument that was not given is None. The next chapter fixes both.
Concept: The attribute name comes from the argument name with the dashes removed. --rate becomes args.rate, and --min-balance becomes args.min_balance.
argparse: types, choices, defaults and flags
Extra settings on add_argument() do the checking and converting that you would otherwise write by hand.
| Setting | Effect |
|---|---|
type=float | Convert the text with this function. Bad text becomes a clear error |
default=4.0 | The value when the argument is not given |
choices=[...] | Only these values are allowed |
action="store_true" | A flag: it takes no value. True if present, False if not |
"-v", "--verbose" | A short and a long name for the same option |
required=True | Make an option compulsory |
nargs="+" | Accept one or more values and collect them in a list |
help="..." | The text shown for this argument by --help |
nargs="+" collects several values, for example a list of account holders.
argparse: help and errors for free
Here is the same parser as a real script that reads the command line. It is not runnable in the page because it needs arguments typed after its name.
Nobody wrote a help screen, yet -h and --help work. This is the real output of python3 interest.py --help:
Square brackets in the usage line mark what may be left out. Normal use, and then three mistakes:
On an error, argparse prints the message to the error stream and ends the program with exit code 2. Your own code never sees the bad value.
Version note: This output is from Python 3.9. From Python 3.10 the heading optional arguments: is printed as options:. Everything else is the same.
argparse: subcommands
Larger tools group their work into subcommands, the way git has git commit and git push. Each subcommand is a small parser of its own with its own arguments. Here is a bank with deposit and withdraw. It is not runnable in the page because it reads the command line.
dest="command" stores the chosen subcommand in args.command. prog="bank" sets the name shown in the help. The real output:
Options of the main parser, such as -v, go before the subcommand. Options of a subcommand go after it. Because main() accepts a list, a test can call main(["deposit", "Rishikesh", "500"]) without a terminal. Python Project Setup shows how to install a script so that it runs as plain bank.
getpass
Never take a password as a command-line argument. It would be saved in the shell history and shown in the list of running programs. getpass.getpass() asks for it and hides what is typed. Not runnable here because it waits for the keyboard.
Reach for it when: a script takes more than one or two arguments, or other people will run it. The help text alone is worth it.
Running other programs
Part 4 of 7 · starting a command and reading its result
Sometimes the job is already done by another program: git, a PDF tool, a backup command. The subprocess module starts it, waits for it and gives you what it printed.
- subprocess.run: run a command and capture its output
- Failures and timeouts:
checkandtimeout - Pass a list, avoid shell=True
subprocess.run
subprocess.run() takes the command as a list: the program first, then each argument as its own item. It waits until the program ends and returns a result object.
The examples run a second Python, because that exists on every machine that can run this page. sys.executable is the path of the Python that is running now, and -c runs the code given in the next item.
| Argument | Meaning |
|---|---|
capture_output=True | Collect the output in result.stdout and result.stderr. Without it, the output goes straight to the screen |
text=True | Decode the output to strings. Without it you get bytes |
check=True | Raise an exception if the exit code is not zero |
timeout=10 | Stop waiting, and stop the program, after this many seconds |
input="text" | Send this text to the program's standard input |
cwd="folder" | Run the program in that folder |
env={...} | Give the program these environment variables |
input feeds the other program. Here the second Python adds up the lines it receives, like total.py in the sys chapter.
Reach for it when: a command-line tool already does what you need. If a Python module can do the same job, prefer the module: shutil.copy() instead of running cp.
Failures and timeouts
By default run() does not complain when the other program fails. It only reports the exit code, and it is easy to forget to look. check=True turns a non-zero exit code into a CalledProcessError, so a failure cannot pass unnoticed.
A program that never ends would make yours wait for ever. timeout sets a limit in seconds. When it is passed, the other program is stopped and TimeoutExpired is raised.
If the program itself cannot be found, Python raises FileNotFoundError before anything runs.
Habit: For commands whose success matters, always write check=True and a timeout.
Pass a list, avoid shell=True
There is a second way to give the command: one string with shell=True. The string is handed to the system shell, the program that reads the commands you type in a terminal. The shell gives special meaning to characters such as ;, |, & and $. If any part of that string comes from a user, they can add a command of their own. This is called shell injection.
Below, a "name" typed by a user contains ;, which means "and then run this" to the shell. The example uses echo, which only prints its arguments, so nothing harmful happens. It needs macOS or Linux.
With the string, the shell ran two commands, and the second one was written by the user. With the list there is no shell at all. Each item goes to the program exactly as it is, so the whole name is just text.
- Pass a list. It is safe, and names with spaces need no quoting.
- If you have a command as one string,
shlex.split("git log -n 5")turns it into a list. - If you really need a shell feature such as a pipe, build the string only from values you wrote yourself, or wrap each outside value in
shlex.quote().
Data formats
Part 5 of 7 · saving, sending, hiding and counting data
Data has to leave the program sooner or later: into a file, to another system, into a password table. And money has to be counted exactly. These modules cover the common formats and the common traps.
- json, csv and configparser: text formats
- pickle: Python objects as bytes, with a warning
- base64 and hashlib: encoding and fingerprints
- Storing passwords: never as plain text
- secrets and uuid: tokens and identifiers
- decimal and rounding: exact money
- fractions and statistics
- random: chance you can repeat
json
JSON is a text format for data that almost every language and web service understands. json.dumps() turns Python data into a JSON string, and json.loads() turns it back. The versions without the s, dump() and load(), work with an open file.
Python's True and None became JSON's true and null. JSON knows only strings, numbers, true, false, null, lists and objects. Other types need help: default=str tells dumps to turn anything it does not know into a string.
Reach for it when: data goes to or comes from a web API, a config file or another language. Two details: dictionary keys always become strings, and tuples come back as lists.
csv
CSV (comma-separated values) is the format spreadsheets import and export: one row per line, columns separated by commas. Do not split the lines yourself. A value may contain a comma inside quotes, and the csv module handles that.
The writer added quotes around the names because they contain a comma, and the reader took them off again. Every value read from a CSV file is a string, so the balance had to go through int().
Reach for it when: data moves to or from a spreadsheet or a bank export. Always open the file with newline="", as the module asks, or blank lines can appear on Windows.
configparser
Settings files often use the INI format: section names in square brackets and key = value lines below them. configparser reads and writes it. Values are strings, and helper methods convert them.
Reach for it when: people need to edit simple settings by hand. Newer projects often use TOML, the format of pyproject.toml. The module that reads it, tomllib, is in the standard library only from Python 3.11, so it is not shown here.
pickle
pickle turns almost any Python object, including objects of your own classes, into bytes and back again. JSON cannot do that. The price is that only Python can read the result.
Security warning: Loading a pickle can run any code that the file's author put into it. Never call pickle.load() or pickle.loads() on data from the internet, from a user, or from anywhere you do not fully control. For data that crosses a boundary, use JSON.
Reach for it when: your own program saves its own Python objects for a short time, for example a cache on the same machine. Not for files you send to others, and not for long-term storage.
base64 and hashlib
base64 rewrites any bytes using only letters, digits and a few signs, so that they can travel through places that accept only text, such as JSON or email. It is an encoding, not a lock. Anyone can turn it back.
A hash is a short fixed-length fingerprint of some data. The same input always gives the same hash, a tiny change gives a completely different one, and there is no way back from the hash to the data. hashlib provides the common hash functions.
One common use is a checksum: hash a file before sending it and again after receiving it. If the two values match, the file arrived unchanged.
Reach for it when: bytes must pass through a text-only channel (base64), or you need to know whether data has changed (hashlib). Use sha256. The older md5 and sha1 are not safe for security work.
Storing passwords
Never store a password as plain text. If the table is ever copied, every account is open. Store a hash instead, and when someone logs in, hash what they typed and compare.
A plain sha256 is not enough, for two reasons. It is very fast, so an attacker can try billions of guesses. And two people with the same password get the same hash. The fix has two parts:
- A salt: random bytes, different for every user, mixed into the hash. It is stored next to the hash and is not secret.
- A slow hash made for passwords.
hashlib.pbkdf2_hmac()repeats the hashing many thousands of times.
The salt and the hash are different on every run, so only their lengths are printed. The last line shows the salt at work: the same password hashed twice gives two different results. hmac.compare_digest() compares in a way that does not leak, through timing, how many bytes were right.
In real projects: Use a library made for this job, such as argon2-cffi or bcrypt (both need pip install), or the login system of your web framework. The example shows the idea with the standard library alone.
secrets and uuid
secrets makes random values that are safe for security: reset links, API keys, one-time codes. The values are different on every run, so the examples print their length or type instead.
A UUID (universally unique identifier) is a 128-bit number written as 36 characters. uuid.uuid4() makes a random one. The chance of two being equal is so small that separate machines can create ids without asking each other.
Reach for it when: you need a token nobody can guess (secrets) or an id that is unique without a central counter (uuid). Do not use the random module for either. Its numbers can be predicted.
decimal for money
Python's float stores numbers in binary. Most decimal fractions, such as 0.1, have no exact binary form, just as one third has no exact decimal form. The stored value is a tiny bit off, and the errors show up in sums.
For a bank that is not acceptable. decimal.Decimal stores numbers in base ten, exactly as written, the way people do sums on paper.
Always build a Decimal from a string or an int. Building it from a float copies the float's error.
Reach for it when: the numbers are money, or any quantity where every digit must be right. Keep float for measurements and science, where a tiny error does not matter and speed does.
decimal: quantize and rounding
Interest and tax produce more digits than a currency has. quantize() rounds a Decimal to the same number of places as the pattern you give it: Decimal("0.01") means two places. You also choose how to round.
The float version rounds 2.675 down, because the float is really 2.67499999... The decimal version rounds it the way you learned at school.
The rounding mode decides what happens to a value exactly half-way between two results.
| Mode | Rule | Typical use |
|---|---|---|
ROUND_HALF_UP | A half goes away from zero | What most people and most invoices expect |
ROUND_HALF_EVEN | A half goes to the nearest even digit | "Banker's rounding". The default. Over many values, the ups and downs cancel out |
ROUND_DOWN | Cut the extra digits off | Never pay out more than was earned |
ROUND_UP | Any extra digit pushes away from zero | Fees that are always rounded up |
A classic problem: split 100 rupees between three people without losing a paisa. Round each share down, then give the remainder to one of them.
Tip: Another common approach is to store money as a whole number of the smallest unit, paise or cents, in an int. Integers are always exact.
fractions and statistics
fractions.Fraction keeps a number as an exact numerator and denominator. Shares of ownership are a natural fit.
statistics has the everyday summary numbers, so you do not have to write them yourself.
The mean is the average. The median is the middle value when the list is sorted, and one very large balance does not pull it up. stdev (standard deviation) says how spread out the values are. The mode is the most common value.
Reach for it when: you need exact ratios (fractions) or a quick summary of a list of numbers (statistics). For large tables of data, people use the third-party package pandas.
random
random makes random choices for games, test data and simulations. The numbers come from a formula, so they only look random. Giving the formula a starting value, called a seed, makes it produce the same sequence every time. That is how a test or a demo can be repeated exactly.
For test data it is tidier to keep a generator of your own, so that seeding it does not affect other code.
Reach for it when: you need sample data, a shuffle or a simulation. Never for passwords, tokens or one-time codes. Use secrets for those.
Time
Part 6 of 7 · dates, time zones and stopwatches
Dates look simple and are full of traps: months of different lengths, leap years, time zones, clocks that change in summer. Let the library do the arithmetic.
- datetime: a recap in one example
- zoneinfo: real time zones
- time.perf_counter: measuring how long code takes
- calendar: months and weekdays
datetime
datetime has a type for a day (date), for a moment (datetime) and for a length of time (timedelta). They are introduced in Python Errors and Files. Here is the whole set in one example: a fixed deposit opened on one day that matures 90 days later.
strftime turns a date into text ("f" for format) and strptime reads text into a date ("p" for parse). date.today() and datetime.now() give the current day and moment. They are not used here because their output changes.
Reach for it when: anything involves a date. Never do date arithmetic on strings or by counting days yourself.
zoneinfo
A datetime without a time zone is called naive: "10:30" with no idea where. Attach a zone and it becomes aware, and can be converted to any other zone. zoneinfo, new in Python 3.9, knows the world's time zones by name, including their summer-time rules.
All four printed values of paid are the same moment, written for different places. In July, London is on summer time, so the gap to India is one hour smaller. The library knows that. You do not have to.
Watch out: zoneinfo reads the time zone database of the operating system. macOS and Linux have one. Windows does not, so there you need pip install tzdata, the one exception to "no installs" on this page. A good rule for programs: store and compare times in UTC, and convert to a local zone only when showing them to a person.
time.perf_counter
To measure how long a piece of code takes, read a clock before and after. time.perf_counter() is the clock made for this: very fine-grained, and it never jumps backwards. Its value means nothing alone. Only the difference between two readings matters.
The time taken differs from run to run, so the line that prints it is left as a comment.
| Function | Use |
|---|---|
time.perf_counter() | Timing code |
time.time() | Seconds since 1 January 1970, as a float. A timestamp to store or send |
time.sleep(0.5) | Pause the program for half a second |
time.monotonic() | A clock that never goes backwards, for timeouts |
Reach for it when: you want a quick "how long did that take". Careful measuring, with the timeit module, is in Python Performance and Memory.
calendar
calendar answers questions about months and weekdays, and can print a month as text.
Reach for it when: you need the last day of a month (for interest that is paid monthly), the weekday of a date, or a month laid out as text.
Handy helpers
Part 7 of 7 · small modules that save a lot of code
The last group is a mix. Some of these modules have a full chapter on another page, so they get only a pointer here. The rest are shown briefly.
- Covered on other pages: collections, itertools, functools and more
- enum: a fixed set of named values
- copy: shallow and deep copies
- pprint and operator
- zipfile and gzip: compressed files
- Quick reference: "I need to ..."
Covered on other pages
These modules are just as much part of the standard library. Each has a proper treatment elsewhere on this site.
| Module | In one line | Read more |
|---|---|---|
collections | Counter counts things, defaultdict fills in missing keys, deque is a fast queue, namedtuple is a tuple with field names | Python Power Features |
itertools | Building blocks for loops: chain, islice, groupby, combinations, accumulate | Python Iterators and Generators |
functools | lru_cache remembers results, partial fixes some arguments, wraps keeps a decorated function's name | Python Power Features and Python Decorators |
dataclasses | @dataclass writes __init__, __repr__ and __eq__ for a class that mostly holds data | Python Classes |
contextlib | @contextmanager turns a small function into something usable with with | Python Power Features |
heapq, bisect | Always get the smallest item quickly, and search or insert in a sorted list | Python Data Structures and Algorithms |
sqlite3, shelve | A real SQL database in one file, and a dictionary that is saved to disk | Python Databases |
urllib, http.server | Fetch a web address, and serve a folder over HTTP for local testing | Python Web APIs |
logging, pdb, traceback, warnings | Recording what a program does and finding out why it went wrong | Python Debugging and Logging |
unittest | The built-in test framework | Python Testing |
threading, multiprocessing, asyncio | Doing several things at once | Python Threads and Processes and Python Async IO |
A one-block taste of the first three, so the names mean something:
enum
An enumeration is a fixed set of named values. Without it, code passes strings such as "savings" around, and a typing mistake like "saving" goes unnoticed. With an Enum, a wrong name is an error at once.
AccountType("fixed") looks a member up by its value, which is how text from a file or a form becomes an enum member.
Reach for it when: a value can only be one of a short, known list: account types, transaction states, days of the week. Compare members with is.
copy
Assigning with = does not copy anything. It gives the same object a second name. copy.copy() makes a shallow copy: a new outer object that still shares the inner ones. copy.deepcopy() copies everything, all the way down.
The shallow copy kept its own owner, but its history is the very same list as the original's, so the new entry shows up in both.
Reach for it when: you need to change a nested structure and keep the original as it was, for example to try a transfer and throw the result away if it fails. Use deepcopy for anything nested.
pprint and operator
pprint ("pretty print") lays out nested data over several lines so that it can be read. It is a good partner for debugging.
operator offers small ready-made functions for use as the key of sorted(), max() and min(). itemgetter("balance") does the same as lambda a: a["balance"], and attrgetter does it for attributes of objects.
Reach for it when: a printed dictionary runs off the screen (pprint), or you sort by a field and want something shorter than a lambda (operator).
zipfile and gzip
zipfile reads and writes .zip archives, which pack many files into one. gzip compresses a single file, and is common for log files and data downloads.
gzip.open() works like open(). Use mode "wt" and "rt" for text.
33,000 characters of repeated text shrank to a few hundred bytes. Real data compresses less than this, but log files often shrink to a tenth of their size.
Reach for it when: you send or store many files as one (zipfile), or want a big text file to take less room (gzip). shutil.make_archive() zips a whole folder in one call.
Quick reference
| I need to ... | Module |
|---|---|
| Check or pull out text by its pattern | re |
| Wrap long text to a width | textwrap |
| Fill a simple template from a file | string.Template |
| Build, inspect or search file paths | pathlib |
| Copy, move or delete files and folders | shutil, os |
| Use a file for a moment and have it cleaned up | tempfile |
| End the program with an exit code, or read piped input | sys |
| Read a setting from outside the code | os.environ |
| Take arguments on the command line | argparse |
| Ask for a password without showing it | getpass |
| Run another program | subprocess |
| Exchange data with a web service or another language | json |
| Read or write spreadsheet data | csv |
| Read a settings file | configparser (or tomllib from Python 3.11) |
| Save my own Python objects for my own program | pickle, never on untrusted data |
| Send bytes through a text-only channel | base64 |
| Fingerprint data or check a download | hashlib |
| Store a password | hashlib.pbkdf2_hmac with a salt, or a dedicated library |
| Make a token, key or one-time code | secrets |
| Make a unique id | uuid |
| Do sums with money | decimal |
| Keep exact ratios | fractions |
| Get the mean, median or spread of some numbers | statistics |
| Make sample data or shuffle a list | random |
| Work with dates and lengths of time | datetime |
| Convert between time zones | zoneinfo |
| Time a piece of code | time.perf_counter |
| Find the last day of a month or print a month | calendar |
| Name a fixed set of choices | enum |
| Copy a nested structure | copy.deepcopy |
| Print nested data readably | pprint |
| Sort by a field without a lambda | operator.itemgetter |
| Count things, or use a queue | collections |
| Combine or slice loops | itertools |
| Cache the results of a function | functools.lru_cache |
| Pack or compress files | zipfile, gzip |
| Keep data in a small database | sqlite3 |
Before you write a helper of your own or install a package, search the library index in the official Python documentation. There is a good chance the module already exists.
Next in the practice path: Python Data Structures and Algorithms.