Learnings

QA and test automation

Playwright: UI and API Test Automation

Playwright drives a real browser and calls HTTP APIs from code, so a program can check your application the way a tester would. This page starts from zero: what UI and API automation mean, how to install Playwright for Python, and a complete set of tests for a tiny bank app. Every test on the page was run with Playwright 1.60 and pytest.

What it is and why

Part 1 of 5 · the ideas, before any code

Manual testing vs automation

What it is and why

A manual test is a person following steps: open the site, type a username, click "Sign in", check that the dashboard appears. An automated test is a small program that performs the same steps and checks the same result, in seconds, as often as you like.

ManualAutomated
First runFast: just do itSlower: someone has to write the test
Every run after thatSame effort againFree, and identical each time
Best forExploring, new features, "does this feel right?"Checks repeated on every release: login, payments, critical flows
FindsSurprises nobody thought ofThings that used to work and broke (regressions)

Regression: a feature that worked yesterday and is broken today because of an unrelated change. Catching regressions is the main job of an automated test suite.

UI testing vs API testing

What it is and why

Most applications have two doors. The UI (user interface) is the web page a person sees and clicks. The API is the set of URLs the page, the mobile app and other programs call behind the scenes to read and change data, usually exchanging JSON.

UI automationAPI automation
What the test doesOpens a browser, clicks and types like a userSends HTTP requests, reads the replies
What it checksWhat the user sees: text, buttons, pages, errorsStatus codes, JSON data, rules, permissions
Example"After signing in, the heading says Welcome""GET /api/accounts/2 returns balance 12000"
SpeedAbout a second per testA few milliseconds per test
Breaks whenThe page layout or wording changesThe data contract changes
How many to writeFew: the journeys that matter mostMany: every rule and every error case

Rule of thumb: test the rules through the API, because it is fast and stable. Test through the UI only what a user must be able to do end to end. This shape, many fast tests below and few slow ones on top, is called the test pyramid.

What Playwright is

What it is and why

Playwright is a free, open-source automation library from Microsoft. One library does both jobs on this page:

  • UI automation: it starts Chromium (the engine of Chrome and Edge), Firefox or WebKit (the engine of Safari) and controls it.
  • API automation: it sends HTTP requests without any browser.

Three features make it friendly for beginners:

FeatureWhat it saves you from
Auto-waitingWriting sleep(3) and hoping the page has loaded. Playwright waits until the button is there and clickable.
Locators by role and labelFragile addresses such as div > form > input:nth-child(2). You say "the field labelled Username".
Codegen and trace viewerGuessing. One tool records your clicks as code, the other replays a failed test step by step.

Playwright exists for JavaScript/TypeScript, Python, Java and .NET. This page uses Python with pytest. The ideas and method names are almost the same in the other languages.

How it works

What it is and why

A UI test goes through a browser. An API test skips the browser and talks to the application directly, which is why it is so much faster.

Words you will meet in every Playwright test:

WordMeaning
BrowserOne running Chromium, Firefox or WebKit
ContextA fresh, private browser profile: its own cookies and logins. Each test gets a new one, so tests cannot disturb each other.
PageOne tab. Almost everything you do is a method on page.
LocatorA description of how to find an element, such as "the button named Sign in"
HeadlessThe browser runs without a visible window. The default, and what servers use. Headed means you can watch.
AssertionA check that must be true or the test fails: expect(...) for pages, assert for data
FixtureA pytest helper that prepares something for a test, for example a ready page

Playwright, Selenium and Cypress

What it is and why

You will hear these three names together. A fair short comparison:

PlaywrightSeleniumCypress
LanguagesJS/TS, Python, Java, .NETMost languagesJS/TS only
BrowsersChromium, Firefox, WebKitAll major browsersChromium family, Firefox, WebKit (experimental)
WaitingAutomaticMostly written by youAutomatic
API testing built inYesNo, add another libraryYes
Known forSpeed, reliability, toolingThe oldest and most widely usedA pleasant interactive runner

For API tests only, tools such as Postman, or pytest with the requests library, do the job too. The reason to use Playwright for both is one tool, one report, and tests that mix API and UI steps, as shown later.

Setup

Part 2 of 5 · install, get an app to test, run a first script

Install

Setup

You need Python 3.9 or newer. Make a project folder with its own virtual environment, then install two things: the Python package and the browsers it controls.

Two steps, both needed: pip install pytest-playwright is the only Python package to install. It brings in playwright (the library), pytest (the test runner) and pytest-base-url as dependencies. But pip does not download a browser. That is what playwright install chromium does. Skip it and the API tests still run, while every UI test fails with a message telling you to run playwright install.

  • playwright install with no name downloads all three browsers. Start with Chromium.
  • Playwright uses its own browser copies, kept apart from the Chrome on your machine, so the browser version always matches the library.
  • On Linux servers, playwright install --with-deps chromium also installs the system libraries the browser needs.

The app under test

Setup

To learn testing you need something to test that will not change under you. Save the file below as bank_app.py. It is a tiny bank written with Python's standard library only, so there is nothing more to install.

DoorAddressWhat it does
UI/Sign-in page. The only user is rishikesh with password secret123.
/dashboardShows the balance and a Deposit button. Sends you back to / if you are not signed in.
APIPOST /api/loginTakes a username and password, returns a token
GET /api/accountsLists the accounts. Needs the token.
GET /api/accounts/2One account. Needs the token.
POST /api/accounts/2/depositAdds money. Needs the token.
Show bank_app.py (about 110 lines; you do not need to understand it to test it)

Start it in one terminal and leave it running. Open the address in your own browser and sign in once by hand, so you know what the tests will be doing.

A toy, on purpose: one user, a fixed token, data kept in memory. Never build real login this way. It exists so the tests have something predictable to check.

First UI script

Setup

In a second terminal (with the virtual environment active), save and run this. It opens the sign-in page, fills the form, clicks the button and takes a screenshot of what it finds.

  • No window appeared: the browser ran headless. Change the launch line to p.chromium.launch(headless=False, slow_mo=500) to watch it, half a second per step.
  • dashboard.png is now in the folder. Open it: that is what the browser saw.
  • There is no sleep anywhere. click() waited for the button, and reading the heading waited for the new page.

First API script

Setup

Now the other door. This script never starts a browser. It sends three HTTP requests and prints each status code and JSON reply.

Status codeMeaningSeen here when
200OKThe request worked
400Bad requestThe deposit amount is not valid
401Not authenticatedThe token is missing or wrong
404Not foundThere is no account with that id
500Server errorThe application crashed: always a bug

These two scripts only print. A test must decide pass or fail by itself. That is what pytest adds next. You can stop bank_app.py now: from here on the tests start it themselves.

UI automation

Part 3 of 5 · tests that use the page like a person

Shared setup: conftest.py

UI automation

pytest reads a file named conftest.py before the tests and makes its fixtures available to all of them. A test asks for a fixture by naming it as a parameter. The pytest-playwright package already provides page (a fresh tab in a fresh context for every test) and playwright. This file adds three more.

FixtureGives the testUsed by
page (built in)A new browser tabUI tests
base_urlThe address of the running bank, so tests can write page.goto("/") and not a full URLBoth
apiAn HTTP client pointed at the bank, with balances resetAPI tests
tokenA login token, fetched through the APIAPI tests that need to be signed in

In a real project the application is already running somewhere, so you delete the server lines and pass the address when you run: pytest --base-url https://staging.example.com.

Testing the sign-in page

UI automation

A test is a function whose name starts with test_, in a file whose name starts with test_. Each one follows the same three steps: arrange (get to the starting point), act (do the thing), assert (check the result).

  • One happy path, the rest are problems. Wrong password and "not signed in" are the cases a hurried manual tester skips.
  • Each test starts clean. The last test gets a new context with no cookies, which is why it is not signed in even though the test before it was.
  • sign_in is an ordinary helper function. Repeated steps belong in one place.

Locators: finding things

UI automation

A locator says how to find an element. Prefer the ones that describe the page the way a user would. They keep working when a developer restyles the page.

LocatorFindsUse it
page.get_by_role("button", name="Sign in")A button, link, heading, checkbox... by what it is and its visible nameFirst choice
page.get_by_label("Username")The form field with that labelForm fields
page.get_by_placeholder("Search")A field by its grey hint textFields with no label
page.get_by_text("Welcome")Any element showing that textMessages, paragraphs
page.get_by_test_id("balance")An element with data-testid="balance"When nothing visible identifies it. Ask developers to add the attribute.
page.locator("#amount")Anything by CSS selector or XPathLast resort: it breaks when the HTML structure changes

Narrow a search by chaining, and pick from several matches explicitly:

Strict by default: if a locator matches two elements and you try to click it, Playwright stops with an error that lists both. That is a feature: it never silently clicks the wrong one.

Inspecting elements with DevTools

UI automation

Before you can locate an element you have to look at it. Every browser has built-in developer tools (DevTools) that show the HTML behind what is on screen. The steps below are for Chrome and Edge; Firefox is almost identical.

  1. Open the inspector on an element: right-click it on the page and choose Inspect. Or press F12 (Cmd+Option+I on a Mac), click the arrow icon at the top left of DevTools, then click the element.
  2. Read the highlighted line in the Elements panel. That is the element's HTML: its tag (input, button), attributes (id, name, data-testid) and text. Hovering over a line highlights the matching area on the page.
  3. Copy an address for it: right-click the highlighted line, open Copy, and pick one of the options in the table.
Copy optionGives, for the Amount field of the bank dashboardUse in a test?
Copy selector#amountFine when it is a short id like this. Long chains such as div > div:nth-child(3) are fragile.
Copy XPath//*[@id="amount"]Same: fine when it is based on an id
Copy full XPath/html/body/label/inputAvoid. It is the path from the top of the page and breaks when anything above the element moves.
Copy JS pathdocument.querySelector("#amount")For the Console, not for tests
Copy elementThe HTML itselfHandy for a bug report

Check a selector before you put it in a test. Two ways, both inside DevTools:

  • In the Elements panel press Ctrl+F (Cmd+F) and type a CSS selector or an XPath. It shows "1 of 1" when exactly one element matches, which is what you want.
  • In the Console panel, $$("css selector") and $x("xpath") return the list of matches.

Find the role and name for get_by_role: select the element, then open the Accessibility tab beside Styles. It shows, for example, Role: button, Name: "Deposit", which is exactly what page.get_by_role("button", name="Deposit") looks for.

Easier still: Playwright has its own picker. Run playwright codegen <url>, press "Pick locator" and click any element: it prints the best locator, usually a role or label one, ready to paste. Use DevTools to understand the page and the picker to write the locator.

CSS selectors and XPath in Playwright

UI automation

Older pages often have no labels or test ids, and many teams already hold a list of XPaths from Selenium days. page.locator() accepts both CSS and XPath. This test signs in to the bank using only those, and passes:

I wantCSSXPath
By id#amount//*[@id='amount']
By attributeinput[name='username']//input[@name='username']
By class.error//*[contains(@class, 'error')]
By exact textNot possible in plain CSS//button[text()='Sign in']
By part of the textNot possible in plain CSS//button[contains(text(), 'Sign')]
Inside somethingform input//form//input
Parent of somethingNot possible in plain CSS//input[@name='username']/..
The second matchli:nth-child(2)(//li)[2]

Trap with a full XPath: Playwright treats a string starting with // or .. as XPath. A copied full XPath starts with a single /, is read as CSS and fails with Unexpected token "/" while parsing css selector. Put xpath= in front, as in the test above.

  • Good XPath is short and names a meaning: an id, a name, a data-testid, a visible text. Bad XPath counts positions: /div[2]/div[1]/span[3].
  • Order of preference: role or label locator, then test id, then a short CSS or XPath, and a copied full XPath never.
  • Everything else still applies to these locators: auto-waiting, expect, strictness.

Actions: doing things

UI automation

ActionWhat a user would do
page.goto("/dashboard")Type an address and press Enter
locator.click()Click. Also dblclick() and hover().
locator.fill("text")Clear a field and type into it
locator.press("Enter")Press a key
locator.check() / uncheck()Tick or untick a checkbox or radio button
locator.select_option("savings")Choose from a dropdown
locator.set_input_files("statement.pdf")Pick a file to upload
page.go_back(), page.reload()Browser buttons
page.screenshot(path="x.png")Take a picture of the page

Before each action Playwright checks that the element exists, is visible, is not moving, is enabled and is not covered by something else. It retries for up to 30 seconds, then fails with a message saying which check did not pass.

expect: checking things

UI automation

expect() is Playwright's assertion for pages. It keeps trying for up to 5 seconds until the condition is true, so it copes with pages that update a moment after a click.

AssertionPasses when
expect(page).to_have_title("Bank: Sign in")The tab title matches
expect(page).to_have_url("/dashboard")The address matches
expect(locator).to_be_visible()The element is on screen. Also to_be_hidden().
expect(locator).to_have_text("5250")Its text is exactly this. to_contain_text() for part of it.
expect(locator).to_have_value("250")A form field holds this value
expect(locator).to_be_enabled()A button can be clicked. Also to_be_disabled(), to_be_checked().
expect(locator).to_have_count(3)The locator matches exactly 3 elements
expect(locator).not_to_be_visible()Any assertion can be reversed with not_

The beginner mistake: assert locator.inner_text() == "5250" reads the text once, right now, and fails if the page has not updated yet. expect(locator).to_have_text("5250") waits. Use expect for anything on a page, and plain assert only for data you already hold, such as an API reply.

A test that waits for the page

UI automation

On the dashboard, clicking Deposit does not load a new page. JavaScript calls the API in the background and changes the balance on screen when the reply arrives. This is how most modern sites behave, and it is where tests written with fixed sleeps go wrong.

The last two checks run right after the click, before the reply has arrived. They pass because expect keeps looking until the number changes.

Never write time.sleep(2) in a test. Too short and the test fails on a slow day. Too long and a thousand tests waste half an hour. Wait for the thing you care about, which is what expect does.

Running tests

UI automation

CommandEffect
pytestRun everything, headless
pytest tests/test_ui_login.pyOne file
pytest -k wrong_passwordOnly tests whose name contains that text
pytest -vOne line per test with its name
pytest -xStop at the first failure
pytest --headed --slowmo 500Show the browser, half a second per step
pytest --browser firefox --browser webkitRun on other browsers (install them first)
pytest --device "iPhone 13"Emulate a phone: screen size, touch, user agent
pytest --screenshot only-on-failureSave a picture when a test fails
pytest --video retain-on-failureKeep a video of failed tests
pytest --tracing retain-on-failureKeep a full trace of failed tests
pytest --base-url https://staging.example.comTest another environment

The whole suite built on this page, UI and API together:

Reading a failure

UI automation

A failing test is the product doing its job. Here the deposit test was changed to expect 5300 when the right answer is 5250.

Read it from the top: the line that failed, what was expected, what was actually there, and a log showing Playwright saw 5000, then 5250, and kept checking for 5 seconds. The two options on the command line left evidence in test-results/:

The trace viewer opens a window with a timeline of every step. Click a step to see the page before and after it, the network requests it caused and the browser console. It is the fastest way to understand a test that fails on a server and passes on your machine.

Codegen: record a test

UI automation

With the bank running, this opens a browser and a second window. Click through the site by hand and the second window writes the Python for each step, choosing good locators for you.

  • Use it to find locators and to get a first draft. The "pick locator" button shows the best locator for any element you point at.
  • Then clean up: recorded code has no assertions unless you add them with the toolbar, repeats itself, and has no helper functions.

To pause a test and inspect the live page, put page.pause() in it and run with --headed, or run PWDEBUG=1 pytest to step through from the start.

API automation

Part 4 of 5 · tests that talk to the server directly

Requests and replies

API automation

An API call is a request (a method, a URL, maybe headers and a body) and a response (a status code, headers and a body, usually JSON). The api fixture from conftest.py is Playwright's request client.

MethodMeansWith the api fixture
GETReadapi.get("/api/accounts", params={"kind": "savings"})
POSTCreate, or perform an actionapi.post("/api/login", data={"username": "..."})
PUT / PATCHReplace / change partapi.put(...), api.patch(...)
DELETERemoveapi.delete("/api/accounts/2")
On the replyGives you
reply.statusThe status code, for example 200
reply.okTrue for any 2xx status
reply.json()The body as a Python dict or list
reply.text()The body as a string
reply.headersThe response headers as a dict

Passing a dict as data= sends it as JSON. Headers, such as the login token, go in headers=.

Testing the accounts API

API automation

Same arrange, act, assert shape as the UI tests, with plain assert because a reply is data you already hold: there is nothing to wait for.

  • @pytest.mark.parametrize runs one test once per value. Four bad amounts make four tests, which is why seven functions count as ten.
  • Check the change, not just the reply. The deposit test reads the account again to prove the balance was really stored.
  • Tests do not depend on each other. The api fixture resets the balances first, so the tests pass in any order.
  • Ten API tests in under a second, including starting the app. The four sign-in UI tests alone took longer. That is the test pyramid in numbers.

What to check in an API test

API automation

CheckQuestion it answersExample on this page
Status codeDid the server say the right thing?assert reply.status == 200
Body valuesIs the data correct?reply.json()["balance"] == 12500
Body shapeAre all the fields there, with the right types?The list of owners
Side effectDid it really change what it claims?Read the account again after the deposit
Bad inputDoes it refuse nonsense politely (4xx, not 500)?Amounts 0, -50, "ten", 2.5
AuthenticationIs it locked without a login?No token gives 401
Missing thingsWhat about an id that does not exist?Account 99 gives 404

Negative tests are the ones that send wrong input on purpose. For an API, write more of them than positive tests: there is one way to be right and many ways to be wrong.

Putting it together

Part 5 of 5 · mixing both, keeping tests tidy, running them automatically

API for setup, UI for the check

Putting it together

The strongest reason to have both in one tool. To test that the dashboard shows a deposit, you do not need to click the deposit through the UI as well. Create the situation through the API in milliseconds, then use the browser only for the part you are testing.

The first test is the pattern to copy: set up by API, verify by UI. The second is explained next.

Faking a server reply

Putting it together

How do you test what the page shows when the server fails, without breaking the server? page.route() intercepts requests the browser makes and lets the test answer them itself. In test_mocked_api_failure_shows_error above, the deposit request never reaches the bank: Playwright replies with a 500 error, and the test checks that the page shows the message and leaves the balance alone.

Inside the handlerEffect
route.fulfill(status=500, json={...})Answer with a made-up reply
route.abort()Pretend the network is down. Also used to block ads or images.
route.continue_()Let the request through unchanged

Use it sparingly. A test with a faked reply proves the page handles that reply. It proves nothing about the real server. Keep fakes for cases that are hard to produce for real: errors, slow replies, empty lists.

Sign in once, reuse everywhere

Putting it together

If 200 UI tests each fill the sign-in form, you spend minutes testing login 200 times. Sign in once, save the browser's cookies to a file (the storage state), and start every other test already signed in.

  • scope="session" means the fixture runs once for the whole run, not once per test.
  • browser_context_args is a fixture from pytest-playwright. Overriding it changes how every test's context is created, here by loading the saved cookies.
  • Put these two fixtures in conftest.py to apply them to a whole folder. Keep the login tests themselves in a folder without them.

Page objects

Putting it together

When the label "Username" changes to "Email", you do not want to edit 40 tests. A page object is a class that knows how to find and use one page. Tests call its methods and never mention locators. A change to the page is then a change in one file.

When to bother: not for the first ten tests. Start with helper functions like sign_in. Move to page objects when the same locators appear in several files.

Project layout

Putting it together

Naming files test_api_* and test_ui_* lets you run one kind: pytest -k api for a fast check on every change, everything before a release. Options you always want, such as --tracing retain-on-failure, can go in pytest.ini on a line starting addopts =.

A real case: Next.js frontend, FastAPI backend

Putting it together

A common setup: the web pages are a Next.js application, and they fetch their data from a Python FastAPI backend. A QA engineer is asked to automate one journey: log in, open a certain page, type a prompt, click Submit, and check that an answer appears.

The key idea: a UI test only needs the address of the frontend. The test drives the browser, and the browser calls FastAPI by itself, exactly as it does for a real user. The test does not care what the frontend or backend is written in.

  1. Get a place to test and a test account. A staging address where both parts are deployed, and a login made for automation. Never a real person's account.
  2. Install, as in Install.
  3. Do the journey once by hand with playwright codegen https://staging.example.com and note the locators it suggests. Inspect anything it cannot name well.
  4. Write the test below, replacing the labels and addresses with the real ones.
  5. Run it with pytest --base-url https://staging.example.com --headed to watch, and without --headed in CI.

This file was run, and passes, against a small stand-in app with the same pages and a prompt endpoint that takes 1.5 seconds to answer. It was not run against a real Next.js and FastAPI application: the labels, addresses and the /api/prompts path are examples.

Concern in this kind of appWhat to do
Password in the test fileNever. Read it from environment variables, as here, and store it as a secret in CI.
Every test logs in againSign in once and reuse the saved state
Login through Google, SSO or a one-time codeHard and fragile to automate through the UI. Ask the developers for a test-only login or a token you can set.
Next.js class names change on every buildDo not locate by class. Use role, label and placeholder, and ask the frontend developers to add data-testid to the prompt box and the answer area.
The answer takes many seconds, or streams in piece by pieceWait for the response with page.expect_response or for a "finished" state on the page, with a longer timeout. Never a fixed sleep.
The answer comes from an AI model and differs every runCheck that an answer appears, is not empty and contains expected words or structure. Do not compare with an exact text.
You want to test the page without waiting for the modelFake the reply with page.route, as the second test does
The backend's own rules: empty prompt, no login, very long inputAPI tests straight against FastAPI. Its /docs page lists every endpoint with the fields it expects.

How to split the work: one or two UI tests for the journey, a handful of faked-reply tests for how the page behaves, and many API tests for the rules. That is the test pyramid applied to this application.

Flaky tests

Putting it together

A flaky test sometimes passes and sometimes fails with no change to the application. Flaky tests are worse than no tests: people stop believing red results. The usual causes and their cures:

CauseCure
Fixed sleeps, or reading a value before the page updatedexpect(...), which waits for the condition
Tests share data: one test's deposit changes another's balanceReset or create fresh data in a fixture, as api does here
Tests must run in a certain orderEach test arranges everything it needs
Locators tied to layout: div:nth-child(3)Role, label and test-id locators
Depending on today's date, random values or a third-party siteFixed test data, and faked replies for services you do not own
A slow shared test environmentLonger timeouts for that environment, never sleeps

Running in CI

Putting it together

CI (continuous integration) is a server that runs your tests automatically on every code change, so a regression is caught minutes after it is written. The steps are the same ones you typed by hand. This is the shape of a GitHub Actions workflow for the project; it was not run for this page, so treat it as a starting point.

CI machines have no screen, which is why headless is the default. When a test fails only in CI, download the uploaded trace and open it with playwright show-trace.

Cheat sheet

Putting it together

I want toWrite or run
Installpip install pytest-playwright, then playwright install chromium
Open a pagepage.goto("/")
Find a buttonpage.get_by_role("button", name="Sign in")
Find a form fieldpage.get_by_label("Username")
Type and click.fill("text"), .click()
Check the pageexpect(locator).to_have_text("...")
Call an APIapi.get(url, headers=...), api.post(url, data={...})
Check an API replyassert reply.status == 200, reply.json()
Fake a server replypage.route(pattern, lambda route: route.fulfill(...))
Watch the browserpytest --headed --slowmo 500
Record a testplaywright codegen <url>
Debug a failurepytest --tracing retain-on-failure, then playwright show-trace trace.zip
Run only API testspytest -k api

For pytest itself (fixtures, parametrize, marks) see Python Testing. Back to the QA path.