AI steps

AI steps let you locate elements and drive flows in plain English, without giving up the determinism of hand-written Playwright:

// Find one element by description
await page.piwiLocator('the email address field').fill('ada@example.com')

// Run a whole flow, with type-safe parameters
await page.piwiRun('sign in as {email}', { email: 'ada@example.com' })

The guiding principle is the LLM is a compiler, not a runtime. The first time a prompt is seen, an agent resolves it once into a committed, deterministic JSON artifact. Every run after that replays that artifact with ordinary Playwright calls — zero LLM calls and zero network in the default replay mode. Your CI stays fast, offline, and reproducible; the model is only ever involved while authoring.

This composes with the capture fixtures

AI steps are a separate opt-in from the capture fixtures, and compose with them. Replayed actions flow through the instrumented page, so they feed traces, reports, and locator healing exactly like hand-written code.

Setup

Extend your test with extendPiwiAi. Compose it over extendPiwiFixtures to get the capture fixtures too:

// tests/fixtures.ts
import { test as base } from '@playwright/test'
import { extendPiwiFixtures, extendPiwiAi } from '@piwitests/reporter'

// AI steps + capture fixtures
export const test = extendPiwiAi(extendPiwiFixtures(base))
export { expect } from '@playwright/test'

page now carries two extra methods:

Method	Returns	Use for
page.piwiLocator(template)	a real, synchronous Locator	locating one element by description
page.piwiRun(template)	Promise<void>	replaying a compiled multi-step flow

Placeholders are type-checked at compile time: a template with {param} requires a matching params object, and a misspelled or missing name is a TypeScript error.

await page.piwiRun('add {qty} of {sku} to the cart', { qty: '2', sku: 'PIWI-1' })

The lifecycle: author once, replay forever

An AI step moves through three modes, set with the ai.mode option or PIWI_AI:

Mode	What it does
replay (default)	Executes the committed artifact read-only. Never calls the model. Fails closed on a missing entry.
resolve	Authors any missing entry by driving the model, verifies it against the live page, and writes it to disk.
heal	Repairs a committed entry that no longer replays (e.g. a renamed element), without re-authoring from scratch.

The intended workflow:

1. Write the test with piwiLocator / piwiRun prompts.
2. Author locally in resolve mode (see below). The agent produces a JSON artifact next to your spec under __piwi__/.
3. Commit that artifact to git alongside the test.
4. Replay everywhere else — locally and in CI — with zero model calls.

Authoring (resolve mode)

Authoring needs a running Piwi dashboard with an AI provider configured (Settings → AI, or the PIWI_AI_PROVIDER environment variables). The reporter sends each iteration to the server's resolution endpoint; the server calls the model and returns one decision, which the reporter compiles deterministically. API keys stay on the server.

Point the reporter at that server and run the suite in resolve mode:

export PIWI_DASHBOARD_URL="https://piwi.example.com"   # the authoring server
export PIWI_API_KEY="…"                                # a reporter API key
export PIWI_AI=resolve

npx playwright test           # or: npx piwi ai resolve --grep "sign in"

piwi ai resolve runs the suite in resolve mode for you (forcing --workers=1 so authoring is serial). Once it finishes, review and commit the generated files.

Authoring drives the real page

In resolve mode each step is executed against the live page with your real parameter values so the flow can advance. Author against a disposable environment, not production.

Replay and CI

In replay mode a missing entry is a hard, actionable failure — the run does not silently pass and does not reach for a model:

piwi AI: no committed entry for "sign in as {email}" in test "checkout".
  expected file: tests/__piwi__/checkout.spec.ts/checkout.sign-in-as-email.a1b2c3d4.json
  run:  piwi ai resolve --grep "checkout"

That means an environment with no model access (a locked-down CI runner, an air-gapped box) can never accidentally author — it either replays a committed artifact or fails. If you would rather a missing entry mark the test fixme (yellow) instead of failing red, set ai.onMiss: 'fixme' (PIWI_AI_ON_MISS=fixme).

Because replay is plain Playwright, there is nothing extra to install or start in CI. Commit the artifacts and your existing test job replays them.

What's inside an artifact — and why it's safe

An entry is data, never code: an allowlisted locator program plus, for a flow, an ordered list of steps and a postcondition oracle. Determinism and safety come from several guarantees:

• Deterministic bytes. The model only ever names an element (its ARIA role + accessible name). The @piwitests/core scorer turns that into the committed locator, so model sampling never changes the file. Two runs that reach the same conclusion produce byte-identical JSON, and a no-op re-resolution leaves your working tree clean.
• No evaluation. Every locator method and action is checked against an allowlist before it touches the page. A tampered or malformed artifact can never become arbitrary execution.
• Drift guard. Each step records the element's role/name at author time. On replay, if the page positively shows that element has drifted (a rename), the flow stops before acting rather than clicking the wrong thing.
• Postcondition oracle. Every flow ends with an assertion the agent chose (an element became visible/hidden, or the URL changed). Replay verifies it, so a subtly wrong flow fails loudly instead of passing.
• Ajax waiting, race-proof. A step can wait for a network response it triggers (an XHR/fetch the agent observed while authoring). The wait is armed before the action fires, so a fast reply can never slip through the gap.

Intent in the dashboard

Each test's replayed AI steps are reported to the dashboard as a small usage manifest: the committed artifacts it exercised (powering the project's AI steps liveness tab) plus intent mappings — each compiled locator paired with the prompt it came from. Two places use them:

• Locator healing: when the failing locator was compiled from a prompt, the healing panel shows it — "Compiled from prompt: 'the email address field'" — so you fix the intent, not just the selector.
• AI diagnosis: the diagnosis context includes an AI Steps section listing the prompts behind the test's locators, so a root cause can be phrased in intent terms ("the element the test calls the email address field was renamed"). Capped by PIWI_AI_MAX_STEP_INTENTS (default 20, 0 disables).

Intent mappings are as private as everything else here: templates keep their {param} placeholders and locators their markers — parameter values never appear.

Privacy

Parameter values are masked out of everything sent to the model. The page snapshot the agent sees has your {param} values replaced with markers, and placeholders survive compilation as markers that are substituted locally at replay. Secrets in parameters never leave your machine. See Privacy & data flow for the full picture.

The piwi ai CLI

piwi ai check     Scan committed entries for orphans, non-canonical files and
                  duplicate templates. Read-only; exits 1 when issues are found.
piwi ai resolve   Author missing entries by running the suite in resolve mode.
piwi ai prune     Delete orphaned/dormant entries.

piwi ai check is offline and CI-friendly — add it as a lint step to catch an entry whose prompt was deleted or renamed, a file that isn't in canonical form, or two prompts that collide:

npx piwi ai check          # exit 1 on any hygiene issue
npx piwi ai check --json   # machine-readable findings

Configuration

Set these under the ai key of your reporter options (in wrapConfig's second argument, or the reporter entry options). Every option has an environment-variable equivalent.

Option	Env var	Default	Purpose
ai.mode	PIWI_AI	replay	replay / resolve / heal.
ai.dir	PIWI_AI_DIR	__piwi__	Per-spec directory holding committed entries.
ai.onMiss	PIWI_AI_ON_MISS	fail	On a replay miss: fail (red) or fixme (yellow).
ai.maxSteps	PIWI_AI_MAX_FLOW_STEPS	20	Max steps the agent may take authoring one flow.
ai.maxSnapshotChars	PIWI_AI_MAX_SNAPSHOT_CHARS	24000	Max ARIA-snapshot characters sent per authoring iteration.
ai.optionalProbeTimeout	PIWI_AI_OPTIONAL_PROBE_TIMEOUT	2000	Existence-probe timeout (ms) for an optional step during replay.
ai.responseWaitTimeout	PIWI_AI_RESPONSE_WAIT_TIMEOUT	Playwright default	Timeout (ms) for a step's waitForResponse, and the authoring settle window.
ai.screenshotFallback	PIWI_AI_SCREENSHOT_FALLBACK	false	Send a screenshot when the ARIA snapshot is empty (vision models only).

Authoring also reads PIWI_DASHBOARD_URL and PIWI_API_KEY (the server that runs the model), and PIWI_AI_UPDATE=true (or --update-ai) forces re-authoring of entries that already exist.

On the server side, two limits bound each authoring iteration — see the AI steps section of the configuration reference:

• PIWI_AI_STEP_MAX_SNAPSHOT_CHARS — snapshot characters the model receives.
• PIWI_AI_STEP_MAX_OUTPUT_TOKENS — output tokens the model may return. Reasoning models spend tokens on hidden chain-of-thought, so raise this (up to 8192) when authoring with one, or the JSON decision can be truncated.

Related

• Reporter — installing and configuring the reporter.
• Capture fixtures — the sibling opt-in that powers healing and performance data.
• AI diagnosis — the dashboard's failure-analysis AI (a different feature that reuses the same provider config).