This is the abridged developer documentation for Axx # Let agents build it. Prove that it works. > Axx is an open-source acceptance testing framework that verifies real software against executable behavioral expectations, independent of the language, architecture or agent that built it. Define what correct means. Let implementations evolve. Verify the behavior that matters. ## Generating software is getting easier. Trusting it isn’t. [Section titled “Generating software is getting easier. Trusting it isn’t.”](#generating-software-is-getting-easier-trusting-it-isnt) Coding agents are changing how quickly software can be created and changed. But generating an implementation and knowing that it meets its requirements are different problems. Code review, unit tests and static analysis remain valuable. None of them shows that the running system delivers the behavior expected of it. As software factories become more autonomous, verifying that behavior independently matters more, not less. **The challenge isn’t generating more software. It’s changing software continuously without losing confidence in what it does.** ## Keep what correct means apart from how it’s built. [Section titled “Keep what correct means apart from how it’s built.”](#keep-what-correct-means-apart-from-how-its-built) ### Intent Software exists to meet requirements, not to run code. Acceptance criteria state the behaviors a system must provide, as scenarios that engineers, product owners and agents can all review, and that stay meaningful while the implementation changes. [How steps are designed →](/next/explanations/step-design/) ### Freedom of implementation Refactor a service, switch frameworks, replace a component or rewrite it in another language. While the external contract holds, the acceptance criteria still apply: axx never loads your code. [Black-box testing →](/next/explanations/black-box-testing/) ### Independent verification Axx runs each scenario against the running system, through its APIs, events, data stores and user interfaces. An agent can use axx to check its own work, but the code it wrote isn’t what defines success. The acceptance criteria are, and your team decides who approves them. [How axx works →](/next/explanations/how-axx-works/) ## Give agents freedom to build. Keep control of what correct means. [Section titled “Give agents freedom to build. Keep control of what correct means.”](#give-agents-freedom-to-build-keep-control-of-what-correct-means) 1. ### Define behavior Write the acceptance criteria: what the system must accomplish, as scenarios made of real steps. 2. ### Implement freely Engineers and coding agents build, change, refactor or replace the implementation. 3. ### Verify independently Axx starts the system, exercises it through its observable interfaces and checks the expected outcomes. 4. ### Iterate with evidence Failures come back as structured data, expected against actual, so the next change fixes the code, not the requirement. Axx is the verification layer of a software factory, not the factory: it doesn’t orchestrate agents or write code. It runs wherever the software is built, from a developer’s terminal to a CI pipeline to an agent’s tool calls. ## Built for real systems. Built for autonomous engineering. [Section titled “Built for real systems. Built for autonomous engineering.”](#built-for-real-systems-built-for-autonomous-engineering) ### Black-box by architecture Scenarios exercise running systems through the boundaries their clients use: HTTP and gRPC, events, databases, files, browsers and apps. They never import a class or depend on a framework, so they outlive refactors and rewrites. [Black-box testing →](/next/explanations/black-box-testing/) ### One scenario across the system Send a request, check the call your service made to a mocked dependency, then the rows it stored and the event it published, all in one scenario, with steps from axx’s packs and no glue code. [See a scenario →](/next/explanations/why-axx/) ### Native interfaces for coding agents Agents search the steps actually installed, validate scenarios without starting anything, work through axx’s MCP server, and read failures as structured data, instead of guessing at a test API or parsing logs. [Axx for agents →](/next/explanations/axx-for-agents/) ### Fast, repeatable feedback `axx run` starts your services, waits until they’re healthy, runs scenarios in parallel and cleans up. Generated fixtures, isolation checks, polling steps and reruns of what failed keep each cycle quick and reliable. [Parallel runs →](/next/guides/parallel-runs/) ### Verification beyond traditional applications Test AI features, MCP servers, A2A agents, mobile and desktop apps, and the models apps run on the device, replaced by stubs axx builds in the model’s own format, whose answers are your mappings. [Stub embedded models →](/next/guides/stub-embedded-models/) ## Acceptance criteria you can execute. [Section titled “Acceptance criteria you can execute.”](#acceptance-criteria-you-can-execute) This scenario registers a parcel through a service’s REST API. In one run, axx checks the request and the response against the service’s OpenAPI document, checks that the service asked its address service (a WireMock mock) about the postcode, and checks what it stored in PostgreSQL, down to a JSON column. features/register-parcels.feature ```gherkin Feature: Register parcels Background: Given the parcels service with the following properties: | url | http://localhost:8400 | | openapi | http://localhost:8400/openapi.json | And the mocked addresses service with the following properties: | url | http://localhost:8081 | And a parcels-db database with the following properties: | url | postgres://localhost:5432/parcels | | user | parcels | | password | parcels | Scenario: A shop registers a parcel Given a POST request to /api/parcels And a request payload using an application/json content example And the request payload properties are: | reference | PX-REG-1001 | | recipient.postcode | "53111" | When the request is executed Then the response status code is 201 And the mocked GET request to /v1/postcodes/DE/53111 named postcode-check was received by addresses And a selection of rows is retrieved from the parcels.parcels table where: | reference | PX-REG-1001 | And the 1st row details property for the selection json properties are: | source | api | | zone | DE-1 | ``` Nothing in it names a class, a framework or a language. It describes what the service does at its boundaries, so the same scenario verifies a refactor, a new framework or a rewrite. Every line is a step from one of axx’s packs: the [step index](/next/references/step-index/) lists them all, and the [parcels example](https://github.com/nimbusxr/axx/tree/main/examples/parcels) runs a full suite like it. ## Go beyond conventional acceptance testing. [Section titled “Go beyond conventional acceptance testing.”](#go-beyond-conventional-acceptance-testing) ### Deterministic AI testing Mock the model APIs your service calls (OpenAI’s and the servers that speak it, Anthropic’s, Gemini’s) and check what it sent them. For the models an app runs itself, axx builds stubs in their native formats, from ONNX and Core ML to TFLite and GGUF, and stands in for the platform’s own: Apple’s Vision, Android’s ML Kit, a browser’s built-in AI. [Test AI features →](/next/guides/test-ai-features/) ### Agent protocol testing Call an MCP server’s tools as an AI assistant does, over stdio or HTTP, with the tools’ own schemas as the contract. Message [A2A agents](/next/guides/test-a2a/) over JSON-RPC, HTTP+JSON or gRPC and check their tasks and artifacts. Mock the servers and agents your service depends on. [Test MCP servers →](/next/guides/test-mcp/) ### Native application verification Drive Android and iOS apps on emulators and simulators, and [macOS, Windows and Linux apps](/next/guides/test-desktop-apps/), by the names people see. Move the phone along a route, tap its notifications, and show a simulator’s or an emulator’s camera a photo, a video or a 3D scene, which the app’s AR (ARKit, ARCore) finds where it is. [Test mobile apps →](/next/guides/test-mobile-apps/) ### Distributed workflows Publish and consume Kafka events with Avro and Schema Registry, seed and query SQL and MongoDB, mock dependencies, check [cloud services](/next/guides/test-cloud-services/) like S3, Pub/Sub and Service Bus, and validate against OpenAPI and AsyncAPI contracts, in the same scenario. [Test Kafka with Avro →](/next/guides/test-kafka-avro/) ## Designed for engineers. Ready for agents. [Section titled “Designed for engineers. Ready for agents.”](#designed-for-engineers-ready-for-agents) Axx comes from years of maintaining acceptance suites for complex enterprise systems, where executable acceptance criteria shaped how teams built and shipped software. Its architecture reflects those lessons: black-box scenarios, step text that never changes, and test data that can’t collide between parallel scenarios. It wasn’t built to make Gherkin work with AI. Engineers use it directly: one binary, feature files, `axx run`. Coding agents use the same tool, with [interfaces made for them](/next/tutorials/with-an-agent/). Axx publishes how coding agents do with it. In its [agent evaluation](/next/explanations/agent-evals/), an agent writes the tests for a parcels service. A suite succeeds only if it passes on the working service and fails on every bug planted in it. * **With axx:** all 72 suites succeeded, across four setups (with its skills, its MCP server, both or neither). * **Without axx:** 14 of 18 did. It’s a small sample, run by the project itself: one model (`openai/gpt-6-luna`), six tasks, three tries each. The tasks, planted bugs, checks and transcripts are all [public](https://github.com/nimbusxr/axx-evals). ## Build faster. Keep the contract. [Section titled “Build faster. Keep the contract.”](#build-faster-keep-the-contract) Whether software is written by an engineer, generated by an agent or rebuilt from scratch, its required behavior still matters. Axx verifies that behavior against the running system. [Start verifying](/next/tutorials/quickstart/)[Explore the documentation](/next/explanations/why-axx/) Axx is open source (Apache-2.0) and in beta (`v0.x`): interfaces may change before `v1.0.0`. **For coding agents:** in a project, the axx command line answers faster than this site: `axx steps` lists every step of the project’s packs, a line each, and `axx steps show ` gives one step’s documentation and examples. On this site, every page has a Markdown twin (add `.md` to its path), [llms.txt](/next/llms.txt) lists them, and a pack’s steps are at `/references/packs/.md`. # Why Axx > Axx verifies running software against executable acceptance criteria, black-box, so implementations can change without losing the definition of correct. It's the behavioral verification layer for agentic software factories. Software is increasingly produced by coding agents, but the responsibility for correctness hasn’t gone away. As implementations become easier to generate and replace, engineering teams need durable, independently executable definitions of expected behavior. A test suite that imports the code it checks changes with that code; a definition of correct shouldn’t. Axx approaches this through black-box acceptance testing. It verifies running systems through their observable interfaces (their HTTP APIs and web apps, their databases, their event streams and the dependencies they call), so implementations can evolve without the acceptance criteria being tied to their internal structure. The result is a verification framework for conventional engineering workflows and for the emerging generation of agentic software factories, where agents build and change the implementation and Axx checks that it still does what it must. Three ideas run through it: * **Intent.** Acceptance criteria state what the system must do, as scenarios people and agents can review. They’re written in Gherkin, and they stay meaningful while the implementation changes. * **Freedom of implementation.** Refactor a service, switch frameworks or rewrite it in another language: while the external contract holds, the same scenarios apply. * **Independent verification.** An agent can use Axx to check its own work, but the code it wrote isn’t what defines success. The scenarios are, and your team decides who approves them. When a scenario passes, the behavior it describes works in the running system, not in a mock of it or in one unit on its own. Dependencies outside the system can be mocked (Axx checks the calls made to them), while the system itself runs for real. ## One scenario covers the whole flow [Section titled “One scenario covers the whole flow”](#one-scenario-covers-the-whole-flow) ```gherkin Feature: Register parcels Background: Given the parcels service with the following properties: | url | http://localhost:8400 | | openapi | http://localhost:8400/openapi.json | And the mocked addresses service with the following properties: | url | http://localhost:8081 | And a parcels-db database with the following properties: | url | postgres://localhost:5432/parcels | | user | parcels | | password | parcels | Scenario: A shop registers a parcel Given a POST request to /api/parcels And a request payload using an application/json content example And the request payload properties are: | reference | PX-REG-1001 | | recipient.postcode | "53111" | When the request is executed Then the response status code is 201 And the mocked GET request to /v1/postcodes/DE/53111 named postcode-check was received by addresses And a selection of rows is retrieved from the parcels.parcels table where: | reference | PX-REG-1001 | And the 1st row details property for the selection json properties are: | source | api | | zone | DE-1 | ``` This one scenario: * builds a request from the service’s own OpenAPI example and sends it; * checks the request and the response against that OpenAPI document; * checks that the service asked its downstream address service (mocked) about the postcode; * checks what the service stored in PostgreSQL, down to a JSON column. Every line is a step from one of Axx’s packs, so there is no step code to write. ## Test the service, not the code [Section titled “Test the service, not the code”](#test-the-service-not-the-code) Axx never loads your code. The service can be written in any language, and the tests keep passing through refactors and rewrites because they only depend on what the service does. The people who wrote the acceptance criteria can read them too. See [Black-box testing](/next/explanations/black-box-testing/). Axx’s packs cover: | Area | What you get | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | REST | Requests and responses validated against your OpenAPI document | | Mocked dependencies | Calls to them verified through WireMock | | SQL | PostgreSQL, MySQL, SQLite and SQL Server: seeds, selections, JSON columns, locks and triggers | | MongoDB | Seeding and querying | | Kafka | Publishing and consuming events, with Avro and Schema Registry | | Logs | The entries your services log, in files or sent over UDP, TCP or HTTP | | Files | The files your services write: their content, their text (PDF, Word, Excel…) and their rows | | Web apps | Real browsers through Playwright, with screenshots, accessibility audits, network control, Lighthouse scores and JavaScript coverage | | Clouds | S3, SQS, SNS, EventBridge and DynamoDB; Cloud Storage, Pub/Sub, BigQuery and Firestore; Blob Storage and Service Bus | ## One binary with nothing to wire up [Section titled “One binary with nothing to wire up”](#one-binary-with-nothing-to-wire-up) Axx needs no JVM, no build plugin and no test-runner glue. `axx run` does the whole job: 1. It starts the services listed in `axx.yaml` and waits until they are healthy. 2. It runs the scenarios in parallel. 3. It stops everything and cleans up, even after a crash or Ctrl-C. `axx up` keeps the system running between runs, so each edit-and-run cycle takes seconds. It is fast. The [example suite](https://github.com/nimbusxr/axx/tree/main/examples/parcels) has 101 scenarios and 888 steps, covering REST, WireMock, PostgreSQL, MongoDB, Kafka, logs, files and a web app in real browsers. Against a running stack, its 38 scenarios that need no browser finish in about a second. ## Axx’s steps handle the hard parts [Section titled “Axx’s steps handle the hard parts”](#axxs-steps-handle-the-hard-parts) The steps of Axx’s packs do the difficult work: * OpenAPI validation * database seeds and JSONB queries * Avro with Schema Registry * waiting for events to arrive * checking calls to mocked services All of them compose the same way. You build something (a request, an event, a selection), act on it, and assert on what came back. A step can name the service it targets (`on parcels-db`) when a scenario uses more than one. Ordinals such as `the 2nd selection` refer back to earlier ones. Once you know the pattern, a pack you have never used reads the same. Step text is public API, and it never changes, so a feature file keeps working across every release. ## Extend it your way [Section titled “Extend it your way”](#extend-it-your-way) Steps come in **packs**. A project uses only the packs it needs (`axx pack add rest sql`). When Axx doesn’t cover something, write a pack of your own in Go. Your steps work on the same scenario context as Axx’s: * the services the scenario registered * its requests and responses * its database selections * its events Use custom packs to reach a system Axx doesn’t cover, or to write scenarios in your team’s own words. `axx pack new ./steps` creates a pack. The first run builds it into Axx and caches the result. After that, your steps appear in `axx steps search`, `axx validate` and the agent skills like any other step. See [Write custom steps](/next/guides/write-custom-steps/). ## Built for coding agents [Section titled “Built for coding agents”](#built-for-coding-agents) Axx is designed so coding agents never have to guess. They search the real step text instead of inventing it, check every line without starting anything, get failures as data with the expected and actual values, and read docs generated from the binary they are running. [Axx for agents](/next/explanations/axx-for-agents/) explains how. ## Test data that doesn’t collide [Section titled “Test data that doesn’t collide”](#test-data-that-doesnt-collide) Parallel scenarios only stay reliable if each one uses its own data. `axx lint` finds colliding ids and keys before they cause flaky failures. `axx fixtures` generates test data checked against your schemas and reports when it drifts. See [Isolate test data](/next/guides/isolate-test-data/) and [Fixture factories](/next/guides/fixture-factories/). ## Where to go next [Section titled “Where to go next”](#where-to-go-next) | If you want to | Go to | | ---------------------------------------------- | -------------------------------------------------- | | Get a passing scenario in about ten minutes | [Quickstart](/next/tutorials/quickstart/) | | Get a task done | [Guides](/next/guides/install/) | | Look up a pack, a step, a command or a setting | [References](/next/references/packs/) | | Understand the design | [How Axx works](/next/explanations/how-axx-works/) | Axx is pre-release (`v0.x`): interfaces may change before `v1.0.0`. # Your first suite > Build an acceptance suite for a realistic service step by step - its API and OpenAPI contract, a dependency it calls, and its database. In this tutorial, we’ll write an acceptance suite for Parcels, an example service that comes with Axx. Parcels is a parcel-delivery service with a PostgreSQL database. Shops ask it for price quotes and register their parcels with it, and it checks every address with an address service, which the example replaces with a mock. Step by step, our suite will: * call the service’s API and check its answers; * check every request and response against the API’s OpenAPI document; * check the call the service makes to the address service; * check what the service writes to its database. You’ll need Docker with Compose v2, Git, and Axx (see the [Quickstart](/next/tutorials/quickstart/)). Plan on about 30 minutes, including a few minutes for Docker to build images the first time. ## Get the example [Section titled “Get the example”](#get-the-example) ```sh git clone https://github.com/nimbusxr/axx.git cd axx/examples/parcels ``` The `infra` directory holds a Docker Compose file that runs Parcels with everything it needs. ## Create the suite [Section titled “Create the suite”](#create-the-suite) Make a directory for our suite: ```sh mkdir my-suite cd my-suite ``` Create `axx.yaml` in it: axx.yaml ```yaml version: 1 services: parcels: dir: ../infra command: docker compose up --build ready: http: url: http://localhost:8400/health timeout: 10m cleanup: docker compose down -v --remove-orphans ``` This tells Axx to start Parcels with Docker Compose, to wait until its health check answers, and to remove the containers when it stops. Next, choose the packs of steps the suite uses. We’ll send requests to Parcels with `rest`, check what it asked the address service’s mock with `mock`, and look in its database with `sql`: ```sh axx pack add rest mock sql ``` ```console created axx-packs.yaml added rest, mock, sql ``` ## Start Parcels [Section titled “Start Parcels”](#start-parcels) ```sh axx up ``` ```console axx: starting services in the background (logs: .axx/logs) up: parcels (stop with `axx down`) ``` The first time, Docker builds the images, so this takes a few minutes. Parcels now keeps running in the background, and every run in this tutorial takes a fraction of a second. ## Ask for a price [Section titled “Ask for a price”](#ask-for-a-price) Create a `features` directory: ```sh mkdir features ``` Then create `features/quotes/quotes.feature`: features/quotes/quotes.feature ```gherkin Feature: Quotes Background: Given the parcels service with the following properties: | url | http://localhost:8400 | Scenario: A shop asks for a price Given a POST request to /api/quotes And a request payload using an application/json empty content template And the request payload properties are: | weightGrams | 1200 | | recipient | {"postcode": "10115", "country": "DE"} | When the request is executed Then the response status code is 200 And the response payload property priceCents is '690' ``` The scenario builds a request body from an empty JSON object: a 1.2 kg parcel to a Berlin postcode. Run it: ```sh axx run ``` ```console axx: preparing rest, mock, sql (once; cached for later runs) axx: ready in 31s axx: reusing services started by `axx up`: parcels warning: services started by `axx up` keep the data of earlier runs; `axx down`, then `axx up`, starts them afresh Feature: Quotes Scenario: A shop asks for a price # features/quotes/quotes.feature:7 ✓ Given the parcels service with the following properties: (background) | url | http://localhost:8400 | ✓ Given a POST request to /api/quotes ✓ And a request payload using an application/json empty content template ✓ And the request payload properties are: | weightGrams | 1200 | | recipient | {"postcode": "10115", "country": "DE"} | ✓ When the request is executed (213ms) log: POST http://localhost:8400/api/quotes -> 200 OK (213ms, 112 bytes) attachment: request body (application/json, 68 B) {"weightGrams":1200,"recipient":{"postcode":"10115","country":"DE"}} attachment: response body (application/json, 112 B) {"zone":"DE-1","serviceLevel":"STANDARD","weightGrams":1200,"priceCents":690,"currency":"EUR","deliveryDays":2} ✓ Then the response status code is 200 ✓ And the response payload property priceCents is '690' 1 scenario (1 passed) 7 steps (7 passed) Finished in 213ms ``` Notice the first lines. Axx prepared itself with the suite’s packs, which it does only once. And it didn’t start anything this time: it used the Parcels service that `axx up` started. The price is 6.90 EUR, in cents. ## Check the API contract [Section titled “Check the API contract”](#check-the-api-contract) Parcels publishes an OpenAPI document that describes its API. Add it to the service in the `Background`: features/quotes/quotes.feature ```gherkin Background: Given the parcels service with the following properties: | url | http://localhost:8400 | | openapi | http://localhost:8400/openapi.json | ``` Run `axx run` again. The scenario still passes. Axx now checks every request and response against that document, so the suite notices when Parcels breaks its contract. We’ll see the check at work in a moment. ## Start from the contract’s example [Section titled “Start from the contract’s example”](#start-from-the-contracts-example) The OpenAPI document also gives an example request for each operation. Add a second scenario at the end of `features/quotes/quotes.feature`: features/quotes/quotes.feature ```gherkin Scenario: Heavier parcels cost more Given a POST request to /api/quotes And a request payload using an application/json content example And the request payload property weightGrams is '4500' When the request is executed Then the response status code is 200 And the response payload properties are: | weightGrams | 4500 | | priceCents | 690 | ``` Run `axx run`. Both scenarios pass. Look at the new scenario’s output: ```console Scenario: Heavier parcels cost more # features/quotes/quotes.feature:18 ✓ Given the parcels service with the following properties: (background) | url | http://localhost:8400 | | openapi | http://localhost:8400/openapi.json | ✓ Given a POST request to /api/quotes ✓ And a request payload using an application/json content example ✓ And the request payload property weightGrams is '4500' ✓ When the request is executed log: POST http://localhost:8400/api/quotes -> 200 OK (3ms, 112 bytes) attachment: request body (application/json, 94 B) {"weightGrams":4500,"serviceLevel":"STANDARD","recipient":{"postcode":"10115","country":"DE"}} attachment: response body (application/json, 112 B) {"zone":"DE-1","serviceLevel":"STANDARD","weightGrams":4500,"priceCents":690,"currency":"EUR","deliveryDays":2} ✓ Then the response status code is 200 ✓ And the response payload properties are: | weightGrams | 4500 | | priceCents | 690 | ``` Notice the request body. We only wrote the weight. Axx started from the example request in the OpenAPI document, which also has a service level and a recipient, and changed the weight. ## Break the contract on purpose [Section titled “Break the contract on purpose”](#break-the-contract-on-purpose) In the new scenario, change the weight to a string: ```gherkin And the request payload property weightGrams is '"heavy"' ``` Run `axx run`: ```console ✗ When the request is executed log: POST http://localhost:8400/api/quotes -> 400 Bad Request (1ms, 125 bytes) attachment: request body (application/json, 97 B) {"weightGrams":"heavy","serviceLevel":"STANDARD","recipient":{"postcode":"10115","country":"DE"}} attachment: response body (application/problem+json, 125 B) {"detail":"weightGrams must be an integer","instance":"/api/quotes","status":400,"title":"Bad Request","type":"about:blank"} OpenAPI validation failed for POST http://localhost:8400/api/quotes (status 400): - validation.request.body.schema.type: $.weightGrams: got string, want integer (POST request body for '/api/quotes' failed to validate schema) To relax a check, set its key (or a parent key) to WARN, INFO or IGNORE with "Given the OpenAPI validation levels are:" or openapi.levels in axx.yaml. ↷ Then the response status code is 200 ↷ And the response payload properties are: | weightGrams | 4500 | | priceCents | 690 | ``` The double quotes make the value a string. The contract says `weightGrams` is an integer, and Axx fails the request step with the reason: `$.weightGrams: got string, want integer`. The steps after it are skipped. Change the value back to `'4500'` and run `axx run` again. Both scenarios pass. ## Check what Parcels asked the address service [Section titled “Check what Parcels asked the address service”](#check-what-parcels-asked-the-address-service) To price a parcel, Parcels asks the address service which delivery zone serves the recipient’s postcode. In this example, the address service is a mock running on port 8081. Let’s check that the call happened. Register the mock in the `Background`, and add a last step to the first scenario. Your file now looks like this: features/quotes/quotes.feature ```gherkin Feature: Quotes Background: Given the parcels service with the following properties: | url | http://localhost:8400 | | openapi | http://localhost:8400/openapi.json | And the mocked addresses service with the following properties: | url | http://localhost:8081 | Scenario: A shop asks for a price Given a POST request to /api/quotes And a request payload using an application/json empty content template And the request payload properties are: | weightGrams | 1200 | | recipient | {"postcode": "10115", "country": "DE"} | When the request is executed Then the response status code is 200 And the response payload property priceCents is '690' And the mocked GET request to /v1/postcodes/DE/10115 named zone-lookup was received by addresses Scenario: Heavier parcels cost more Given a POST request to /api/quotes And a request payload using an application/json content example And the request payload property weightGrams is '4500' When the request is executed Then the response status code is 200 And the response payload properties are: | weightGrams | 4500 | | priceCents | 690 | ``` Run `axx run`. Both scenarios pass: ```console ✓ Then the response status code is 200 ✓ And the response payload property priceCents is '690' ✓ And the mocked GET request to /v1/postcodes/DE/10115 named zone-lookup was received by addresses ``` Axx asked the mock which requests it had received, and found Parcels’ `GET /v1/postcodes/DE/10115`. ## Check the database [Section titled “Check the database”](#check-the-database) Now we’ll register a parcel through the API and check that Parcels stored it. Create `features/parcels/parcels.feature`: features/parcels/parcels.feature ```gherkin Feature: Parcels Background: Given the parcels service with the following properties: | url | http://localhost:8400 | | openapi | http://localhost:8400/openapi.json | And a parcels-db database with the following properties: | url | postgres://localhost:5432/parcels | | user | parcels | | password | parcels | Scenario: A registered parcel is stored Given a POST request to /api/parcels And a request payload using an application/json content example And the request payload properties are: | reference | PX-TUTORIAL-1 | | sender | my-shop | When the request is executed Then the response status code is 201 And a selection of rows is retrieved from the parcels.parcels table where: | reference | PX-TUTORIAL-1 | | sender | my-shop | And the selection has 1 row ``` The `Background` now also registers Parcels’ database. Run `axx run`, and look at the end of the output: ```console ✓ Then the response status code is 201 ✓ And a selection of rows is retrieved from the parcels.parcels table where: | reference | PX-TUTORIAL-1 | | sender | my-shop | ✓ And the selection has 1 row 3 scenarios (3 passed) 26 steps (26 passed) Finished in 22ms ``` The new scenario registered a parcel through the API. Then it selected the rows of the `parcels.parcels` table with that reference and sender, and found exactly one: Parcels stored what it was sent. Run `axx run` once more. This time the new scenario fails: ```console ✓ When the request is executed log: POST http://localhost:8400/api/parcels -> 409 Conflict (1ms, 135 bytes) attachment: request body (application/json, 211 B) {"reference":"PX-TUTORIAL-1","sender":"my-shop","weightGrams":1200,"serviceLevel":"STANDARD","recipient":{"name":"Ada Lovelace","street":"Invalidenstrasse 116","city":"Berlin","postcode":"10115","country":"DE"}} attachment: response body (application/problem+json, 135 B) {"detail":"parcel PX-TUTORIAL-1 is already registered","instance":"/api/parcels","status":409,"title":"Conflict","type":"about:blank"} ✗ Then the response status code is 201 Expected status code <201> but was <409>. expected: 201 actual: 409 ``` The parcel from the first run is still in the database, and references are unique. Acceptance tests run against the service’s real data, so data a scenario creates stays behind. [Isolate test data](/next/guides/isolate-test-data/) shows how suites deal with that. For now, stopping Parcels clears everything. ## Stop Parcels [Section titled “Stop Parcels”](#stop-parcels) ```sh axx down ``` ```console stopped: parcels ``` Axx stopped Parcels and ran its cleanup, which removed the containers and their data. The next `axx up` starts from empty databases. ## What you’ve done [Section titled “What you’ve done”](#what-youve-done) You wrote a suite for a service with a database and a dependency, and checked four things from the outside: * the answers of its API; * its requests and responses against its OpenAPI contract; * the call it makes to another service; * what it writes to its database. You didn’t write any test code: every line is a step Axx provides. The complete suite for Parcels, with events, a document store and more, is in `examples/parcels/acceptance`. Next, in [Test with an agent](/next/tutorials/with-an-agent/), we’ll hand an acceptance criterion to a coding agent and review the scenario it writes. # Quickstart > Install Axx and run your first passing scenario against a small web server on your machine, in about ten minutes. In this tutorial, we’ll install Axx, create a project, and write a scenario that tests a small web server running on your machine. We’ll run it and watch it pass, then break it on purpose to see what a failure looks like. It takes about ten minutes. You’ll need a terminal, [Go](https://go.dev/dl/) 1.27 or newer to install Axx, and Python 3, which we’ll use as the web server. ## Install Axx [Section titled “Install Axx”](#install-axx) Install Axx with Go: ```sh go install github.com/nimbusxr/axx/cmd/axx@latest ``` Check that your shell finds it: ```sh axx version ``` Axx prints its version. If your shell says `command not found`, add Go’s `bin` directory to your `PATH` (`export PATH="$PATH:$(go env GOPATH)/bin"`) and try again. ## Create a project [Section titled “Create a project”](#create-a-project) Make a directory for the project and let Axx set it up: ```sh mkdir hello-axx cd hello-axx axx init ``` ```console axx: preparing rest (once; cached for later runs) axx: ready in 31s create axx.yaml create axx-packs.yaml create features/smoke/smoke.feature create .github/workflows/acceptance.yml create AGENTS.md create .gitignore create .gitattributes create .agents/skills (5 skills) Codex keeps its MCP servers in ~/.codex/config.toml; `axx mcp install --agent codex --scope user` adds axx there. next: put your service's command, port and health check in axx.yaml and the smoke feature, then `axx doctor` and `axx run` ``` Axx created a configuration file, `axx.yaml`, and a first feature file, `features/smoke/smoke.feature`. We’ll change both. It also created `axx-packs.yaml`, the list of the packs of steps the project uses. It lists `rest`, whose steps send HTTP requests, and that’s all we need. The first time, Axx prepared itself with that pack, which it won’t need to do again. We won’t need the other files in this tutorial: they are for CI and for coding agents. ## Give the web server something to serve [Section titled “Give the web server something to serve”](#give-the-web-server-something-to-serve) Our web server will serve the files in this directory. Create a small JSON file for it: ```sh echo '{"message": "Hello, axx"}' > hello.json ``` ## Tell Axx how to start the server [Section titled “Tell Axx how to start the server”](#tell-axx-how-to-start-the-server) Open `axx.yaml`. At the bottom, replace the whole `services:` section with this one: axx.yaml ```yaml services: hello-axx: command: python3 -m http.server 8000 ready: http: url: http://${sys:local.host}:8000/ ``` Axx now knows how to start the server, and how to tell when it’s ready: when `http://localhost:8000/` answers. ## Write the scenario [Section titled “Write the scenario”](#write-the-scenario) Replace everything in `features/smoke/smoke.feature` with: features/smoke/smoke.feature ```gherkin Feature: Hello axx Background: Given the hello-axx service with the following properties: | url | http://${sys:local.host}:8000 | Scenario: The service says hello Given a GET request to /hello.json When the request is executed Then the response status code is 200 And the response payload property message is 'Hello, axx' ``` Every line is a step Axx already knows. The `Background` registers the server as a service named `hello-axx`. The scenario sends it a request and checks the answer. ## Check the scenario [Section titled “Check the scenario”](#check-the-scenario) Before running anything, ask Axx to check every line: ```sh axx validate ``` ```console 1 file, 1 scenario, 5 steps: ok ``` Axx matched each line to one of its steps, without starting the server. ## Run it [Section titled “Run it”](#run-it) ```sh axx run ``` ```console axx: starting hello-axx (logs: .axx/logs/services.log) Feature: Hello axx Scenario: The service says hello # features/smoke/smoke.feature:7 ✓ Given the hello-axx service with the following properties: (background) | url | http://${sys:local.host}:8000 | ✓ Given a GET request to /hello.json ✓ When the request is executed log: GET http://localhost:8000/hello.json -> 200 OK (4ms, 26 bytes) attachment: response body (application/json, 26 B) {"message": "Hello, axx"} ✓ Then the response status code is 200 ✓ And the response payload property message is 'Hello, axx' 1 scenario (1 passed) 5 steps (5 passed) Finished in 4ms ``` Notice what happened. Axx started the web server, waited until it answered, ran the scenario, and stopped the server again. Under the request step, the `log:` line shows the request Axx sent, and the attachment shows the response it got back. ## Break it on purpose [Section titled “Break it on purpose”](#break-it-on-purpose) Change the last line of `features/smoke/smoke.feature` so it expects a different message: ```gherkin And the response payload property message is 'Hello, world' ``` Run it again: ```sh axx run ``` ```console axx: starting hello-axx (logs: .axx/logs/services.log) Feature: Hello axx Scenario: The service says hello # features/smoke/smoke.feature:7 ✓ Given the hello-axx service with the following properties: (background) | url | http://${sys:local.host}:8000 | ✓ Given a GET request to /hello.json ✓ When the request is executed log: GET http://localhost:8000/hello.json -> 200 OK (2ms, 26 bytes) attachment: response body (application/json, 26 B) {"message": "Hello, axx"} ✓ Then the response status code is 200 ✗ And the response payload property message is 'Hello, world' Response payload property message is not Hello, world expected: "Hello, world" actual: "Hello, axx" Failed scenarios: ✗ The service says hello # features/smoke/smoke.feature:7 rerun: axx run features/smoke/smoke.feature:7 1 scenario (1 failed) 5 steps (1 failed, 4 passed) Finished in 3ms ``` Axx marks the step that failed and shows what it expected next to what it got. It also prints a command that reruns only this scenario. Change the line back to `'Hello, axx'` and run `axx run` once more. The scenario passes again. ## What you’ve done [Section titled “What you’ve done”](#what-youve-done) You installed Axx, told it how to start a service, wrote a scenario, ran it, and read a failure. That’s the loop you’ll use with every Axx suite. Next, in [Your first suite](/next/tutorials/first-suite/), we’ll test a real service with a database and a dependency it calls. # Test with an agent > Hand an acceptance criterion to a coding agent, watch it write and run the scenario with Axx, and review what it wrote. In this tutorial, we’ll give a coding agent an acceptance criterion in plain words and watch it write a scenario, check it and run it with Axx. Then we’ll review what it wrote. We’ll use [Claude Code](https://claude.com/claude-code) and the `hello-axx` project from the [Quickstart](/next/tutorials/quickstart/). You’ll need both. ## Teach the agent Axx [Section titled “Teach the agent Axx”](#teach-the-agent-axx) `axx init` installed Axx’s skills in `.agents/skills` in the Quickstart. The `hello-axx` directory had no sign of Claude Code then, so connect it now. In the `hello-axx` directory, give Claude Code the Axx MCP server and link the skills where it reads them: ```sh axx mcp install --agent claude axx skills install ``` ```console create .mcp.json: the axx MCP server for Claude Code installed 5 skills to .agents/skills (0 files updated) linked for Claude Code in .claude/skills ``` The skills teach an agent how to find Axx’s steps, write a scenario, run it and read a failure. The MCP server gives it tools to do so, and the `AGENTS.md` that `axx init` wrote points agents at the same rules. In a repository that already uses Claude Code, `axx init` does all of this itself. ## Ask for a test [Section titled “Ask for a test”](#ask-for-a-test) Start Claude Code in the project: ```sh claude ``` It asks whether to use the `axx` server from `.mcp.json`; allow it. Give it the criterion, not step text: ```text Add an acceptance test for this: asking for a file that doesn't exist returns 404. Use axx, and run it. ``` ## Watch what it does [Section titled “Watch what it does”](#watch-what-it-does) The agent’s wording will differ from run to run, but you’ll see it work through the same loop. Claude Code asks before each command and each tool call, so you can follow along. Notice that it: 1. reads the project’s steps with the `steps_search` tool (or `axx steps`) instead of guessing their text; 2. checks a line it’s unsure of with `step_explain` (`axx explain`); 3. writes a feature file; 4. starts the server with `env` (`axx up`), runs the scenarios with `scenarios_run` (`axx run --compact`), which also reports what `axx validate` and `axx lint` find, and stops the server again. ## Review what it wrote [Section titled “Review what it wrote”](#review-what-it-wrote) Open the feature file the agent created. In our run it was `features/missing-file.feature`: features/missing-file.feature ```gherkin Feature: Missing files Asking for a file the service does not have is answered with Not Found. Background: Given the hello-axx service with the following properties: | url | http://${sys:local.host}:8000 | Scenario: Asking for a file that does not exist returns 404 Given a GET request to /no-such-file-404.json When the request is executed Then the response status code is 404 ``` Read it against the criterion you gave. Each line says what it checks, so you can tell at a glance whether the agent tested what you asked for. That review is the part that stays yours. ## Run it yourself [Section titled “Run it yourself”](#run-it-yourself) ```sh axx run ``` The end of the output shows both scenarios, the Quickstart’s and the agent’s: ```console 2 scenarios (2 passed) 9 steps (9 passed) ``` ## What you’ve done [Section titled “What you’ve done”](#what-youve-done) You gave an agent a criterion in plain words. It found the steps, wrote the scenario, checked it and ran it, and you confirmed it by reading it. [Set up agents](/next/guides/set-up-agents/) connects other agents and Axx’s MCP server, and [Axx for agents](/next/explanations/axx-for-agents/) explains why Axx works this way. # Agent evaluations > How coding agents do with axx and without it, on the same tasks, held to the same checks, with bugs planted in the service. Coding agents write acceptance tests for a parcels service (REST and OpenAPI, PostgreSQL, MongoDB, Kafka, a downstream service mocked with WireMock), from acceptance criteria, as a developer would ask them to. Each task runs under five conditions: with axx and none of its aids (`none`), with its agent skills (`skills`), its MCP server (`mcp`), both (`both`), and without axx at all, tests of any kind run by a script (`plain`). Every suite is held to the same **core checks**: it passes against the correct service, twice and against variants that differ only where the contract allows, and it fails against every bug planted in the service. A suite that passes them all would catch those bugs in a real project. Everything is public in [nimbusxr/axx-evals](https://github.com/nimbusxr/axx-evals): the tasks, the planted bugs, the verifier, and every run’s transcripts. ## openai/gpt-6-luna [Section titled “openai/gpt-6-luna”](#openaigpt-6-luna) 2026-10-03, axx 0.1.12, opencode through OpenRouter ([the run](https://github.com/nimbusxr/axx-evals/actions/runs/37149658056)). ### With axx and without [Section titled “With axx and without”](#with-axx-and-without) The 6 tasks that have a version without axx (`kafka-registered-event`, `mock-address-check`, `mongo-tracking-view`, `openapi-reject-invalid`, `rest-crud-happy-path`, `sql-manifest-import`), each condition’s suites held to the same core checks: pass against the correct service twice and against its correct variants, and fail against every planted bug. **With axx, 72 of 72 suites passed every core check (100%)**, across none, skills, mcp, both; **without axx, 14 of 18 (78%)**. Without axx: 3 failed against the correct service, 1 failed against a correct variant. The agents wrote a median of 76 lines per suite with axx and 170 without. | On these tasks | none | skills | mcp | both | plain (no axx) | | ---------------------------------- | --------- | --------- | --------- | --------- | -------------- | | **Passed every core check** | **18/18** | **18/18** | **18/18** | **18/18** | **14/18** | | Failed against the correct service | 0 | 0 | 0 | 0 | 3 | | Failed against a correct variant | 0 | 0 | 0 | 0 | 1 | | Lines written per suite (median) | 60 | 86 | 80 | 83 | 170 | | Agent minutes per trial | 1.0 | 1.2 | 1.3 | 1.3 | 1.3 | | Requests learning axx per trial | 4.4 | 3.2 | 4.2 | 3.6 | 0.0 | | Cost per trial (¢) | 1.08 | 1.45 | 1.01 | 1.23 | 0.86 | | **Cost per passing suite (¢)** | **1.08** | **1.45** | **1.01** | **1.23** | **1.10** | Lines written are the lines of the files the agent added or changed: features and seeds with axx, test code and scripts without. Requests learning axx are those whose tool calls mostly read its steps, docs, skills or help. The cost per passing suite is what every trial cost, divided by the suites that passed every core check. The run’s report lists every failed suite and why. ### Scores by task [Section titled “Scores by task”](#scores-by-task) The share of each task’s trials that passed every check; the core score holds every condition to the same checks, and the score adds what only an axx suite has (`axx validate`, the features’ readability, `axx lint` where the task asks). | Task | none | skills | mcp | both | plain | | ------------------------ | --------- | --------- | --------- | --------- | -------- | | `fix-broken-feature` | 3/3 | 3/3 | 3/3 | 3/3 | - | | `init-first-feature` | 3/3 | 3/3 | 3/3 | 3/3 | - | | `kafka-registered-event` | 3/3 | 3/3 | 3/3 | 3/3 | 0/3 | | `mock-address-check` | 3/3 | 3/3 | 3/3 | 3/3 | 2/3 | | `mongo-tracking-view` | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | | `openapi-reject-invalid` | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | | `parallel-unique-data` | 3/3 | 3/3 | 3/3 | 3/3 | - | | `rest-crud-happy-path` | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | | `sql-manifest-import` | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | | **Core score** | **100.0** | **100.0** | **100.0** | **100.0** | **77.8** | | **Score** | **100.0** | **100.0** | **100.0** | **100.0** | **77.8** | # Axx for agents > The design choices that make Axx usable by coding agents - discoverable steps, validation without side effects, compact structured output, stable contracts, and docs served from the binary. Coding agents are now a primary user of test tools: they write most new tests, run them, and act on the result. Axx is designed for that user alongside people. The principle: an agent should never have to guess. ## Never guess a step [Section titled “Never guess a step”](#never-guess-a-step) Agents invent plausible step text, and Gherkin punishes near-misses. Axx makes the real steps cheap to find and cheap to check: * `axx steps` and the `steps_search` MCP tool without a query list every step of the project in one line each, which an agent reads once; `axx steps show ` and `step_explain` give one step’s documentation and examples. Searches return the steps that fit, briefly, and the MCP tool sends a step it returned before as its id only, so what an agent learns does not fill its context twice. * The skills send agents to `axx steps`, which lists the project’s steps, custom ones included, rather than shipping pages of them: agents read pages a chunk at a time, about ten times the tokens of `axx steps show `. * `axx validate` and `axx explain` check every line without starting anything, and suggest the closest real steps for a near-miss. * The `steps_try` MCP tool runs steps in a live scenario that stays open between calls, so an agent sees what a step does before it writes it down; the packs’ tools look at that scenario, such as the web-core pack’s `web_page`, which shows the page as the steps name its elements. * Step text is public API: it never changes, so what an agent learned stays true. ## Never guess what happened [Section titled “Never guess what happened”](#never-guess-what-happened) * **Exit codes** classify the outcome: a failed scenario (`1`), a broken setup (`2`), an invented step (`3`), a service that did not start (`4`). * **Error codes** (`AXX-Exxxx`) are stable and come with a hint and a link. * **Failures are data.** Assertion failures carry expected and actual values, the matched step definition, pack context such as the last HTTP exchange, and the exact command that reruns the scenario. * **`--json` everywhere**, in one envelope, with a frozen schema. ## Spend tokens on the problem [Section titled “Spend tokens on the problem”](#spend-tokens-on-the-problem) When Axx detects an agent and its output is not a terminal, it switches to compact output: failures and one summary line. A green run of a thousand scenarios costs one line of context. Details are one call away (`axx run --json`, or the `failure_context` MCP tool). ## Stay in sync with the installed version [Section titled “Stay in sync with the installed version”](#stay-in-sync-with-the-installed-version) Documentation drifts; binaries do not. The MCP server, `axx steps`, the skills and the reference pages on this site are all generated from the binary (ADR 0005), so an agent that asks the tool gets answers for the version in the repository, including that project’s custom packs. ## A fast, safe loop [Section titled “A fast, safe loop”](#a-fast-safe-loop) `axx up` keeps the system under test running between runs, so an agent’s edit-run loop costs seconds instead of minutes. `axx run features/x.feature:LINE` reruns exactly one scenario. Validation, explanation and step search have no side effects and need no running services. ## Guardrails in the instructions [Section titled “Guardrails in the instructions”](#guardrails-in-the-instructions) The skills and the AGENTS.md section encode the rules that keep agent-written tests honest: * one scenario per acceptance criterion, named after the behavior; * unique data in every scenario, and fixture factories where data repeats (optional, and `axx lint` hints at them); * assert on observable outcomes, never loosen an assertion to go green; * no fixed sleeps, use readiness checks and polling steps. ## Docs for machines [Section titled “Docs for machines”](#docs-for-machines) Every page on this site has a Markdown twin (append `.md`), advertised with ``. [`/llms.txt`](/next/llms.txt) indexes them, and [`/.well-known/agent-skills/index.json`](/next/.well-known/agent-skills/index.json) lists the skills. Each page also has *Copy as Markdown* and *Open in Claude* buttons. `axx init` sets up the skills and the MCP server for the agents a repository already uses, in the repository’s own files, and `axx doctor` says what an agent lacks. Read [Set up agents](/next/guides/set-up-agents/), and see it work in [Test with an agent](/next/tutorials/with-an-agent/). How much this helps is measured: [Agent evaluations](/next/explanations/agent-evals/) runs coding agents on the same tasks with axx and without it, against a service with planted bugs. # Black-box acceptance testing > Why Axx tests services from the outside, through the same interfaces their clients use, and what that buys you over in-process tests. An **acceptance test** checks that the system does what was agreed: the acceptance criteria of a story, written so the people who agreed on them can read the test. A **black-box** test checks it the way a client would, through the system’s public interfaces, with no knowledge of its insides. Axx does both. A scenario sends real requests to a running service, puts real rows in its real database, publishes real events to its broker, and checks what comes back and what the service did to the world. ## Why from the outside [Section titled “Why from the outside”](#why-from-the-outside) * **The test survives refactoring.** Rename a class, swap a framework, rewrite the service in another language: the scenarios still describe the same behavior and still run. * **It tests what ships.** The artifact under test is the one you deploy (a container, a binary), started the way you start it, configured the way you configure it. Serialization, framework wiring, database constraints and migrations are all in the loop. * **It needs no access to the code.** The service does not depend on Axx or on any test library. A team can write acceptance tests for a service in any language, and an agent can write them without reading the implementation. * **It reads like the requirement.** A scenario is the acceptance criterion in structured English. When it fails, the name says which promise broke. ## What it costs, and how Axx pays it [Section titled “What it costs, and how Axx pays it”](#what-it-costs-and-how-axx-pays-it) Black-box suites have a reputation for being slow and flaky. The causes are specific, and so are the fixes: | Cause | What Axx does | | ---------------------------------------- | -------------------------------------------------------------------------- | | Starting the system for every test class | starts services once per run, or once per session with `axx up` | | Tests run one at a time | runs scenarios in parallel by default | | Tests share and corrupt data | makes data isolation checkable with `axx lint` and fixture identities | | Waiting with fixed sleeps | readiness checks for services, polling steps for asynchronous results | | Brittle hand-written payloads | payloads from OpenAPI examples and fixture factories | | Opaque failures | expected and actual values, the last request and response, a rerun command | ## Where it fits [Section titled “Where it fits”](#where-it-fits) Black-box acceptance tests sit on top of unit and integration tests; they do not replace them. Use them for the behavior a client or another team relies on: the API contract, the events you publish, the data you persist, the calls you make to dependencies. Keep edge cases of pure logic in unit tests, where they are cheaper. ## Contracts in both directions [Section titled “Contracts in both directions”](#contracts-in-both-directions) A service has two contracts: the API it provides and the APIs it consumes. Axx checks both from the outside: * **Provided**: every request and response is validated against your OpenAPI document ([OpenAPI as contract](/next/explanations/openapi-contract/)). * **Consumed**: dependencies are replaced by WireMock mocks, which can themselves be validated against the provider’s OpenAPI document ([Mock dependencies](/next/guides/mock-dependencies/)). # How Axx works > What happens between typing axx run and reading the result - configuration, step matching, service lifecycle, parallel execution, packs, the world and reporting. Axx is one Go binary with its own Cucumber executor. It reads Gherkin, matches every step to a definition, starts the system under test, runs the scenarios against it from the outside, and reports. ```text axx.yaml ─┐ features ─┼─▶ load & validate ─▶ match steps ─▶ start services ─▶ run scenarios ─▶ report ───────────────▶ stop services packs ─┘ (registry) (lifecycle) (workers) (pretty, junit, html…) (cleanup) ``` ## 1. Load [Section titled “1. Load”](#1-load) Axx finds `axx.yaml` by walking up from the working directory, applies the selected profile and `axx.local.yaml`, expands `${env:...}` and `${sys:...}`, and validates the result against the [schema](/next/references/config/). It then parses the feature files with the official Cucumber Gherkin parser and applies the path, line, tag and name filters. ## 2. Match [Section titled “2. Match”](#2-match) Every step line is matched against the **registry**: the steps of the packs the project loads. Steps are [Cucumber Expressions](https://github.com/cucumber/cucumber-expressions) with custom parameter types such as `{ordinal}` and `{service}`. Optional segments like `[[ on {service}]]` register every variant of a step. A line that matches nothing is *undefined*; a line that matches two definitions is *ambiguous*. `axx validate`, `axx explain` and `axx run --dry-run` are this phase alone: they find both without starting a service or running a step. In a run, a scenario stops at an undefined or ambiguous step, and the report suggests the closest real steps or lists the candidates. ## 3. Start the services [Section titled “3. Start the services”](#3-start-the-services) The lifecycle manager starts `services` one after another in the order `axx.yaml` declares them, or, when services declare `dependsOn`, in dependency order with independent services in parallel. Each runs in its own process group. It polls every `ready` check until they pass or time out. With `active.enabled`, only the services the selected scenarios’ tags call for start, with the services they depend on. With `axx up`, a background supervisor keeps them running and later runs skip this step. ## 4. Run [Section titled “4. Run”](#4-run) Scenarios run in parallel on `run.workers` workers; steps within a scenario run in order. Scenarios tagged with a `run.exclusive` tag run one at a time after the parallel phase. Each scenario gets a fresh **world**: its registered services and the state each pack keeps (requests built and responses received, selections, events, browser pages). Nothing in the world is shared between scenarios, which is why parallel runs are safe as long as the *external* data is unique. See [Scenario isolation](/next/explanations/scenario-isolation/). ## 5. Packs [Section titled “5. Packs”](#5-packs) A **pack** is a bundle of steps, parameter types and hooks, its settings in `axx.yaml` and its tools for agents (`axx mcp`), plus the per-scenario context its steps share. The packs are `rest`, `mock`, `grpc`, `jsonrpc`, `graphql`, `sql`, `mongo`, `redis`, `kafka`, `amqp`, `mqtt`, `nats`, `websocket`, `sse`, `asyncapi`, `logs`, `files`, `mail`, `cli`, the web packs (`web-core`, `web-screenshots`…), the cloud service packs (`aws-s3`, `gcp-pubsub`, `azure-blob`…), and `core` for shared parameter types. A project lists the packs it uses in `axx-packs.yaml`. The `axx` binary is the core alone: on first use, it builds a copy of itself with the project’s packs, caches it and runs it in its place ([Choose packs](/next/guides/use-packs/)). ## 6. Report and stop [Section titled “6. Report and stop”](#6-report-and-stop) Reporters follow the run as it goes, as [Cucumber Messages](https://github.com/cucumber/messages) events and scenario results, and finish their reports after the last scenario: pretty, progress or compact on the console, TeamCity service messages for the editors’ test runners, and JUnit, HTML, Cucumber JSON, NDJSON messages and the JSON agent report to files. Then services are stopped (signal, grace period, kill) and every `cleanup` runs, including after a crash or an interrupt. The exit code summarizes the worst outcome ([exit codes](/next/references/error-codes/#exit-codes)). ## What Axx is not [Section titled “What Axx is not”](#what-axx-is-not) * **Not a unit test framework.** Axx tests a running system through its public interfaces. It never loads your code. * **Not a load-testing tool.** Parallelism is for speed, not for generating traffic. # OpenAPI as the contract > Why Axx treats your OpenAPI document as the single source of truth for an API, validates every exchange against it, and builds payloads from its examples. An API has one contract, and it should live in one place. In Axx that place is the OpenAPI document. Scenarios do not restate the contract; they rely on it. ## The document is the source of truth [Section titled “The document is the source of truth”](#the-document-is-the-source-of-truth) Some contract-testing approaches make the tests the source of truth and generate stubs or documents from them. Then the published OpenAPI document and the tested behavior can disagree, and nobody notices until a client breaks. Axx goes the other way: the document the service publishes is the contract, and the acceptance suite proves the service honors it. When a REST service is registered with an `openapi` property, every request a scenario sends and every response it receives is validated against that document. A scenario that checks only the status code still catches a missing required field, a wrong type or an undocumented status, because the validator checks everything the scenario did not. ## Examples are test data [Section titled “Examples are test data”](#examples-are-test-data) Documents already carry examples for their operations. Axx uses them as the starting payload (`a request payload using an application/json content example`), so a scenario lists only the values it is about. Two good effects follow: * The examples in your published documentation are exercised on every run, so they stay correct. * When a schema gains a field, you update the example once instead of every scenario. ## Strict by default, relaxed on purpose [Section titled “Strict by default, relaxed on purpose”](#strict-by-default-relaxed-on-purpose) Every validation rule starts at `ERROR`. A negative test, which sends an invalid request on purpose to check that the service rejects it, relaxes exactly the rule it breaks, in that scenario only: ```gherkin Given the OpenAPI validation levels are: | validation.request.body.schema.maximum | IGNORE | ``` That line comes from a scenario that sends a 31 kg parcel to check that the service refuses parcels over 30 kg: the request breaks the document’s `maximum`, and nothing else is relaxed. The relaxation is visible in the feature file, next to the behavior that needs it. Suite-wide relaxations in `axx.yaml` exist for documents you do not control; using them for your own contract defeats the point. ## Both sides of a dependency [Section titled “Both sides of a dependency”](#both-sides-of-a-dependency) A contract has two sides, and Axx checks both. Your service’s own document is checked by the REST steps: the provider side, where your service keeps its promise to its clients. The documents of the services your service calls are checked by the WireMock image that mocks them: the consumer side, where your service calls its dependencies correctly and the mocks answer as the real services would, so a stub cannot promise something the real API never does. The two sides have separate settings. Relaxing a rule of your own contract for a negative test says nothing about a dependency’s contract, and a stub that answers off-contract on purpose says nothing about yours. The provider validates its implementation against its document; the consumer validates its calls and mocks against the same document. Neither side needs the other’s tests. See [Validate against OpenAPI](/next/guides/validate-openapi/) for the steps and settings, and [Mock dependencies](/next/guides/mock-dependencies/) for the consumer side. # Scenario isolation and test data > Why Axx shares infrastructure between scenarios but never data, how that makes parallel runs safe, and the tools that enforce it. A fast black-box suite shares infrastructure: one database, one broker, one WireMock, one running service for hundreds of scenarios running in parallel. It stays correct only if scenarios never share **data**. Axx is built around that rule. ## What is isolated for you [Section titled “What is isolated for you”](#what-is-isolated-for-you) Each scenario has its own **world**: the services it registered, the requests it built and the responses it received, its selections and its events, and its browser pages, with their own cookies and storage. Nothing in the world leaks into another scenario, and a scenario always runs on a single worker. REST requests share connections, but no cookies: a request sends only the cookies its scenario gives it ([Send REST requests](/next/guides/send-rest-requests/#cookies)). ## What you isolate [Section titled “What you isolate”](#what-you-isolate) Everything outside Axx persists: rows in the database, documents in MongoDB, events on topics, requests in the WireMock journal, entries in logs, files in folders and buckets. Those are visible to every scenario, including ones running at the same moment and ones in the next run. So: 1. **Every scenario owns its data.** References, ids, keys and emails belong to one scenario: `PX-REG-1004`, not `test`. A scenario that registers a parcel for the sender `shop-example` and then counts that sender’s parcels will pass alone and fail beside any other scenario that does the same. 2. **Assert on your own data only.** Select rows by your ids; name mock request patterns by URLs that contain your ids; consume events by your keys. 3. **Do not rely on cleanup.** Data from earlier runs, failed and interrupted ones included, is still there. A scenario must pass beside everyone else’s data. A scenario that inserts a fixed id (a seed, a registration with a fixed reference) needs that id to be free, so it runs again only on a fresh environment: `axx run` starts from empty databases when the services’ `cleanup` removes their data (as `docker compose down -v` does), and against `axx up`, restart with `axx down` and `axx up`. ## Enforcing it [Section titled “Enforcing it”](#enforcing-it) Conventions drift, so Axx makes them checkable: * **`axx lint`** extracts ids from seeds, payloads and features and fails when a value appears where it must be unique ([Isolate test data](/next/guides/isolate-test-data/)). * **Fixture identities**: an `identity:` in a factory spec derives unique values for every fixture and generates the matching lint rule ([Fixture factories](/next/guides/fixture-factories/)). * **Randomized order**: `axx run --order random` exposes scenarios that only pass after another one ran. ## When sharing is unavoidable [Section titled “When sharing is unavoidable”](#when-sharing-is-unavoidable) Some scenarios change the environment for everyone: a database trigger that makes inserts fail, a global feature flag, a dependency that is stopped on purpose. Tag them with a tag listed in `run.exclusive` (for example `@isolated`). Axx runs them one at a time after the parallel phase, so they never overlap with another scenario ([Run in parallel](/next/guides/parallel-runs/)). ## Why not reset the database between scenarios [Section titled “Why not reset the database between scenarios”](#why-not-reset-the-database-between-scenarios) Truncating tables or restoring snapshots makes scenarios serial (a reset in one breaks another that is running), slow, and blind to problems that only appear with realistic data volumes. Unique data costs a naming convention and gives you parallel runs against a long-lived environment, including `axx up` sessions that last all day. # Step design > Why Axx steps are shaped the way they are - variants, parameter types, tables versus arguments - and why step text never changes. The steps of Axx’s packs follow a few rules, so once you know one pack you can guess the others, and an agent can too. ## One step, several variants [Section titled “One step, several variants”](#one-step-several-variants) The steps that address a service, a database or a broker come in up to four forms: the plain step, a form that names the service, a form with an ordinal for the 2nd or 3rd request, selection or event, and both together. The plain form covers the common case, a scenario with one service and one request, so most scenarios read like the acceptance criterion. The other forms appear only when a scenario needs them, and they read the same way in every pack ([How steps read](/next/references/steps/) has the grammar): ```gherkin Scenario: A reference can only be registered once Given a 1st ordered POST request to /api/parcels And a request payload using an application/json content example for 1st ordered request And the request payload property reference is 'PX-REG-1004' for 1st ordered request And a 2nd ordered POST request to /api/parcels And a request payload using an application/json content example named 'Express parcel' for 2nd ordered request And the request payload property reference is 'PX-REG-1004' for 2nd ordered request When the 1st ordered request is executed And the 2nd ordered request is executed Then the 1st ordered response status code is 201 And the 2nd ordered response status code is 409 ``` ## Parameter types [Section titled “Parameter types”](#parameter-types) Besides Cucumber’s built-in types (`{int}`, `{word}`, `{string}`, …), Axx adds types that let steps read naturally: `a 2nd selection`, `within 5s`, `on parcels`, `on parcels-db`. `{filepath}` marks a value that names a file, such as a seed or a schema, so editors can link it to the file. They are listed in [How steps read](/next/references/steps/#parameter-types). ## Tables for many values, arguments for one [Section titled “Tables for many values, arguments for one”](#tables-for-many-values-arguments-for-one) Steps that set or check one value take it inline (`the request header Accept is 'application/json'`); their plural twins take a two-column table (`the request headers are:`). Tables use JSONPath keys for payloads, so nested and array values need no extra steps. A table’s rows are its data: every row is a property and its value, or whatever its step’s columns say. It may start with a row naming those columns, as the reference shows them (`| JSONPath | value |`), or go straight to its rows; axx sets such a row aside, for every table of two columns or more. ## Step text is public API [Section titled “Step text is public API”](#step-text-is-public-api) Once a step is released, its text never changes. New behavior gets new steps; old steps can be deprecated with a pointer to the replacement, never renamed. That is what keeps feature files working across releases, and what lets an agent trust `axx steps search` today and next year. Every build checks that each step in a frozen catalog of step text is still defined, word for word. ## Designing your own steps [Section titled “Designing your own steps”](#designing-your-own-steps) The same rules make [custom steps](/next/guides/write-custom-steps/) easy to use: * Give every step a stable `ID`, its documentation (`Doc`) and a complete example line (`Examples`). * Add an optional `on` part that names the service, with its pack’s parameter type (`[[ on {dbService}]]`), when a step talks to a service that can appear twice. * Report failed expectations with expected and actual values, not just a message. # How the web-core pack works > How Axx drives real browsers - Playwright's browsers on your machine, a browser context for every scenario, steps that wait and find things the way people do - and why it works that way. The `web-core` pack runs acceptance scenarios in real browsers. It stands on [Playwright](https://playwright.dev), the browser automation Microsoft maintains for Chromium, Firefox and WebKit, and it keeps to what Axx is for: scenarios people can read, black-box tests of what your service does ([Black-box testing](/next/explanations/black-box-testing/)). ## The browsers run on your machine [Section titled “The browsers run on your machine”](#the-browsers-run-on-your-machine) The browsers run where Axx runs, as they do for any Playwright project: Playwright’s builds of Chromium, Firefox and WebKit, which the pack downloads the first time it uses each one, or the Google Chrome and Microsoft Edge installed on the machine. Axx drives them through the Playwright driver, Node.js and Playwright’s own package, which it downloads once and checks against pinned hashes. So: * The browsers reach your web apps at the addresses you would: `localhost`, whether their service runs in Compose or on your machine, under a debugger or not. * Watching a run is watching browser windows on your screen, with their DevTools, as when you use the app yourself. * Scenarios run the same everywhere: the pack pins one Playwright version, so its browsers are the same builds on every laptop and in CI, and it gives the browser a language and a time zone (`en-US` and `UTC`) unless the web app says otherwise, rather than the machine’s. ## Playwright’s tools, in steps [Section titled “Playwright’s tools, in steps”](#playwrights-tools-in-steps) Playwright’s own tools debug the scenarios: its Inspector, where a run pauses, its recorder and its trace viewer. They speak Playwright’s languages, and the pack teaches them the steps’ instead: the Playwright driver it downloads is patched, for the Playwright version it pins, so that * the Inspector shows the feature file, at the step a paused scenario waits before, and every browser action belongs to its step; * picking an element names it as steps do, `the "Weight (grams)" field`, and the Inspector finds what such a name finds; * the recorder writes the steps people take, with the steps of the pack; * a trace groups the browser’s actions by step, keeps the feature file, and names elements as steps do in the trace viewer your editor opens. What people see names what steps find: an element that the recorder can only find by its position or its markup has no name for people either, and the recording says so. ## Isolation [Section titled “Isolation”](#isolation) Playwright isolates tests in **browser contexts**: a context is a fresh browser profile, with its own cookies, storage, cache, permissions and pages, and nothing of one is visible to another. The pack keeps one browser per engine for the whole run, and gives every scenario a context of its own, for each web app it registers. Scenarios run side by side without seeing each other’s sessions, and a context is gone when its scenario ends ([Scenario isolation](/next/explanations/scenario-isolation/)). That is also why the pack has no way to sign in once and share the session with every scenario: shared, one scenario signing out would sign out the others, and scenarios would depend on each other again. A scenario signs in, or starts with its own copy of a session ([Sign in to web apps](/next/guides/web-sign-in/)). ## The current page [Section titled “The current page”](#the-current-page) A scenario acts on one page at a time, the current page, as a person looks at one tab. Opening a page makes it current; a link that opens a new tab makes the new tab current, as browsers show it; closing it goes back to the tab it came from. With several web apps, the one whose page was opened last is current. A dialog the page opens blocks the page, for Axx as for people: until a step answers it, other steps fail and say so, rather than hang or go on behind it. ## Waiting instead of sleeping [Section titled “Waiting instead of sleeping”](#waiting-instead-of-sleeping) Pages change after the fact: a price arrives, a message goes, a button is enabled once the form is valid. Every action waits for its element to be visible, enabled and still; every check retries until the page shows what it expects, 10 seconds or `within {duration}`. A step that has to wait longer than that says what the page shows instead, and a scenario never needs a pause. ## Names first, selectors when needed [Section titled “Names first, selectors when needed”](#names-first-selectors-when-needed) Steps find things the way people do: a field by its label, a button by its name, text anywhere. This keeps scenarios readable to the people who own the acceptance criteria, keeps them working when the markup changes, and makes accessibility part of the test: a field a step cannot find by its label is one a screen reader cannot name either. Steps look inside frames the same way, since people see no frames. Selectors (`css=`, `xpath=`, `testid=`) go where names are not enough: rows of a list, a canvas, a widget of a library with no accessible names. They take the name’s place in the same steps, so there is one set of steps to know. ## Secrets [Section titled “Secrets”](#secrets) Values a step takes from the environment, `${env:…}`, are secrets: passwords, tokens, session cookies. They are masked in failure messages, in the failure report, and in saved traces, where they would otherwise sit in the recorded actions, the snapshots of the page, and the form data and headers of its requests. Screenshots and videos show the page as it is. ## Packs that build on it [Section titled “Packs that build on it”](#packs-that-build-on-it) The `web-core` pack drives the browsers and holds the steps every web app needs. What only some projects need are packs of their own, which build on it as `aws-s3` builds on `aws-core`: a project adds the ones it uses ([Choose packs](/next/guides/use-packs/)). | Pack | | | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | | `web-screenshots` | pages that look as designed, each platform with its own screenshots ([Compare screenshots](/next/guides/web-screenshots/)) | | `web-a11y` | WCAG audits with axe-core, and what a screen reader reads ([Check accessibility](/next/guides/web-accessibility/)) | | `web-network` | what the pages fetch: requests that fail, answer or take their time, recordings, websockets ([Control what pages fetch](/next/guides/web-network/)) | | `web-lighthouse` | Lighthouse scores and the Core Web Vitals ([Audit pages with Lighthouse](/next/guides/web-lighthouse/)) | | `web-coverage` | how much of the web apps’ JavaScript the scenarios run ([Measure JavaScript coverage](/next/guides/web-coverage/)) | They use the pages the `web-core` pack opens, found the way its steps find things: a screenshot of `the "Price" element` is of the element the other steps name so. The tools some of them bring (axe-core, Lighthouse) are downloaded the first time, as the browsers are. ## What Playwright cannot do [Section titled “What Playwright cannot do”](#what-playwright-cannot-do) A few limits are Playwright’s (or the browsers’) and hold in the pack too: * Firefox has no mobile emulation: a `device` that is a phone needs `chromium` or `webkit`. * WebKit shows some downloads in the page, text files among them, unless their link has the `download` attribute. * Pages learn the browser’s location on secure addresses only (`https://`, or `localhost`). * Google Chrome and Microsoft Edge exist for x86-64 Linux only; on ARM Linux machines, use Chromium, Firefox and WebKit. # Check emails > Check the emails your services send, by their recipients, subject, text, HTML, attachments and headers, read from Mailpit or any mailbox over POP3 or IMAP; and the mail they send through a provider's API. Services email people: a label to a shop, a delivery notice, a password reset. Your service sends its mail as it does in production, by SMTP, to the mail server of your test environment, and the `mail` pack reads the mailbox that catches it. ```gherkin Scenario: A shop gets its parcel's label by email Given a seeds/mail-registered.yaml db seed And a POST request to /api/parcels And a request payload using an application/json content example And the request payload property reference is 'PX-MAIL-9701' And the request payload property sender is 'juniper-and-jay' When the request is executed Then the response status code is 201 And within 10s the shops mailbox has an email where: | to | orders@juniper-and-jay.example | | subject | Parcel PX-MAIL-9701 registered | | attachment | PX-MAIL-9701-label.zpl | ``` Add the pack to the project with `axx pack add mail` ([Choose packs](/next/guides/use-packs/)). ## Which mailbox [Section titled “Which mailbox”](#which-mailbox) How your service sends its mail decides where a test finds it: | Your service sends | In the tests | The check | | ---------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | | by **SMTP**, to a relay or a provider’s SMTP endpoint | a mail server that catches it: [smtp4dev](https://github.com/rnwood/smtp4dev), which can also stub refusals, or [Mailpit](https://mailpit.axllent.org), GreenMail, Inbucket | a mailbox with an `imap://` or `pop3://` `url`, or Mailpit’s, read through its API | | by SMTP, and it must really arrive | a real inbox, in a staging environment | a mailbox with an `imaps://` `url` | | through a **provider’s HTTP API** (SendGrid, SES, Mailgun, Postmark, Resend) | WireMock, against the provider’s contract | the [mock pack](/next/guides/mock-dependencies/): the request’s payload properties | Point your service’s SMTP settings at the mail server of the test environment, as the parcels example does in its Compose file: compose.yaml ```yaml services: mail: image: rnwood/smtp4dev:3.15.0 ports: ['1025:25', '1143:143', '8025:80'] # SMTP, IMAP, its web interface app: environment: PARCELS_SMTP_ADDR: mail:25 ``` ## Register the mailbox [Section titled “Register the mailbox”](#register-the-mailbox) ```gherkin Background: Given the shops mailbox with the following properties: | url | imap://${sys:local.host}:1143 | | username | axx | | password | axx | ``` * **`url`:** Mailpit’s address (`http://` or `https://`), or `pop3://`, `pop3s://`, `imap://` or `imaps://` and the mail server’s host. * **`username` and `password`:** sign in over POP3 and IMAP, or to a Mailpit behind a password. The password is masked. * **`folder`:** the IMAP folder, `INBOX` unless it says otherwise. smtp4dev takes any username and password over IMAP, and its `INBOX` has every email it received. Mailpit answers POP3 once it is given a password file (`--pop3-auth-file`, or `MP_POP3_AUTH`). ## Check an email [Section titled “Check an email”](#check-an-email) ```gherkin Then within 10s the shops mailbox has an email where: | to | orders@larkspur-lane.example | | subject | Parcel PX-MAIL-9702 delivered | | html | PX-MAIL-9702 was delivered in Leipzig | ``` | Row | What it checks | | --------------- | ---------------------------------------------- | | `to`, `cc` | an address the email went to, or was copied to | | `from` | the address it came from | | `subject` | its subject, all of it | | `text` | text its plain-text body contains | | `html` | text its HTML body contains, tags and all | | `attachment` | the file name of an attachment | | `header ` | a header’s value | Addresses compare without regard to case, and `text` and `html` take runs of spaces and line breaks as one space. Subjects and names in other charsets are decoded first. The check waits for an email that meets every row: 10 seconds, or the time `within` gives. When none does, it lists the emails that arrived. ## Refuse mail [Section titled “Refuse mail”](#refuse-mail) What does your service do when a mail server refuses its email? Stub the refusal in the mail server, as you stub an HTTP dependency’s failures in WireMock’s mappings: by the data a scenario uses. smtp4dev runs a JavaScript expression for each recipient, and `error(code, message)` refuses it with that SMTP code: compose.yaml ```yaml services: mail: image: rnwood/smtp4dev:3.15.0 environment: # quince-and-quill's mail server is busy: it asks to be tried again later. ServerOptions__RecipientValidationExpression: >- recipient.toLowerCase().endsWith("@quince-and-quill.example") ? error(451, "Mailbox busy, try again later") : true ``` A scenario whose shop is quince-and-quill then sees how your service copes: ```gherkin Scenario: A registration email the shop's mail server refuses does not stop the registration Given a seeds/mail-refused.yaml db seed And a POST request to /api/parcels And a request payload using an application/json content example And the request payload property sender is 'quince-and-quill' When the request is executed Then the response status code is 201 ``` * A 4xx code, like 451, is a failure the sender should retry; a 5xx, like 550, one it should not. * The stub refuses only the recipients it names, so scenarios running beside it are not affected: give each refusal a recipient of its own. * `CommandValidationExpression` refuses at other stages, such as the sender’s `mail from:` (smtp4dev passes the command’s verb in lowercase), and `MessageValidationExpression` refuses whole messages. ## Keep scenarios apart [Section titled “Keep scenarios apart”](#keep-scenarios-apart) A check only looks at the emails that arrived since its scenario started, and axx never deletes or marks what it reads. Scenarios running in parallel share the mailbox, so each checks its own mail by a recipient or a subject unique to it, such as a shop or a parcel reference of its own. For a run, axx reads each mailbox the checks name from when the services are up: * **Mailpit and IMAP** say when each email arrived. * **POP3** doesn’t, so axx takes what the mailbox held when it started reading as old. See the [mail pack’s reference](/next/references/packs/mail/) for every step. # Check files > Check the files your services write to a folder or to cloud storage, whether they are CSV, JSON, PDF, Word or Excel files, by their exact content, their JSON properties, their text or the rows of their table. Services write files for other systems and for people: an import report that a shop’s system collects, a workbook for its staff, a handover note the driver signs. The `files` pack checks the files a service writes to a folder, such as an export directory or a volume it shares with Axx. The storage packs have the same checks for the objects in a bucket or a container. ```gherkin Scenario: The import report says what became of every line Given a seeds/manifest-heron.yaml db seed Then within 10s the exports folder has a file named manifests/M-HERON-0501/report.csv And the manifests/M-HERON-0501/report.csv file in the exports folder has a row where: | line | ML-HER-0501-3 | | status | REJECTED | | reason | unknown service level | ``` Add the pack to the project with `axx pack add files` ([Choose packs](/next/guides/use-packs/)). ## Register the folder [Section titled “Register the folder”](#register-the-folder) Register the folder under a name, usually in the `Background`. Its `path` is relative to the directory of `axx.yaml`, or absolute, and it can use `${env:..}` and `${sys:..}`: ```gherkin Background: Given the exports folder with the following properties: | path | ../infra/exports | ``` The folder may appear only when the service first writes to it. A service in a container writes to a folder of the host through a bind mount: infra/compose.yaml ```yaml services: exports: image: busybox:1.37 volumes: ['./exports:/exports'] command: ['sh', '-c', 'rm -rf /exports/* && chmod 777 /exports'] app: build: ../app environment: PARCELS_EXPORT_DIR: /var/lib/parcels/exports volumes: ['./exports:/var/lib/parcels/exports'] depends_on: exports: condition: service_completed_successfully ``` A folder keeps what earlier runs wrote to it, so a check could pass on a file from the run before. The `exports` service empties the folder before the service starts, and lets a service that runs as a non-root user write to it. Name the files you check after data unique to the scenario, such as a manifest ID: scenarios run in parallel. ## Put files in a folder [Section titled “Put files in a folder”](#put-files-in-a-folder) A scenario can put files in a folder, as a service or an app would find them there: a copy of a file of the project, or a file with the content of a doc string, where `${env:..}` and `${sys:..}` are expanded. The folders a file is in are made, and a file already there is replaced. ```gherkin Given the incoming/M-HERON-0501.csv file in the imports folder is a copy of the manifests/M-HERON-0501.csv file And the incoming/M-HERON-0501.json file in the imports folder has the content: """json {"manifest": "M-HERON-0501", "shop": "heron-garden", "lines": 3} """ ``` `is emptied` removes everything in a folder, hidden files and subfolders too, and keeps the folder: ```gherkin Given the imports folder is emptied ``` It empties a folder of the project, or one a service or an app owns; a folder elsewhere on the machine is a person’s, and the step refuses it. Scenarios run in parallel, so empty a folder only one scenario uses at a time, such as an app’s: an app’s files are its own in each scenario. ## Check a file [Section titled “Check a file”](#check-a-file) A check names a file by its path in the folder, with `/` between folders, and without spaces. Every check waits for the file to be there and to meet the check: 10 seconds, or the time `within` gives. A `*` in the path stands for any characters but a slash, for a file named by something a check cannot know, like the time it was saved: `"snap-*.json"` names the one file it matches, and a check waits while none or several do. ```gherkin Then within 10s the exports folder has a file named manifests/M-HERON-0501/report.csv And the manifests/M-HERON-0501/report.csv file in the exports folder is identical to the reports/M-HERON-0501.csv file And the manifests/M-HERON-0501/summary.json file in the exports folder has the following properties: | manifest | M-HERON-0501 | | imported | 2 | | rejected | 1 | And the manifests/M-WREN-0503/handover.pdf file in the exports folder contains "Parcels to collect: 2" ``` * `is identical to` compares the file byte for byte with a file of the project, found through `resources`. * `has the following properties:` reads JSON, like the other JSON property steps: a path and the value as text, `null` for null and `undefined` for absent. * `contains` reads the file’s text, whatever its type (below). ## Check what a service takes away [Section titled “Check what a service takes away”](#check-what-a-service-takes-away) A service that imports a file often moves or deletes it, and an app clears what it has dealt with. `has no file named` checks that a file is gone, and `is empty` that a folder has no files, in its subfolders too. Hidden files, like `.DS_Store`, are not counted, and a folder that is not there is empty. Both wait for the files to go: 10 seconds, or the time `within` gives. ```gherkin Then within 10s the imports folder has no file named incoming/M-HERON-0501.csv And the imports folder is empty ``` A check that something is gone passes too when nothing happened at all, so check what the service did as well, such as the report it wrote. `axx lint` names the scenarios whose only checks are of what is gone. ## The text of PDF, Word and Excel files [Section titled “The text of PDF, Word and Excel files”](#the-text-of-pdf-word-and-excel-files) `contains` reads a file by its type, which its extension tells (or else its content): | File | Its text | | -------------------------------------- | -------------------------------------------------------------------- | | `.pdf` | the text of its pages, line by line, as the pages show it | | `.docx` | its paragraphs, its tables (a line per row), its headers and footers | | `.xlsx` | the cells of every sheet, a line per row, as the workbook shows them | | `.xml`, `.html` | the text between the tags; the markup counts too | | `.csv`, `.json`, `.txt` and other text | the text itself | Case matters. Runs of spaces and line breaks count as one space, so a check does not depend on how a PDF spaces its columns: ```gherkin Then the manifests/M-WREN-0503/handover.pdf file in the exports folder contains "PX-WRN-5201 Marie Curie, 01067 Dresden 1200 g STANDARD" ``` A PDF of scanned images has no text to read, and neither has a PDF that needs a password to open. Older `.doc` and `.xls` files are not read. ## Rows of CSV files and workbooks [Section titled “Rows of CSV files and workbooks”](#rows-of-csv-files-and-workbooks) `has a row where:` reads the file as a table whose first row names the columns: a CSV file (with commas or semicolons), a TSV file, or an Excel workbook (its first sheet). One row must have every value of the step’s `| column | value |` table: ```gherkin Then within 10s the manifests/M-OSPREY-0502/report.xlsx file in the exports folder has a row where: | line | ML-OSP-0502-2 | | reference | PX-OSP-5102 | | status | IMPORTED | ``` Cells compare as text, as the workbook shows them: a price formatted with two decimals is `6.90`, not `6.9`. An empty value matches an empty cell. ## Images, compared with their screenshots [Section titled “Images, compared with their screenshots”](#images-compared-with-their-screenshots) `looks like the … screenshot` compares an image a service or an app saves (PNG or JPEG) with its screenshot, pixel by pixel, anti-aliasing aside: ```gherkin Then within 10s the manifests/M-PLOVER-0504/labels/PX-PLV-5301.png file in the exports folder looks like the "label-PX-PLV-5301" screenshot ``` The first time, there is no screenshot: the step takes it and fails, so that you look at it and keep it (`screenshots/label-PX-PLV-5301.linux.png`). Each platform has its own, named after it, since apps draw their images their own way on each. When the image differs, the step attaches the screenshot, the image and their difference. `packs.files.screenshots` in `axx.yaml` sets their `folder` (default `screenshots`), the share of pixels that may differ (`tolerance`), whether to take them all again (`update`), and the `platforms` a project keeps them for. ## When a check fails [Section titled “When a check fails”](#when-a-check-fails) A failed check says what the file holds instead: the lines of its text nearest the text you expected, the rows of its table, or the columns it has. ```text The manifests/M-HERON-0501/report.csv file in the exports folder did not meet the expectation within 10s: no row has line=ML-HER-0501-3, status=IMPORTED. Its 3 rows are: line=ML-HER-0501-1, status=IMPORTED line=ML-HER-0501-2, status=IMPORTED line=ML-HER-0501-3, status=REJECTED ``` ## Objects in cloud storage [Section titled “Objects in cloud storage”](#objects-in-cloud-storage) The storage packs, `aws-s3`, `gcp-storage` and `azure-blob`, have the same checks for their objects ([Test cloud services](/next/guides/test-cloud-services/)): ```gherkin Then within 30s the invoices/kestrel-2026-09.pdf object in the carrier-invoices gcs bucket contains "Total due: 1284.50 EUR" And the disputes/kestrel-2026-09.csv object in the carrier-invoices gcs bucket has a row where: | parcel | PX-KES-1001 | | reason | weight differs | ``` # Check logs > Prove that something did not happen by asserting the log entry that says so, reading logs from files or receiving them over UDP, TCP or HTTP. It is hard to prove that something did not happen: waiting and seeing nothing proves only that nothing arrived yet. Flip it to a positive. Have your service log its decision (“registration refused; not announced”, “line ML-KES-0413-1 rejected: duplicate reference”) and assert that entry. The `logs` steps read what your services log, whether it lands in a file or is sent over the network. ## Register a log [Section titled “Register a log”](#register-a-log) A log is where a service’s lines are. Register it in the `Background`, with a `url`: ```gherkin Background: Given the parcels log with the following properties: | url | udp://0.0.0.0:5140 | And the console log with the following properties: | url | file://.axx/logs/services.log | ``` | url | Axx | | ------------------------------------------------------- | ----------------------------------------------------------------------------------------- | | `file:///var/log/app.log`, `file://logs/app.log` | reads what is appended to the file; a relative path is relative to `axx.yaml` | | `udp://0.0.0.0:5140` | listens; each datagram is one or more lines | | `tcp://0.0.0.0:5150` | listens; newline-delimited or octet-counted syslog (RFC 6587) messages | | `http://0.0.0.0:5160/logs`, `https://0.0.0.0:5161/logs` | listens; the body of each POST or PUT to that path (https uses a self-signed certificate) | A log only counts what it receives after the scenario registers it. Scenarios run in parallel and share logs, so match on data unique to the scenario: a reference, an order id. For the network schemes Axx is the log service: your service, or the thing that ships its logs, sends lines to Axx. Axx opens those listeners before it starts the services, for the log steps of the scenarios in the run, so services can connect and send from the moment they start; `axx up` keeps them open between runs. Containers reach them at `host.docker.internal` (on Linux, add `extra_hosts: ["host.docker.internal:host-gateway"]` to the service). ## Point your logs at Axx [Section titled “Point your logs at Axx”](#point-your-logs-at-axx) **The console of the services Axx starts** needs nothing: Axx writes it to `.axx/logs/services.log`, every compose container’s lines prefixed with its name, so `file://.axx/logs/services.log` reads it. **A log file** a service writes, for example to a mounted volume: infra/compose.yaml ```yaml app: volumes: - ./logs:/var/log/parcels ``` ```gherkin Given the parcels log with the following properties: | url | file://../infra/logs/app.log | ``` **Syslog over UDP or TCP.** A service’s syslog handler (Python’s `SysLogHandler`, logback’s `SyslogAppender`) can send to `host.docker.internal:5140`, and so can Docker for any container’s console, without changing the service: infra/compose.yaml ```yaml app: logging: driver: syslog options: syslog-address: tcp://host.docker.internal:5150 ``` The example’s service sends every log line as a UDP datagram when `PARCELS_LOG_UDP` is set, which is what its `parcels` log receives. **A log forwarder over HTTP**, such as Fluent Bit or Vector posting to Axx: infra/fluent-bit.conf ```ini [OUTPUT] Name http Match * Host host.docker.internal Port 5160 URI /logs Format json_lines ``` ## Assert entries [Section titled “Assert entries”](#assert-entries) Patterns are regular expressions (Java syntax), searched in the log’s text rather than matched against whole lines. Every step waits for its entries: 10 seconds, or the time `within` gives. ```gherkin Then the parcels log has an entry matching 'msg="registration refused; not announced" reference=PX-EVT-4003' Then within 30s the parcels log has an entry matching 'depot notified reference=PX-DSP-2001' ``` Several entries in one log; each row needs a match of its own, so the same pattern twice needs two matches: ```gherkin Then the console log has entries matching: | msg="manifest line processed" line=ML-FJORD-0101-1 status=REJECTED reason="weight exceeds 30000 g" | | msg="manifest line processed" line=ML-FJORD-0101-2 status=REJECTED reason="unknown service level" | ``` A number of matches, for example one per retry; more than that fails: ```gherkin Then the parcels log has 2 entries matching 'msg="storing parcel failed, retrying" reference=PX-DBF-3002' ``` Entries in different logs, for example your service’s decision and the call its dependency received: ```gherkin Then the logs have entries matching: | parcels | msg="registration refused; not announced" reference=PX-ADR-1104 | | console | GET /v1/postcodes/DE/00012 | ``` ## Multi-line entries [Section titled “Multi-line entries”](#multi-line-entries) Patterns match across the whole text, so an entry can span lines, such as a stack trace. `^` and `$` match at the start and end of each line, `\n` matches a line break, and `(?s)` lets `.` match line breaks too: ```gherkin Then the parcels log has an entry matching '(?s)registration failed reference=PX-7.*IllegalStateException: address service unavailable' ``` In a table cell, Gherkin turns `\n` into a real line break, which matches a line break just the same. A log that gathers several services’ output, such as `.axx/logs/services.log`, can put another service’s lines between the lines of an entry, since each service writes on its own. There, match a single line of the entry. When a step fails, it shows what the log received since the scenario registered it, so you can see what your service said instead. # Configure services > Register the REST APIs, mocks, databases and brokers a scenario talks to, name them, and keep URLs and credentials out of feature files. A **service** is something a scenario talks to: your REST API, a WireMock server, a SQL or MongoDB database, a Kafka cluster. Scenarios register services by name, usually in a `Background`, and later steps use them. ## Register services in a Background [Section titled “Register services in a Background”](#register-services-in-a-background) ```gherkin Feature: Register parcels Background: Given the parcels service with the following properties: | url | http://localhost:8400 | | openapi | http://localhost:8400/openapi.json | And the mocked addresses service with the following properties: | url | http://localhost:8081 | And a parcels-db database with the following properties: | url | postgres://localhost:5432/parcels | | user | parcels | | password | parcels | And a tracking-db mongo database with the following properties: | url | mongodb://localhost:27017/parcels?authSource=admin | | user | parcels | | password | parcels | And the events kafka service with the following properties: | brokers | localhost:9092 | ``` Each kind of service has its own registration step and properties, listed on its pack’s page: [REST](/next/references/packs/rest/) (an `openapi` property turns on [validation](/next/guides/validate-openapi/)), [Mocks](/next/references/packs/mock/), [SQL](/next/references/packs/sql/), [MongoDB](/next/references/packs/mongo/) and [Kafka](/next/references/packs/kafka/). ## The default service and named services [Section titled “The default service and named services”](#the-default-service-and-named-services) The first service of a type registered in a scenario is the default. Steps without a service name use it: ```gherkin Given a GET request to /api/parcels/PX-REG-1001 When the request is executed Then the response status code is 200 ``` When a scenario uses two services of the same type, name the one you mean with the `on ` form of the step: ```gherkin Given a GET request to /api/parcels/PX-REG-1001 on parcels When the request is executed on parcels Then the response status code is 200 on parcels ``` Most steps that address a service have both forms (a few mock steps always name it); see [How steps read](/next/references/steps/#services). ## Keep values out of feature files [Section titled “Keep values out of feature files”](#keep-values-out-of-feature-files) Values in service tables and in `axx.yaml` are interpolated ([syntax](/next/references/config/#interpolation)). Put what differs between machines, and every secret, in properties and environment variables: axx.yaml ```yaml properties: local.host: localhost db.password: ${env:PARCELS_DB_PASSWORD:-parcels} ``` ```gherkin Given a parcels-db database with the following properties: | url | postgres://${sys:local.host}:5432/parcels | | user | parcels | | password | ${sys:db.password} | ``` Override a property for one run with `axx run -D local.host=docker`. ## Differences between environments [Section titled “Differences between environments”](#differences-between-environments) Use a **profile** for settings that differ in CI or on another machine: axx.yaml ```yaml profiles: ci: properties: local.host: docker ``` Select it with `axx run --profile ci` or `AXX_PROFILE=ci`. A profile can also live in its own file, and personal overrides in an `axx.local.yaml`; [Finding and merging files](/next/references/config/#finding-and-merging-files) has the order. ## Resource paths [Section titled “Resource paths”](#resource-paths) Steps that take a file (`seeds/manifest-kestrel.yaml`, `kafka/scan-delivered.json`, `schemas/depot-scan.avsc`) look for it next to the feature file first, then in the `resources` directories in order, then in the directory of `axx.yaml`. A feature keeps the files only it uses beside it ([Organize a suite by feature](/next/guides/organize-by-feature/)), and files several features share go in a `resources` directory: axx.yaml ```yaml resources: [acceptance, ../shared-fixtures] ``` If a file is not found, the step fails with [`AXX-E0301`](/next/references/error-codes/#axx-e0301) and lists the directories it searched. # Debug failures > Go from a red axx run to the cause - read the exit code, rerun one scenario, explain a step, read expected and actual values, find service logs, and fix flaky parallel scenarios. ## Start from the exit code [Section titled “Start from the exit code”](#start-from-the-exit-code) | Code | What happened | First move | | ----- | ------------------------------- | ----------------------------------------------------------------------------------------- | | `1` | a scenario failed | read the failing step’s expected and actual values | | `2` | usage or configuration error | read the `hint`; config errors point at `axx.yaml:line:col`; run `axx doctor` | | `3` | undefined or ambiguous step | `axx validate`, then `axx explain ""` | | `4` | a service did not start or stop | the error shows the service’s last output lines; the full log is `.axx/logs/services.log` | | `130` | interrupted | the run was cancelled (Ctrl-C); run again | Every error also has a stable code (`AXX-E0408`, say) that links to its entry in the [error code reference](/next/references/error-codes/). ## Read the failure [Section titled “Read the failure”](#read-the-failure) ```console $ axx run ... Scenario: The service says hello # features/smoke/smoke.feature:7 ✓ Given the hello-axx service with the following properties: (background) | url | http://${sys:local.host}:8000 | ✓ Given a GET request to /hello.json ✓ When the request is executed log: GET http://localhost:8000/hello.json -> 200 OK (2ms, 26 bytes) attachment: response body (application/json, 26 B) {"message": "Hello, axx"} ✓ Then the response status code is 200 ✗ And the response payload property message is 'Hello, world' Response payload property message is not Hello, world expected: "Hello, world" actual: "Hello, axx" Failed scenarios: ✗ The service says hello # features/smoke/smoke.feature:7 rerun: axx run features/smoke/smoke.feature:7 ``` Each failure names the step, the expected and actual values, and the command that reruns only that scenario. Steps that talk to a service attach what they sent and received. For the same failure as data, with the last request and response, use `axx run --json` ([the run report](/next/references/json-output/#the-run-report)). ## Rerun one scenario [Section titled “Rerun one scenario”](#rerun-one-scenario) ```sh axx up # keep the services running while you iterate axx run features/register-parcels.feature:17 # the scenario on (or containing) line 17 ``` The line of any step inside the scenario works too. ## See what a web scenario did [Section titled “See what a web scenario did”](#see-what-a-web-scenario-did) A failed scenario of the [`web-core` pack](/next/guides/test-web-apps/) keeps a Playwright trace: each step, its actions, the page before and after them, and the network. In the editor, open it from the scenario’s test ([Open traces and videos](/next/guides/set-up-your-editor/#open-traces-and-videos)). To look at the page where the scenario fails, run it again with *Debug* in the editor: it pauses there, in Playwright’s Inspector ([Pause a scenario](/next/guides/watch-web-browsers/#pause-a-scenario)). From a terminal: ```sh axx run --workers 1 --set packs.web-core.pauseOnFailure=true features/shop-portal.feature:21 ``` *Watch* shows the page as the scenario runs, without pausing ([Watch a run](/next/guides/watch-runs/)). ## Undefined and ambiguous steps [Section titled “Undefined and ambiguous steps”](#undefined-and-ambiguous-steps) ```console $ axx explain 'the mocked request named postcode-check was received 2 times' undefined: no step matches did you mean: the mocked request named {word} was received exactly {int} time(s) (mock.count.exactly) the mocked request named {word} was received at least {int} time(s) (mock.count.atLeast) the mocked request named {word} was received at most {int} time(s) (mock.count.atMost) ... search all steps with `axx steps search ` ``` * **Undefined**: the text differs from every step. Use the suggestion or `axx steps search`. Look for extra spaces, `a` versus `an`, singular versus plural, and missing quotes around `{string}` values. An ordinal goes where each step has it: `the 2nd ordered response status code is 200`, but `the response payload property status is 'REGISTERED' for 2nd ordered response`; a line with it in the other place gets the right one as its first suggestion. Never change a step definition to match your text. * **Ambiguous**: two steps match. Make the line more specific, usually by naming the service with `on `. ## Common causes [Section titled “Common causes”](#common-causes) * **Expected X, got Y, and the product looks right.** Another scenario running at the same time may have used the same ids. Check that the data is unique (`axx lint`), then check the value’s type: the step’s entry on its [pack’s page](/next/references/packs/) says how it reads values (`'5'` and `'"5"'` differ). * **OpenAPI validation error on `the request is executed`.** The request or the response violates the document; the message names the rule, such as `validation.response.body.schema.required`. Fix the payload or the service. For a deliberate negative test, relax that rule in the scenario ([Validate against OpenAPI](/next/guides/validate-openapi/#validation-levels)). * **Timeout.** A step exceeded `run.timeouts.step`. For asynchronous behavior use a polling step (`within 10s a selection of at least 1 row ...`) instead of a sleep or a longer timeout. * **Passes alone, fails in the full run.** Shared data or shared state. Make the data unique; if the scenario really must run alone, tag it for `run.exclusive` ([Run in parallel](/next/guides/parallel-runs/)). * **A service never becomes ready.** Check `services..ready` (URL, port, timeout). Run `axx up`, then `curl` the health URL yourself, and read `.axx/logs/services.log`. ## Debug the service itself [Section titled “Debug the service itself”](#debug-the-service-itself) Set a breakpoint in your service under test and run the scenario against it: ```sh axx run --attach parcels features/register-parcels.feature:17 # you start parcels from your IDE; axx waits for it axx run --debug=parcels features/register-parcels.feature:17 # axx starts parcels with its debug command ``` [Manage the services under test](/next/guides/manage-services/#debug-a-service) shows how to configure `services..debug`. ## Stop in step code [Section titled “Stop in step code”](#stop-in-step-code) To see exactly what a step does, why it fails, or what your service sent back, set breakpoints in the step’s own Go code, whether it comes from one of Axx’s packs or from your own, and step through it: ```sh axx run --debug-steps features/register-parcels.feature:17 ``` Axx builds itself with debug information (the first time; later runs reuse the build), starts under [Delve](https://github.com/go-delve/delve), Go’s debugger, and waits for a debugger on port 2345 (`--debug-steps=` picks another). The scenario starts when one attaches. Step timeouts are off while you debug, and the exit code is the run’s as usual. * **GoLand, or IntelliJ IDEA with the Go plugin:** click the gutter icon of a scenario and choose *Debug*. The axx plugin attaches the Go debugger for you. From a terminal run, start the *Debugger: axx-steps* configuration (the axx plugin creates it the first time you debug, and `axx ide intellij` writes it). * **VS Code with the Go extension:** use the debug button of a scenario in the gutter or the Testing view. To attach to a run you started in a terminal, use *axx: attach to steps* (written by `axx ide vscode`). * **Anything else:** `dlv connect 127.0.0.1:2345`. Jump from a step in a feature file to its code with go-to-definition ([Set up your editor](/next/guides/set-up-your-editor/)). It opens the Go code that defines the step when that code is on your machine (your own packs, or Axx built from source), otherwise the step’s entry in a reference page. There is nothing to install: Axx builds itself and Delve with the Go toolchain it downloads, as it does for packs, and keeps them for later runs. To use a Delve of your own, set `AXX_DLV` to its path. ## Logs [Section titled “Logs”](#logs) | File | Contents | | -------------------------- | --------------------------------------------------------------------------- | | `.axx/logs/services.log` | output of every service Axx started | | `.axx/logs/supervisor.log` | what the background process of `axx up` logged; read it when `axx up` fails | Add `-v` or `-vv` to any command for more detail from Axx itself. ## Don’t [Section titled “Don’t”](#dont) * Don’t loosen an assertion to make a test pass. Find out whether the product is wrong first. * Don’t add fixed sleeps. Use readiness checks and polling steps. * Don’t edit generated files (`axx-lint.generated.yaml`, generated fixtures); change their source and regenerate. # Fixture factories > Generate schema-valid seeds, event payloads and mock bodies from a compact factory spec with axx fixtures, and catch drift when a schema changes. Suites accumulate dozens of fixture files that share one schema: Kafka payloads for one Avro record, seed datasets for one set of tables, mock bodies for one API. Most of each file is boilerplate, and the day the schema gains a required field, every file needs the same edit. A **factory** keeps the shared shape once and each fixture as only its differences. `axx fixtures` expands them into ordinary files. ## The files [Section titled “The files”](#the-files) Everything lives next to the fixtures it produces: ```text kafka/ depot-scans.factory.yaml # family, schema, identities depot-scans.prototype.yaml # the shared shape scan-delivered.fixture.yaml # one fixture: only its differences scan-out-for-delivery.fixture.yaml scan-delivered.json # generated scan-out-for-delivery.json # generated ``` kafka/depot-scans.factory.yaml ```yaml # yaml-language-server: $schema=https://axx.nimbusxr.us/schemas/v0/axx-factory.schema.json factory: family: avro schema: ../schemas/depot-scan.avsc # relative to this file identity: - path: scanId # unique across every fixture ``` kafka/depot-scans.prototype.yaml ```yaml data: parcelRef: PX-EXAMPLE-1 location: Leipzig scannedAt: "2026-05-06T10:15:00Z" ``` kafka/scan-delivered.fixture.yaml ```yaml data: scanId: SC-FIXTURE-DELIVERED status: DELIVERED ``` The fixture’s file name is its name, and `scan-delivered.json` is what features reference, exactly as if you had written it by hand: kafka/scan-delivered.json (generated) ```json { "scanId": "SC-FIXTURE-DELIVERED", "parcelRef": "PX-EXAMPLE-1", "status": "DELIVERED", "location": "Leipzig", "scannedAt": "2026-05-06T10:15:00Z" } ``` ## Identities [Section titled “Identities”](#identities) An `identity:` entry names a field that tells fixtures apart, like `scanId` above. A fixture that gives it a value keeps that value; one that leaves it out gets one derived from its name (`prefix:` goes in front, and `format: uuid-name-based` makes it a UUID). Generation fails when two fixtures, of any factories, have the same value in identity fields of the same name, and names both fixtures. * **Fields of different names never collide.** A shop’s `key` and a pickup’s `id` can both be `alder-stationery`. So can two identity fields of one fixture: each identity is checked on its own. * **A namespace groups identities by what they identify.** Give identities a `namespace:` when their field names say too little. Different namespaces keep fields of one name apart: a depot’s `id` and a shop’s `id` can both be `leipzig`. One namespace compares fields of different names: a SQL seed’s `shops.shop_id` and a registration payload’s `shopId` that both create shops. depots/depots.factory.yaml ```yaml identity: - path: id namespace: depots # a depot's id may equal a shop's ``` Before namespaces, identity values were unique across every identity field of every factory. To keep that, give every identity the same namespace. ## Reference other fixtures [Section titled “Reference other fixtures”](#reference-other-fixtures) A value can be another fixture, or part of one, with `$ref`: the path of its `*.fixture.yaml` (or `*.factory.yaml`) file, relative to the file that refers to it (a leading `/` starts at the directory of `axx.yaml`, or `fixtures.baseDir`), then `#` and a JSON Pointer into it. seeds/handover.fixture.yaml ```yaml data: parcel: $ref: ../parcels/express-berlin.fixture.yaml # the whole fixture parcelId: $ref: ../parcels/express-berlin.fixture.yaml#/id # one value of it ``` A reference sees the fixture as it is generated: its prototype, its defaults, and the identities derived for it. Keys next to `$ref` override what it brings. References that loop fail and name the loop; a path that names no fixture says what it was resolved to, and the fixtures it most likely meant. ## Generate and check [Section titled “Generate and check”](#generate-and-check) ```sh axx fixtures generate # write the fixture files, the manifest and lint rules axx fixtures check # CI: regenerate in memory and compare, without writing ``` * **Generation is a development-time step.** Nothing is generated during `axx run`; features cannot tell a generated fixture from a hand-written one. * **Output is deterministic.** The same spec always produces the same bytes, and every output is validated against the schema (the Avro schema here, so a `status` that is not one of its symbols fails) before it is written. * **The tool only touches files it owns.** `axx-fixtures.manifest.yaml` records them with their checksums. A managed file that was edited by hand is refused, never overwritten: move the change into the spec. * **Identities become lint rules.** Generation refuses two fixtures that claim one identity value (`AXX-E0902`), and a fixture that omits one gets a value derived from its name, so it is unique by construction. Each `identity:` entry also produces a rule in `axx-lint.generated.yaml`, which `lint.include` pulls into [`axx lint`](/next/guides/isolate-test-data/), so a hand-written file beside the generated ones that reuses a value fails too. The rule reads the files in the format the factory writes: JSON, YAML, XML, or a seed’s YAML, flat XML or CSV. When the schema gains a required field, generation fails and names every fixture that lacks it. Add it once to the prototype (or a `defaults:` entry) and regenerate. ## Where a value comes from [Section titled “Where a value comes from”](#where-a-value-comes-from) A generated value comes from the fixture, a prototype overlay, the prototype, a `defaults:` entry or the schema’s default, and it may get there through a `$ref`, an expression or an identity. `axx fixtures explain` names the source to edit, and the place in it: ```sh axx fixtures explain kafka/scan-delivered.json location ``` ```text location = "Leipzig" in kafka/scan-delivered.json, fixture scan-delivered of kafka/depot-scans.factory.yaml set by the prototype at data.location in kafka/depot-scans.prototype.yaml ``` The path is dotted, with indices (`recipient.postcode`, `items[0].sku`); a dataset’s starts with its table (`parcels.manifest_lines[0].weight_grams`). A fixture’s `*.fixture.yaml` works in place of the file it generates. With `--json`, the origins come as a list, outermost first: a `$ref`, then the source of what it points at. A payload the cloud wraps and encodes on its way to your service, such as a Pub/Sub push envelope with the message base64-encoded in it, is not a fixture to build: publish the message itself, and the emulator’s push subscription delivers the envelope ([Send a service messages](/next/guides/test-cloud-services/#send-a-service-messages)). ## Families [Section titled “Families”](#families) | Family | Produces | Schema | | ---------- | ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | `avro` | Kafka payload JSON | an `.avsc` file | | `json` | any JSON: mock bodies, request payloads, messages | JSON Schema, an OpenAPI component (`api.yaml#/components/schemas/Name`), or an AsyncAPI message’s payload (`asyncapi.yaml#/components/messages/Name`) | | `yaml` | record-shaped YAML | the same as `json` | | `xml` | XML documents | an XSD | | `protobuf` | canonical proto-JSON | a `.proto` file or a descriptor set, with the message (`.proto#pkg.Message`, `.desc#pkg.Message`) | | `dataset` | SQL seeds: YAML datasets, flat XML or CSV directories | optional SQL DDL (a file or a directory of migrations) | A JSON Schema can reference definitions in other local files, by a path relative to the schema holding the `$ref`, with or without a fragment: `"$ref": "address.schema.json"` or `"$ref": "common/money.schema.json#/$defs/Money"`. A failure a referenced file’s rule causes names that file. Remote schemas are not fetched: save them next to yours. For `dataset`, the prototype is a row template per table, and identities are `table.column` paths enforced per row. `options: { format: xml }` or `{ format: csv }` in the factory writes flat XML or a CSV directory instead of YAML. The parcels example generates its rejected manifest lines as flat XML: seeds/manifests/xml/manifests-xml.factory.yaml ```yaml factory: family: dataset options: { format: xml } schema: ../../../../infra/postgres/init/01-schema.sql ``` ## Project settings [Section titled “Project settings”](#project-settings) axx.yaml ```yaml fixtures: sources: ["../infra/wiremock"] # extra roots, e.g. mock bodies served by a container output: { ignored: true } # generated files are gitignored, not committed conformance: # lint hand-written files against a schema too - name: seeds match the database schema filePatterns: ["seeds/*.yaml"] schemaType: dataset schemaRef: ../infra/postgres/init/01-schema.sql ``` With `output: { ignored: true }`, generated files are derived on demand instead of committed: Axx maintains an exact `.gitignore` in each output directory, and a fresh clone runs `axx fixtures generate` once before `axx run`. The specs, the manifest and the generated lint rules stay committed; they are what reviewers read. In CI, check before you generate: ```sh axx fixtures check # the sources, the committed manifest and any generated files agree axx fixtures generate # write the ignored outputs this checkout lacks axx run ``` A fresh clone checks clean: the ignored outputs it lacks pass when the committed manifest records what the sources produce. Changed sources whose manifest was not generated again fail, and so does a generated file edited by hand. The other order would hide a stale manifest, because generate brings it up to date. `conformance` rules check files that no factory owns, so hand-written seeds are held to the same schema. ## Adopt existing files [Section titled “Adopt existing files”](#adopt-existing-files) You do not need to write specs for fixtures you already have. Adoption reads existing files, puts the values they all share into the prototype, keeps each file’s differences in its own `*.fixture.yaml`, and proposes identity candidates for you to review. It only writes anything if regenerating from the new spec reproduces the originals exactly. Try it first with `--dry-run`, which does everything but write: ```sh axx fixtures adopt --family json --schema openapi/parcels.yaml#/components/schemas/Parcel \ --files 'mocks/parcels/*.json' --factory parcel-bodies --dry-run ``` ```text decoded 3/3 files (family: json) prototype: 4 field(s) (the values every adopted file shares) identity candidates, not declared: id (distinct in all 3), trackingNumber (distinct in all 3); declare the ones that identify a fixture with --identity would write mocks/parcels/parcel-bodies.factory.yaml + 3 *.fixture.yaml + parcel-bodies.prototype.yaml semantic round-trip: 3/3 deep-equal. 0 file(s) will be reformatted by `axx fixtures generate`, 0 semantic changes. next: axx fixtures generate ``` * **The prototype takes only what every file shares.** A value two files of three happen to have stays in those two files, so no fixture’s customer or tenant becomes every new fixture’s default. `--prototype common` takes each field’s most common value instead. * **Identities are yours to declare.** A candidate is a string field whose value is distinct in every file, which is what an identity looks like, but not proof of one: an email or a domain can be distinct in three examples by chance. Declare the ones that identify a fixture with `--identity` (repeatable); the factory file lists the others as comments. * **A refusal says what would change.** When regenerating would not reproduce the originals, adoption lists the paths that would differ and writes nothing. Seed layouts whose maps are keyed by document ID (`orgs: {alpha: {...}}`) often do this: adopt the records as a factory of their own, then reference them from the layout with `$ref`. ```sh axx fixtures adopt --family json --schema openapi/parcels.yaml#/components/schemas/Parcel \ --files 'mocks/parcels/*.json' --factory parcel-bodies --identity id axx fixtures adopt --into parcel-bodies --files 'mocks/returns/*.json' # more files into the same factory axx fixtures generate ``` `--into` adds files to an existing factory, with its prototype and identities. # Install Axx > Install the Axx CLI with Homebrew, the install script, go install or the container image, verify the download and keep it up to date. Axx is a single static binary for Linux, macOS and Windows on amd64 and arm64. It has no runtime dependencies. Docker is only needed if the services you test start with Docker Compose. Pre-release Axx is in beta. Every `0.x` release is marked as a pre-release on GitHub. ## Choose a channel [Section titled “Choose a channel”](#choose-a-channel) ### Homebrew (macOS, Linux) [Section titled “Homebrew (macOS, Linux)”](#homebrew-macos-linux) ```sh brew install nimbusxr/tap/axx ``` ### Install script (Linux, macOS, CI) [Section titled “Install script (Linux, macOS, CI)”](#install-script-linux-macos-ci) ```sh curl -fsSL https://axx.nimbusxr.us/install.sh | sh ``` The script picks the newest release (pre-releases included, since every `0.x` release is one), downloads the archive for your OS and architecture, and checks it against `checksums.txt`. It installs `axx` in `/usr/local/bin` when that is writable, otherwise in `~/.local/bin`. Set `AXX_INSTALL_DIR` to choose the directory, and `AXX_VERSION` to install a given version (`0.1.0`) or the `nightly` build. ### Install script (Windows) [Section titled “Install script (Windows)”](#install-script-windows) ```powershell irm https://axx.nimbusxr.us/install.ps1 | iex ``` The script installs the newest release in `%LOCALAPPDATA%\axx\bin`, after checking it against `checksums.txt`, and adds that directory to your user `PATH`. ### Go [Section titled “Go”](#go) ```sh go install github.com/nimbusxr/axx/cmd/axx@latest ``` Needs Go 1.27 or newer. The binary lands in `$(go env GOPATH)/bin`. ### Container image [Section titled “Container image”](#container-image) ```sh docker run --rm -v axx-cache:/home/nonroot -v "$PWD:/work" -w /work ghcr.io/nimbusxr/axx validate ``` The image contains only the `axx` binary (on a distroless base). The `axx-cache` volume keeps the packs Axx prepares for the project ([Choose packs](/next/guides/use-packs/)); without it, every run prepares them again. It suits commands that do not start services (`validate`, `steps`, `explain`, `schema`) and runs against services that are already up (`axx run --no-start`). It cannot run `docker compose` for you. ### Release archives [Section titled “Release archives”](#release-archives) Download `axx___.tar.gz` (`.zip` on Windows) from [GitHub releases](https://github.com/nimbusxr/axx/releases), extract `axx` and put it on your `PATH`. A rolling `nightly` pre-release is built from `main` every day. ## Verify the download [Section titled “Verify the download”](#verify-the-download) Release checksums are signed with Sigstore (keyless), and archives carry an SBOM and build provenance. ```sh cosign verify-blob checksums.txt \ --bundle checksums.txt.sigstore.json \ --certificate-identity-regexp '^https://github.com/nimbusxr/axx/' \ --certificate-oidc-issuer https://token.actions.githubusercontent.com sha256sum --ignore-missing -c checksums.txt gh attestation verify axx_0.1.0_linux_amd64.tar.gz --repo nimbusxr/axx ``` ## Check the installation [Section titled “Check the installation”](#check-the-installation) ```sh axx version axx doctor ``` `axx doctor` checks the binary, `axx.yaml`, the step packs, your feature files and the commands your services need. It exits `0` when nothing failed (warnings are allowed) and `4` otherwise. ## Shell completion [Section titled “Shell completion”](#shell-completion) ```sh axx completion zsh > "${fpath[1]}/_axx" # zsh axx completion bash > /etc/bash_completion.d/axx # bash axx completion fish > ~/.config/fish/completions/axx.fish ``` ## Editors [Section titled “Editors”](#editors) To see Axx’s steps in feature files as you write them, set up IntelliJ IDEA, VS Code or another editor with [Set up your editor](/next/guides/set-up-your-editor/). ## Upgrade [Section titled “Upgrade”](#upgrade) Upgrade with the channel you installed from (`brew upgrade`, rerun the script, or `go install ...@latest`). Step text never changes between versions, so feature files keep working. Before `v1.0.0`, a minor version bump (`0.1` to `0.2`) may change a flag or a configuration key; read the [changelog](https://github.com/nimbusxr/axx/blob/main/CHANGELOG.md) first. After upgrading, refresh the agent skills so their step index matches the new binary: ```sh axx skills install ``` # Isolate test data > Use axx lint to guarantee that ids, keys and names in seeds, payloads and features never collide, so scenarios can share infrastructure and run in parallel. Scenarios run in parallel against one database, one broker and one set of mocks. They stay independent only if each one owns its data. Two seed files that both insert the parcel `PX-KES-1001`, or two scenarios that both publish events keyed `PX-TRK-3002`, produce failures that come and go with scheduling. `axx lint` finds those collisions before a run does. ## Write rules [Section titled “Write rules”](#write-rules) Rules live under `lint:` in `axx.yaml`. Each one extracts values from files and says how unique they must be: axx.yaml ```yaml lint: config: baseDir: . # patterns are relative to this directory mode: error # error fails the lint; warn only reports rules: - name: Parcel references in seeds filePatterns: ["seeds/*.yaml"] regex: '^\s+-?\s*reference:\s*"([^"]+)"' description: Every seeded parcel and manifest line has its own reference validation: cross-file-unique - name: Manifest line ids filePatterns: ["seeds/*.yaml"] regex: '^\s+-?\s*id:\s*"([^"]+)"' validation: cross-file-unique - name: Depot scan ids filePatterns: ["seeds/*.json"] type: jsonpath jsonPath: "scans[*].scanId" validation: cross-file-unique ``` A `jsonpath` rule reads each file by its extension, as JSON, YAML, XML or CSV, so it needs no pattern for the file’s layout. A seed’s rows read the same in YAML, flat XML and CSV: `$['parcels.manifest_lines'][*]['id']` is the `id` of every manifest line, whichever format the seed is in. Each rule key is described in the [configuration reference](/next/references/config/#lint). A rule that scans files but extracts no value from them checks nothing, so `axx lint` warns about it instead of reporting ok: most often its pattern misses the files’ format, like a regex anchored at `reference:` for YAML list items, whose lines start with `- `(the rules above allow it with `-?`). Generated rules can be merged in with `include`: axx.yaml ```yaml lint: include: [axx-lint.generated.yaml] # written by `axx fixtures generate` ``` ## One scenario, several files [Section titled “One scenario, several files”](#one-scenario-several-files) A `cross-file-unique` rule compares file with file: a value two files share is a collision. That holds while each scenario’s data is in a file of its own. Some scenarios need more than one file. The couriers’ app’s list refresh signs a courier in with one parcel, then gives the same courier a second parcel while the scenario runs, from a second seed file. Both files name the courier, and one scenario owns it. Such a scenario keeps its files in a folder of its own, named after it, and the rule names that folder with a pattern that ends in `/`. All the files below a folder that pattern matches count as one file: they may share values with each other, and with nothing else. ```plaintext features/courier-app/seeds/ courier-app-today.yaml one scenario's seed: one file courier-app-refresh/ one scenario's seeds: one folder, counted as one file at-sign-in.yaml given-since.yaml ``` axx.yaml ```yaml lint: rules: - name: Couriers in seeds filePatterns: - "features/*/seeds/*.yaml" # each scenario's file - "features/*/seeds/*/" # each scenario's folder, as one file regex: '"courier": "([^"]+)"' description: Every courier's parcels are one scenario's validation: cross-file-unique ``` ```gherkin Scenario: A courier pulls the list down for a parcel given to them since Given a seeds/courier-app-refresh/at-sign-in.yaml db seed # ... When a seeds/courier-app-refresh/given-since.yaml db seed ``` A courier in `courier-app-refresh/` and in `courier-app-today.yaml` is reported as two files sharing a value (`a folder's files count as one`), and so is a courier two folders share. * **One folder, one scenario.** A folder says its files are one scenario’s, and the rule takes its word: two scenarios’ files in one folder may share values unnoticed. A folder the pattern matches that is no scenario’s, like one that keeps the fixtures a [factory](/next/guides/fixture-factories/) generates, goes in `excludePatterns` (`"features/manifest-import/seeds/manifests/**"`); its own rules check it. * **A folder pattern takes every file below the folder**, whatever its kind; `excludePatterns` remove the ones the rule should not read. * **Every rule over the seeds names the folders.** A rule without the folder pattern checks the folder’s files one by one, or not at all when its file pattern does not reach them (`features/*/seeds/*.yaml` does not). * **A folder claims its files.** A file a file pattern matches too (`seeds/*.yaml` reaches into subfolders) still counts as its folder’s; folders inside a matched folder are part of it. * `cross-file-unique` and `file-unique` count the folder as one file; `global-unique` counts every occurrence, as always. `excludePatterns` still remove files one by one. ## Run it [Section titled “Run it”](#run-it) ```sh axx lint ``` ```console FAIL Parcel references in seeds (cross-file-unique, 52 files): 1 duplicate value Every seeded parcel and manifest line has its own reference value "PX-KES-1001" appears in 2 files (cross-file-unique) [AXX-E0820] seeds/manifest-kestrel-resend.yaml:4:17 reference: "PX-KES-1001" seeds/manifest-kestrel.yaml:4:17 reference: "PX-KES-1001" ok Manifest line ids (cross-file-unique, 52 files) ok Depot scan ids (cross-file-unique, 1 file) ok kafka/depot-scans.factory.yaml: scanId uniqueness (cross-file-unique, 2 files) ok requests/registrations.factory.yaml: reference uniqueness (cross-file-unique, 2 files) ok Services registered before use (13 files) ok SQL selection and trigger ordinals (13 files) ok Stale selections (13 files) ok REST request ordinals (13 files) ok REST payload values (13 files) hint: seeds/ has 39 hand-written .yaml files of one shape (parcels.parcels) that no fixture factory generates; a factory would keep what they share in one place (optional; `axx fixtures adopt --help`) hint: seeds/ has 5 hand-written .yaml files of one shape (parcels.manifest_lines) that no fixture factory generates; a factory would keep what they share in one place (optional; `axx fixtures adopt --help`) axx lint: 10 rules, 70 files: 1 error, 0 warnings ``` A new seed file reused a reference that `seeds/manifest-kestrel.yaml` already inserts. The rules from `axx-lint.generated.yaml` run too, and so do the built-in checks of the features. They catch: * a When after a Then: a scenario that does something and checks it, then does something else and checks that, which is several scenarios in one (write one per behavior, with what an earlier round did as Given steps); * a REST, SQL, MongoDB or Kafka step that comes before the scenario registers a service of its pack; * SQL selections and triggers, and REST requests, addressed by ordinals they cannot have (a `3rd ordered` request after only one, or a second request added without its ordinal); * a step without an ordinal that checks the first selection while a later one goes unchecked (`the selection has 1 row` after a second retrieval); * request payload values in single quotes, which in a table are part of the value. They are warnings, and `axx validate` reports them as well. The `hint:` lines are suggestions, never errors or warnings. When the scenarios read three or more hand-written `.json` or `.yaml` files from one directory, whose top-level keys are the same and that no [fixture factory](/next/guides/fixture-factories/) generates, `axx lint` suggests one: it would keep what the files share in one place and each file’s differences in its own. Factories are optional; `axx fixtures adopt` turns the files into one without changing a value. `axx lint`, `axx validate` and a passing `axx run` also hint at scenarios whose checks prove little: those that check only a success status (a 2xx response, or a command’s exit code 0), which says it was accepted, not what it did, and those that check only that something did not happen, which also passes when the action never ran. A hint names the scenarios; whether one needs another check is for its author to judge. `axx lint` exits with `3` when an `error`-mode rule finds a duplicate, like `axx validate` does for undefined steps. Run it in CI next to `axx validate`, and start new rules in `warn` mode while you clean up existing data. `axx lint --mode error` makes every rule an error, the built-in checks of the features too, so CI fails on an ordinal that would fail every run, before any service starts. ## Tips [Section titled “Tips”](#tips) * **Prove a rule bites.** After writing a rule, plant a duplicate value and check that `axx lint` fails. A pattern that matches no files still passes (its line only says `no files matched`). * **Name values after the scenario.** `PX-DBF-3001` tells a reader which feature owns it; `test` tells nobody anything. * **Declare identities once.** With [fixture factories](/next/guides/fixture-factories/), an `identity:` in the factory spec generates the matching lint rule for you. * **Some scenarios cannot share.** A scenario that changes a table’s triggers affects everyone. Tag it with a tag from `run.exclusive` so it runs alone ([Run in parallel](/next/guides/parallel-runs/)). [Scenario isolation](/next/explanations/scenario-isolation/) explains the reasoning behind shared infrastructure with isolated data. # Manage the services under test > Declare how Axx starts, waits for, stops and cleans up the services under test, keep them running with axx up, attach services you run yourself, and start only the services a run needs. `services:` in `axx.yaml` describes the system under test: your service and anything it needs, such as databases, brokers and mocks. `axx run` starts them, waits until they are ready, runs the scenarios, stops them and runs their cleanup. A service is what runs on its own and waits for whoever connects to it, by an address. A program its user starts, uses and ends is not a service, even when it is what you test: a command-line tool, or an MCP server its clients run over stdio. The step that uses it runs it, as its user would: the cli pack’s and the mcp pack’s `command`. ## Declare a service [Section titled “Declare a service”](#declare-a-service) axx.yaml ```yaml services: infra: dir: ./infra # relative to axx.yaml command: docker compose up --wait postgres kafka ready: tcp: localhost:5432 cleanup: docker compose down -v --remove-orphans api: dependsOn: [infra] command: ./gradlew bootRun # or npm start, go run ./cmd/api, ... env: SPRING_PROFILES_ACTIVE: acceptance ready: http: url: http://localhost:8080/actuator/health timeout: 120s interval: 1s stop: signal: SIGTERM grace: 20s ``` Each key is described in the [configuration reference](/next/references/config/#services). A command that exits `0` before the service is ready is fine, which is what `docker compose up -d` or `--wait` does: Axx keeps polling the `ready` checks. If you also run a compose file yourself while you develop, give the stack Axx starts from it a name of its own, in `command` and `cleanup` alike: `docker compose -p parcels-axx up`. Compose otherwise names both stacks after the folder, so Axx’s takes over yours, and `down -v` deletes your stack’s volumes, your local data with them. `axx init` names the stack it writes so. Service output goes to `.axx/logs/services.log`. When a service fails to start, the error shows the last lines of its output and a stable code from [`AXX-E0400` to `AXX-E0415`](/next/references/error-codes/#service-lifecycle). ## Keep services running between runs [Section titled “Keep services running between runs”](#keep-services-running-between-runs) ```sh axx up # start every enabled service (or: axx up api) and wait until ready axx run # reuses the running services: no start, no stop axx run --tags @smoke axx env # which services are running, left over from a killed run, or not cleaned up axx down # stop the services and run their cleanup ``` This is the fastest local loop, and the one agents should use. The services keep their data from run to run, and `axx run` warns so when it reuses them: a scenario that creates something (registers a parcel by its reference, say) finds what an earlier run created and fails where a fresh start would pass. `axx down`, then `axx up`, starts them afresh; check a change on fresh services before you trust its result. ## When a run cannot start them [Section titled “When a run cannot start them”](#when-a-run-cannot-start-them) An agent whose sandbox keeps it from Docker cannot start services that run with `docker compose`. `axx host`, started in the project on your machine, starts them for its runs: each run asks the host for its services, the host starts them as `axx.yaml` says and cleans up after the run, and it starts only a definition you approved. [Run tests from an agent’s sandbox](/next/guides/run-from-a-sandbox/) shows how. ## When a run is stopped by force [Section titled “When a run is stopped by force”](#when-a-run-is-stopped-by-force) A run that is killed before it can stop its services (stopped by force from an editor, say, or crashed) leaves them in `.axx/run/state.json`. The next run cleans up first, as `axx down` does: it stops what is left, runs the cleanups, says so, and starts from a clean slate. What belongs to a run still going, or to `axx up`, is left alone: a run then starts no service and says what is left (`AXX-E0415`). ## When a cleanup fails [Section titled “When a cleanup fails”](#when-a-cleanup-fails) A cleanup can fail: `docker compose down` without access to Docker, say. The run reports it (`AXX-E0411`), and `.axx/run/state.json` keeps the cleanup, so what it should have removed is not forgotten. Until it succeeds: * runs and `axx up` try it again first, and start no service while it fails, saying what is left (`AXX-E0415`): containers, volumes and requests recorded by a mock would be where the next run starts from; * `axx down` runs the cleanup again, and says `cleaned up after an earlier run` once it succeeds; * `axx doctor` lists what is left, and so do `axx env` and the MCP `env` tool’s `status`, with each service `running`, `left over` from a run that was killed, or `not cleaned up`. ## Run a service yourself [Section titled “Run a service yourself”](#run-a-service-yourself) To run the service from your IDE (with breakpoints, hot reload, a profiler), tell Axx not to start it. Axx still starts everything else and waits for its readiness checks: ```sh axx run --attach api ``` `--no-start` skips starting, stopping and cleaning up every service; use it when the whole system is already running somewhere. ## Debug a service [Section titled “Debug a service”](#debug-a-service) Give the service a debug command and a debugger to wait for: axx.yaml ```yaml services: api: command: ./gradlew bootRun debug: command: ./gradlew bootRun -PappJvmArgs=-agentlib:jdwp=transport=dt_socket,server=n,address=localhost:5005,suspend=n debugger: type: java # java, go, nodejs or python port: 5005 mode: ide-listens # the IDE listens, the service connects (default for java) onUnavailable: retry # retry (default), fail, or fallback to the normal command retry: {attempts: 10, delay: 3s} ``` ```sh axx run --debug # every service with a debug block axx run --debug=api # only api axx up --debug=api ``` With `mode: ide-listens`, start a listening debugger in your IDE first (in IntelliJ IDEA: a *Remote JVM Debug* configuration in *Listen to remote JVM* mode on port 5005), then run Axx. With `mode: app-listens` (delve, `--inspect`, debugpy), the service opens the port and you attach; the [parcels example](https://github.com/nimbusxr/axx/tree/main/examples/parcels) runs its Go service under Delve this way. If no debugger is listening, `onUnavailable` decides whether to wait, fail with [`AXX-E0409`](/next/references/error-codes/#axx-e0409), or run without debugging. ## Start only what a run needs [Section titled “Start only what a run needs”](#start-only-what-a-run-needs) In a repository with several services, start only the ones the selected scenarios need: axx.yaml ```yaml active: enabled: true onNoTags: fallback # no selected scenario has a tag: start every enabled service (or: error) services: orders: command: ./gradlew :orders:bootRun active: {tags: ["@orders"]} billing: command: ./gradlew :billing:bootRun active: {tags: ["@billing", "@payments"]} wiremock: command: docker compose up wiremock # no active.tags: always starts ``` A service with `active.tags` starts when any selected scenario carries one of its tags, and the services in its `dependsOn` start with it. Services without `active.tags` always start. `axx run --tags @billing` starts `billing` and `wiremock`, not `orders`. A service can also wait for what the scenarios use, with `active.uses`: it starts when a selected scenario uses a step of one of its packs, no tag needed. A service only the mobile apps call, say: axx.yaml ```yaml services: push-gateway: command: docker compose up push-gateway active: {uses: [mobile-android, mobile-ios]} ``` # Mock dependencies > Replace the services your service calls with WireMock, verify the requests it sent, and check both sides of each dependency's OpenAPI contract with the Axx WireMock image. Your service calls other services. In an acceptance test you replace them with [WireMock](https://wiremock.org/) mocks, which gives you two things: the responses are under your control, and you can check exactly what your service sent, down to the signatures of its webhooks ([Test webhooks and tokens](/next/guides/webhooks-and-tokens/)). ## Define the stubs [Section titled “Define the stubs”](#define-the-stubs) Stubs are ordinary WireMock mapping files. Axx does not create stubs; it verifies traffic. Mount the mappings into a WireMock container: infra/wiremock/mappings/postcode-undeliverable.json ```json { "name": "postcode-undeliverable", "priority": 1, "request": { "method": "GET", "urlPathPattern": "/v1/postcodes/[A-Z]{2}/999[^/]*" }, "response": { "status": 200, "headers": { "Content-Type": "application/json" }, "body": "{\"postcode\": \"{{request.pathSegments.[3]}}\", \"country\": \"{{request.pathSegments.[2]}}\", \"deliverable\": false, \"reason\": \"no delivery to this postcode\"}", "transformers": ["response-template"] } } ``` infra/compose.yaml (excerpt) ```yaml services: address-service: image: wiremock/wiremock:3.13.0 ports: ["8081:8080"] volumes: ["./wiremock:/home/wiremock"] ``` Configure your service to call `http://localhost:8081` (or the compose service name) instead of the real dependency. ## Verify what your service sent [Section titled “Verify what your service sent”](#verify-what-your-service-sent) Register the mock in the scenario, trigger the behavior, then check the requests WireMock received: ```gherkin Feature: Address check Background: Given the parcels service with the following properties: | url | http://localhost:8400 | And the mocked addresses service with the following properties: | url | http://localhost:8081 | Scenario: The address service is asked with the API key Given a POST request to /api/parcels And a request payload using an application/json content example And the request payload properties are: | reference | PX-ADR-1101 | | recipient.postcode | "53111" | When the request is executed Then the response status code is 201 And the mocked GET request to /v1/postcodes/DE/53111 named postcode-check was received by addresses And the mocked request named postcode-check was received exactly 1 time And the header X-Api-Key for mocked request named postcode-check on addresses is 'example-address-key' ``` `the mocked GET request to /v1/postcodes/DE/53111 named postcode-check was received by addresses` registers a request pattern (method and exact URL, including the query string) under a name you choose, and checks it was received at least once. Later steps refer to it by that name; the [mock pack reference](/next/references/packs/mock/) has the ones that check counts, absence and headers. ```gherkin Scenario: Invalid registrations never reach the address service Given the OpenAPI validation levels are: | validation.request.body.schema.minimum | IGNORE | And a POST request to /api/parcels And a request payload using an application/json content example And the request payload properties are: | reference | PX-ADR-1103 | | weightGrams | 0 | | recipient.postcode | "12489" | When the request is executed Then the response status code is 400 And the mocked GET request to /v1/postcodes/DE/12489 named skipped-check was not received ``` ### What a request sent [Section titled “What a request sent”](#what-a-request-sent) When requests share a URL, like every courier booking going to `/v1/collections`, check what a request sent: the properties of its JSON body, or the fields of its form. The checks stay with the named request, so its counts count only the requests that sent that: ```gherkin Scenario: The courier collects an express parcel without being told who it goes to Given the mocked courier service with the following properties: | url | http://localhost:8082 | And a POST request to /api/parcels And a request payload using an application/json content example named 'Express parcel' And the request payload property reference is 'PX-REG-1401' When the request is executed Then the response status code is 201 And the mocked POST request to /v1/collections named collection was received by courier And the payload properties for mocked request named collection on courier are: | reference | PX-REG-1401 | | deliverTo.postcode | "75001" | | recipient | undefined | And the mocked request named collection on courier was received exactly 1 time ``` * **A property is a JSONPath** (`deliverTo.postcode`, `$.lines[0].reference`), and values compare as text: `800` matches the number 800, `"75001"` the string. * **`undefined` checks that something was left out**, such as a recipient’s name the courier must never get. * **Form fields** of a form-encoded body are checked the same way: `the form fields for mocked request named pickup-notice are:`, with a field and its value a row. When a request’s query changes from call to call, such as a request ID, name the request by its path, and check the query parameters that matter: ```gherkin Then the mocked POST request to path /v1/collections named collection was received by courier And the query parameters for mocked request named collection on courier are: | slot | same-day | | reference | undefined | ``` A request named by its path matches whatever its query string; its query parameters narrow it, and its counts, like its body’s properties. ### What your service did not send [Section titled “What your service did not send”](#what-your-service-did-not-send) A check that something did not happen also passes when the service never got that far, so check what it did do too, like the response. Every scenario’s requests count, so the check names the scenario’s own data: ```gherkin Scenario: The courier is not called about a cancelled standard parcel Given a seeds/cancel-standard.yaml db seed And the mocked courier service with the following properties: | url | http://localhost:8082 | And a DELETE request to /api/parcels/PX-REG-1602 When the request is executed Then the response status code is 204 And none of the mocked DELETE requests to path /v1/collections on courier have the query parameters: | reference | PX-REG-1602 | ``` * **`none of the mocked ... requests to path ... have ...`** checks that no request to the path has every row of its table: `the query parameters`, `the payload properties` or `the form fields`. Put the scenario’s own data in the table, next to what must not be there. * **`the mocked ... request to path ... named ... was not received`** checks that no request went to a path at all, whatever its query string. Use it for a path that is the scenario’s own, like one with its parcel’s reference. Journals persist WireMock keeps its request journal between scenarios, and between runs for as long as it keeps running, and scenarios run in parallel. Match on something unique to the scenario (a postcode or an id in the URL, a header, a property of the body) so one scenario never counts another scenario’s requests. A count such as `exactly 1 time` also counts the requests of earlier runs while the mock keeps running, as it does between runs with `axx up`. See [Isolate test data](/next/guides/isolate-test-data/). ## Mock gRPC dependencies [Section titled “Mock gRPC dependencies”](#mock-grpc-dependencies) The Axx WireMock image also carries [WireMock’s gRPC extension](https://wiremock.org/docs/grpc/), so a gRPC dependency is mocked like a REST one. Give the mock a `grpc` folder with the descriptor set of the dependency’s services (`protoc --include_imports --descriptor_set_out=rating.dsc rating.proto`), and stub each method as a POST to `/./`, with its request and answer as proto JSON: infra/rating/mappings/rating-switzerland.json ```json { "name": "rating-switzerland", "request": { "method": "POST", "urlPath": "/parcels.rating.v1.Rates/Quote", "bodyPatterns": [{"matchesJsonPath": "$[?(@.country == 'CH')]"}] }, "response": {"status": 200, "jsonBody": {"surchargeCents": 1250, "deliveryDays": 3}} } ``` * **Another status than OK** is a response header: `"headers": {"grpc-status-name": "NOT_FOUND", "grpc-status-reason": "the carrier does not deliver to AQ"}`. * **The mock steps check the calls** as they check REST requests: WireMock records each call as a POST to its method’s path, with its request as JSON. ```gherkin Scenario: A parcel beyond the EU is priced by the partner carrier Given the mocked rating service with the following properties: | url | http://localhost:8084 | And a POST request to /api/quotes And a request payload using an application/json content example named 'Express abroad' And the request payload properties are: | weightGrams | 2350 | | recipient.postcode | "8001" | | recipient.country | CH | When the request is executed Then the response status code is 200 And the mocked POST request to path /parcels.rating.v1.Rates/Quote named rate was received by rating And the payload properties for mocked request named rate on rating are: | country | CH | | weightGrams | 2350 | | serviceLevel | EXPRESS | ``` Every scenario’s calls to a method share its path, so the payload check names data of the scenario’s own, here the weight. The gRPC extension runs when WireMock’s root has a `grpc` folder; the OpenAPI validation does not apply to gRPC calls. To call a gRPC service yourself, see [Call gRPC services](/next/guides/test-grpc/). ## Mock GraphQL services and subgraphs [Section titled “Mock GraphQL services and subgraphs”](#mock-graphql-services-and-subgraphs) The Axx WireMock image mocks a GraphQL service, or a federated subgraph, **field by field from its schema**. Set `GRAPHQL_SCHEMA_SOURCE` to its SDL, and the mock answers `POST /graphql` by running each operation against the schema: a field’s value comes from its parent’s value when that has it, and otherwise from a stub. Whatever fields a query selects, it gets an answer, which is what a gateway’s changing query plans need. infra/compose.yaml (excerpt) ```yaml shop-directory: image: ghcr.io/nimbusxr/axx-wiremock: # a version with GraphQL mocks environment: GRAPHQL_SCHEMA_SOURCE: /home/wiremock/graphql/shops.graphql volumes: - './shop-directory:/home/wiremock' ``` Stubs are ordinary WireMock mappings, matched with WireMock’s matchers: * **A field** is a POST to `/graphql//`, whose body is `{"arguments": {...}, "source": {...}}` (the field’s arguments, and its parent’s value). * **An entity** of a subgraph (`_entities`) is a POST to `/graphql/_entities/`, whose body is its representation, such as `{"__typename": "Shop", "id": "alder-and-ash"}`. * **The stub’s JSON body is the value.** A `graphql-error` header makes the field an error instead, with `graphql-error-code` as its `extensions.code`. * **A field without a value is null,** and an entity without a stub too. infra/shop-directory/mappings/shop-alder-and-ash.json ```json { "name": "shop-alder-and-ash", "request": { "method": "POST", "urlPath": "/graphql/_entities/Shop", "bodyPatterns": [{"equalToJson": {"id": "alder-and-ash"}, "ignoreExtraElements": true}] }, "response": {"status": 200, "jsonBody": {"name": "Alder & Ash", "tier": "STANDARD"}} } ``` A subgraph’s SDL makes the mock a subgraph: it answers `_service { sdl }` and `_entities`, so a gateway composes and plans with it. The journal records each operation the mock received, with its query and variables, so the mock steps check what was asked: ```gherkin And the mocked POST request to path /graphql named shop-lookup was received by shop-directory And the payload properties for mocked request named shop-lookup on shop-directory are: | variables.representations[0].id | alder-and-ash | ``` A stub of a whole operation on `/graphql` (for example, matching `$.query`) wins over the field-by-field answers, for an outage or an answer the schema cannot give. The OpenAPI validation does not apply to GraphQL calls. To send operations to a GraphQL service yourself, see [Call GraphQL services](/next/guides/test-graphql/). ## Mock AI models [Section titled “Mock AI models”](#mock-ai-models) The Axx WireMock image mocks the models a service asks, in the format of the request: OpenAI’s API (and the servers that speak it), Anthropic’s, Gemini’s, Bedrock’s and Ollama’s, with their streams and their errors. A stub’s `model-request` matcher says which requests it answers (`about` a text of the conversation), and its `model-answer` transformer renders its answer, written the same way for every provider: infra/models/mappings/address-note.json ```json { "request": {"customMatcher": {"name": "model-request", "parameters": {"about": "Birkenallee 3"}}}, "response": { "transformers": ["model-answer"], "jsonBody": {"json": {"name": "Mara Lindqvist", "street": "Birkenallee 3", "postcode": "01067", "city": "Dresden", "country": "DE"}} } } ``` The mock pack’s model steps check what the service asked the model: the texts, the tools and the schema. See [Test AI features](/next/guides/test-ai-features/). ## Mock MCP servers and A2A agents [Section titled “Mock MCP servers and A2A agents”](#mock-mcp-servers-and-a2a-agents) When your service calls another team’s MCP server, or asks another team’s A2A agent, the Axx WireMock image mocks them from their contract, as it mocks GraphQL subgraphs, with a mapping file per answer: infra/compose.yaml (excerpt) ```yaml partner-carrier: image: ghcr.io/nimbusxr/axx-wiremock: # a version with MCP and A2A mocks environment: MCP_SERVER_SOURCE: /home/wiremock/mcp/partner-carrier.yaml A2A_AGENT_CARD_SOURCE: /home/wiremock/a2a/agent-card.json volumes: - './partner-carrier:/home/wiremock' ``` * **An MCP server** is mocked from a description of what it lists, in MCP’s own field names: its `tools` (with their input and output schemas), `resources`, `resourceTemplates` and `prompts`. It answers at `/mcp` (`MCP_PATH`), in MCP 2026-07-28 and in 2025-11-25, 2025-06-18 and 2025-03-26. * **A tool call** is answered by a stub of `POST /mcp/tools/`, whose body is the call’s arguments and whose JSON body is the tool’s result: `structuredContent`, `content` and `isError`. Arguments that break the tool’s input schema get a tool error, as from a real server, and a stub’s result is checked against the tool’s output schema. * **Resources and prompts** are stubs of `POST /mcp/resources` and `POST /mcp/prompts/`. * **An A2A agent** is mocked from its card: the mock serves the card and answers the JSON-RPC and HTTP+JSON interfaces it lists, streams included. A message is answered by a stub of `POST /a2a/messages`, whose body has the message’s `text`, and whose JSON body is the agent’s answer: a `reply`, or a task’s `state`, `message` and `artifacts`. A reply to a task continues it, since the mock remembers the tasks it answered. infra/partner-carrier/mappings/shipment-status-px-ai-8105.json ```json { "name": "MCP: the partner carrier's shipment_status of PX-AI-8105, which cleared customs", "request": { "method": "POST", "urlPath": "/mcp/tools/shipment_status", "bodyPatterns": [{"equalToJson": {"reference": "PX-AI-8105"}}] }, "response": { "status": 200, "jsonBody": { "structuredContent": { "reference": "PX-AI-8105", "status": "CLEARED_CUSTOMS", "location": "Basel", "message": "PX-AI-8105 cleared customs in Basel", "estimatedDelivery": "2026-10-01" } } } } ``` infra/partner-carrier/mappings/where-is-px-a2a-9206.json ```json { "name": "A2A: the partner carrier agent's answer about PX-A2A-9206, which cleared customs", "request": { "method": "POST", "urlPath": "/a2a/messages", "bodyPatterns": [{"matchesJsonPath": {"expression": "$.text", "contains": "PX-A2A-9206"}}] }, "response": { "status": 200, "jsonBody": { "state": "completed", "message": "PX-A2A-9206 cleared customs in Basel and arrives on Thursday." } } } ``` A mistake is loud: a description or card that is not one, or a stub for a tool the description does not list, stops WireMock at startup, and a call no stub answers gets an error that says so. The mock pack checks what your service called and asked: ```gherkin Then the mocked partner-carrier mcp server's shipment_status tool was called with the following arguments: | reference | PX-AI-8105 | And the mocked partner-carrier a2a agent was sent a message containing 'PX-A2A-9206' 1 time ``` All settings are in the [extension’s README](https://github.com/nimbusxr/axx/tree/main/extensions/wiremock-openapi). To test your own MCP server or A2A agent, see [Test MCP servers](/next/guides/test-mcp/) and [Test A2A agents](/next/guides/test-a2a/). ## Check the dependency’s contract [Section titled “Check the dependency’s contract”](#check-the-dependencys-contract) A mock that answers something the real API never would makes a test pass for the wrong reason, and a service that calls its dependency wrongly only finds out in production. The `ghcr.io/nimbusxr/axx-wiremock` image is WireMock with Axx’s OpenAPI validation extension. It checks every call to the mock against the dependency’s OpenAPI document: your service’s request, and the stub’s response. infra/compose.yaml (excerpt) ```yaml services: address-service: image: ghcr.io/nimbusxr/axx-wiremock:0.1 ports: ["8081:8080"] environment: OPENAPI_SPEC_SOURCE: /var/openapi/address-service.yaml OPENAPI_VALIDATION_MODE: report volumes: - ./wiremock:/home/wiremock - ./openapi:/var/openapi:ro ``` `OPENAPI_SPEC_SOURCE` is the dependency’s document (a path inside the container or a URL). A stub can name another one with `"metadata": {"openApiSpecSource": "..."}`, or opt out with `"openApiValidation": false`. In `report` mode your service gets the stub’s answer unchanged, as it would from the real dependency, and Axx reports what broke the contract: * A mock step that checks the call fails, and names the rule: ```console ✗ And the mocked GET request to /v1/postcodes/DE/44444 named zone-lookup was received by addresses The GET /v1/postcodes/DE/44444 request named zone-lookup broke the contract of the mocked addresses service (/var/openapi/address-service.yaml): - validation.response.body.schema.additionalProperties (response): property 'surcharge' is not defined in the schema and the schema does not allow additional properties ``` `(response)` means the stub broke the contract; `(request)` means your service did. * A call that breaks the contract and that no scenario checks fails the run, after the scenarios: ```console Run failures: ✗ Calls to the mocked addresses service (http://localhost:8081) broke its contract, and no scenario checked them: (mock) GET /v1/postcodes/DE/55555 (/var/openapi/address-service.yaml) - validation.response.body.schema.additionalProperties (response): property 'surcharge' is not defined in the schema and the schema does not allow additional properties ``` Axx only reads WireMock’s request journal, so scenarios running in parallel cannot affect each other’s checks. It reads the journals of the mocks the run’s scenarios register: register the mocked service in the `Background` of every feature whose scenarios make the calls, so Axx knows to look. ### Relax a check [Section titled “Relax a check”](#relax-a-check) Every rule is an `ERROR` until you relax it. Relaxing is how you tell Axx a deviation is known: `WARN` logs the finding on the checking step instead of failing it, `INFO` and `IGNORE` drop it. The rules have the same keys as your own service’s contract (`validation.request.body.schema.required`, `validation.response.status.unknown`, …), and a key also covers the keys below it. The settings are separate: [`the OpenAPI validation levels are:`](/next/guides/validate-openapi/) never relaxes a dependency’s contract. Relax where the deviation lives: | Where | Relaxes | Use it for | | -------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | | `"openApiValidationLevels": {"validation.response.body": "WARN"}` in a stub’s metadata | the calls that stub answers, in every scenario and in the end-of-run check | a stub whose answer is off-contract wherever it is used | | `OPENAPI_VALIDATION_LEVELS` on the container, e.g. `validation.response.body.schema.additionalProperties=WARN` | every call to that mock | a known gap between the dependency’s document and its behavior | | the scenario step below | the calls that scenario’s mock steps check | a scenario that exercises an off-contract call on purpose, next to the behavior it checks | In the example, the address service starts sending a field its document does not list yet, and quoting must keep working. The relaxation sits in that scenario, which checks the call: ```gherkin Scenario: Quotes keep working when the address service sends a field it has not documented Given the OpenAPI validation levels for the mocked addresses service are: | validation.response.body.schema.additionalProperties | WARN | And a POST request to /api/quotes And a request payload using an application/json content example named 'Domestic parcel' And the request payload property recipient.postcode is '"80995"' When the request is executed Then the response status code is 200 And the response payload property zone is 'DE-8' And the mocked GET request to /v1/postcodes/DE/80995 named zone-lookup was received by addresses ``` The scenario step only reaches calls the scenario checks with a mock step: the end-of-run check cannot tell which scenario made a call. When a scenario relaxes a mocked service but checks none of its calls, Axx warns at the end of that scenario, and if one of its calls breaks the rule anyway, the run report names the scenario that meant to relax it. The more specific place wins. Without the extension’s `report` mode (`fail`, its default), a call that breaks the contract also gets an HTTP 500 listing the findings; Axx reports it the same way. All settings, the admin endpoints and how the image is versioned are in the [extension’s README](https://github.com/nimbusxr/axx/tree/main/extensions/wiremock-openapi). Keep mock response bodies schema-valid as the provider’s API evolves with [fixture factories](/next/guides/fixture-factories/): the `json` family can use an OpenAPI component as its schema, which is how the example generates `postcode-remote.json` from `infra/wiremock/__files/postcodes.factory.yaml`. # Organize a suite by feature > Give each feature a folder of its own and keep in it everything only that feature uses, its seeds, payloads, messages, screenshots and fixture factories, so one folder holds the whole feature. A feature is what people read, change, run and delete as one. Give each feature a folder named after it, and keep in that folder the files only that feature uses: the seeds it starts from, the payloads it sends, the messages it publishes, the files it uploads or expects, its screenshots, and the fixture factories that generate any of them. Someone working on the feature opens one folder and has all of it. ```plaintext acceptance/ axx.yaml features/ manifest-import/ manifest-import.feature one file, the same on every platform seeds/manifest-kestrel.yaml seeds/manifests/xml/ a fixture factory, with the seeds it generates courier-app/ courier-app-android.feature one file for each platform, when they differ courier-app-ios.feature seeds/ what both platforms start from routes/grimmaische-strasse.gpx shop-portal/ shop-portal.feature screenshots/quote.chromium-linux.png kafka/ what several features share schemas/ infra/ the services' own setup: compose, WireMock mappings ``` That is the [parcels example](https://github.com/nimbusxr/axx/tree/main/examples/parcels/acceptance), laid out this way. ## One file, or one for each platform [Section titled “One file, or one for each platform”](#one-file-or-one-for-each-platform) A feature that is the same on every platform is one file named after the feature, `manifest-import.feature`, whichever platforms run it: a browser, a desktop or a phone is chosen by its tags, a profile or the run. When the platforms differ, with screens of their own and steps that differ, each platform gets its own file in the feature’s folder, named after the feature and the platform: `courier-app-android.feature` and `courier-app-ios.feature`. Both read the folder’s files. A feature on one platform only says so the same way: `incidents-android.feature`. File names stay unique across the suite, so a search for `courier-app-ios` finds one file, not one of several `ios.feature`. ## How steps find a feature’s files [Section titled “How steps find a feature’s files”](#how-steps-find-a-features-files) A step that takes a file looks for it next to its feature file first, then in the `resources` directories of `axx.yaml`, then in the directory of `axx.yaml`. So the feature names its files from its own folder, and its paths stay short: features/manifest-import/manifest-import.feature ```gherkin Given a seeds/manifest-kestrel.yaml db seed ``` A file several features use goes in a folder of its own outside `features/` (`kafka/`, `schemas/`), where every feature finds it. Moving a file there is a decision you can see in a review, not an accident of where it was first written. Screenshots follow the same rule: a check keeps its screenshot in a `screenshots` folder next to the feature file, unless `axx.yaml` sets a folder (`packs.web-screenshots.folder` and the other packs’ `screenshots.folder`). A screenshot kept in the project’s `screenshots` folder before is still found there. ## What stays outside the features [Section titled “What stays outside the features”](#what-stays-outside-the-features) What the services under test read is their setup, not a feature’s data: the WireMock mappings and the bodies WireMock serves, compose files, database migrations and init scripts. They live with the services (in the parcels example, `infra/`), even when one feature is the only one that needs a mapping. The feature’s own files are the ones its steps name. ## Why [Section titled “Why”](#why) * **One folder holds the feature.** Its scenarios and everything they read are side by side: nothing to hunt for across `seeds/`, `requests/` and `screenshots/`. * **Changes stay in one place.** A change to a feature touches its folder, and a review shows the scenarios with the data they read. * **Data stays with its scenarios.** A seed in a feature’s folder is that feature’s: another feature does not pick it up by accident, which keeps [scenarios isolated](/next/explanations/scenario-isolation/). What features share sits apart, on purpose. * **Deleting a feature deletes its data.** No seeds are left behind that nothing reads, and no file has to be checked for other users first. * **Short paths and local names.** `seeds/wen.yaml` only has to be unique in its folder, not across the suite. * **Agents read one folder.** An agent adding a scenario or fixing one finds the feature and its files together, and the next feature’s folder shows it the pattern to follow. * **A folder runs, moves and has an owner.** `axx run features/manifest-import` runs the feature; the folder moves to another suite whole; a `CODEOWNERS` line can give it to a team. # Run in parallel > Control how many scenarios run at once, keep scenarios that cannot share infrastructure on their own with run.exclusive, and randomize order to flush out hidden dependencies. Axx runs scenarios in parallel by default, one per CPU. Parallelism is what keeps a suite of hundreds of black-box scenarios fast, and it only works if scenarios do not step on each other. ## Workers [Section titled “Workers”](#workers) axx.yaml ```yaml run: workers: auto # default: the number of CPUs; or a number ``` ```sh axx run --workers 4 axx run -w 1 # one at a time, for debugging an ordering problem ``` Each scenario runs start to finish on one worker. Steps inside a scenario always run in order. Workers share one machine; to share a run between machines, CI’s runners, give each a shard of it ([`--shard`](/next/guides/run-in-ci/#share-a-run-between-runners)). ## Scenarios that must run alone [Section titled “Scenarios that must run alone”](#scenarios-that-must-run-alone) Some scenarios change shared state in a way no other scenario can tolerate: they install database triggers, change a feature flag for the whole service, or restart a dependency. Tag them and list the tag in `run.exclusive`: axx.yaml ```yaml run: exclusive: ["@isolated"] ``` ```gherkin @isolated Feature: Database failures Background: Given a parcels-db database with the following properties: | url | postgres://localhost:5432/parcels | | user | parcels | | password | parcels | Scenario: A brief database failure does not fail the registration Given a before insert trigger on the parcels.parcels table will raise a 40001 exception 1 time where: | reference | PX-DBF-3001 | ``` Axx runs every other scenario in parallel first, then the exclusive ones one at a time. Tags are inherited, so tagging the feature covers every scenario in it. ## Order [Section titled “Order”](#order) ```sh axx run --order random # a new random order each run axx run --order random:4242 # repeat a specific order ``` `run.order` in `axx.yaml` sets the default (`defined` unless you change it). A suite that passes in `defined` order but fails in a random one has scenarios that depend on each other. ## Timeouts [Section titled “Timeouts”](#timeouts) axx.yaml ```yaml run: timeouts: step: 60s # default 10m scenario: 5m # default: none hook: 2m # default 2m ``` Durations are strings such as `90s` or `5m`, or a number of seconds. A step that runs over is cancelled and reported with `error.kind: timeout`. For behavior that takes time, use a polling step (`within 10s ...`) instead of a longer timeout. ## When parallel runs fail [Section titled “When parallel runs fail”](#when-parallel-runs-fail) A scenario that passes alone (`axx run features/x.feature:12`) but fails in a full run almost always shares data with another scenario. In order of preference: 1. Give it unique data, and enforce that with [`axx lint`](/next/guides/isolate-test-data/). 2. Assert on its own data only (filter selections by its ids, name mock request patterns by its URLs). 3. If it truly cannot share, tag it for `run.exclusive`. Lowering `--workers` hides the problem instead of fixing it. # Reports > Choose console output, write JUnit, HTML, Cucumber JSON, Cucumber Messages and agent reports, and get machine-readable results with --json. ## Console output [Section titled “Console output”](#console-output) | Format | What you see | When | | ---------- | ---------------------------------------------------------- | ----------------------------------------- | | `pretty` | every scenario and step, tables, logs, failures, a summary | the default | | `progress` | one character per scenario, then failures and a summary | long suites | | `compact` | failures only, and one summary line | automatic for coding agents (`--compact`) | ```console $ axx run --format progress features/register-parcels.feature ...F.... Feature: Register parcels Scenario: A reference can only be registered once # features/register-parcels.feature:69 ... Failed scenarios: x A reference can only be registered once # features/register-parcels.feature:69 rerun: axx run features/register-parcels.feature:69 8 scenarios (1 failed, 7 passed) 96 steps (1 failed, 1 skipped, 94 passed) Finished in 0.6s ``` Problems that belong to no single scenario are listed under `Run failures:` after the failed scenarios, and fail the run. The one Axx reports today is a call to a [mocked dependency](/next/guides/mock-dependencies/#check-the-dependencys-contract) that broke its contract when no scenario checked the call. JUnit reports them as an `axx run` test suite, the agent report as `runErrors`, and Cucumber Messages in `TestRunFinished`. `--compact` is turned on automatically when an agent is detected (`CLAUDECODE`, `CODEX_SANDBOX`, `GEMINI_CLI`, `CURSOR_AGENT`, `AGENT` or `AI_AGENT` is set) and stdout is not a terminal. `--no-color` or `NO_COLOR` turns colors off. ## Report files [Section titled “Report files”](#report-files) Add `--format NAME:FILE` once per report: ```sh axx run \ --format pretty \ --format junit:build/axx/junit.xml \ --format html:build/axx/report.html ``` | Name | Output | Use it for | | --------------- | ----------------------------------------------------------------------- | ----------------------------------------------------------------------- | | `junit` | JUnit XML, one test case per scenario | CI test tabs (GitHub, GitLab, Jenkins) | | `html` | a single self-contained HTML report | humans, as a CI artifact | | `cucumber-json` | the classic Cucumber JSON format | tools that read Cucumber reports | | `messages` | [Cucumber Messages](https://github.com/cucumber/messages) as NDJSON | the Cucumber ecosystem, custom tooling | | `agent` | the compact JSON report below | agents, scripts | | `teamcity` | TeamCity service messages: a live tree of features, scenarios and steps | IDE test runners (the axx IntelliJ plugin and VS Code extension run it) | The same list can be the default in `axx.yaml`: axx.yaml ```yaml run: reporters: - pretty - junit: build/axx/junit.xml - html: build/axx/report.html ``` An unknown name fails with [`AXX-E0600`](/next/references/error-codes/#axx-e0600). ## Machine-readable results [Section titled “Machine-readable results”](#machine-readable-results) `axx run --json` prints one [JSON envelope](/next/references/json-output/) whose `data` is the run report. `--format agent:FILE` writes the same report to a file while the console shows another format. Only failures are listed, so the report stays small however large the suite is. [JSON output](/next/references/json-output/#the-run-report) describes every field. The exit code says what kind of result it was without parsing any output; see [exit codes](/next/references/error-codes/#exit-codes). # Run commands > Run commands in scenarios, such as a command-line tool you ship or an operator's admin command, and check their exit code, what they print and what they print as errors. Some of what a system does is done from the command line: an admin command an operations team runs where the service runs, a command-line tool you ship to customers, a migration or an export. The `cli` pack runs them in scenarios and checks how they end and what they print. ```gherkin Scenario: The desk cannot cancel a parcel the courier has picked up Given a seeds/admin-picked-up.yaml db seed When the admin command is run with 'cancel PX-ADM-6102' Then the admin command's exit code is 3 And the admin command's error output contains 'PX-ADM-6102 is IN_TRANSIT: it can no longer be cancelled' And the admin command's output is empty ``` Add the pack to the project with `axx pack add cli` ([Choose packs](/next/guides/use-packs/)). ## Register the command [Section titled “Register the command”](#register-the-command) Register the command under a name, usually in the `Background`: ```gherkin Background: Given the admin command with the following properties: | command | docker compose exec -T app parcels admin | | dir | ../infra | | timeout | 30s | ``` | Property | What it is | | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `command` | The program and its first arguments (required). They are split like a service’s `command` in `axx.yaml`: double quotes group words, and there is no shell, so no pipes, redirects or `$VARIABLES`. | | `dir` | The folder it runs in, relative to the directory of `axx.yaml`. By default, an empty folder of the scenario’s own. | | `env.` | An environment variable, on top of those axx runs with. | | `timeout` | How long it may run: `1m` unless it says otherwise. | A program named by a relative path, such as `bin/parcels`, is found from the directory of `axx.yaml`. A bare name, such as `psql`, is looked up on the `PATH`. Values can use `${env:..}` and `${sys:..}`, and `${env:..}` values are masked in logs, attachments and failures. A command that belongs to a service can run where the service runs: in its container, through `docker compose exec -T` (`-T` because no terminal is attached). The parcels example keeps the command in a property, so a run of the service on the host can replace it with `-D`: axx.yaml ```yaml properties: parcels.admin: docker compose exec -T app parcels admin ``` ```gherkin Given the admin command with the following properties: | command | ${sys:parcels.admin} | | dir | ../infra | ``` ## Run it [Section titled “Run it”](#run-it) Run the command with more arguments, which are split the same way, and, if it reads them, lines of input (its stdin): ```gherkin When the admin command is run with 'label PX-ADM-6101' ``` ```gherkin When the admin command is run with 'cancel -' and the input: """ PX-ADM-6103 PX-ADM-6104 """ ``` A command that ends with a failing exit code does not fail the step that runs it: that is for the checks to say. Only a command that cannot start, or runs past its timeout, fails the step, and it shows what the command printed. A command that runs past its timeout is stopped with every process it started. A command that starts a process elsewhere, as `docker compose exec` does in a container, stops, but the process it started may run on. ## Check what it did [Section titled “Check what it did”](#check-what-it-did) The checks look at the command’s last run in the scenario: ```gherkin Then the admin command's exit code is 0 And the admin command's output is: """ cancelled PX-ADM-6103 cancelled PX-ADM-6104 """ ``` | Check | What it compares | | -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | `exit code is {int}` | The exit code. | | `output contains {string}` | Text anywhere in what it printed. Runs of spaces and line breaks count as one space; a failure shows the nearest line. | | `output has a line matching {string}` | A line that matches the regular expression (Java syntax). | | `output is:` | The whole of what it printed, with `\r\n` read as `\n` and one final line break left out. | | `output is empty` | Nothing printed at all. | | `output has the following properties:` | JSON it printed: a path and its value, like the other JSON property steps. | | `output is identical to the {filepath} file` | Byte for byte, a file of the project. | Each check but the last two also reads the error output (stderr): `the admin command's error output contains '...'`. A command’s JSON output is checked by its properties: ```gherkin Then the admin command's output has the following properties: | shop | pebble-and-pine | | parcels[0].reference | PX-ADM-6105 | | parcels[1].serviceLevel | EXPRESS | ``` And output a printer or another system reads, such as a shipping label, is compared with a file: ```gherkin Then the admin command's output is identical to the labels/PX-ADM-6101.zpl file ``` An exit code of 0 says the command succeeded, not what it did. Check what it did too: what it printed, or a row it changed. `axx lint` points out a scenario whose only check is a success. ## Keep scenarios apart [Section titled “Keep scenarios apart”](#keep-scenarios-apart) Scenarios run in parallel, so a command must not write over another scenario’s files. By default a command runs in an empty folder of the scenario’s own: the folder is removed when the scenario passes and kept when it fails, for you to look at, and the scenario’s log says where it is. A command that works on shared data, like the admin command on the parcels database, gets data unique to the scenario, like every other step ([Isolate test data](/next/guides/isolate-test-data/)). See the [cli pack’s reference](/next/references/packs/cli/) for every step. # Run tests from an agent's sandbox > Let an agent whose sandbox keeps it from Docker run acceptance tests on clean services. You start axx host on your machine; each run the agent starts gets its services from it, and runs only what you approved. Some agent harnesses run the agent in a sandbox that may not use Docker, because the Docker socket would let it out of the sandbox. Its `axx run` then cannot start your services, and you end up running the tests yourself and pasting the output back. `axx host` starts the services on your machine for the agent’s runs, without giving the agent that way out. ## How it works [Section titled “How it works”](#how-it-works) * You start `axx host` in the project, on your machine, outside the sandbox. Your editor can start it for you ([below](#start-it-from-your-editor)). It listens on a loopback port and writes `.axx/host.json`. * The agent runs `axx run` as it always does. Axx finds `.axx/host.json` in the project folder you share with the sandbox, and asks the host for the services the run needs. * The host cleans up anything an earlier run left behind, then starts the services as `axx.yaml` says: `docker compose ... up --build`, so the agent’s changes are built. It waits until they are ready and sends their output back to the run’s `.axx/logs/services.log`. * The scenarios run in the sandbox, under its rules: every request a step sends, every browser it opens. * When the run ends, the host stops the services and runs their cleanup. It does the same when the run’s connection ends, so a run that is killed leaves nothing behind. * Runs take turns. While one has the services, the next waits and says which run it waits for. ```console $ axx host axx host: serving /Users/you/shop on 127.0.0.1:53123 (Ctrl-C stops it) approved: axx.yaml's services: api, db; compose.yaml waiting for runs run 1: axx run --compact features/orders run 1: starting api, db run 1: ready in 38.2s run 1: 12 scenarios: 11 passed, 1 failed; stopped and cleaned up in 6.0s ``` In the sandbox, the run looks as it always does: ```console $ axx run --compact features/orders axx: asking the axx host (127.0.0.1:53123) for api, db axx: the axx host started api, db (logs: .axx/logs/services.log) ``` Ctrl-C stops the host. A run that has the services at that moment loses them, and the host stops them and cleans up before it exits. The [command reference](/next/references/cli/axx-host/) lists its flags. ## What you approve [Section titled “What you approve”](#what-you-approve) The host never runs a command that a run sends. A run names services, and the host runs what the definition you approved says. The definition is: * `axx.yaml`’s services: their commands, cleanups, readiness checks and environment; * each compose project they run, as `docker compose config` resolves it, with its includes, its `.env` and its variables; * the project files their commands run, such as `./scripts/start.sh` or the `server.js` of `node server.js`. Starting the host approves the definition as it is. If the agent changes it, the next run waits for you: ```console run 5: axx run --compact run 5 (axx run --compact): the services' definition changed since you approved it --- approved +++ proposed @@ -23,6 +23,11 @@ target: 8080 published: "18080" protocol: tcp + volumes: + - type: bind + source: /var/run/docker.sock + target: /var/run/docker.sock + bind: {} networks: default: warning: services.api: compose service api mounts the Docker socket (/var/run/docker.sock): it controls Docker on this machine approve? [y/N] ``` The warnings name what in the definition reaches past a container onto your machine: * privileged containers, added capabilities and devices; * your machine’s network, process or other namespaces; * mounts of files outside the project, and builds from outside it; * files from outside the project read into a container’s environment or secrets; * the Docker socket; * services that run on your machine outside any container. Code the agent changes then runs on your machine. How you answer decides what happens next: * **`y`**: the run goes on, and the approval is kept. Runs with that definition don’t ask again. * **`n`**, or Enter: the run fails with [`AXX-E0501`](/next/references/error-codes/#axx-e0501), which tells the agent you refused the change. The host refuses that definition again, without asking you, until it restarts, so an agent that runs again doesn’t ask you again. A newer change is asked about. * **No answer within a minute**: the run fails with [`AXX-E0507`](/next/references/error-codes/#axx-e0507), and the question stays open. Answer it when you can: the agent’s next run has your answer. If the agent changes the definition again before you answer, the host asks about the newer one instead. Lines about other runs are printed above the question, and the question is asked again below them. Anything typed before the question appears is dropped, so a stray `y` approves nothing. A host that doesn’t run in a terminal, such as one started in the background, can’t ask you: it says so as it starts, and refuses every change. Changes to the app’s own code need no approval: compose builds it and runs it in containers. A Dockerfile’s steps run in build containers too, and what a build may use of your machine (its context, privileges, network) is set in the compose file, which the definition covers. Approvals are kept in your own configuration directory, outside the project, where the agent cannot write them. Starting the host again doesn’t approve anything new: if the definition changed while no host ran, the host asks you about it as it starts. ## Check the sandbox first [Section titled “Check the sandbox first”](#check-the-sandbox-first) The host depends on one thing: the sandbox reaching your machine’s loopback addresses. That’s how the tests reach the services, and how Axx reaches the host. Check it once. Start a server in your terminal: ```sh python3 -m http.server 8765 ``` Then ask the agent to run `curl -sI http://127.0.0.1:8765/`. If it gets `HTTP/1.0 200 OK`, the sandbox reaches your machine. Databases and brokers don’t speak HTTP, so check one of their ports too: start the services in your terminal (`docker compose up -d`) and ask the agent to run `nc -z -w 2 127.0.0.1 5432 && echo open`, with your service’s port. If the checks fail, the agent probably runs in a container, a virtual machine or a remote machine, which has loopback addresses of its own. `axx host` doesn’t serve those. The project folder the agent edits must be the one on your disk, not a copy: the host builds what the agent changed there. ## Start it from your editor [Section titled “Start it from your editor”](#start-it-from-your-editor) The [VS Code extension and the IntelliJ plugin](/next/guides/set-up-your-editor/#the-axx-host) can start the host when you open the project, and stop it when you close it: * An approval is a notification with **Show Diff**, **Approve** and **Refuse**, and the diff opens in the editor’s diff view. It works as the terminal’s question does: it stays open until you answer. Only a click on that notification approves. Nothing else in the editor can, including an agent’s editor tools. * The agent’s runs show in the test results as they run, scenario by scenario. * The status bar shows what the host is doing: idle, a run’s services starting or running, or waiting for your approval. ## Everyone’s runs share the host [Section titled “Everyone’s runs share the host”](#everyones-runs-share-the-host) * Your own runs go through the host while it runs, whether you start them in a terminal or in your editor. All runs use the same ports and the same compose project, so they take turns. * `axx up` asks the host for the services and keeps them until `axx down`. Runs in between reuse them, as they do without a host. * With the [MCP server](/next/guides/set-up-agents/), `steps_try` gets services of its own from the host when none are up, and `scenarios_run` ends them first, so that every run starts clean. `env` works through `axx up` and `axx down`. * `axx doctor` says when a host serves the project, and leaves the checks of the services’ commands and of Docker to the machine the host runs on. * If the host stops during a run, the run loses its services ([`AXX-E0502`](/next/references/error-codes/#axx-e0502)). A host that was killed leaves `.axx/host.json` behind. Runs then warn that it doesn’t answer, and start the services themselves. The host’s errors are [`AXX-E0500` to `AXX-E0506`](/next/references/error-codes/#axx-host). ## What your security team should know [Section titled “What your security team should know”](#what-your-security-team-should-know) * The agent can reach one more loopback port. What it can send there is a run’s request: names of services from `axx.yaml`, and the configuration file, profile and settings it runs with. The host reads the configuration from the project itself, and runs only a definition you approved. * `.axx/host.json` holds a token, readable only by your user. The agent, which runs as you, reads it; other users’ processes can’t. * Approvals come only from you: from the host’s terminal, or from the editor that started the host. Never through the port. * The app’s code, which the agent writes, runs in containers on your machine, with what the approved compose file gives those containers. Containers can reach the network unless the compose file keeps them from it, for example with an `internal: true` network. # Run in CI > Run Axx in GitHub Actions with setup-axx, in GitLab CI with the install script, or anywhere with the container image, and publish JUnit and HTML reports. In CI, Axx does the same thing it does on a laptop: start the services, wait until they are ready, run the scenarios, stop everything. The only extra work is installing Axx and keeping the reports. ## GitHub Actions [Section titled “GitHub Actions”](#github-actions) `axx init` writes this workflow to `.github/workflows/acceptance.yml`: .github/workflows/acceptance.yml ```yaml name: acceptance on: pull_request: push: branches: [main] permissions: contents: read jobs: acceptance: runs-on: ubuntu-latest steps: - uses: actions/checkout@v5 - uses: nimbusxr/setup-axx@v0 - run: axx run --format junit:build/axx/junit.xml --format html:build/axx/report.html - uses: actions/upload-artifact@v7 if: always() with: name: axx-report path: build/axx/ ``` GitHub-hosted Ubuntu runners include Docker and Compose, so services that start with `docker compose up` work as they do locally. `nimbusxr/setup-axx` installs the latest release (pre-releases included) and adds it to `PATH`. Its `version` input installs another: a version such as `0.1.0`, or `nightly`. `@v0` is the action’s own version: its tag `v0` follows the action’s 0.x releases while Axx is in beta. ## GitLab CI [Section titled “GitLab CI”](#gitlab-ci) Install Axx with the install script. When the services start with Docker Compose, run the job with Docker-in-Docker and point the tests at the `docker` host through a profile: .gitlab-ci.yml ```yaml acceptance: image: docker:27 services: - docker:27-dind variables: DOCKER_TLS_CERTDIR: "/certs" before_script: - apk add --no-cache curl - curl -fsSL https://axx.nimbusxr.us/install.sh | sh script: - axx run --profile ci --format junit:build/axx/junit.xml --format html:build/axx/report.html artifacts: when: always paths: [build/axx/] reports: junit: build/axx/junit.xml ``` axx.yaml ```yaml profiles: ci: properties: local.host: docker # containers publish their ports on the dind host ``` This works when your features and `axx.yaml` use `${sys:local.host}` instead of a hard-coded `localhost`. The job runs as root, so the install script puts `axx` in `/usr/local/bin`, which is on `PATH`. ## Any other CI [Section titled “Any other CI”](#any-other-ci) Use the install script, or the container image for commands that do not start services: ```sh docker run --rm -v axx-cache:/home/nonroot -v "$PWD:/work" -w /work ghcr.io/nimbusxr/axx validate ``` ## Keep Axx’s cache [Section titled “Keep Axx’s cache”](#keep-axxs-cache) The first command in a project prepares Axx with the packs in `axx-packs.yaml` ([Choose packs](/next/guides/use-packs/)). On a fresh runner, that happens on every run. Keep the prepared builds between runs to skip it. In GitHub Actions, add this step before `axx run`: ```yaml - uses: actions/cache@v6 with: path: ~/.cache/axx/builds key: axx-${{ runner.os }}-${{ hashFiles('**/axx-packs.yaml', '**/axx-packs.lock') }} restore-keys: axx-${{ runner.os }}- ``` GitLab caches only paths inside the project, so move Axx’s cache there: .gitlab-ci.yml ```yaml acceptance: variables: XDG_CACHE_HOME: "$CI_PROJECT_DIR/.cache" cache: key: files: [axx-packs.yaml] paths: [.cache/axx/builds] ``` A stale cache is harmless: when the packs or Axx’s version change, Axx prepares again. ## Catch problems early [Section titled “Catch problems early”](#catch-problems-early) The exit code says what kind of failure it was, without parsing output: `1` is a failed scenario, `3` an undefined step, `4` a service that did not start ([exit codes](/next/references/error-codes/#exit-codes)). Two cheap guards catch the last two before any service starts: ```sh axx validate # exit 3 on undefined steps, in seconds axx doctor # exit 4 if a service's command is missing ``` Use `--fail-fast` to stop scheduling new scenarios after the first failure when you prefer a quick red over a full report. ## Reports [Section titled “Reports”](#reports) The workflows above write JUnit and HTML reports with `--format`. [Reports](/next/guides/reports/) lists every format, and `run.reporters` in `axx.yaml` can make them the default. `--rerun-file build/axx/rerun.txt` also records the failed scenarios for a follow-up run ([Rerun what failed](/next/guides/tags-and-filtering/#rerun-what-failed)). ## Share a run between runners [Section titled “Share a run between runners”](#share-a-run-between-runners) A suite whose scenarios take long on one runner (a device’s or a browser’s, which run a scenario at a time) can run on several at once: `--shard 2/3` runs the second third of the scenarios the run selects, every third in their files’ and lines’ order, from the second. The shards of a run, `1/3` to `3/3`, run each scenario once between them, the same on every machine: .github/workflows/acceptance.yml ```yaml jobs: acceptance: runs-on: ubuntu-latest strategy: fail-fast: false matrix: shard: [1, 2, 3] steps: - uses: actions/checkout@v5 - uses: nimbusxr/setup-axx@v0 - run: axx run --shard ${{ matrix.shard }}/3 --format junit:build/axx/junit.xml ``` Each runner starts the services for its own shard. A shard that selects nothing passes. ## Run scenarios in parallel safely [Section titled “Run scenarios in parallel safely”](#run-scenarios-in-parallel-safely) CI runners have fewer CPUs than laptops, and `run.workers` defaults to the number of CPUs. If scenarios that pass locally fail in CI, check that they use unique data ([Isolate test data](/next/guides/isolate-test-data/)) before lowering `--workers`. # Seed and query SQL > Load datasets into PostgreSQL (and MySQL, SQLite or SQL Server), assert on rows and JSON columns, wait for asynchronous writes, and inject database faults with triggers. The SQL steps put a database in a known state before a scenario and check what your service wrote afterwards. PostgreSQL is first-class; MySQL, SQLite and SQL Server are supported. ## Register the database [Section titled “Register the database”](#register-the-database) ```gherkin Background: Given a parcels-db database with the following properties: | url | postgres://localhost:5432/parcels | | user | parcels | | password | parcels | ``` Native URLs (`postgres://`, `mysql://`, `sqlserver://`, `sqlite:`) and JDBC-style URLs (`jdbc:postgresql://`, `jdbc:mysql://`, `jdbc:mariadb://`, `jdbc:sqlserver://`, `jdbc:sqlite:`) are accepted. Name a table with its schema, like `parcels.parcels`, or set `currentSchema` in a PostgreSQL URL. The drivers are pure Go and come with the `sql` pack. JSONB containment and triggers need PostgreSQL, and row locks a database other than SQLite; seeds, selections and row counts work on every database. ## Seed [Section titled “Seed”](#seed) ```gherkin Given a seeds/manifest-kestrel.yaml db seed Given a seeds/dispatching.yaml db seed on parcels-db ``` A seed is a dataset keyed by table. YAML is the usual format: seeds/manifest-kestrel.yaml ```yaml parcels.manifest_lines: - id: "ML-KES-0412-1" manifest_id: "M-KESTREL-0412" reference: "PX-KES-1001" sender: "kestrel-books" weight_grams: 850 service_level: "STANDARD" recipient: '{"name": "Emmy Noether", "street": "Bunsenstrasse 3", "city": "Goettingen", "postcode": "37073", "country": "DE"}' - id: "ML-KES-0412-2" manifest_id: "M-KESTREL-0412" reference: "PX-KES-1002" sender: "kestrel-books" weight_grams: 2300 service_level: "EXPRESS" recipient: '{"name": "Lise Meitner", "city": "Hamburg", "postcode": "20095", "country": "DE"}' ``` Flat XML (`.xml`), JSON (`.json`), Excel (`.xlsx`, one sheet per table) and CSV dataset directories (one `.csv` per table, in the order `table-ordering.txt` lists when there is one) work the same way. The `[null]` and `[UNIX_TIMESTAMP]` placeholders are supported, and so are times counted from now, such as `[DAY,NOW]`, `[DAY,PLUS,1]` or `[HOUR,MINUS,2]` (in `DAY`, `HOUR`, `MIN` or `SEC`). A time counted from now is the machine’s local time. PostgreSQL also gets its offset, so a `timestamptz` column holds that very moment whatever time zone the machine is in, and a `timestamp` column holds the local wall-clock time. Values are escaped, so text containing `'` is safe. Scenarios share the database and run in parallel, so give every seeded row an id that belongs to one scenario. [Isolate test data](/next/guides/isolate-test-data/) shows how to enforce that with `axx lint`. A seed inserts its rows in file order, in one transaction. It never updates or deletes existing rows, so a row whose key already exists fails the step and nothing from that file is written. ## Select rows and assert [Section titled “Select rows and assert”](#select-rows-and-assert) A selection is a query with equality conditions. Later steps check its size and contents: ```gherkin Then a selection of rows is retrieved from the parcels.manifest_lines table where: | manifest_id | M-KESTREL-0412 | And the selection has 2 rows And the selection has more than 1 row And the selection has fewer than 5 rows ``` A scenario can hold several selections. They are numbered in the order they are retrieved, and later steps address them by ordinal (`the 2nd selection`); in `a 2nd selection of rows is retrieved` the ordinal is only a label. Leaving the ordinal out means the first: ```gherkin Then a selection of rows is retrieved from the parcels.manifest_lines table where: | manifest_id | M-KESTREL-0412 | And a 2nd selection of rows is retrieved from the parcels.manifest_lines table where: | id | ML-KES-0412-1 | And the 2nd selection has 1 row ``` ### JSON columns [Section titled “JSON columns”](#json-columns) Assert on properties inside a JSON or JSONB column of a selected row: ```gherkin Then a selection of rows is retrieved from the parcels.parcels table where: | reference | PX-REG-1001 | And the 1st row details property for the selection json properties are: | source | api | | zone | DE-1 | And the 1st row details property for the selection json properties match: | zone | ^DE-.*$ | ``` Or select by JSONB containment (PostgreSQL): ```gherkin Then a selection of rows is retrieved from the parcels.parcels table where the details jsonb column contains: | manifestId | M-KESTREL-0412 | And the selection has 2 rows ``` Selections are numbered in the order a scenario retrieves them, and a step without an ordinal checks the first. After a second retrieval, name the one a step checks: `the 2nd selection has 1 row`. `axx validate` warns about a step that checks the first while a later selection goes unchecked. In `json properties are:` and `json properties match:`, paths are JSONPaths such as `zone` or `items[0].sku`, and values are compared as text ([the rules](/next/references/packs/sql/#sqljsonare)). The containment step takes dotted names (`recipient.city`) and compares each value as a JSON string, so it never matches a number or a boolean. ## Wait for asynchronous writes [Section titled “Wait for asynchronous writes”](#wait-for-asynchronous-writes) When the service writes in the background (importing a manifest, say), poll instead of sleeping: ```gherkin Then within 10s a selection of at least 2 rows is retrieved from the parcels.manifest_lines table where: | manifest_id | M-KESTREL-0412 | | status | IMPORTED | And the selection has 2 rows ``` The step queries again every 500 ms until the selection has at least that many rows, and moves on as soon as the rows appear; it fails when the time runs out with fewer. `the selection has 2 rows` then checks that no more than those came back. ## Inject faults with triggers [Section titled “Inject faults with triggers”](#inject-faults-with-triggers) To test how your service handles database errors, install a trigger that raises an SQL state for matching inserts: ```gherkin @isolated Scenario: A brief database failure does not fail the registration Given a before insert trigger on the parcels.parcels table will raise a 40001 exception 1 time where: | reference | PX-DBF-3001 | And a POST request to /api/parcels And a request payload using an application/json content example And the request payload property reference is 'PX-DBF-3001' When the request is executed Then the response status code is 201 And the before insert trigger on the parcels.parcels table was raised 1 time ``` The trigger raises a serialization failure (`40001`) once; the service retries and the second insert succeeds. Without `1 time` the trigger raises on every matching insert. Another variant inserts the row and then raises (`... will insert and raise a 23505 exception where:`). Triggers change a shared table, so tag those scenarios with a tag in `run.exclusive` (for example `@isolated`) to run them alone after the parallel phase. See [Run in parallel](/next/guides/parallel-runs/). ## Row locks [Section titled “Row locks”](#row-locks) ```gherkin Scenario: A parcel cannot be changed while the depot dispatches it Given a seeds/dispatching.yaml db seed And the rows in the parcels.parcels table are locked where: | reference | PX-DSP-2001 | And a PATCH request to /api/parcels/PX-DSP-2001 And a request payload using an application/json content example named 'Heavier' When the request is executed Then the response status code is 409 Scenario: A parcel can be changed again once the depot lets go of it Given a seeds/dispatched.yaml db seed And the rows in the parcels.parcels table are locked where: | reference | PX-DSP-2002 | And a PATCH request to /api/parcels/PX-DSP-2002 And a request payload using an application/json content example named 'Heavier' When the row locks are released And the request is executed Then the response status code is 200 ``` Lock rows to test timeouts and contention in your service. The lock holds until `the row locks are released` or the end of the scenario. ## Named databases [Section titled “Named databases”](#named-databases) Every step has an `on ` form for scenarios with more than one database, for example `a selection of rows is retrieved from the parcels.parcels table on parcels-db where:` and `the selection on parcels-db has 1 row`. # Seed and query MongoDB > Register a MongoDB database, insert documents from JSON seed files, and assert on the documents your service wrote. The MongoDB steps put documents in place before your service reads them, and check the documents it writes. ## Register the database [Section titled “Register the database”](#register-the-database) ```gherkin Background: Given a tracking-db mongo database with the following properties: | url | mongodb://localhost:27017/parcels?authSource=admin | | user | parcels | | password | parcels | ``` All three properties are required and support `${env:...}` and `${sys:...}`. The URL must include the database name; `authSource` defaults to it. The first MongoDB database registered in a scenario is the default. ## Write a seed file [Section titled “Write a seed file”](#write-a-seed-file) A seed is a JSON object that maps collection names to arrays of documents. Extended JSON (`{"$oid": "..."}`, `{"$date": "..."}`) is supported: seeds/scans-in-transit.json ```json { "scans": [ { "scanId": "SC-3001-2", "parcelRef": "PX-TRK-3001", "status": "IN_TRANSIT", "location": "Hamburg hub", "scannedAt": {"$date": "2026-05-05T06:40:00Z"} }, { "scanId": "SC-3001-1", "parcelRef": "PX-TRK-3001", "status": "PICKED_UP", "location": "Berlin depot", "scannedAt": {"$date": "2026-05-04T16:10:00Z"} } ] } ``` ## Seed and use it [Section titled “Seed and use it”](#seed-and-use-it) ```gherkin Scenario: Tracking shows the latest scan, however the scans arrive Given a seeds/scans-in-transit.json mongo db seed And within 10s a selection of at least 1 document is retrieved from the tracking collection where: | _id | PX-TRK-3001 | | scanCount | 2 | When a GET request to /api/parcels/PX-TRK-3001/tracking And the request is executed Then the response status code is 200 And the response payload properties are: | status | IN_TRANSIT | | lastLocation | Hamburg hub | | scanCount | 2 | ``` The service builds its tracking summaries from the scans in the background, so the scenario [waits for the summary](#wait-for-asynchronous-writes) before it calls the API. To seed a specific database when the scenario registers more than one: ```gherkin Given a seeds/scans-in-transit.json MongoDB seed for tracking-db ``` `a seeds/scans-in-transit.json mongo db seed for tracking-db` is the same step with the other spelling. The file path resolves against the `resources` directories in `axx.yaml`. See the [mongo pack reference](/next/references/packs/mongo/). ## Query and assert [Section titled “Query and assert”](#query-and-assert) A document selection finds documents with equality conditions. Dotted field paths reach into nested documents, and [the mongo pack reference](/next/references/packs/mongo/#mongofind) says how values are read: ```gherkin Then a selection of documents is retrieved from the scans collection where: | parcelRef | PX-TRK-3001 | | status | PICKED_UP | And the selection has 1 document And the 1st document for the selection properties are: | scanId | SC-3001-1 | | location | Berlin depot | | scannedAt | 2026-05-04T16:10:00.000Z | ``` Properties are JSONPaths (`location`, `history[0].status`), compared as text; `null` means null and `undefined` means the field is absent ([the rules](/next/references/packs/mongo/#mongodocare)). `the 1st document for the selection properties match:` takes Java regular expressions instead. Selections are numbered in the order a scenario retrieves them, and a step without an ordinal checks the first. After a second retrieval (the tracking view before and after a scan, say), name the one a step checks: `the 1st document for the 2nd selection properties are:`. `axx validate` warns about a step that checks the first while a later selection goes unchecked. Like the SQL selections, they are numbered in the order they are retrieved (`the 2nd selection has 3 documents`), and `has more than` / `has fewer than` compare counts. ### Wait for asynchronous writes [Section titled “Wait for asynchronous writes”](#wait-for-asynchronous-writes) ```gherkin Then within 10s a selection of at least 1 document is retrieved from the tracking collection where: | _id | PX-TRK-3001 | | scanCount | 2 | ``` The step polls every 500 ms until enough documents match, and fails when the time runs out with fewer. The documents it found are the selection, for later steps to check (`the selection has 1 document`, or its properties). Values that parse as JSON are typed, so `2` matches the number 2. Every query step has an `on ` form for scenarios with more than one MongoDB database. ## Keep documents apart [Section titled “Keep documents apart”](#keep-documents-apart) Seeded documents stay in the database after the scenario, and scenarios run in parallel. Give each scenario’s documents their own keys (`PX-TRK-3001` above) and query by them. [Scenario isolation](/next/explanations/scenario-isolation/) explains the approach. # Send REST requests > Payloads from files, the redirects the REST pack follows and the ones it leaves to the scenario, cookies, and the headers custom steps give a request. A scenario registers a REST service, adds requests to it, executes them and checks their responses ([rest pack](/next/references/packs/rest/)). This page is about what happens in between: the payload a request sends, the redirects it follows, the cookies it sends, and the headers a custom step gives it. To sign a request as a webhook is signed, or authorize it with a token, see [Test webhooks and tokens](/next/guides/webhooks-and-tokens/). The REST pack tests APIs. To test a web app’s pages, forms or sign-in, use the web-core pack: a real browser keeps cookies, follows redirects and submits forms, as the app’s users do ([Test web apps](/next/guides/test-web-apps/)). ## Payloads from files [Section titled “Payloads from files”](#payloads-from-files) A payload starts from an example of the service’s OpenAPI specification, from an empty template the payload property steps fill, or from a file of the project, sent as it is: ```gherkin Scenario Outline: A shop's order system registers the parcels it exports Given a POST request to /api/parcels And a request payload using an application/json requests/.json resource When the request is executed Then the response status code is 201 And the response header Location is '/api/parcels/' Examples: | order | reference | | order-7731 | PX-REG-1301 | | order-7732 | PX-REG-1302 | ``` * **The file is found like the other files steps name:** relative to the `resources` directories, or to `axx.yaml`’s directory. * **A JSON payload can still change.** The payload property steps edit it, like a payload from an example. A property inside an object the payload lacks creates the object: `recipient.name` and `recipient.postcode` fill an empty template’s `recipient` ([ADR 0010](https://github.com/nimbusxr/axx/blob/main/docs/adr/0010-payload-properties-create-missing-objects.md)). * **A form payload file is a JSON object.** With `application/x-www-form-urlencoded`, its properties are sent form-encoded. * **No OpenAPI specification is needed,** which suits a callback or a webhook that the service has no contract for. A [fixture factory](/next/guides/fixture-factories/) keeps such files in one place and checks them against a schema. The parcels example generates the two files above with `requests/registrations.factory.yaml`, from the `ParcelRequest` schema of the service’s OpenAPI contract. ## Several requests in one scenario [Section titled “Several requests in one scenario”](#several-requests-in-one-scenario) Requests are numbered in the order a scenario adds them. The first is the default, which steps without an ordinal use; every other request is added, executed and checked with its ordinal: ```gherkin Scenario: A registered parcel can be looked up by its reference Given a POST request to /api/parcels And a request payload using an application/json content example And the request payload property reference is 'PX-REG-1401' And a 2nd ordered GET request to /api/parcels/PX-REG-1401 When the request is executed And the 2nd ordered request is executed Then the response status code is 201 And the 2nd ordered response status code is 200 And the response payload property reference is 'PX-REG-1401' for 2nd ordered response ``` * **Add a request before using it:** a step that uses the 2nd request fails unless a 2nd was added before it. `axx validate` says so, with the step that adds it. * **A second `a GET request to …` adds the 1st again**, and fails: write it as `a 2nd ordered GET request to …`. `axx validate` shows the line to write. * **A check names its response:** `the 2nd ordered response status code is 200`, or `… for 2nd ordered response` at the end of the other response steps. A check without one reads the 1st response. ## Find an item in a list [Section titled “Find an item in a list”](#find-an-item-in-a-list) A property can pick items out of a list with a JSONPath filter: `parcels[?(@.reference=='PX-REG-1401')].status` is the status of that parcel. The value a check compares is the list of what matched, so one match reads `'["delivered"]'`. ```gherkin Then the response payload property parcels[?(@.reference=='PX-REG-1401')].status is '["delivered"]' And the response payload property parcels[?(@.service=~/Next\sday\sair/)].reference is '["PX-REG-1401"]' ``` * **A path is one word**, without spaces. To match a text with spaces, use `=~` and a regular expression, where `\s` stands for the space. * **`=~` matches the whole text**, as Jayway’s JSONPath does: `/Next\sday/` doesn’t match “Next day air”, while `/Next\sday.*/` does. ## Redirects [Section titled “Redirects”](#redirects) | Request | Redirects it follows | The response the steps check | | ----------------------------------- | ----------------------------------------- | -------------------------------- | | `GET`, `HEAD` | 301, 302, 303, 307 and 308 | the one the redirects lead to | | `POST`, `PUT`, `PATCH`, `DELETE`, … | `303 See Other`, with a `GET` | after a 303, the one it leads to | | `POST`, `PUT`, `PATCH`, `DELETE`, … | none of the others: 301, 302, 307 and 308 | the redirect itself | A redirect the pack follows is not kept: its status, its `Location` header and the cookies it sets are not the response’s, and the response steps cannot check them. The scenario’s log shows the request as the step sent it and the status at the end, like `GET http://localhost:8080/api/parcels/PX-REG-1010 -> 200 OK (12ms, 412 bytes)`. The `GET` that follows a 303 has no body and no `Content-Type`. It has the request’s other headers, `Authorization` and `Cookie` only when the redirect stays on the same host or goes to a subdomain of it. ### Redirects left to the scenario [Section titled “Redirects left to the scenario”](#redirects-left-to-the-scenario) A `POST`, `PUT`, `PATCH` or `DELETE` answered with 301, 302, 307 or 308 has the redirect as its response. Check its status and its `Location` header, and send the request it leads to as the next ordered request. Say a service moved its API from `/api/v1` and answers the old paths with `308 Permanent Redirect`: ```gherkin Scenario: A cancellation sent to the old API cancels the parcel through its redirect Given a 1st ordered DELETE request to /api/v1/parcels/PX-REG-1010 And a 2nd ordered DELETE request to /api/parcels/PX-REG-1010 When the 1st ordered request is executed And the 2nd ordered request is executed Then the 1st ordered response status code is 308 And the response header Location is '/api/parcels/PX-REG-1010' for 1st ordered response And the 2nd ordered response status code is 204 ``` The scenario sends both requests, then checks both responses: the redirect, where it leads, and the cancellation there. ## Cookies [Section titled “Cookies”](#cookies) The REST pack keeps no cookies. A `Set-Cookie` response header is never sent back: not with the scenario’s later requests, and not to where a redirect leads. A request carries the cookies its scenario gives it, in its `Cookie` header (`the request header Cookie is '...'`), and nothing else, so scenarios running side by side never share cookies. ## Headers from custom steps [Section titled “Headers from custom steps”](#headers-from-custom-steps) A custom step changes a request the pack’s steps built, before it is executed, through the request’s headers: `Header()` returns the headers it is sent with, and `Set`, `Add` and `Del` change them. `SetHeader(name, value)` does what the request header steps do. [Write custom steps](/next/guides/write-custom-steps/#change-a-rest-request) has an example. # Set up agents > Give coding agents the Axx skills, the Axx MCP server and the AGENTS.md section. axx init sets them up for the agents a repository uses, axx mcp install connects one more, and axx doctor checks them. Three pieces give a coding agent everything it needs: the skills, the MCP server and the `AGENTS.md` section. They reinforce each other, and `axx init` sets up all three for the agents the repository already uses. [Test with an agent](/next/tutorials/with-an-agent/) shows them in use. ## What axx init sets up [Section titled “What axx init sets up”](#what-axx-init-sets-up) `axx init` works in the repository only and lists every file it writes. It never writes to your home directory. | When the repository has | `axx init` writes | | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | anything | the `AGENTS.md` section and the skills in `.agents/skills/` | | `.claude/`, `CLAUDE.md` or `.mcp.json` (Claude Code) | `.mcp.json`, links the skills into `.claude/skills/`, and imports `AGENTS.md` into `CLAUDE.md` where Claude Code reads that instead ([AGENTS.md](#agentsmd)) | | `.codex/` (Codex) | `.codex/config.toml` | | `.cursor/` (Cursor) | `.cursor/mcp.json` | | `.vscode/` (VS Code) | `.vscode/mcp.json` | | `.gemini/` or `GEMINI.md` (Gemini CLI) | `.gemini/settings.json` | In a repository with a `CLAUDE.md` and a `.cursor/` directory: ```console create axx.yaml create axx-packs.yaml create features/smoke/smoke.feature create .github/workflows/acceptance.yml create AGENTS.md update CLAUDE.md (imports AGENTS.md for Claude Code) create .gitignore create .gitattributes create .agents/skills (5 skills) create .claude/skills (linked for Claude Code) create .mcp.json (the axx MCP server for Claude Code) create .cursor/mcp.json (the axx MCP server for Cursor) Codex keeps its MCP servers in ~/.codex/config.toml; `axx mcp install --agent codex --scope user` adds axx there. next: put your service's command, port and health check in axx.yaml and the smoke feature, then `axx doctor` and `axx run` ``` Axx adds its server to the files that exist: other servers and settings stay where they are, and a file that has the axx server already is left alone, so running `axx init` again changes nothing. A file Axx cannot read back exactly, such as JSON with comments, is left alone too: its line says so, and `axx mcp install --agent ` shows what to add by hand. Codex usually reads its servers from your home directory, which `axx init` never writes, so it prints the command that does. `axx init --no-agents` skips the skills and the MCP servers. Commit what `axx init` wrote, so every contributor and every CI agent gets it. ## Skills [Section titled “Skills”](#skills) ```sh axx skills install # this repository axx skills install --scope user # your home directory, for every repository axx skills list ``` `install` writes the skills to `.agents/skills/` (read by Codex, Cursor, Gemini CLI and Copilot) and, when the repository uses Claude Code (a `.claude/` directory, `CLAUDE.md` or `.mcp.json`), links them into `.claude/skills/` for it. `--claude` links them anyway, and `--no-claude` never; with `--scope user`, they are linked when your home directory has `.claude/`. A `.claude/` that holds nothing but these links does not count as using Claude Code, so `axx doctor` does not ask a Codex repository for a Claude Code MCP server. `axx init` installs them the same way, and marks them as generated in `.gitattributes`, so GitHub collapses their diffs in pull requests. Commit them so every contributor and every CI agent gets them. | Skill | Teaches | | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | `axx-acceptance-tests` | the write, validate, run loop; rules that keep tests reliable; the step index | | `axx-test-data` | seeds, payloads and mock bodies in [fixture factories](/next/guides/fixture-factories/): `axx fixtures`, identities, adoption, `axx lint`, CI | | `axx-setup` | `axx init`, services and readiness, CI, agent integration | | `axx-custom-steps` | custom steps as Go packs | | `axx-debugging` | reading failures, exit codes and logs | The skills carry no pages of steps: they send agents to `axx steps`, which lists your project’s steps, your custom packs’ included, a line each, and to `axx steps show ` for one step’s documentation. Agents read pages a chunk at a time, about ten times the tokens. Rerun `axx skills install` after upgrading Axx; files you edited are kept unless you pass `--force`, and files the skills no longer have are removed. The same skills are published at [`/.well-known/agent-skills/index.json`](/next/.well-known/agent-skills/index.json). ## MCP server [Section titled “MCP server”](#mcp-server) `axx mcp` serves Axx over the [Model Context Protocol](https://modelcontextprotocol.io) on stdio. Because it is the installed binary, its answers match your Axx version and your project’s steps. | Tool | Does | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `steps_search` | without a query, the catalog: every step of the project, one line each; with one, the steps that fit it (id, expression, table columns and an example; 6 by default), a step it returned before as its id only | | `step_explain` | with a line, how it matches (or the closest steps: first the line written out when it keeps an expression’s notation, or with its ordinal moved where the step has it); with an id or several, the steps’ documentation and examples | | `feature_validate` | check feature files or feature text without running; warnings come from `axx lint`’s feature checks (a scenario that checks several behaviors in turn among them), and hints name scenarios whose checks prove little | | `lint_run` | run `axx lint`: values such as seed ids that collide across files, with `file:line`, and the rules that found something | | `scenarios_run` | run scenarios (paths, tags, names); returns failures with expected and actual, the warnings `feature_validate` and `lint_run` would give, and for a passing run their hints | | `failure_context` | logs, attachments and the last request and response of one failure; without a run ID, of the latest run | | `env` | `up`, `down` or `status` of the services: which are running, left over from a killed run, or not cleaned up (`axx env` on the command line) | | `config_show` | the effective `axx.yaml`, with secrets redacted, and its packs (`axx config show`) | | `scaffold` | starter contents for a feature or an `axx.yaml` | | `steps_try` | try steps in a live scenario that stays open between calls, until `restart` | Packs add tools of their own, which look at the scenario `steps_try` keeps open. The `web-core` pack’s: | Tool | Does | | ---------- | --------------------------------------------------------------------------------------------------------------------------------- | | `web_page` | the page the session’s web app is on: its elements as steps name them, what a screen reader reads, script errors and a screenshot | An agent tries the steps it is unsure of, looks at the page they led to, then writes them into the feature: ```json {"steps": "Given the portal web app with the following properties:\n | url | http://localhost:8400/portal |\nWhen the \"/quote\" page is opened"} ``` The services must be running (`env` `up`). The session’s browsers close when the agent restarts it or the server stops. It also serves the `axx.yaml` JSON Schema as a resource and a `write-acceptance-tests` prompt. Pass `--profile ci` (or any profile) to apply it to every tool call. ### Connect an agent [Section titled “Connect an agent”](#connect-an-agent) `axx mcp install --agent ` adds the server to one agent’s configuration: in the repository by default, or with `--scope user` in your home directory, which only this command writes, and only when you ask. Like `axx init`, it keeps what the file holds and leaves a file that has axx alone. `--dry-run` shows what it would write. ```sh axx mcp install --agent cursor # .cursor/mcp.json axx mcp install --agent codex --scope user # ~/.codex/config.toml ``` | `--agent` | `--scope project` (the default) | `--scope user` | | ---------------------------------- | ------------------------------- | ----------------------------------------------------------------------- | | `claude` (Claude Code) | `.mcp.json` | prints `claude mcp add --scope user axx -- axx mcp` | | `codex` (Codex) | `.codex/config.toml` | `~/.codex/config.toml` (`$CODEX_HOME/config.toml` when set) | | `cursor` (Cursor) | `.cursor/mcp.json` | `~/.cursor/mcp.json` | | `vscode` (VS Code, GitHub Copilot) | `.vscode/mcp.json` | prints `code --add-mcp '{"name":"axx","command":"axx","args":["mcp"]}'` | | `gemini` (Gemini CLI) | `.gemini/settings.json` | `~/.gemini/settings.json` | Claude Code keeps your own servers with the rest of its state, and VS Code in its profile, so for them Axx prints the agent’s own command instead of editing those files. Codex loads a project’s `.codex/config.toml` only once you trust the project. This is what each agent’s file gets. #### Claude Code [Section titled “Claude Code”](#claude-code) .mcp.json ```json { "mcpServers": { "axx": { "command": "axx", "args": ["mcp"] } } } ``` Claude Code asks before it starts a server from a project’s `.mcp.json`. Or install the skills and the server together as a plugin: ```text /plugin marketplace add nimbusxr/axx /plugin install axx@nimbusxr ``` #### Codex [Section titled “Codex”](#codex) .codex/config.toml or \~/.codex/config.toml ```toml [mcp_servers.axx] command = "axx" args = ["mcp"] ``` #### Cursor [Section titled “Cursor”](#cursor) .cursor/mcp.json ```json { "mcpServers": { "axx": { "command": "axx", "args": ["mcp"] } } } ``` #### VS Code (GitHub Copilot) [Section titled “VS Code (GitHub Copilot)”](#vs-code-github-copilot) .vscode/mcp.json ```json { "servers": { "axx": { "type": "stdio", "command": "axx", "args": ["mcp"] } } } ``` #### Gemini CLI [Section titled “Gemini CLI”](#gemini-cli) .gemini/settings.json ```json { "mcpServers": { "axx": { "command": "axx", "args": ["mcp"] } } } ``` ## Check the setup [Section titled “Check the setup”](#check-the-setup) `axx doctor` checks, for the agents the repository uses (and Codex, when your machine has it), that the skills are installed and that the agent finds the axx MCP server, in the repository’s file, your own, or (Claude Code) the plugin. What is missing is a warning with the command that fixes it: ```console ok agent skills 5 skills ok Claude Code skills 5 skills ok Claude Code MCP axx server in .mcp.json warn Cursor MCP no axx server in .cursor/mcp.json → `axx mcp install --agent cursor` adds it ``` ## Agents in a sandbox without Docker [Section titled “Agents in a sandbox without Docker”](#agents-in-a-sandbox-without-docker) Some harnesses run the agent in a sandbox that may not use Docker, so its runs cannot start the services. Start `axx host` in the project on your own machine: the agent’s runs, `axx up` and MCP sessions get their services from it, clean for each run, and it starts only what you approved. [Run tests from an agent’s sandbox](/next/guides/run-from-a-sandbox/) shows how. ## AGENTS.md [Section titled “AGENTS.md”](#agentsmd) `axx init` adds a managed section to `AGENTS.md`, the file most agents read first. Running `axx init` again updates the section in place and leaves the rest of the file alone: AGENTS.md ```markdown ## Acceptance tests (axx) axx (github.com/nimbusxr/axx, "axxeptance") verifies running software against executable acceptance criteria (Gherkin), black-box. - Writing tests: `axx steps` lists every step, a line each (read it once; `axx steps show ` gives one step's documentation); write one scenario per acceptance criterion with those steps, never invented ones, written out (a {name} replaced with a value, a(n) as "a" or "an", [[...]] kept without the brackets or left out); `axx up` once, then `axx run --compact`, which also reports what validate and lint find; `axx down` when done. - Use the axx MCP tools when you have them: `steps_search` without a query is the step list, then `scenarios_run` (it reports what `feature_validate` and `lint_run` find) and `failure_context`. - If `.agents/skills` has the axx skills, they teach the details: axx-acceptance-tests, axx-test-data (fixture factories), axx-debugging. `axx skills install` installs or updates them. - Feature files are acceptance criteria a person can read, in plain language, with the steps as written. No programming constructs in Gherkin. - Every scenario uses unique data (ids, names, keys): scenarios run in parallel and data persists. - Check what the service did (a response property, a row, an event) and why it refused, not only status codes. - `services:` in axx.yaml are what runs on its own, reached by an address; a program its user starts (a CLI, an MCP server over stdio) is run by its step's `command`, as its user would, and is not a service. - Steps come from the packs in `axx-packs.yaml` (`axx pack list`, `axx pack add `). Configuration is in `axx.yaml` (`axx schema --outline` lists its keys); features in `features/`. - Repeating payloads, seeds or mock bodies: fixture factories generate them (`axx fixtures`; optional, strongly recommended where data repeats); `axx fixtures adopt` turns hand-written ones into one. ``` Claude Code reads `AGENTS.md` only where no `CLAUDE.md`, `.claude/CLAUDE.md` or `CLAUDE.local.md` is in the folder or one above it; where one is, it reads those instead. So in a repository that uses Claude Code, `axx init` adds `@AGENTS.md`, Claude Code’s import, to the project’s `CLAUDE.md` (or to `.claude/CLAUDE.md`, or to a new `CLAUDE.md` when the one Claude Code reads is your own `CLAUDE.local.md` or a folder above), and Claude Code reads the section with it. A `CLAUDE.md` that imports `AGENTS.md` already, or links to it, is left alone. ## Docs for agents [Section titled “Docs for agents”](#docs-for-agents) * Every page on this site has a Markdown twin: append `.md` to the path (`/guides/set-up-agents.md`). Pages advertise it with ``. * [`/llms.txt`](/next/llms.txt) indexes the site; [`/llms-full.txt`](/next/llms-full.txt) and [`/llms-small.txt`](/next/llms-small.txt) contain it in one file. * The `axx.yaml` schema is at [`/schemas/v0/axx.schema.json`](/next/schemas/v0/axx.schema.json); `axx schema` prints the same thing offline. # Set up your editor > See Axx's steps in feature files as you write them - problems flagged as you type, step completion, docs on hover and highlighted parameters - in IntelliJ IDEA, VS Code or any editor with LSP support. Axx’s steps are defined inside the `axx` binary, not in your project, so an editor cannot find them on its own. `axx lsp`, a language server built into the binary, tells the editor about them. The IntelliJ plugin and the VS Code extension start it for you. In feature files you get: * undefined, ambiguous and misused steps, and Gherkin syntax errors, flagged as you type, with the closest real steps; * completion of step text, with the step’s parameters as placeholders; * the step’s documentation, and the values of its parameters, on hover; * go to a step’s definition: the Go code that defines it when that code is on your machine (your custom packs, or axx built from source), otherwise the step’s entry in a reference page; * the files that steps name: Ctrl/Cmd+click a seed, a payload, a schema, an OpenAPI document or a log’s `file://` url to open the file, get its path completed as you type, and a warning when it does not exist (with a pointer to `axx fixtures generate` for a fixture that is not generated yet). These are the steps’ `{filepath}` parameters and file properties, and the Examples cells that fill them in; the file is found the way the step finds it, in the `resources` directories, next to `axx.yaml` or at an absolute path. A log’s file may appear only during the run, so it is never flagged; * highlighted parameter values and Scenario Outline ``. The steps are the ones your project loads, custom packs included ([Choose packs](/next/guides/use-packs/)). The server finds the project’s `axx.yaml` itself, even when it is in a subdirectory of the folder you opened. Pre-release The IntelliJ plugin is not listed on the JetBrains Marketplace, and the VS Code extension is not on the Visual Studio Marketplace. Build them from the repository: `./gradlew buildPlugin` in `ide/intellij`, `npm ci && npm run package` in `ide/vscode`. ## IntelliJ IDEA [Section titled “IntelliJ IDEA”](#intellij-idea) Needs IntelliJ IDEA 2026.1.4 or later (or another JetBrains IDE of that version). 1. Install the **axx** plugin from the JetBrains Marketplace: *Settings | Plugins | Marketplace*, search for axx. 2. Install the **Gherkin** plugin too: it highlights Gherkin keywords, and the axx plugin needs it for the run icons in the gutter, for running a feature file or directory, and for breakpoints on steps. The Gherkin plugin, and Cucumber+ if you have it, look for step definitions in code and would mark every Axx step as undefined, so the axx plugin turns those inspections off in Axx projects. 3. Open a `.feature` file. The server appears in the **Language Services** widget in the status bar, where you can restart it. If `axx` is not on your `PATH`, set **axx executable** in *Settings | Tools | axx*. ## VS Code [Section titled “VS Code”](#vs-code) 1. Install the **axx** extension: *Extensions: Install from VSIX…* with the `.vsix` file `npm run package` writes. 2. Disable the official Cucumber extension in Axx workspaces. It looks for step definitions in code and marks every Axx step as undefined. 3. Open a `.feature` file. The extension highlights Gherkin itself and starts the server once the workspace is trusted. If `axx` is not on your `PATH`, set `axx.path` to its location. The **axx** output channel shows the server’s log. ## Other editors [Section titled “Other editors”](#other-editors) Any editor with LSP support can run `axx lsp` for `*.feature` files. It talks over stdio and finds the project from its working directory. In Neovim 0.11 or later, which gives `.feature` files the `cucumber` file type: init.lua ```lua vim.lsp.config('axx', { cmd = { 'axx', 'lsp' }, filetypes = { 'cucumber' }, root_markers = { 'axx.yaml', '.git' }, }) vim.lsp.enable('axx') ``` ## Run and debug scenarios [Section titled “Run and debug scenarios”](#run-and-debug-scenarios) Both plugins run scenarios from the editor, with the results in the IDE’s test view: features, scenarios and steps, each one linked to its line, with expected and actual values for failed assertions. * **IntelliJ IDEA:** click the run icon in the gutter of a Feature, Rule, Scenario or Scenario Outline line, or of an Examples row, or choose *Run* on a feature file or directory. *Rerun Failed Tests* is in the test view’s toolbar. * **VS Code:** use the run buttons in the gutter or the Testing view. ### With profiles [Section titled “With profiles”](#with-profiles) A run from the editor can apply the project’s [profiles](/next/references/config/#finding-and-merging-files), in order, as `axx run --profile ios,watch` does: * **IntelliJ IDEA:** the axx run configuration has a **Profiles** list with the project’s profiles: check the ones to apply, and move them up and down to set their order. To start every run from the gutter, the editor and the project view with them, check them in the axx configuration template (*Run | Edit Configurations | Edit configuration templates | axx*); those runs are named with them, like `features (ios)`. For one file or directory, *More Run/Debug | Modify Run Configuration…* in its context menu opens its configuration, with the list. * **VS Code:** the gear next to *Run*, *Debug* and *Watch* in the Testing view picks the profiles all three apply, kept in the workspace’s `axx.profiles` setting. They apply in the order the setting lists them; a project without one of them runs without it. *Debug* instead of *Run* stops at breakpoints in the Go code of the steps ([Stop in step code](/next/guides/debug-failures/#stop-in-step-code)). It needs a Go debugger: GoLand or IntelliJ IDEA with the Go plugin, or the Go extension in VS Code. Without one, *Debug* runs the scenarios without stopping in step code; breakpoints on steps still pause. ## Watch a run [Section titled “Watch a run”](#watch-a-run) Scenarios can run with what they do on your screen: their browsers in windows, an iOS simulator in Device Hub, an Android emulator in its window. They run one scenario at a time, slowed down so you can follow what happens ([Watch a run](/next/guides/watch-runs/)). * **IntelliJ IDEA:** choose *Watch* next to *Run* and *Debug*, in the gutter menu of a Feature, Rule, Scenario or Examples row line, or in the context menu of a feature file or directory. To watch every run of an **axx** run configuration, check **Watch the browsers** in it. * **VS Code:** run with the **Watch** profile: from the menu next to the run button in the Testing view, or with *Execute Using Profile* on a gutter button. Watching runs `axx run --workers 1 --watch --slowdown 300ms`. Set the slowdown in milliseconds, `0` for none: **Slowdown when watching the browsers** in *Settings | Tools | axx*, or `axx.watch.slowdown` in VS Code. To stop before a step, click in the gutter of its line to set a breakpoint, then *Debug*. The run pauses before that step, and Playwright’s Inspector opens next to the browser: resume the run, step through its actions, pick an element or record new ones. When debugging, a failed scenario also pauses there, at the step that failed. *Debug* adds `--debug-steps --workers 1 --set packs.web-core.pauseOnFailure=true` to the run, and `--pause-at` for each breakpoint, as in `axx run features/.feature --pause-at features/.feature:`. *Watch* ignores breakpoints. While a scenario is paused, put the cursor on a step to highlight the element it names in the page; the status bar says what the step finds, or why it would fail. To try a step in the paused scenario, run it: *Run Step in Paused Scenario* in the editor’s context menu (Ctrl+Alt+Shift+R in IntelliJ IDEA), or the **Run in paused scenario** link above each step in VS Code. *Insert Recorded Steps*, in the editor’s context menu, inserts the steps you recorded in the Inspector below the cursor. ## Open traces and videos [Section titled “Open traces and videos”](#open-traces-and-videos) When a scenario keeps a Playwright trace (a failed one does, by default) or a video, the editor adds it to the scenario’s test: * **IntelliJ IDEA:** the scenario’s output in the test view ends with an *Open trace* or *Play video* link, and the scenario’s context menu has **Open Trace** and **Play Video**. * **VS Code:** right-click the scenario in the Testing view, or its gutter button, for **Open Trace** and **Play Video**. The scenario’s test output has links too. A trace opens in an editor tab, in Playwright’s trace viewer. The viewer comes with the Playwright driver Axx downloads, and the editor serves it with the trace from your machine, on `127.0.0.1`: nothing but the viewer and the files runs named. So traces open after the run too, offline. Without the driver’s viewer, the editor shows the file in your file manager: drop it on [trace.playwright.dev](https://trace.playwright.dev). A video plays in an editor tab. In IntelliJ IDEA without its built-in browser (JCEF), traces open in your default browser, and videos in your video player. In VS Code, **Open in Browser** at the top of each tab opens its page in your default browser. ## The axx host [Section titled “The axx host”](#the-axx-host) For an agent whose sandbox keeps it from Docker, [`axx host`](/next/guides/run-from-a-sandbox/) starts the project’s services on your machine. Both plugins can run it for you: * **VS Code:** turn on `axx.host.start` in your user settings, and the host starts when a workspace folder whose `axx.yaml` declares services opens. A workspace’s own settings can’t turn it on, because an agent can write them. **axx: Start axx host** and **axx: Stop axx host** start and stop it by hand. * **IntelliJ IDEA:** check **Start the axx host when an axx project opens** in *Settings | Tools | axx*. It’s an IDE setting, kept outside the project, so an agent can’t turn it on. *Tools | Start axx Host* and *Tools | Stop axx Host* start and stop it by hand. The plugin stops the host when the project closes. Stopping it stops the services of a run in progress and cleans up. While the host runs: * A run whose services changed, or the host as it starts, asks you in a notification, which lists what in the change reaches past a container. It stays until you answer, as the terminal’s question does. **Show Diff** opens the approved and the proposed definition in the diff view. **Approve** lets runs start the services, and **Refuse** fails the run that waits. Only a click there answers: no command or action approves, so neither other plugins nor an agent’s editor tools can. If you closed the notification, click the host in the status bar, or use **Show axx host approval** (*Tools* menu in IntelliJ IDEA, Command Palette in VS Code), to see it again. * The runs the host serves that the editor didn’t start, such as the agent’s, show in the test results as they run, named after the run (`axx host: run 4 (axx run --compact)`): a tab of the Run tool window in IntelliJ IDEA, which opens for them without taking the focus, and a test run of their own in VS Code’s Testing view. * Your own runs from the editor go through the host too, and take their turn with the agent’s. * The status bar shows what the host is doing: ready, a run’s services starting or running, waiting for your approval, or already running in a terminal. A host you started in a terminal serves the editor’s runs too, and asks there. * The host’s log is in the **axx host** output channel in VS Code, and in IntelliJ IDEA’s log (*Help | Show Log in Finder/Explorer*). ## Projects and their packs [Section titled “Projects and their packs”](#projects-and-their-packs) Each Axx project in the editor (each `axx.yaml`) gets the steps of its own packs, so a repository with several projects works as one with a single project does. When a project’s `axx.yaml`, its `axx-packs.yaml` or the code of a pack of its own changes, the server prepares the project again and loads its steps: a step you add to your pack is known once you save it. Until the new steps are ready, the ones before them answer, and if they cannot load, the editor says why and keeps them. After you upgrade Axx, restart the server: from the Language Services widget in IntelliJ, with **axx: Restart language server** in VS Code, or by restarting the editor. # Stub embedded models > Replace a model your app or service runs itself with a stub: a model of the real one's format and contract (ONNX, TFLite, Core ML, ExecuTorch, OpenVINO, MNN, ncnn, TensorFlow, Keras, TensorFlow.js, TorchScript, PyTorch Mobile, GGUF, safetensors, CTranslate2, whisper.cpp, Parakeet, LiteRT-LM, MNN-LLM, MLC (WebLLM), Tesseract, Piper, Foundation Models), a fraction of its size, whose answers are its mappings. The app's runtime loads it as it loads the real model. A model the platform provides (Vision, Speech, Foundation Models and Apple's server model, ML Kit, Gemini Nano, a browser's built-in AI: its Prompt API, Summarizer, Writer, Rewriter, Translator, LanguageDetector, Proofreader and speech recognition; the Windows AI APIs) gets a stub of its API, and a Windows or Linux desktop's voice and Linux's Tesseract a stub of their own format. Some models your apps and services run themselves: an image classifier on a phone (LiteRT on Android, Core ML on iOS), a model a service loads with ONNX Runtime, a language model it runs with llama.cpp. These are embedded models. A real one fills a simulator’s or an emulator’s memory, takes seconds to load, and answers as it likes. It also will not, on demand, give the answer that sends a claim to a person. A stub replaces the model. It is a model of the real one’s format, with the same inputs and outputs, and its answers are its mappings. The app’s runtime loads it as it loads the real model, and an input like one of the mappings’ gets that mapping’s answer. The app’s test build or test deployment puts the stub where the app gets its model, the way a compose file points a service at WireMock. Nothing in the app changes. A model the platform provides (Vision, Speech and Foundation Models on iOS and macOS; Android’s ML Kit, Gemini Nano and speech recognizer; a browser’s built-in AI; the Windows AI APIs) has no file of the app’s to replace: a stub of its API answers the app’s calls in its test runs instead. A Windows or Linux desktop’s voice, and Linux’s Tesseract, take a stub of their own format, a Piper voice or a Tesseract language ([Models the platform provides](#models-the-platform-provides)). Models a service reaches over HTTP (OpenAI, Anthropic, Gemini, Bedrock, Ollama) are mocked with WireMock instead: see [Test AI features](/next/guides/test-ai-features/). ## Four steps, for every model [Section titled “Four steps, for every model”](#four-steps-for-every-model) Every model is stubbed the same way, whatever its format and wherever the app runs it: 1. **[Write its contract](#1-write-the-contract)**, once, from the real model: what the app relies on from it. 2. **[Map inputs to answers](#2-map-inputs-to-answers)**: the pictures (or other inputs) your scenarios give the app, and what the model answers each. 3. **[Build the stub](#3-build-the-stub)** with `axx models build`, and keep it with the app’s sources. 4. **[Put it where the app gets its model](#4-put-it-where-the-app-gets-its-model)**, in the test build or the test deployment. Then the scenarios test what the app does with each answer, with the steps they already use. There are no steps for models: the app runs its model, and the scenarios see what it does. List the `models-core` pack in the project’s `axx-packs.yaml`, and give each model a directory of the acceptance project: ```text models/evidence-check/ contract.json what the app relies on from the model (step 1) mappings/ damage-photos.json the inputs it answers, and its answers (step 2) default.json the answer to every other input ``` ## 1. Write the contract [Section titled “1. Write the contract”](#1-write-the-contract) The contract is what the app relies on from the model: * its inputs and outputs (names, types, shapes); * an image input’s layout; * a classifier’s labels; * how an integer output stands for scores (`quant`); * the version of its format, so the app’s runtime loads the stub as it loads the model. Write it from the real model, once, and keep it with the project. No test needs the real model after that. ```sh axx models contract mobilenetv2-7.onnx --task classification --labels labels.txt \ -o models/evidence-check/contract.json ``` | Format | Models | What the contract takes from the model | | -------------------------- | --------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ONNX | `.onnx` | Its inputs and outputs, IR version, operator sets, operators and metadata, and the files beside it its weights are in (external data). A classifier’s labels come from `--labels`. A speech recognizer (Whisper’s encoder, `encoder_model*.onnx`, its decoder beside it) is a `speech-to-text` model’s: its input a sound’s log-mel spectrogram of the size its `preprocessor_config.json` gives, its decoder’s inputs and outputs, and the tokens its `generation_config.json` ends and suppresses, with its tokenizer in `model-files`. A language model’s decoder (`input_ids`, `attention_mask` and its cache, `past_key_values`, in; `logits` and `present` out, as Hugging Face’s exporters and ONNX Runtime GenAI write them) is a `text-generation` model’s: the token that ends its turn comes from its folder’s configs, and its tokenizer (`tokenizer.json`, beside it or in the folder above, as the model’s repository has it) goes in `model-files`. An image model’s image processor (`preprocessor_config.json`, beside it or in the folder above, as transformers.js and Optimum keep it) says how it fits a picture to its input, in RGB: stretched to its `size` of a height and a width (ViT’s, BiRefNet’s), or its shorter side resized to its `shortest_edge` and its middle cropped to its `crop_size` (ConvNeXT’s, CLIP’s: `crop` its share). | | TFLite (LiteRT) | `.tflite` | Its first subgraph’s inputs and outputs (their quantization too), its signature, operators, metadata buffers and the files appended to it. A classifier’s labels come from its metadata, when it has them. | | ExecuTorch | `.pte` | Its forward method’s inputs and outputs, by their places (`input0`, `output0`…: an app gives and gets them in order), its other methods’ (React Native ExecuTorch’s text recognizers have a method for each width of a strip of text), and the kernels and delegates it calls. Only the program is read: the head of the file, before its weights. A stub has every method, each of its own shapes. A language model’s program (forward of its tokens and their position, and methods that give its settings, `get_max_seq_len`, `get_eos_ids`…, as ExecuTorch’s LLM export writes it) is a `text-generation` model’s: its settings are the real program’s, the token that ends its turn is one of its `get_eos_ids`, and its tokenizer (`tokenizer.json`, beside it or in the two folders above, as the model’s repository has it) goes in `model-files`. A speech recognizer’s program (`encode` of a sound’s samples, `decode` of a token, its position and the encoder’s state, as React Native ExecuTorch exports Whisper) is a `speech-to-text` model’s: its input `encode`’s sound (16 kHz), and its tokenizer in `model-files`. | | TensorFlow SavedModel | `saved_model.pb` | Its meta graph for serving (its tags) and its signature (`serving_default`, or its first): its inputs and outputs by their keys, which TF Serving and `signatures[...]` call it by, and the names of their tensors in its graph (`serving_default_inputs:0`, `StatefulPartitionedCall:0`), which a session of TensorFlow 1 runs it by; the operators of its graph and its functions, and the version of TensorFlow that saved it. Its variables are not read. | | TensorFlow\.js | `model.json` (its shards beside it) | A graph model’s signature (as TensorFlow\.js’s converter writes it from version 2): its inputs and outputs by their keys, in their order (`predict` gives and gets them in it), and the operators of its graph. A graph model of no signature (of the converter before version 2, as tfjs-models’ older models are): its placeholders and the nodes no node takes, by their names, as TensorFlow\.js runs it; its outputs’ shapes come from `--output`. Its integers are int32s, as TensorFlow\.js has them. A layers model (Keras’s layers, `tf.loadLayersModel`) is not read. | | Keras | `.keras`, `.h5` | A Functional or Sequential model’s config (a `.keras` file’s `config.json`, a `.h5` file’s `model_config`, of Keras 3 or Keras 2): its inputs (their names, types and shapes), its outputs by their layers’ names, and their shapes where its last layers say them (a `Dense`’s units, a `Reshape`‘s shape); the version of Keras that saved it, and its layers. Its weights are not read. Other outputs’ shapes, and a subclassed model’s inputs and outputs, come from `--input` and `--output`. | | TorchScript | `.pt`, `.torchscript` | Its forward’s tensors (its parameters by their names, its outputs by their places: `output0`, `output1`…), its class, and the operators its code calls. TorchScript does not say its tensors’ types and shapes: they come from what the model’s folder has (an Ultralytics export’s config, inside it: its pictures’ size, its task and labels; Triton’s `config.pbtxt` beside or above it), or from `--input` and `--output`. | | PyTorch Mobile | `.ptl` (a module’s `_save_for_lite_interpreter`: Android’s and iOS’s PyTorch Lite runtimes, `LiteModuleLoader`) | As TorchScript’s, from the code the file keeps beside its bytecode, and the operators its bytecode calls (`aten::conv2d`, `aten::add.Tensor`…), those a selective build of PyTorch Mobile made for the model alone has. | | MNN | `.mnn` | Its inputs (their names, types and dimensions), its outputs’ names (those it names, or the tensors no op takes, as MNN’s loader has them), and the ops it uses. MNN does not say its outputs’ types and shapes: they come from `--output`. | | ncnn | `.param` (its `.bin` beside it) | Its inputs (its `Input` layers’ blobs, of the dimensions they give: a picture’s of no batch, CHW, as ncnn’s `from_pixels` gives it), its outputs’ names (the blobs no layer takes), and the layers it uses. ncnn does not say its outputs’ shapes, nor (pnnx’s models) its inputs’: they come from `--input` and `--output`, of no batch, as ncnn’s blobs are. | | OpenVINO | `.xml` (its `.bin` beside it) | Its inputs (its Parameters) and outputs (its Results, by their tensors’ names), its IR version and the operations it uses; a classifier’s labels, and whether the app gives a picture’s channels reversed (BGR), from the `config.json` beside it (Open Model Zoo’s). | | Core ML | `.mlmodel`, `.mlpackage` | Its inputs (an image’s size and colors, or a multiarray), its outputs (a classifier’s label and probabilities, with its class labels, or multiarrays), its specification’s version and metadata. A detector’s pipeline (a network of candidates, then a non-maximum suppression: Create ML’s object detectors, YOLO’s Core ML exports, as Vision runs them) is a `detection` model’s of its suppression’s labels, its outputs the objects it keeps (`confidence`, `coordinates`); the contract carries its suppression and its description. A text encoder (its tokens’ ids, `input_ids`, as Hugging Face’s exporters and coremltools convert BERT’s and MiniLM’s) matches texts with its tokenizer: the `tokenizer.json` beside it or in the folder above, or `--vocab vocab.txt`. | | safetensors (transformers) | `model.safetensors` (a folder of it) | A language model’s folder, as Hugging Face’s repositories have it: its vocabulary’s size and context length (`config.json`), the token that ends its turn (`generation_config.json`’s, or `config.json`‘s), and its tokenizer (`tokenizer.json`) in `model-files`. A text embedding model’s folder (an encoder’s `config.json`: BERT’s, MPNet’s…; or sentence-transformers’ `modules.json`) is an `embedding` model’s of its hidden size, or of its last Dense module’s features (EmbeddingGemma’s: 768 to 3072 to 768). A speech recognizer’s folder (Whisper’s: a `config.json` of `model_type` `whisper`, its `generation_config.json` and `preprocessor_config.json`) is a `speech-to-text` model’s: its sizes, the tokens its transcripts begin and end with and the ones it suppresses, its features’ size, and its tokenizer in `model-files`. | | CTranslate2 | `model.bin` (a folder of it) | A Whisper’s (its spec’s, `WhisperSpec`: faster-whisper’s models): its sizes (its variables’), its settings (its scalars), the tokens its transcripts begin and end with (its tokenizer’s, `tokenizer.json` beside it, which goes in `model-files`) and those its `config.json` suppresses. | | whisper.cpp | `ggml-*.bin` | A Whisper’s model file, as whisper.cpp’s converters write it: its sizes, the type of its weights, the tokens whisper.cpp begins and ends its transcripts with, and its mel filters and vocabulary (tiktoken’s, by their ranks), which go in `model-files`. | | Parakeet (parakeet.cpp) | `.gguf` (Parakeet’s, as parakeet.cpp’s converters write NVIDIA’s CTC, TDT and hybrid TDT-CTC models) | Its features (their mels and sample rate) and its vocabulary’s size. Its task is `speech-to-text`, its input the sound (`"features": "parakeet"`): parakeet.cpp computes the features from the file’s own frontend, so a stub has a frontend of its own, whatever the real model’s. | | OpenAI Whisper | `.pt` (OpenAI’s checkpoints: `tiny.en.pt`…) | A checkpoint’s dims, the tokens the `openai-whisper` package begins and ends its transcripts with and suppresses, and its vocabulary: the package’s `whisper/assets/multilingual.tiktoken` (or `gpt2.tiktoken`, of an English-only model), which goes in `model-files`. Copy it beside the checkpoint first. A TorchScript module, also a `.pt` file, is told apart by its content. | | Tesseract | `.traineddata` | The contract is written by hand: a grayscale image input of a fixed size, and the CTC output’s labels (the characters it reads, space, and the blank `""`). Its task is `text-recognition`. The stub reads an image of the contract’s size as one line, as it is (page segmentation mode 13, which its config sets, and an app’s `--psm 13` keeps): an app gives it crops of one line, as an app reads a field of a form or a label. | | Piper | `.onnx` (with `.onnx.json` beside it) | The voice’s inputs and output, sample rate and `phoneme_id_map`, and its sidecar config. Its task is `text-to-speech`; a mapping’s `text` is found anywhere in what the app speaks, in any case (Piper reads a text case-folded), and its `audio` is a WAV, resampled to the voice’s rate. The output waveform is fixed in the contract (use `--output` when the real model’s is open: Piper’s voices say as long as they speak). Piper’s runtimes (its Python package, its command-line program) leave a pause after each sentence (0.2 s), and scale a waveform to its loudest sample. | | GGUF (llama.cpp) | `.gguf` | A language model’s: its task (`text-generation`), its context length, the token that ends its turn, and its metadata (tokenizer, chat template), which goes in `model-files`. An embedding model’s (one that pools its tokens’ outputs: BERT’s, bge-m3, nomic-embed, EmbeddingGemma, Qwen3-Embedding) is an `embedding` model’s: its embedding’s size and its pooling (mean, the first token’s, the last’s). A reranker’s (a classifier’s, `cls`, of RANK pooling: bge-reranker, jina-reranker, Qwen3-Reranker) is a `classification` model’s of one label, `relevant`, its output `score` the logit of the mapping’s `score`: llama.cpp’s server answers a BERT reranker’s logit and node-llama-cpp its sigmoid, and both answer a Qwen3 reranker’s probability (its yes and no, softmaxed), the mapping’s `score` itself. Its tokenizer is BPE, SentencePiece’s (BPE or unigram: XLM-R’s) or WordPiece. | | LiteRT-LM | `.litertlm` | An on-device language model’s, as Android and iOS apps run them with LiteRT-LM (Gemma’s): its KV cache’s length, its prefill signatures, the token that ends its turn, and its LLM metadata and tokenizer (SentencePiece’s, as Gemma’s, or Hugging Face’s, as Qwen’s and SmolLM’s), which go in `model-files`. | | MNN-LLM | `config.json` (of its folder: `llm_config.json`, `llm.mnn`, `tokenizer.txt` or `tokenizer.mtok`) | An on-device language model’s, as MNN’s LLM engine runs it (`Llm::createLLM`: Android’s and iOS’s MNN apps): its context length (`max_all_tokens`), the token that ends its turn (its chat template’s, or its tokenizer’s stop token that ends a turn), and its configs and tokenizer, which go in `model-files`. axx tokenizes as MNN’s engine does, of each of its tokenizers: Hugging Face’s BPE (`tokenizer.txt` of type 3: its expression of bytes), tiktoken’s (type 1: the longest token first), SentencePiece’s (type 0: a vocabulary MNN’s export wrote of spaces, which MNN’s encoder writes as their bytes) and its pipeline (`tokenizer.mtok`, a Hugging Face tokenizer’s in binary), held to MNN’s tokens of Qwen2.5’s, Llama 3.2’s, MiniCPM4’s and Gemma 3’s. | | MLC (WebLLM) | `mlc-chat-config.json` (of its folder: `tensor-cache.json`, `tokenizer.json`, the weights’ shards) | A language model as MLC LLM compiles it (WebLLM’s, in a page, on its GPU; MLC LLM’s apps’): its architecture and quantization (the model library compiled for them runs the stub, as it runs the real model), its context window, the token that ends its turn (its chat template’s), its manifests of its weights’ records and its tokenizer, which go in `model-files`. The architectures of WebLLM’s models: llama’s (SmolLM2, Llama 3, TinyLlama), Mistral’s, Qwen 2’s, 3’s and 3.5’s, Phi-3’s (Phi-3.5, Phi-4 mini, Phi-3.5 vision’s text), Phi-2’s (and Phi-1.5’s), Gemma’s, Gemma 2’s and Gemma 3’s, OLMo 2’s, StableLM’s and GPT-NeoX’s (RedPajama), of each quantization WebLLM’s models come in: `q4f16_1`, `q4f32_1`, `q0f16`, `q0f32` (and `q3f16_1`). Not Ministral 3’s: MLC’s YaRN turns the two halves of each of its heads’ pairs by different angles, so that no head of it scores a key by its offset alone, as a stub’s heads do. | | Foundation Models | (a Swift type) | None: the stub stands in for a type of the app’s own of the `LanguageModel` protocol, and its contract is written by hand (the system’s model, `SystemLanguageModel`, gets a stub of its API instead: `axx models contract ios.foundation-models`). Its format is `foundation-models`, of `details.name`, the stub’s Swift type, and `details.capabilities`, what it can do as `LanguageModelCapabilities` names it (without it, the system model’s: `guidedGeneration`, `toolCalling` and `vision`); its task is `text-generation`. It has no inputs or outputs: its stub answers a prompt. | A contract of an app’s own Foundation Models model, written by hand: models/notes/contract.json ```json { "format": { "type": "foundation-models", "details": { "name": "NotesModelStub" } }, "task": "text-generation" } ``` A model the platform provides has no file to read: its API names it, and `axx models contract` writes the API’s contract ([Models the platform provides](#models-the-platform-provides)): ```sh axx models contract ios.vision.text -o models/label-reader/contract.json ``` A model that does not say a tensor’s type and shape (a TorchScript module’s, an MNN or ncnn model’s outputs’), or that the app gives otherwise (an open batch the app always gives one picture), takes them from `--input` and `--output`, by the tensor’s name, in quotes (shells read `[` and `?`): ```sh axx models contract photo-check.pt --task classification --labels labels.txt \ --input 'images=float32[1,3,224,224]' --output 'output0=float32[1,1000]' \ -o models/photo-check/contract.json ``` When the model says none and the flags do not either, the command names each tensor and the flags that give it. What the model carries besides its graph (a TFLite model’s metadata and label files) goes in a `model-files` folder beside the contract, and the stub carries it too: an app that reads its labels from the model’s metadata reads the stub’s. A language model’s contract keeps its tokenizer, so a stub reads a prompt in the real model’s tokens and writes its answers in them: its metadata (1.7 MB for SmolLM2’s, up to 10 MB for the largest vocabularies) goes in `model-files`. axx tokenizes as the model’s runtime does: * a GGUF model’s as llama.cpp does, with its byte-level BPE tokenizers (Llama 3, Qwen 2.5, SmolLM2 and the others llama.cpp splits words for alike), SentencePiece (Mistral, Gemma 3, Phi-3) and WordPiece (BERT’s embedding models: bge, all-MiniLM, nomic-embed), held to llama.cpp’s tokens of those models; * an ONNX, ExecuTorch or safetensors model’s as Hugging Face’s tokenizers do (`tokenizer.json`), with BPE models’ (byte-level, or of `▁` for spaces with their bytes’ fallback), their normalizers (NFC, NFKC) and their pre-tokenizers’ splits, held to transformers.js’s tokens of Llama 3.2, Gemma 3, Phi-4, LFM2, SmolLM2, Qwen 2.5 and Qwen 3. An ONNX language model’s stub runs in transformers.js (pages, Electron apps, Node.js services) and in ONNX Runtime GenAI (Windows, Android and iOS apps, services), with the real model’s configs: each runtime keeps the model’s cache its own way, and the stub keeps up with both. A safetensors text embedding model’s stub embeds in sentence-transformers and in transformers’ `AutoModel`, 4.57 and 5 alike, exactly as its mappings say. A safetensors language model’s stub runs in Hugging Face’s transformers (`AutoModelForCausalLM`’s `generate`, and the `text-generation` pipeline), 4.57 and 5 alike, in the real model’s folder. An ExecuTorch language model’s stub runs in ExecuTorch’s text LLM runner (React Native ExecuTorch, ExecuTorch’s Android and iOS libraries), with the real model’s tokenizer: its prompt in one call, or a token a call (as the runner gives a program of fixed shapes its prompt), and a conversation the runner continues from chat to chat. An ExecuTorch speech recognizer’s stub (Whisper’s, as React Native ExecuTorch runs it) transcribes a sound of any length up to 30 seconds. A Whisper’s stub, of its weights (safetensors, CTranslate2’s `model.bin`, whisper.cpp’s model file, OpenAI’s checkpoint), transcribes in transformers (its pipeline, in float32, float16 and bfloat16, on CPUs and on a Mac’s GPU), in faster-whisper (CTranslate2, in float32 and in int8, which rounds every layer’s inputs and weights to 8 bits, as faster-whisper runs on CPUs), in whisper.cpp (on CPUs and on Metal, of its flash attention) and in `openai-whisper` (`whisper.load_model` of the stub’s path: of a model’s name, it downloads the real one), of the language the runtime detects or the app gives, with timestamps or without. A Tesseract stub is a small, float LSTM `.traineddata`, with whole-word unichars and no dictionaries. Tesseract’s own recognizer matches the picture and decodes the answer. The model’s config reads the whole picture as one raw line (page segmentation mode 13) and does not try a second, inverted recognition. An application that sets its own page segmentation mode overrides that config, as it does with any traineddata. A Piper voice stub is an ONNX model with its `.onnx.json` beside it. The sidecar keeps the real voice’s settings but uses `phoneme_type: "text"`, so codepoints are the phonemes and no eSpeak installation is needed. Each mapping’s `audio` WAV is the answer the real PiperVoice synthesizes, resampled to the voice’s rate. ```sh axx models contract SmolLM2-360M-Instruct-Q4_K_M.gguf -o models/description-triage/contract.json axx models contract Llama-3.2-1B-Instruct-ONNX/onnx/model_q4f16.onnx -o models/customs-draft/contract.json ``` A model that reads a text (a text classifier or embedder: BERT’s, MediaPipe’s) has its tokens’ ids for input: its contract’s `text` says which input, and how the app tokenizes the text, with the vocabulary it carries. `axx models contract` reads it from a TFLite model’s metadata (MediaPipe’s WordPiece and regex tokenizers), and from an ONNX model’s folder (its `tokenizer.json`, Hugging Face’s: WordPiece’s, as BERT’s normalizer has it, or BPE’s), with its task: a classifier of the labels its `config.json` names, or an embedder of its `last_hidden_state` and `sentence_embedding`. Give another ONNX model’s with `--vocab vocab.txt`, its WordPiece vocabulary. An input of a sound’s samples (a classifier of sounds: YAMNet, a Create ML sound classifier) gets `audio`, its sample rate: `axx models contract` reads it from a TFLite model’s metadata and from a Create ML sound classifier’s (whose labels it reads from the classifier at the end of its pipeline); give it others’ (`"audio": {"sampleRate": 16000}`). An input shaped like an image gets an image layout. Check what the app does with a picture before it gives it to the model, and say so in the contract: | `image` key | What it says | | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `layout` | The order of the input’s dimensions: `NCHW`, `NHWC`, or `CHW` and `HWC` without a batch. | | `channels` | `RGB` (the default), `BGR` or `gray`. | | `resize` | How the app fits a picture to the input’s size: `center-crop` (the default: the middle of it, at the input’s proportions) or `stretch`. | | `crop` | The share of the fitted picture’s middle the app keeps, of each side: torchvision’s and timm’s evaluation transforms (the shorter side resized to 256, its middle 224 kept) keep `0.875`. The default keeps all of it. | How the app scales the pixels (0 to 255, 0 to 1, a mean and a deviation for each channel) makes no difference: matching undoes it. A detector’s contract says what each of its outputs holds, with `detection`. `axx models contract` reads it from a TFLite model’s metadata (MediaPipe’s detectors, TensorFlow Lite Model Maker’s), and `--task detection --labels labels.txt` names the outputs of others: | `detection` | What the output holds | Detectors | | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------- | | `boxes`, `classes`, `scores`, `count` | The objects, first: their boxes, their labels’ indexes (the labels the `classes` output’s), their scores, and how many there are. | SSD and EfficientDet with their post-processing (TensorFlow’s object detection API, the TFLite detection post-processing) | | `anchors` | Each anchor’s box and its labels’ scores; the app suppresses the overlapping ones. | YOLO (v8 and later), as ONNX and TFLite export it | | `locations`, `classScores` | Each anchor’s box, encoded against the anchor (`anchors`: the anchors’ file, the box coder’s scales), and each anchor’s labels’ scores. The app decodes the boxes. | MediaPipe’s object detector (EfficientDet-Lite) | A box’s values are in the order `box.order` gives (`ymin,xmin,ymax,xmax`, TensorFlow’s, by default for `boxes`; `cx,cy,width,height` in pixels, YOLO’s, for `anchors`), as shares of the input picture’s size or, with `box.pixels`, in its pixels. MediaPipe’s tasks stretch a picture to the model’s input: give its contract’s image `resize: stretch`. A classifier’s output says what it holds with `activation`: * `softmax`: probabilities that add up to 1 (the default); * `sigmoid`: a probability for each label; * `none`: logits, which the app turns into probabilities. ## 2. Map inputs to answers [Section titled “2. Map inputs to answers”](#2-map-inputs-to-answers) A mapping is a JSON file of the `mappings` folder, as WireMock’s stubs are, or one of several under `"mappings"`: models/evidence-check/mappings/damage-photos.json ```json { "mappings": [ { "name": "A crushed carton on the porch", "input": {"image": "../../../features/damage-claims/evidence/crushed-box.png"}, "output": {"label": "carton", "score": 0.91} }, { "name": "A screenshot of the shop's order page", "input": {"image": "../../../features/damage-claims/evidence/order-screenshot.png"}, "output": {"label": "web site", "score": 0.88} } ] } ``` A mapping’s `input` is what it answers: * `image`: a picture (PNG, JPEG, GIF or WebP) beside the mapping or anywhere; * `audio`: a sound (a WAV file), for an input of a sound’s samples (its first samples, as many as the input has, at its sample rate, resampled from the file’s), or of its features (a speech recognizer’s log-mel spectrogram, which the stub computes from the sound as the app’s processor does); * `tensor`: the input’s values, as nested arrays or a JSON file of them; * `text`: a text the prompt has, for a language model, or the text a text classifier or embedder (BERT’s) reads. An input of several keys that answer alike lists them: `{"text": ["green gate", "wooden box"]}` answers a prompt that has either, and `{"image": ["parcel-front.jpg", "parcel-side.jpg"]}` either photo (a tensor’s values, a list of numbers, are one key). Each is a key of its own, which no other key of the stub may be. Its `output` is the answer, in the terms of the model’s task: * **A classifier** answers `label` (with `score`, 1 by default) or `scores`, by label. * **A detector** answers `objects`: each object’s `label`, its `box` (`x`, `y`, `width` and `height`, shares of the picture’s width and height from its top left corner) and its `score` (1 by default). The stub gives them as the detector’s outputs hold them, and nothing else. * **A segmenter** answers `mask`: a picture (beside the mapping, stretched to the output’s size) whose gray levels are each pixel’s label’s index (0 for the first label); or, for a segmenter of one label or none (a selfie segmenter’s person, a text detector’s probability map: PaddleOCR’s), each pixel’s confidence, 255 being 1. An output of `activation: none` holds logits, which the app takes a sigmoid of (a background remover’s: BiRefNet’s, RMBG’s): the stub gives each pixel the logit of its confidence, ±20 for 0 and 1. A segmenter of each label’s own confidence (a text detector’s of its text’s regions and the links between their characters: CRAFT’s, as EasyOCR and React Native ExecuTorch run it) answers `masks`: a picture a label, `{"region": "door-region.png", "affinity": "door-links.png"}`, each pixel’s gray level the label’s confidence; a label without one is 0. * **An embedder** (`embedding`) answers `embedding`: the values of each output of its size, or a name. An output of each token’s (`last_hidden_state`) has them at every token, so the app’s mean of them is the embedding. A name is a unit vector of its own (of SHA-256 of the name), so the texts of the mappings that answer one name are one embedding apart (their cosine 1), and those of two names nearly at right angles: a search that should find an article for a question maps both to one name. * **A language model** (`text-generation`) answers `text`, what it writes; `json` or `toolCalls`, a JSON value or an array of calls it writes compactly (a structured output); or `refusal`, its refusal as text. * **A text-to-speech model** answers `audio`: a WAV file, resampled to the voice’s sample rate and written as the model’s fixed waveform, with silence after it or its end trimmed. * **A text recognizer** (`text-recognition`: CRNN’s, PaddleOCR’s, EasyOCR’s) answers `text`, what it reads, and `score` (1 by default): its output’s steps hold each character’s class, a blank between two alike, then blanks, as CTC decoding reads them. Its contract’s labels are its classes, the blank an empty one (`--labels`: a file of a character a line, the blank an empty line). * **A speech recognizer** (`speech-to-text`) answers `text`, what it hears said. It writes it after a space, as Whisper does, and, when the app asks for timestamps, from 0 to where the sound ends. A text with a symbol the recognizer’s runtime never lets it write (Whisper’s `generation_config.json` suppresses `:`, `#`, brackets and others) stops the build, named. * **Any model but a language model** can give its outputs’ values themselves, as `tensors`, by output name. An output an answer does not give is zeros. One mapping is the default. It answers every input no other mapping matches. A model cannot refuse an input, so a stub always has one: models/evidence-check/mappings/default.json ```json { "name": "Any other photo: the model is unsure", "default": true, "output": {"label": "carton", "score": 0.3} } ``` A language model’s mappings, in the parcel-claims example: models/description-triage/mappings/descriptions.json ```json { "mappings": [ { "name": "The customer changed their mind", "input": {"text": "changed their mind"}, "output": {"json": {"transitDamage": false, "summary": "The customer changed their mind."}} } ] } ``` ### Which input matches a mapping [Section titled “Which input matches a mapping”](#which-input-matches-a-mapping) **A text** a text classifier or embedder reads matches the first mapping whose text it has, as words, in the model’s tokens: the stub compares them with the input’s ids. A BPE tokenizer’s tokens of a text depend on what is around it (a space before it, a comma after it), so the stub compares every way the text may be tokenized. **A prompt** matches the first mapping whose text it has, as words: `changed their mind` matches “The customer changed their mind and sent it back”, wherever the app puts the shop’s description in its prompt (its system prompt, its chat template, quotes around it), but not “changed their minds”. Letters’ cases count. A prompt that has no mapping’s text gets the default’s answer. The stub writes the answer token by token, then ends its turn, at any temperature, and through any grammar the answer keeps to. A language model reads its answer as it writes it, so an answer may not have the text of a mapping before it: the build stops and names both. **Any other input** matches the mapping whose input it is most like, when it is alike enough. The contract’s `match.similarity` sets how alike, from 0 to 1 (0.9 by default). Similarity is measured on a reduced form of each input: * **A picture** is pooled to a grid of at most 16×16, each channel standardized: a phone camera’s noise, another resolution or a slightly different crop still match. * **A sound** is reduced to its spectrum: its power in 32 bands as wide in mels (finer low, as hearing is), averaged over frames of about 256 samples, as logs, standardized. The sound a little later in the app’s window, or quieter or louder, still matches. * **A speech recognizer’s features** (a sound’s log-mel spectrogram, Whisper’s) are pooled to 16 bands over each tenth of a second, each tenth’s bands centered and scaled to the same loudness (what is said, not how loud), a tenth of no sound (silence, a faint hum) left out, then the whole scaled alike. The same recording, louder, quieter or a little noisy, matches; the same speaker’s other words do not. * **Any other tensor** is standardized as a whole. The same input gets the same answer every time. A mapping’s picture is the picture a scenario gives the app (the photo it uploads, the camera’s scene), so use the same file in both. ## 3. Build the stub [Section titled “3. Build the stub”](#3-build-the-stub) ```sh axx models build models/evidence-check -o ../infra/models/evidence-check.onnx ``` The stub is a model of the contract’s format, with the real model’s inputs, outputs and version, and the files it carries. It is a fraction of the real model’s size: a stub of an ONNX MobileNetV2 (14 MB) is 30 KB, of a TFLite MobileNet (4.3 MB) 38 KB, of Apple’s Core ML MobileNetV2 (24.7 MB) 65 KB, of SmolLM2-360M-Instruct (270 MB) 6.7 MB. A language model’s stub is most of all its vocabulary: up to 35 MB for the largest (Gemma’s 262,000 tokens). An ONNX language model’s stub is its graph alone, 28 to 52 KB, and an ExecuTorch one’s its program alone, 59 KB for SmolLM2’s (173 MB): its tokenizer and configs stay the real model’s files, in its folder. A stub of an ONNX model whose weights are in files beside it (`model_q4f16.onnx_data`) comes with empty files of their names: a runtime that loads them with the model (transformers.js, as the model’s `config.json` says) finds them, and the stub has no weights in them. A mistake stops the build and names the mapping: * an unknown label; * a tensor of the wrong size; * a picture of one color, which is like no input; * two mappings of the same input; * a second default, or none. Keep the stub with the app’s sources, where step 4 puts it, as you keep generated files. A stub is built the same, bit for bit, on every machine: build it again when its contract or mappings change (with the project’s other generation steps), and have CI build it and check that nothing changed. The app’s build then needs neither axx nor the real model. ## 4. Put it where the app gets its model [Section titled “4. Put it where the app gets its model”](#4-put-it-where-the-app-gets-its-model) The stub takes the real model’s place where the app already gets its model. Only the test build or the test deployment changes: | The app gets its model | The stub goes | Example | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | From its own package: an Android app’s assets, an iOS app’s bundle | Into the test build’s package, under the real model’s name | The parcels example’s couriers’ apps | | From its own files: a model it downloaded the first time it needed it, and keeps | Into the app’s files, before the step that starts the app: a folder of the app’s (the files pack’s `owner: app:`) and its “is a copy of” step. A mobile app is reset before each scenario, and starts only when a step starts it | The parcels example’s couriers’ Android app (its delivery notes’ model, and its damage reports’ MNN-LLM model, a folder of files) | | From a file its configuration names: a service’s model path | Into the test deployment, at that path | The services of the parcel-claims, carrier-billing, customs-clearance and parcel-support examples | | From the system: a model a distribution’s package installs (Debian’s tesseract-ocr: `/usr/share/tesseract-ocr/5/tessdata/eng.traineddata`) | Over the package’s file, in the test deployment’s container (a file mounted in its place) | The parcel-support example’s service (its label reader) | | From a model server that serves the model to it over the network (TF Serving: `//saved_model.pb`; OpenVINO Model Server: `//model.xml`) | Into the test deployment’s model folder, as the model’s version. The app then reaches its model over HTTP, which the WireMock image’s model mock answers too | | | From an engine built for its GPU (TensorRT’s `.engine` or `.plan`, which Triton’s TensorRT backend serves as `model.plan`), built from an ONNX model | Built, from the ONNX stub of the ONNX model’s contract, as the deployment builds the real model’s engine (`trtexec --onnx`, with its shapes and precision), on the GPU and TensorRT the test deployment runs: an engine is of one GPU and one version of TensorRT. An engine of half precision rounds the stub’s values, its answers the same | | | From a URL its configuration names, which it downloads the model from | To WireMock, which serves the stub at that URL: a mapping whose response’s `bodyFileName` is the stub | | | From the platform, through its API (Vision’s requests, ML Kit’s clients, Foundation Models’ sessions of the system’s model and of Apple’s server model, a browser’s `LanguageModel`) | A stub of the API (`.axxmodel`), into the app’s `axx-models` folder, before the step that starts the app or opens its first page: see [Models the platform provides](#models-the-platform-provides) | The parcels example’s couriers’ apps (their photo checks; the iOS app’s delivery notes and missed-delivery cards) and its web portal | | From a type of its own of Foundation Models’ `LanguageModel` protocol (iOS, macOS and visionOS 27), which its sessions take (`LanguageModelSession(model:)`) | Into the test build’s sources: the stub is a Swift type of the protocol, which the test build gives its sessions in place of the app’s own | The parcels example’s couriers’ iOS app (its notes model, on an iPhone without Apple Intelligence) | | From a model’s folder (transformers.js’s: `config.json`, the tokenizer’s files, `onnx/model_q4f16.onnx`; ONNX Runtime GenAI’s: `genai_config.json`, the tokenizer’s files, `model.onnx`; transformers’: `config.json`, the tokenizer’s files, `model.safetensors`; faster-whisper’s: `config.json`, `tokenizer.json`, `vocabulary.txt`, `model.bin`; MNN-LLM’s: `config.json`, `llm_config.json`, `llm.mnn`, `llm.mnn.weight`, `embeddings_bf16.bin` (or the embeddings in `llm.mnn.weight`), `tokenizer.txt` or `tokenizer.mtok`; MLC’s, which a web app’s WebLLM fetches from the URL its app config gives the model: `mlc-chat-config.json`, `tensor-cache.json`, `tokenizer.json`, `params_shard_*.bin`) | Into the test build’s copy of the folder, in place of its model’s file (and, of a transformers language model’s, its `config.json`), beside the real tokenizer and configs. An MNN-LLM stub is the whole folder: `axx models build -o /llm.mnn` writes the stub’s files and, beside them, the real `config.json` and tokenizer. An MLC stub is the whole folder too (`axx models build -o `), served where the test build’s app config says the model is, and the real model’s library runs it | The parcels example’s shop portal (its tracking page’s WebLLM model too), the parcel-support example’s service (faster-whisper, sentence-transformers and transformers) | A framework that runs the app’s own model gets it the same way: the couriers’ iOS app runs its model with Vision, from its bundle. A sound classifier an app runs with Sound Analysis (`SNClassifySoundRequest(mlModel:)`, a Core ML model of a sound’s samples, as Create ML makes them) is the app’s too: the couriers’ iOS app hears the doorbell or a knock with one when no one is home. Sound Analysis cuts what the microphone hears into the model’s windows and runs the stub on each, as it runs the real model, and a mapping’s recording matches each window of it. **An Android app.** The debug build’s assets (`src/debug/assets`) have the stub, under the real model’s name, and a release build fetches the real model. The couriers’ app runs it with ML Kit’s custom image labeling, which reads the labels from the model’s metadata and stretches a photo to the model’s input (its contract says `resize: stretch`): ```sh axx models build models/photo-check-android -o ../courier/android/app/src/debug/assets/photo-check.tflite ``` Its damage check is the company’s own PyTorch model, saved for PyTorch Mobile, which PyTorch Android Lite runs: the debug build’s assets have its stub, and a release build takes the company’s model from the path its build gives (`-PdamageCheckModel=`): ```sh axx models build models/damage-check-android -o ../courier/android/app/src/debug/assets/damage-check.ptl ``` **An iOS app.** The stub is kept with the app’s sources (`Courier/Models/PhotoCheck.mlmodel`), and a build phase compiles it into the app in a Debug build (`xcrun coremlcompiler compile`, as Xcode compiles a model), and the real model in a Release build: ```sh axx models build models/photo-check-ios -o ../courier/ios/Courier/Models/PhotoCheck.mlmodel ``` **A model of the app’s own, of Foundation Models’ protocol.** From iOS 27 a session takes a model of the app’s (`LanguageModelSession(model:)`, of the `LanguageModel` protocol). An app that ships a type of its own of the protocol gets a stub of that type: a Swift file, kept with the app’s sources, of the type the contract names (`format.details.name`, and `format.details.capabilities`, what the app’s model can do: the system model’s unless it says), which the Debug build compiles and gives its sessions, and which Release builds leave out (`EXCLUDED_SOURCE_FILE_NAMES = CourierNotesModelStub.swift`). The couriers’ iOS app summarizes a long delivery note with a model of its own on an iPhone without Apple Intelligence: in a Release build, SmolLM2 (a GGUF it bundles) on llama.cpp, behind its type `CourierNotesModel`; in a Debug build, which leaves that type and llama.cpp out, the stub: ```sh axx models build models/delivery-note-own-ios -o ../courier/ios/Courier/Models/CourierNotesModelStub.swift ``` Courier/DeliveryNotes.swift (excerpt) ```swift let system = SystemLanguageModel.default if system.isAvailable { return try await respond(LanguageModelSession(model: system, instructions: instructions), note) } // An iPhone without Apple Intelligence: the app's own model. #if DEBUG let own = CourierNotesModelStub() // the stub #else guard let own = CourierNotesModel() else { return nil } // SmolLM2, on llama.cpp #endif return try await respond(LanguageModelSession(model: own, instructions: instructions), note) ``` Its scenario stubs the system’s model as an iPhone without Apple Intelligence (see [Models the platform provides](#models-the-platform-provides)), so that the app takes its own. The stub builds with Xcode 27 (Swift 6.4), and runs on iOS, macOS and visionOS 27: an older Xcode builds its file empty. Sessions of the system’s model (`SystemLanguageModel`) need no stub in the app’s sources: see [Models the platform provides](#models-the-platform-provides). **A service.** The test deployment mounts the stubs at the paths the service loads its models from. The parcel-claims service checks photos with ONNX Runtime, and triages the shops’ descriptions of damage with a language model it runs with llama.cpp: infra/compose.yaml (excerpt) ```yaml claims: environment: CLAIMS_PHOTO_MODEL: /models/evidence-check.onnx CLAIMS_TRIAGE_MODEL: /models/description-triage.gguf volumes: - ./models/evidence-check.onnx:/models/evidence-check.onnx:ro - ./models/description-triage.gguf:/models/description-triage.gguf:ro ``` **A web page.** The parcels example’s shop portal drafts a parcel’s customs description with a small language model the page runs itself (transformers.js, on ONNX Runtime’s WebAssembly), from the folder the service serves its models from. The test deployment mounts a folder of the real model’s configs and tokenizer, and the stub in place of its ONNX file: ```sh axx models build models/customs-draft -o ../infra/portal-models/customs-draft/onnx/model_q4f16.onnx ``` It cuts a photo of a parcel’s contents out of its background the same way (transformers.js’s `background-removal`, of BiRefNet\_lite, whose image processor’s configuration the folder keeps beside the stub); a screenshot of the cutout checks the page: ```sh axx models build models/contents-cutout -o ../infra/portal-models/contents-cutout/onnx/model_fp16.onnx ``` infra/compose.yaml (excerpt) ```yaml app: environment: PARCELS_PORTAL_MODELS: /var/lib/parcels/portal-models volumes: - './portal-models:/var/lib/parcels/portal-models:ro' ``` Then the scenarios test what the app does with each answer, with the steps they already use: ```gherkin Scenario: A photo that shows no parcel goes to a person Given a seeds/PX-4111.yaml dynamodb seed And a POST request to /api/claims And a request payload using an application/json content example And the request payload properties are: | parcel | PX-4111 | | reason | DAMAGED | And the request is executed When the evidence/order-screenshot.png file is uploaded to the claim-evidence s3 bucket as claims/CLM-4111/order-screenshot.png Then within 30s the claim-reviews s3 bucket has an object named CLM-4111/order-screenshot.png And the claims dynamodb table has an item where: | id | CLM-4111 | | status | REFERRED | | note | the photo shows no parcel: it looks like a web site (88%) | ``` ### The examples’ stubs [Section titled “The examples’ stubs”](#the-examples-stubs) Each of axx’s examples runs its stubs embedded, in the real runtime inside the app’s own process (or on the device, or in the page), the way the app runs the real model, and its acceptance suite passes with them: | Example | What the model does | Runtime | Format | | ------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | | [`parcel-claims`](https://github.com/nimbusxr/axx/tree/main/examples/parcel-claims) | checks that an evidence photo shows a parcel | ONNX Runtime, in the service | ONNX | | | triages a damage claim’s description | llama.cpp, in the service | GGUF | | | transcribes a customer’s voice note | whisper.cpp, in the service | whisper.cpp’s (ggml) | | [`carrier-billing`](https://github.com/nimbusxr/axx/tree/main/examples/carrier-billing) | checks that a repack’s photo shows a parcel | TensorFlow (its C library), in the service | TensorFlow’s SavedModel | | [`customs-clearance`](https://github.com/nimbusxr/axx/tree/main/examples/customs-clearance) | names what an inspection’s photo shows | OpenVINO (its C library), in the service | OpenVINO’s IR | | [`parcel-support`](https://github.com/nimbusxr/axx/tree/main/examples/parcel-support) | hears whether anyone speaks in a voicemail | MediaPipe (its audio classifier), in the service | TFLite (YAMNet) | | | transcribes a voicemail (a telephone’s 8 kHz) | faster-whisper (CTranslate2); OpenAI’s whisper, transformers and parakeet.cpp (its command), its profiles | CTranslate2’s; OpenAI’s `.pt`; safetensors; Parakeet’s GGUF (NVIDIA’s Parakeet TDT-CTC 110M) | | | finds the help-centre article that answers a message | sentence-transformers; llama.cpp (llama-cpp-python), its profile | safetensors; GGUF | | | reranks the articles near a message | llama.cpp (llama-cpp-python), in the service | GGUF | | | drafts a reply from the article | transformers; ONNX Runtime GenAI, its profile | safetensors; ONNX (GenAI’s model builder’s) | | | speaks the confirmation a call-back plays to a customer who left a voicemail | Piper (its command-line program), in the service | Piper’s voice (`.onnx` and its `.onnx.json`) | | | reads the reference on a parcel’s label, from a photo of its reference line | Tesseract (the system’s: Debian’s tesseract-ocr), from the service | Tesseract’s traineddata | | | says what a customer’s photo of their parcel shows | PyTorch (TorchScript); ncnn, MNN, ExecuTorch (XNNPACK) and Keras (on PyTorch), its profiles | TorchScript; ncnn’s `.param`; `.mnn`; `.pte`; `.keras` | | [`parcels`](https://github.com/nimbusxr/axx/tree/main/examples/parcels) | checks a proof-of-delivery photo, in the couriers’ Android app | ML Kit (custom image labeling, of the app’s own model) | TFLite | | | checks a proof-of-delivery photo, in the couriers’ iOS app | Core ML (Vision) | Core ML | | | refuses a proof-of-delivery photo of the parcel cut off at its edge, in the couriers’ iOS app | Core ML (Vision), the app’s own object detector (Create ML’s) | Core ML (a detector’s pipeline) | | | summarizes a recipient’s delivery note on an iPhone without Apple Intelligence, in the couriers’ iOS app | the app’s own model, of Foundation Models’ `LanguageModel` protocol (SmolLM2 on llama.cpp, in a Release build) | a Swift type of `LanguageModel` | | | hears the doorbell or a knock when no one is home, and records the attempt with it, in the couriers’ iOS app (the microphone step’s recording, played into the app’s own capture) | Core ML (Sound Analysis), the app’s own sound classifier (Create ML’s) | Core ML (a pipeline classifier) | | | the same, in the couriers’ Android app | MediaPipe (its audio classifier), of the app’s YAMNet | TFLite | | | reports a parcel damaged with a photo of the damage, which the company’s own damage classifier checks, in the couriers’ Android app | PyTorch Android Lite (PyTorch Mobile’s lite interpreter) | PyTorch Mobile (`.ptl`) | | | writes out a courier’s note of a parcel’s damage for the damage report, in the couriers’ Android app | MNN’s LLM engine (MNN-LLM, built into the app’s native code), of Gemma 3 270M | MNN-LLM | | | drafts a parcel’s customs description, in the shop portal’s page | transformers.js (ONNX Runtime Web) | ONNX | | | cuts a photo of a parcel’s contents out of its background, in the shop portal’s page | transformers.js (ONNX Runtime Web), its background removal (BiRefNet\_lite) | ONNX (a mask of logits) | | | checks a recipient’s photo of a damaged parcel, in the tracking page | TensorFlow\.js | TensorFlow\.js graph model | | | suggests why a recipient sends a parcel back, from their own words, in the tracking page | WebLLM (SmolLM2-135M, on the browser’s GPU through WebGPU) | MLC (WebLLM), of float32s (`q4f32_1`), which every GPU runs | | | summarizes a recipient’s delivery note, in the couriers’ Android app | LiteRT-LM; Gemini Nano (ML Kit’s Prompt API), on a phone that has it | `.litertlm`; a stub of its API (`.axxmodel`) | | | summarizes a recipient’s delivery note, in the couriers’ iOS app | Foundation Models (iOS 27), the system’s model | a stub of its API (`.axxmodel`) | | | drafts the card a courier leaves when no one is home, in the couriers’ iOS app | Foundation Models (iOS 27), Apple’s server model (Private Cloud Compute) | a stub of its API (`.axxmodel`) | | | tells a photo of the recipient’s signed note from a proof-of-delivery photo, reads the parcel’s label in one, sees a dog beside the parcel and blurs out the people in it, in the couriers’ iOS app | Vision: `VNClassifyImageRequest`, `VNRecognizeTextRequest`, `VNRecognizeAnimalsRequest`, `VNDetectHumanRectanglesRequest`, `VNGeneratePersonSegmentationRequest` | stubs of their APIs (`.axxmodel`) | | | the same, and refuses a photo of the parcel cut off at its edge, in the couriers’ Android app | ML Kit: image labeling, text recognition, object detection, selfie segmentation | stubs of their APIs (`.axxmodel`) | | | reads a parcel’s reference off the depot’s label camera’s scan, keeps a scan only when it shows a parcel and blurs out the people in it, in the depot desk’s Mac builds (AppKit, SwiftUI) | Vision: `VNRecognizeTextRequest`, `VNClassifyImageRequest`, `VNDetectHumanRectanglesRequest`, `VNGeneratePersonSegmentationRequest` | stubs of their APIs (`.axxmodel`) | | | takes the clerk’s dictated handover note, and sums up the day for the next shift, on the Mac or on Apple’s servers, in the depot desk’s Mac builds | Speech (`SFSpeechRecognizer`, of `AVAudioEngine`’s audio); Foundation Models (macOS 27), the system’s model and Apple’s server model | stubs of their APIs (`.axxmodel`) | | | reads the handover rules aloud, in the depot desk’s Windows builds (Windows Forms, WPF, WinUI); reads a parcel’s reference off a scan, keeps a scan only when it shows a parcel, and sums up the day for the next shift, in its WinUI build | SAPI (System.Speech), with a Piper voice; the Windows AI APIs: `TextRecognizer`, `ImageDescriptionGenerator`, `LanguageModel` (Phi Silica) | a Piper voice stub; stubs of their APIs (`.axxmodel`) | | | reads a parcel’s reference off a scan with the system’s Tesseract, and reads the handover rules aloud through the desktop’s speech service, in the depot desk’s GTK builds | Tesseract’s language data; speech-dispatcher, with a Piper voice | a Tesseract stub (`eng.traineddata`); a Piper voice stub | | | takes a courier’s dictated note, in the couriers’ iOS and Android apps (the microphone step’s recording, played into the app’s own capture) | Speech (`SFSpeechRecognizer`, of `AVAudioEngine`’s audio); Android’s `SpeechRecognizer` | stubs of their APIs (`.axxmodel`) | | | puts a recipient’s delivery instructions in English, checks their spelling and takes them spoken, in the tracking page | the browser’s built-in AI: `LanguageDetector`, `Translator`, `Proofreader`, `SpeechRecognition` | stubs of their APIs (`.axxmodel`) | | | fills in the address a customer wrote, drafts a message to a parcel’s recipient and makes it shorter, and puts a long delivery note in a line, in the shop portal | the browser’s built-in AI: the Prompt API (`LanguageModel`, JSON), `Writer`, `Rewriter`, `Summarizer` | stubs of their APIs (`.axxmodel`) | ### Models the platform provides [Section titled “Models the platform provides”](#models-the-platform-provides) A model the platform provides is the platform’s: the app calls an API (Vision’s `VNRecognizeTextRequest`, ML Kit’s `TextRecognition`, Foundation Models’ sessions of `SystemLanguageModel` and of `PrivateCloudComputeLanguageModel`, a browser’s `LanguageModel`), and the platform runs its own model, in no file of the app’s. Such a model gets a stub of its API: a `.axxmodel` file, built from a contract and mappings as every stub is, which the scenario puts in the app’s `axx-models` folder. A pack of the platform’s then answers the app’s calls from it, in the app’s test runs: | Pack | Where it answers | How | | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `models-ios` | A Debug app on the iOS Simulator (iOS 16 or later; Foundation Models: iOS 27 and Xcode 27) | A library the simulator loads into the app when axx launches it (`DYLD_INSERT_LIBRARIES`), which axx compiles with Xcode and caches | | `models-macos` | A Mac app’s test build, signed without the hardened runtime or with its two exceptions (below), on macOS 15 or later (Foundation Models: macOS 27) | The same library, which macOS loads into the app when axx launches it (`DYLD_INSERT_LIBRARIES`) | | `models-linux` | A Linux app in the desktops `desktop-linux` runs | Tesseract’s documented `TESSDATA_PREFIX`, and speech-dispatcher’s configuration in the app’s home, which speaks with Piper | | `models-windows` | A Windows app in the desktops `desktop-windows` runs; for the Windows AI APIs, a self-contained Windows App SDK build with package identity | A SAPI voice of axx’s, which speaks with Piper, made the user’s default voice for the scenario; and a library of axx’s in place of the app’s own Windows AI API files, which axx builds with MSVC and C++/WinRT | | `models-android` | A debuggable app on the Android emulator (Android 8 or later) | An agent attached to the app as it starts (ART’s tooling interface, `am start --attach-agent`), which axx builds with the SDK’s NDK and caches | | `models-web` | The browsers the `web-core` pack runs | A script the browser runs in the app’s pages before their own (Playwright’s init scripts) | The app’s code does not change, and nothing changes in its release builds or on other devices. Each pack needs `models-core`, `app-core` and its platform’s pack (`mobile-ios`, `desktop-macos`, `desktop-linux`, `desktop-windows`, `mobile-android`, `web-core`); the `files` pack copies the stub. **A Mac app’s test build.** macOS ignores `DYLD_INSERT_LIBRARIES` in an app it signs with the hardened runtime, which would leave the app with Apple’s models instead of the stubs. A Mac app’s test build is signed without it, or with Apple’s two exceptions for it, in its Debug entitlements: `com.apple.security.cs.allow-dyld-environment-variables` and `com.apple.security.cs.disable-library-validation`. axx checks the app before it starts it, and stops the scenario if macOS would keep the library out (`AXX-E0704`). A Mac app’s `axx-models` folder is in its `./` files, its home’s `Library/Application Support`. A Mac has no test’s grant of speech recognition or the microphone, as a simulator has: a stubbed recognizer, and the microphone axx feeds, are authorized. **1. The contract** is the API’s: name it, and `axx models contract` writes it. A labeler’s takes the labels its mappings answer: ```sh axx models contract ios.vision.text -o models/label-reader/contract.json axx models contract android.mlkit.image-labeling --labels labels.txt -o models/hazards/contract.json ``` **2. The mappings** are as every stub’s: pictures or recordings and their answers, or texts a prompt has, and the default. **3. Build it**, and keep it with the acceptance project: ```sh axx models build models/label-reader -o models/label-reader.axxmodel ``` **4. Put it in the app’s `axx-models` folder** with the files pack’s steps, before the step that starts the app (or opens its first page). A mobile app reads its folder when it starts, and again when it restarts; a browser’s page, when it opens. ```gherkin Scenario: A recipient's long delivery note is summarized for the courier Given a seeds/courier-app-ios-note.yaml db seed And the courier-models folder with the following properties: | owner | app:courier | | path | axx-models | And the delivery-note.axxmodel file in the courier-models folder is a copy of the models/delivery-note.axxmodel file When the courier app is launched ``` | Pack | API (`axx models contract `) | What the app calls | Task | | ---------------- | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------- | | `models-ios` | `ios.speech` | `SFSpeechRecognizer`: URL and PCM buffer requests | `speech-to-text` | | | `ios.vision.classify` | `VNClassifyImageRequest` | `classification` | | | `ios.vision.text` | `VNRecognizeTextRequest` | `text-recognition` | | | `ios.vision.humans` | `VNDetectHumanRectanglesRequest` | `detection` | | | `ios.vision.animals` | `VNRecognizeAnimalsRequest` (Vision’s `Cat` and `Dog`) | `detection` | | | `ios.vision.person-segmentation` | `VNGeneratePersonSegmentationRequest` | `segmentation` | | | `ios.foundation-models` | `LanguageModelSession` of `SystemLanguageModel`: its availability, responses and streams | `text-generation` | | | `ios.private-cloud-compute` | `LanguageModelSession` of `PrivateCloudComputeLanguageModel`, Apple’s server model: its availability, responses and streams. A session of a model of the app’s own is Foundation Models’, untouched | `text-generation` | | `models-macos` | `macos.speech` | `SFSpeechRecognizer`: URL and PCM buffer requests | `speech-to-text` | | | `macos.vision.classify` | `VNClassifyImageRequest` | `classification` | | | `macos.vision.text` | `VNRecognizeTextRequest` | `text-recognition` | | | `macos.vision.humans` | `VNDetectHumanRectanglesRequest` | `detection` | | | `macos.vision.animals` | `VNRecognizeAnimalsRequest` (Vision’s `Cat` and `Dog`) | `detection` | | | `macos.vision.person-segmentation` | `VNGeneratePersonSegmentationRequest` | `segmentation` | | | `macos.foundation-models` | `LanguageModelSession` of `SystemLanguageModel`: its availability, responses and streams | `text-generation` | | | `macos.private-cloud-compute` | `LanguageModelSession` of `PrivateCloudComputeLanguageModel`, Apple’s server model, as on iOS | `text-generation` | | `models-windows` | `windows.ai.text-recognizer` | The Windows AI APIs’ `TextRecognizer` (the Windows App SDK’s `Microsoft.Windows.AI.Imaging`): its ready state, and a picture’s text, one line of its words | `text-recognition` | | | `windows.ai.image-description` | `ImageDescriptionGenerator`: a picture’s description, of every kind, streamed as its progress | `text-generation` (a picture’s mappings) | | | `windows.ai.object-extractor` | `ImageObjectExtractor`: its mask of a picture (and its hints’), of the picture’s size, whatever the hint | `segmentation` | | | `windows.ai.language-model` | `LanguageModel`, Phi Silica (`Microsoft.Windows.AI.Text`): responses, of a context’s system prompt too, streamed as their progress (not embedding vectors) | `text-generation` | | | `windows.ai.text-summarizer` | `TextSummarizer`: texts, paragraphs, and conversations (each message a line, `participant: message`) | `text-generation` | | | `windows.ai.text-rewriter` | `TextRewriter`: its tones, and a custom tone (the text, then the tone, a line each) | `text-generation` | | | `windows.ai.text-to-table` | `TextToTableConverter`: an answer’s `json` is the table’s rows, each an array of its columns | `text-generation` | | `models-android` | `android.speech` | `SpeechRecognizer`, and its on-device factory | `speech-to-text` | | | `android.mlkit.image-labeling` | ML Kit’s `ImageLabeling`, of its own model | `classification` | | | `android.mlkit.text-recognition` | ML Kit’s `TextRecognition` | `text-recognition` | | | `android.mlkit.object-detection` | ML Kit’s `ObjectDetection` (its five categories) | `detection` | | | `android.mlkit.segmentation` | ML Kit’s selfie `Segmentation` | `segmentation` | | | `android.gemini-nano` | ML Kit’s Prompt API (`Generation`): responses and streams | `text-generation` | | `models-web` | `web.speech` | `SpeechRecognition` and `webkitSpeechRecognition` | `speech-to-text` | | | `web.language-model` | The Prompt API’s `LanguageModel`: prompts, history, clones and streams | `text-generation` | | | `web.summarizer` | `Summarizer` | `text-generation` | | | `web.writer` | `Writer` | `text-generation` | | | `web.rewriter` | `Rewriter` | `text-generation` | | | `web.translator` | `Translator`: translations and streams | `text-generation` | | | `web.language-detector` | `LanguageDetector`: an answer’s `text` is the language (confidence 1), or its `json` the detector’s list of languages and confidences | `text-generation` | | | `web.proofreader` | `Proofreader`: an answer’s `text` is the corrected text (its correction, the part that changed), or its `json` the proofreader’s own result | `text-generation` | **A Linux desktop’s models.** A Linux app takes its system’s models from files its distribution installs, not from an API: Tesseract’s language data, and the voices of the desktop’s speech service, speech-dispatcher. Their stubs are those files, built from a contract of their format: a Tesseract language (`eng.traineddata`, of the `tesseract` format) and a Piper voice (its `.onnx` and `.onnx.json`, of the `piper` format). In the app’s `axx-models` folder, `models-linux` makes the app’s Tesseract read the language from there (`TESSDATA_PREFIX`), and gives the app’s speech-dispatcher the voice, which Piper speaks (its `piper` program, on the desktop’s PATH) with no sound. **A Windows desktop’s voice.** A Windows app speaks with SAPI, through System.Speech’s `SpeechSynthesizer` or SAPI’s own `SpVoice`, in the voice the user chose. With a Piper voice stub in the app’s `axx-models` folder (its `./` files, its home’s `AppData\Roaming`), `models-windows` adds a voice of its own to SAPI’s, `axx voice`, and makes it the user’s default for the scenario: an engine of axx’s, which speaks with Piper, built with the .NET SDK (8 or later) into axx’s cache. SAPI lists the voices in `HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Speech\Voices\Tokens` only, so axx runs as an administrator, as GitHub’s Windows runners do, or as a user who may create keys there; otherwise the app’s start stops with `AXX-E0704`. The scenario’s end takes the voice out and puts the user’s own default back. **The Windows AI APIs.** A Windows app calls the Windows AI APIs (`TextRecognizer`, `ImageDescriptionGenerator` and `ImageObjectExtractor`, of `Microsoft.Windows.AI.Imaging`; the language model, Phi Silica, and its skills, of `Microsoft.Windows.AI.Text`) through the Windows App SDK, whose files a self-contained build (`WindowsAppSDKSelfContained`) has in its own folder: its manifest maps the APIs’ classes to them, `Microsoft.Windows.AI.Imaging.dll` and `Microsoft.Windows.AI.Text.dll` among them. With a stub of an API in the app’s `axx-models` folder, `models-windows` puts a library of its own in that file’s place as the app starts, and the app’s own back as the scenario ends: a class the folder has a stub of is the library’s, and any other the app’s file’s. The language model’s classes are the library’s together, with a stub of the model or of any of its skills: a skill takes the model it is of, whose availability the model’s stub gives, or a skill’s. axx builds the library once for the app, against the app’s own `.winmd` files, with C++/WinRT (its release, which axx downloads, pinned) and MSVC: Visual Studio’s or its Build Tools’ “Desktop development with C++”, with a Windows 11 SDK. The library answers as Windows does, which answers only an app with package identity whose package declares the `systemAIModels` capability: an unpackaged app is refused (`UnauthorizedAccessException`), as Windows refuses it, with or without a stub. An app that is not packaged takes identity from a package of its build’s folder (packaging with external location), which its build registers, and Windows registers a package of no signature in Developer Mode only. A stub of a PC without the model gives its reason in the contract’s `format.details.availability`, in Windows’ words: `NotSupportedOnCurrentSystem` (`GetReadyState`’s answer, `EnsureReadyAsync` fails, `CreateAsync` throws, with the errors Windows gives), or `NotReady`, a model `EnsureReadyAsync` makes ready. **What the app says.** An app’s speech is heard with the step `the {word} app's speaker plays the {string} file`: an utterance of the app’s has the file’s sound. The voice stub speaks a mapping’s recording for a text that has the mapping’s key, so the file is that recording, as the voice speaks it. ```gherkin Scenario: The desk reads the handover rules aloud Given the voice.onnx file in the desk-models folder is a copy of the models/handover-voice.onnx file And the voice.onnx.json file in the desk-models folder is a copy of the models/handover-voice.onnx.json file And the depot app is launched And the "Handover" element is clicked in the depot app When the "Read aloud" button is clicked in the depot app Then within 20s the depot app's speaker plays the "recordings/handover-rules.wav" file ``` A pack answers the APIs its app’s folder has stubs of; the app’s other calls reach the platform as they would. A folder has one stub of an API; a stub that cannot be read, or a second of an API, stops the app’s start (`AXX-E0704`). **A device without the model.** A stub of `ios.foundation-models` (or `macos.foundation-models`) can be an iPhone (or a Mac) without Apple Intelligence: its contract’s `format.details.availability` gives the reason, in Apple’s words (`deviceNotEligible`, `appleIntelligenceNotEnabled` or `modelNotReady`), and `SystemLanguageModel`’s `availability` and `isAvailable` give it. A stub of `ios.private-cloud-compute` can say Apple’s server model is not there (`deviceNotEligible` or `systemNotReady`), as `PrivateCloudComputeLanguageModel`’s `availability` does. Its mappings are a default only, which the app, told the model is not there, does not ask. An iOS Simulator otherwise has the Mac’s model when the Mac has Apple Intelligence, so a scenario of an iPhone without it says so: models/no-apple-intelligence-ios/contract.json ```json { "format": {"type": "platform", "details": {"api": "ios.foundation-models", "availability": "deviceNotEligible"}}, "task": "text-generation" } ``` The Android emulator has no Gemini Nano: a scenario without its stub is a phone without it. **Speech.** The app’s microphone hears a recording when a step says so: it waits until the app listens, and the app’s stubbed recognizer answers its mapping. ```gherkin Scenario: A recipient says their instructions Given a seeds/portal-instructions-dictated.yaml db seed And the "/track/PX-WEB-5704" page is opened When the "Dictate" button is clicked And the parcels app's microphone hears the "recordings/behind-the-gate.wav" file Then the "Instructions for your courier" field has the value "Behind the green gate, please." ``` The step matches the recording to the speech stub’s mappings as the stub’s keys were made (resampled to its rate), and the recognizer answers it as a final result, with confidence 1; partial results and timings are not played. In a browser, the page’s own microphone (`getUserMedia`) records it too, then silence: a page that records its user for a model of its own (a Whisper it runs itself) hears it with no speech stub, and a `SpeechRecognition` started on that microphone’s track answers its mapping. The page hears it through Web Audio, which Firefox runs on Linux only with a sound server: on a CI machine with none, start PulseAudio with a sink that plays to nothing (`pulseaudio --start`, then `pactl load-module module-null-sink`). On a phone, the app’s own capture hears it, then silence: on iOS, `AVAudioEngine`’s input taps, `AVAudioRecorder`’s files and `AVCaptureSession`’s audio outputs; on Android, `AudioRecord` (converted to its rate, channels and format) and `MediaRecorder`‘s files. An app that runs a sound model of its own on its microphone hears the recording with no speech stub: the couriers’ apps hear the doorbell or a knock when no one is home, with the app’s own sound classifier (Sound Analysis on iOS, MediaPipe on Android). **What each pack answers**, and what it does not: * **iOS.** Vision’s requests through image and sequence request handlers (the newer Swift Vision request types are not answered). Text recognition answers one observation, of the whole picture. Detections’ boxes are shares of the picture; a segmentation’s mask is the contract’s size. A recognition of a recording, a URL or PCM buffers (non-interleaved floats); `CMSampleBuffer` speech input fails, named. Physical devices and distribution builds are not answered. * **Android.** ML Kit’s vision clients of a `Bitmap` (`InputImage.fromBitmap`; other sources fail, named), and the Prompt API of ML Kit (not every Google AI SDK). Text recognition answers one block of the whole picture; detections have no tracking IDs. ML Kit’s clients of the app’s own model (custom image labeling) run its model, as any app’s model: stub that model instead. * **Language models.** Text responses, and a stream’s answer in one piece. Pictures in prompts, tools, guided generation and token counts are not played. * **Browsers.** The app’s origin, its frames of that origin and its popups. Workers and frames of other origins are not answered. Natural Language’s `NLModel` runs the app’s own model, but only a text classifier or word tagger that Create ML made, whose parameters are in Create ML’s own format, which Apple does not document: axx writes no stubs of them. An app that runs a text model with Core ML itself (a BERT’s or MiniLM’s, converted) is stubbed as any Core ML model. ## How a stub answers [Section titled “How a stub answers”](#how-a-stub-answers) A stub’s graph reduces its input as matching does, scores it against each mapping’s input and against the similarity threshold, and gives the outputs of the best: | Format | Its graph | | ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ONNX | Standard operators (`AveragePool`, `MatMul`, `ArgMax`, `Gather`…), at the real model’s operator set, from 7 on. A language model’s is a decoder of the real one’s inputs and outputs, from operator set 13 on: each call writes its tokens in the cache’s tensors that grow, and who wrote them (a prompt, or the model), after the conversation’s positions the attention mask says it has (a cache that grows each call, as transformers.js keeps it, or a buffer of the conversation’s greatest length, as ONNX Runtime GenAI does); the others it gives back as they were (LFM2’s convolutions’ state). It finds the first mapping whose text the conversation has before the prompt’s end, and gives its answer’s next token, as far along the answer as the call is past the prompt. Tokens are kept in halves, so a cache of half-precision floats holds them exactly. A speech recognizer’s is two: its encoder matches the sound’s features to the mappings’ and gives the mapping in its hidden state; its decoder writes that mapping’s transcript after the tokens the runtime forces (its language, its task), then its end. | | TFLite | Builtin operators at their first version, which every LiteRT has (`AVERAGE_POOL_2D`, `FULLY_CONNECTED`, `ARG_MAX`, `GATHER`…). | | ExecuTorch | The ONNX stub’s graph, each operator one or more calls of ExecuTorch’s portable kernels (`avg_pool2d`, `mm`, `argmax`, `index_select`…), its tensors’ memory planned, of the contract’s fixed shapes. A language model’s is the real one’s methods of settings, and a forward of portable kernels’ calls whose planned memory keeps the conversation’s tokens, and who wrote each (a prompt, or the model: the token the call before gave, at the position after it), from call to call; it finds the first mapping whose text the conversation has before the prompt’s end, and gives its answer’s next token. A speech recognizer’s `encode` pads the sound with silence, computes Whisper’s log-mel features of it (as the app’s runtime would, in portable kernels) and matches them to the mappings’ as an ONNX recognizer’s encoder does; its `decode` writes the mapping’s transcript after the tokens the app forces, a call each. An app built with only its real model’s kernels (ExecuTorch’s selective build) lacks the stub’s: `axx models build -v` names them. | | TensorFlow SavedModel | The ONNX stub’s graph, each operator as TensorFlow’s (`AvgPool`, `BatchMatMulV2`, `ArgMax`, `GatherV2`…), a graph of constants and no variables, with the real model’s tags and signature, its inputs and outputs the real model’s tensors’ names (outputs of one node an `IdentityN` of that name: `StatefulPartitionedCall:0`, `:1`…). It is one file, `saved_model.pb`: TF Serving and `tf.saved_model.load` load the folder it is in. | | TensorFlow\.js | The SavedModel stub’s graph, its integers int32s, its constants in one shard beside `model.json` (`group1-shard1of1.bin`, which `axx models build -o model.json` writes), with the real model’s signature; of a model of no signature, none, its placeholders and outputs the real model’s nodes’ names, and no other node one no node takes. | | Keras | A Functional model of the layers every Keras has (`AveragePooling2D`, `LayerNormalization`, `Dense`, `GlobalMaxPooling1D`, `ReLU`…), its weights in HDF5: the input reduced as matching reduces it, its correlation with each key, the best’s one-hot, and the answers’ table’s product with it; a text’s tokens found by an `Embedding` and a causal `Conv1D`; a sound’s log by Keras 3’s `STFTSpectrogram` (of one sample a frame) or Keras 2’s `tf.math.log`. It is the file the real model is, of its version of Keras: a `.keras` file of Keras 3, or a `.h5` file of Keras 3 or Keras 2 (tf.keras, tf\_keras), which `load_model` loads. No layer runs Python code (`Lambda`). | | TorchScript | The ONNX stub’s graph as a frozen module’s code (`torch.avg_pool2d`, `torch.matmul`, `torch.argmax`, `torch.index_select`…), which TorchScript compiles when it loads it, its tables the module’s constants, of the real module’s class. `torch.jit.load`, LibTorch and Triton’s PyTorch backend load it. | | PyTorch Mobile | The TorchScript stub’s module, and its forward’s bytecode (version 8), which the lite interpreter runs without compiling code: each call’s arguments loaded from registers and constants, then its operator (`aten::avg_pool2d`, `aten::matmul`, `aten::argmax`, `aten::index_select`…). `axx models build -v` names the operators the real model’s bytecode does not call, which a selective build lacks. | | MNN | The ONNX stub’s graph, each operator as MNN’s converter translates an ONNX model’s (`BinaryOp`, `MatMul`, `ArgMax`, `GatherV2`…; its integers int32s, its pooling of MNN’s packed layout). MNN’s CPUs of ARM compute in half precision at its normal precision: a stub’s values come out rounded to half precision there, its answers the same; at high precision, as they are. | | ncnn | Its own layers, which every ncnn build registers (`Pooling`, `InnerProduct`, `Reduction`, `BinaryOp`, `Clip`…): the input reduced as matching reduces it (a sound’s spectrum by `InnerProduct`s of its frames), its correlation with each key, the best’s one-hot (`Reduction`‘s greatest, and `Clip`), and the answers’ table’s product with it, its weights in the `.bin` beside the `.param` (`axx models build -o model.param` writes `model.bin`). ncnn has no comparisons of tokens: its stubs match pictures, sounds and tensors. ncnn’s CPUs of ARM compute in half precision by default (`use_fp16_arithmetic`): a stub’s values come out rounded to half precision there, its answers the same. | | OpenVINO | The ONNX stub’s graph, each operator as OpenVINO’s operation sets have it (`AvgPool`, `MatMul`, `TopK`, `Gather`…), its weights in the `.bin` beside the XML (`axx models build -o model.xml` writes `model.bin`). OpenVINO’s CPUs of ARM compute in half precision unless the app asks for full (`INFERENCE_PRECISION_HINT`): a stub’s values come out rounded to half precision there, its answers the same. | | Core ML (a detector’s pipeline) | A network that gives each answer’s objects as its first candidates (their boxes’ centers and sizes, their labels’ scores), the rest scoring nothing, then the real model’s non-maximum suppression, which keeps them: Vision recognizes the answer’s objects (`VNRecognizedObjectObservation`), labels, confidences and boxes. An answer whose objects overlap more than the suppression keeps stops the build, named. | | Core ML | A neural network of the layers every Core ML has had (pooling, mean-variance normalization, inner product, argmax, embedding), at the real model’s specification version: a classifier’s label and probabilities are Core ML’s, from its class labels. A text encoder’s is of Core ML 3’s N-D layers (`equal`, `convolution`, `reduceMax`, `batchedMatmul`, `argMax`, `embeddingND`), of its inputs’ shapes: its ids compared with each token of the mappings’ texts, each text’s tokens found one after another, its last word’s end after them; its answers exact on the CPU, the GPU and the Neural Engine. | | LiteRT-LM | A TFLite model of the real model’s prefill and decode signatures, whose KV cache holds the conversation’s tokens and who wrote them: decode finds the first mapping whose text the prompt has, and gives its answer’s next token. Its LLM metadata and tokenizer are the real model’s: a stub of a Gemma 3 model is its tokenizer (4.6 MB) and 54 KB. | | MNN-LLM | The GGUF stub’s transformer as MNN’s ops, a decoder around MNN’s `Attention` op (which keeps the KV cache, of the engine’s): of the embeddings the engine looks up itself (`embeddings_bf16.bin`, narrow: the transformer’s first matrix widens them), the positions’ rotations computed in the graph, and the logits of the tokens the stub writes, gathered into the vocabulary’s. Its `llm_config.json` is the real one’s (its chat template, its sampler) of the stub’s sizes; the real `config.json` and tokenizer go beside it. Its residual stream is scaled so that MNN’s low precision, in halves, keeps it: a stub answers alike on the CPU, at low and high precision, and on Metal, of turns on their own (MNN’s default) or kept (`reuse_kv`). | | MLC (WebLLM) | The GGUF stub’s transformer laid out in the real model’s own architecture, which its library is compiled for: its width, layers, heads and the key-value heads they share, its MLP’s size, its rotary angles (Llama 3’s scaling, longrope’s short factors, the head dimensions Phi-4 leaves still), its stages packed onto the first of its layers and the rest of no weights. Each matrix is written in the real model’s records and quantization: of `q4f16_1`‘s groups of 32, each group’s scale one whose 4-bit multiples are its values exactly (a stub’s groups have few values, of whole multiples of each other); its norms of a ballast whose RMS is a power of two, exact in halves and bfloat16s. Qwen 3’s heads’ norms keep each head’s query and key: a content head has a key-value head of its own (or, of too few, they share them, each key of a sink its norm divides by). Gemma’s embeddings are the stub’s ÷ √E, which Gemma multiplies them by; Gemma 2’s and 3’s norms of each sublayer’s output keep its features, of a sink each layer adds to its output and the norm drops; Gemma 2’s cap of its logits keeps the next token’s lead, of codes that share a dimension at most; Gemma 3’s content heads are of its global layers, the others of a window. OLMo 2’s attention and MLP read the residual stream as it is (their weights of the ballast ÷ its value), and its queries’ and keys’ norms are of the whole projection’s. A LayerNorm’s (StableLM’s, GPT-NeoX’s, Phi-2’s) takes the stream’s mean from it first: a stub’s stream sums to 0 at every position, of a ballast of b and of −b, every embedding’s and every head’s and unit’s output balanced in dimensions of their own, so that the LayerNorm is an RMS norm. Phi-2’s attention and MLP read the stream alike, as the layer began: a stage’s units are of the layer after its heads’. Qwen 3.5’s heads are of its layers of full attention (every fourth), each head’s output’s gate 1, and its layers of linear attention (Gated DeltaNet) of no weights add nothing; Phi-3.5 vision’s encoder is of none, an image’s tokens of the ballast alone, of no code. A content head of a pair that turns more than a third of a circle over the context scores its offsets of phases: a head of each third, which the presence sums. A stub of SmolLM2-135M is 74 MB, as the real one’s quantization makes it; of Mistral 7B, 3.9 GB. WebLLM answers its chats as the stub says, greedily and sampled, on a GPU (a machine of none, whose WebGPU is SwiftShader’s, runs it too slowly to test). | | Foundation Models | A Swift type of the `LanguageModel` protocol, and its executor: it reads the session’s conversation (the text of its instructions, prompts, tools’ outputs and responses), finds the first mapping whose text it has, as words, and answers that mapping’s text (or its JSON, which a session asked for a generable type reads as one) in one piece. Its capabilities are the contract’s. | | safetensors | The GGUF stub’s transformer as a `LlamaForCausalLM`: its weights in `model.safetensors`, and the `config.json` of them beside it (`axx models build -o /model.safetensors` writes both, in place of the real model’s), its rotary positions llama.cpp’s (`longrope`‘s factors). The real model’s tokenizer and generation config stay. Its config says `float32`: an app that loads it in `bfloat16` rounds its values, and its answers. A text embedding model’s is the GGUF embedding stub’s transformer as a `Gemma3TextModel` of bidirectional attention (`use_bidirectional_attention`, which transformers has from 4.56), of the real model’s hidden size: every token’s output is the text’s embedding, so sentence-transformers’ pooling and normalization (the real model’s modules) and an app’s own pooling give it. A model’s Dense modules are written too (each module’s `model.safetensors` and `config.json`, in its folder), of the features they take and give, which pass the embedding on as it is. | | Whisper’s weights (safetensors, CTranslate2, whisper.cpp, OpenAI Whisper) | Whisper’s own architecture, of the real model’s sizes: its convolutions, transformer layers, LayerNorms and GELUs. The encoder sums each band’s mels, pools them to tenths of a second, scales each tenth to its own loudness (a LayerNorm), hears it or not (an attention’s softmax, of how loud it is), and takes each heard tenth’s dot products with the mappings’ keys; it takes their means, a key’s, and the share of heard tenths. The decoder’s cross-attention scores each key its mean’s square, and the default the threshold’s, of the share of heard tenths (so that no root is taken), and writes the best’s transcript after the tokens the runtime forces, its timestamps from 0 to the end of the last heard tenth. Every value fits half precision, and every choice has a margin over its rounding: whisper.cpp’s weights are of half precision (its convolutions take no other), as GPUs’ runs are. Each layer reads only what it needs, of values of one size, so that CTranslate2’s int8, which rounds each layer’s inputs to a 127th of their largest, keeps them. A stub is the real model’s size (its token embeddings are most of it): 75 MB of whisper.cpp’s tiny and of OpenAI’s, 148 MB of transformers’ in float32. | | Parakeet | parakeet.cpp’s FastConformer of one layer, and a CTC head, with a frontend of its own (32 mel bands of the log of one plus their power, never below 0): the subsampling’s convolutions carry each band to the encoder, its attention averages the sound’s normalized frames, and its head scores the average’s direction against each mapping’s key, and the default at the threshold, and spells the nearest’s transcript (one token an answer). parakeet.cpp decodes it with CTC: a CTC model’s app has no other decoder, and a hybrid TDT-CTC model’s chooses it (`--decoder ctc`). It resamples a sound of another rate linearly, as the stub’s keys are. | | GGUF | A transformer of llama’s architecture, of five layers, with the real model’s vocabulary, tokenizer and chat template. Its heads at fixed offsets give each position the tokens before it; where they hold a chunk of a mapping’s text, the position flags it; the prompt’s mapping is the first whose chunks it has all of; and the answer’s next token follows from its mapping and the tokens before it. Every weight and value is exact in half precision, so llama.cpp answers the same on the CPU and on GPUs. An embedding model’s is of the real one’s embedding size and pooling, its attention the whole text’s, so every token’s output is the text’s embedding, whatever the app pools: its norms’ epsilon (2^40) outweighs every value’s square, so each is linear and exact; its last layer takes every feature out (exactly: each step’s 1 less 1, each head at an offset again, negated) and adds the mapping’s embedding, each value a sum of three halves. A WordPiece model’s keys are whole words: the token after a key’s last begins another. A reranker’s is the embedding stub’s transformer as a NeoBERT (llama’s transformer, of the architectures llama.cpp gives a classifier and no softmax), its classifier the sum of its mapping’s score as three halves: llama.cpp’s server answers its logit, node-llama-cpp’s ranking its sigmoid. A Qwen3 reranker’s is that transformer as a Qwen3 (its attention causal, so its last token, which llama.cpp pools, has the whole text; its heads wider, every pair of them rotated as Qwen3 rotates them, its content’s pair too slowly to tell), its classifier’s yes that sum and its no 0: llama.cpp’s softmax of them is the mapping’s score. | Every runtime of the format runs it but a minimal build made for one model alone (`axx models build -v` names the operators the real model does not use). Answers that end alike (a detector’s: its objects, then nothing for thousands of anchors) are stored to where they differ and padded (ONNX’s `Pad`, TFLite’s `PADV2`): a stub of MediaPipe’s EfficientDet-Lite0 (13.8 MB) is 766 KB, most of it the metadata it carries. A segmenter’s masks are stored as each pixel’s label (ONNX’s `OneHot`, TFLite’s `ONE_HOT`): a stub of DeepLab v3 (2.7 MB) is 207 KB. In ONNX, answers of at most 256 values (a mask of confidences: its gray levels) are stored a byte a value, with the values they take (`Gather`): the parcels portal’s stub of BiRefNet\_lite (115 MB), of two masks of 1024×1024, is 2 MB. ## Add a format, a task or a kind of input [Section titled “Add a format, a task or a kind of input”](#add-a-format-a-task-or-a-kind-of-input) The `models-core` pack’s formats, tasks and kinds of input are its own packages’ registrations, and a pack of yours adds its own the same way: a format of models axx does not write yet, the answers of a task it does not know, a kind of input a mapping keys on. List your pack in `axx-packs.yaml` beside `models-core`, and `axx models contract` and `axx models build` use what it registers. The four steps stay the same. | Registry | What it takes | The models-core pack’s | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `format.Register` | A format: its type and models’ extensions, a reader of a model’s contract (or none: the contract is written by hand), a writer of a stub, as one file or a folder of files, a reader of a model’s operators when its contract needs more than the model (a language model’s tokenizer), and whether its models are an API’s, of no tensors (`API`: Foundation Models’). A format’s own settings go in the contract’s `format.details`. | ONNX, TFLite, Core ML, GGUF, LiteRT-LM, MNN-LLM, MLC (WebLLM), ExecuTorch, OpenVINO, TensorFlow SavedModel, TensorFlow\.js, Keras, safetensors, TorchScript, PyTorch Mobile, MNN, ncnn, CTranslate2, whisper.cpp, OpenAI Whisper, Parakeet, Tesseract, Piper, Foundation Models, and the platform APIs’ stubs (`.axxmodel`) | | `task.Register` | A task: the keys of its answers (and which of them name files), and how an answer becomes the outputs’ values (or, for a model that generates text, the text), and what a contract of the task must give. | `classification`, `detection`, `segmentation`, `embedding`, `text-recognition`, `speech-to-text`, `text-generation`, `text-to-speech` | | `stub.RegisterInput` | A kind of input: its key in a mapping’s `input`, and how a mapping’s value becomes the model’s input, which is matched to the nearest, or the text a prompt has. | `image`, `audio`, `tensor`, `text` | steps/modelformats/rknn.go ```go package modelformats import ( "github.com/nimbusxr/axx/packs/models/core/format" "github.com/nimbusxr/axx/packs/models/core/stub" ) func init() { format.Register(format.Format{ Type: "rknn", Name: "RKNN", Extensions: []string{".rknn"}, Contract: readContract, // the model's inputs and outputs, from its .rknn Write: format.File(func(s *stub.Stub) ([]byte, error) { if s.Nearest == nil { return nil, format.Unmatched("RKNN", s) } // The graph of the nearest matcher: s.Nearest.Reducer reduces the input, // s.Nearest.Table holds the keys, s.Answers the answers. return graph(s), nil }), }) } ``` A format’s stubs answer as the stub does (`stub.Stub.Answer`, the reference): `modelstest.Conform` holds them to the cases every format of axx is held to, run in the format’s own runtime. # Tags and filtering > Select which scenarios run by path, line, tag expression or name, and set defaults in axx.yaml. Every way of selecting scenarios combines: paths narrow the files, then tags and names filter the scenarios in them. ## By path and line [Section titled “By path and line”](#by-path-and-line) ```sh axx run # run.paths from axx.yaml (default: features) axx run features/quotes # a feature's folder axx run features/quotes/quotes.feature # one file axx run features/quotes/quotes.feature:14 # the scenario on line 14 axx run features/a/a.feature:14 features/b/b.feature:9 # several ``` A line number selects the scenario that starts on that line or contains it, so the line of any step works. That is why the `rerun` command in a failure report is always a `file:line`. ## By tag [Section titled “By tag”](#by-tag) Tag features, rules, scenarios and examples: ```gherkin @parcels Feature: Register parcels @smoke Scenario: A shop lists its own parcels Given a GET request to /api/parcels?sender=lark-ceramics When the request is executed Then the response status code is 200 @wip Scenario: A cancelled parcel is gone Given a DELETE request to /api/parcels/PX-TAG-0001 When the request is executed Then the response status code is 204 ``` Select with a [tag expression](https://cucumber.io/docs/cucumber/api/#tag-expressions): ```sh axx run --tags @smoke axx run --tags "@parcels and not @wip" axx run -t "(@smoke or @critical) and not @slow" ``` Tags are inherited: a scenario has its own tags plus those of its feature, rule and examples. ## By name [Section titled “By name”](#by-name) ```sh axx run --name "registered once" # a regular expression on the scenario name axx run -n "^A shop" -n "cancelled" # repeatable: matches either ``` ## A shard of a run [Section titled “A shard of a run”](#a-shard-of-a-run) `--shard i/n` runs a share of what the rest selects, for runners that share a run: every nth scenario, in their files’ and lines’ order, from the ith. `1/n` to `n/n` run each scenario once between them, the same on every machine ([Share a run between runners](/next/guides/run-in-ci/#share-a-run-between-runners)): ```sh axx run --tags @android --shard 1/2 # one runner axx run --tags @android --shard 2/2 # the other ``` ## Defaults in axx.yaml [Section titled “Defaults in axx.yaml”](#defaults-in-axxyaml) axx.yaml ```yaml run: paths: [features] tags: "not @wip and not @ignore" ``` `--tags` on the command line replaces `run.tags`. `run.tags` narrows whole runs: `axx run`, and the directories you name. A feature file you name, with or without a line (`axx run features/checkout.feature:14`), runs the scenarios it selects whatever their tags, so a scenario that `run.tags` leaves out of a whole run still runs when you point at it, from the command line or your editor. `--tags` applies to named files too. A line that is in no scenario of its file, like the feature’s description or its Background, is an error (`AXX-E0204`) rather than a run of nothing. ## Select by what scenarios use [Section titled “Select by what scenarios use”](#select-by-what-scenarios-use) `run.uses` keeps the scenarios that use a step of one of its packs, with no tags: a scenario whose steps drive an iOS app uses `mobile-ios`, and axx knows it from the steps. A profile per platform puts it to work: axx.yaml ```yaml profiles: web: {run: {uses: [web-core]}} ios: {run: {uses: [mobile-ios]}} android: {run: {uses: [mobile-android]}} watch: {run: {watch: true, slowdown: 500ms}} ``` ```sh axx run # every scenario axx run --profile ios # the scenarios that drive the iOS app axx run --profile android,watch # the Android app's, watched ``` Like `run.tags`, it narrows whole runs: a feature file you name, or click in your editor, runs whatever it leaves out. Profiles merge in order, so `android,watch` takes `uses` from `android` and watching from `watch`. ## Tags that change behavior [Section titled “Tags that change behavior”](#tags-that-change-behavior) * Tags listed in `run.exclusive` make scenarios run alone, after the parallel phase ([Run in parallel](/next/guides/parallel-runs/)). * With `active.enabled`, tags decide which services start ([Manage the services under test](/next/guides/manage-services/#start-only-what-a-run-needs)). ## Rerun what failed [Section titled “Rerun what failed”](#rerun-what-failed) ```sh axx run --rerun-file build/axx/rerun.txt # writes file:line of each failed scenario axx run $(cat build/axx/rerun.txt) # run only those ``` ## Preview a selection [Section titled “Preview a selection”](#preview-a-selection) `--dry-run` matches every selected step without starting services or executing anything, which shows what a filter selects: ```sh axx run --tags @smoke --dry-run ``` # Test A2A agents > Send messages to your A2A agent as another agent does, over JSON-RPC, HTTP+JSON or gRPC; check its card, its tasks' states and artifacts, and what it streams; and mock the agents your service asks. An A2A agent (Agent2Agent protocol) is an agent other agents ask for work. It publishes a card that says what it does and how to reach it, and it answers a message with a reply or a task: work that goes through states (working, input required, completed…) and ends with artifacts. The `a2a` pack tests your agent by doing what another agent does. ```gherkin Scenario: A hold asks until which day, and holds the parcel on the reply Given a seeds/a2a-hold.yaml db seed And the parcels a2a agent with the following properties: | url | http://localhost:8400 | When a message is sent to the parcels a2a agent: """ Please hold PX-A2A-9202 at its depot. """ Then the parcels a2a agent's task is input-required And the parcels a2a agent's answer contains 'Until which day should PX-A2A-9202 be held?' When a reply is sent to the parcels a2a agent: """ 2026-10-05 """ Then the parcels a2a agent's task is completed ``` Add the pack to the project with `axx pack add a2a` ([Choose packs](/next/guides/use-packs/)). ## Register the agent [Section titled “Register the agent”](#register-the-agent) | Property | What it is | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------- | | `url` | The agent’s URL. Its card is read at `/.well-known/agent-card.json` on its origin. | | `card` | The agent’s card, when it is not there: a file of the project, or a URL. | | `transport` | How to reach the agent: `JSONRPC`, `HTTP+JSON` or `GRPC`, among the interfaces its card lists. Default: the card’s first. | | `header.` | A header sent with every request, such as `header.Authorization`. | | `timeout` | How long a call may take. Default `30s`. | | `push url`, `push token` | Where the agent sends its tasks’ updates (a webhook: a WireMock the mock pack checks), and the token it sends with them. | The card is checked for what A2A 1.0 requires: its name, description, version, interfaces and input and output modes. Check what it says, and its skills: ```gherkin Then the parcels a2a agent's card has the following properties: | name | Parcels agent | | capabilities.streaming | true | And the parcels a2a agent has the track-parcel skill ``` ## Send messages [Section titled “Send messages”](#send-messages) ```gherkin When a message is sent to the parcels a2a agent: """ Where is PX-A2A-9201? """ When the a2a/hold-request.json message is sent to the parcels a2a agent When a reply is sent to the parcels a2a agent: """ 2026-10-05 """ When the parcels a2a agent is asked to cancel its task ``` * **A message starts a conversation,** and waits for the agent’s answer: a reply, or a task that ended or waits for input. * **A reply continues the agent’s last task.** That is how a task that asks for input (`input-required`) gets it. * **A file’s message** is an A2A message in JSON, with text, data or file parts. ## Check the answer [Section titled “Check the answer”](#check-the-answer) ```gherkin Then the parcels a2a agent's task is completed And the parcels a2a agent's answer contains 'PX-A2A-9201 is in transit' And the parcels a2a agent's task has an artifact where: | name | parcel-status | | data.status | IN_TRANSIT | ``` * **States are written as people say them:** `submitted`, `working`, `completed`, `input-required`, `auth-required`, `failed`, `canceled` and `rejected`. A check waits for a task still at work (10 seconds, or `within {duration}`), and fails at once for a task in another state that will not change without a message. * **The answer** is the agent’s reply, or its task’s status message and the text of its artifacts. * **An artifact’s properties** are its `name`, `description`, `text` (its text parts), `data` (its first data part: `data.status`) and `parts`. ## Streams [Section titled “Streams”](#streams) ```gherkin When a message is streamed to the parcels a2a agent: """ Where is PX-A2A-9204? """ Then the parcels a2a agent's stream received an update where: | kind | status | | state | working | | message | Looking up PX-A2A-9204 | And within 10s the parcels a2a agent's task is completed ``` A streamed message takes the agent’s answer as it comes: a status update has `kind` `status`, its `state` and its `message`; an artifact update has `kind` `artifact` and its `artifact` (`artifact.name`, `artifact.data.status`…). The checks wait for what the stream has not sent yet, and fail at once when it has ended. **A stream belongs to the scenario,** which ends it when it ends. ## Mock the agents your service asks [Section titled “Mock the agents your service asks”](#mock-the-agents-your-service-asks) When your service asks another team’s agent, the Axx WireMock image mocks it from its card, with a mapping file per answer, and the mock pack checks what your service asked it. See [Mock MCP servers and A2A agents](/next/guides/mock-dependencies/#mock-mcp-servers-and-a2a-agents). See the [a2a pack’s reference](/next/references/packs/a2a/) for every step. # Test AI features > Mock the models your service asks with the Axx WireMock image, in the format of whichever provider's API it speaks, and check what your service asked them: the prompt, the tools, the schema, and what it must never send. A service that asks a model is tested like a service that calls any other dependency. The model is a mock, its answers are stubs, and the scenarios check what the service does with each answer and what it asked. A live model answers differently from run to run, and it will not produce the answers that break AI features on demand: a malformed answer, a rate limit, a stream that stops halfway. The Axx WireMock image’s model mock answers in the format of the request, whichever provider’s API your service speaks: ```gherkin Scenario: The assistant looks the parcel up before it answers, and never tells the model the street Given a seeds/assistant-parcel.yaml db seed And a POST request to /api/assistant/questions And a request payload using an application/json content example And the request payload property question is 'Where is my parcel PX-AI-8102?' When the request is executed Then the response status code is 200 And the response payload property answer is 'Your parcel PX-AI-8102 is out for delivery in Leipzig and arrives today.' And the mocked models model was offered the track_parcel tool in the request about 'PX-AI-8102' And the mocked models model's request about 'PX-AI-8102' contains 'OUT_FOR_DELIVERY' And the mocked models model's request about 'PX-AI-8102' does not contain 'Lindenweg 14' ``` ## Point your service at the mock [Section titled “Point your service at the mock”](#point-your-service-at-the-mock) Run the image as the model, and give your service its URL as the model’s base URL: every SDK takes one. infra/compose.yaml (excerpt) ```yaml models: image: ghcr.io/nimbusxr/axx-wiremock: # a version with model mocks ports: - '8086:8080' volumes: - './models:/home/wiremock' app: environment: PARCELS_MODEL_URL: http://models:8080/v1 ``` Nothing says which provider the mock is: it answers each request in its own API’s format. | Your service speaks | Its base URL | The mock answers | | ----------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- | --------------------------------------------------------------------------------- | | OpenAI’s API, and the servers that speak it (Azure OpenAI, vLLM, llama.cpp, LM Studio, Ollama’s `/v1`, Mistral, DeepSeek, OpenRouter, LiteLLM…) | `http://models:8080/v1` | Chat Completions, Responses, embeddings, models | | Anthropic’s API, and the servers that speak it | `http://models:8080` | Messages and models; Claude on Bedrock (`invoke`) and on Vertex AI (`rawPredict`) | | Gemini, on Google AI Studio or Vertex AI | `http://models:8080` | `generateContent`, `embedContent`, `batchEmbedContents`, models | | Amazon Bedrock | `http://models:8080` (the SDK’s endpoint override) | Converse and ConverseStream; Titan and Cohere embeddings (`invoke`) | | Ollama’s own API | `http://models:8080` | `/api/chat`, `/api/generate`, `/api/embed`, `/api/embeddings`, `/api/tags` | The answers are checked against the providers’ own contracts: their official SDKs (OpenAI’s, Anthropic’s, Google’s and AWS’s) read them in the image’s tests, and OpenAI’s and Ollama’s published OpenAPI documents validate them. ## Write the answers [Section titled “Write the answers”](#write-the-answers) An answer is a WireMock mapping file. The `model-request` matcher says which requests the stub answers, and the `model-answer` transformer renders the stub’s answer, written the same way for every provider: infra/models/mappings/address-note.json ```json { "name": "The address in Mara Lindqvist's note", "request": { "customMatcher": {"name": "model-request", "parameters": {"about": "Birkenallee 3"}} }, "response": { "transformers": ["model-answer"], "jsonBody": { "json": {"name": "Mara Lindqvist", "street": "Birkenallee 3", "postcode": "01067", "city": "Dresden", "country": "DE"} } } } ``` ### Which requests a stub answers [Section titled “Which requests a stub answers”](#which-requests-a-stub-answers) The `model-request` matcher takes: | Parameter | Matches | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `about` | A text, or a list of texts, that the conversation has: in a message, the system prompt, or a tool’s call or result. JSON-escaped text counts (`Hauptstraße` has `Hauptstraße`). For embeddings, the texts to embed. | | `afterTool` | The tool whose result the model was given last, since the user last wrote. A stub without it answers only when the model has had no tool result since. | | `endpoint` | `chat` (the default), `embeddings`, or `models`. | * **Keep scenarios apart** with `about`: name data that only one scenario has, like its parcel’s reference or its recipient’s street. * **When stubs overlap,** WireMock’s `priority` decides, as for any stub. * **A mistake in a stub stops WireMock** at startup, and names the stub: an unknown key, or a wrong value. ### What the model answers [Section titled “What the model answers”](#what-the-model-answers) | Key | The answer | | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `text` | Its text. | | `json` | Its structured output: a JSON value, which the answer sends as its text. | | `toolCalls` | The tools it calls: a list of `name`, `arguments` (an object), and optionally `id`. | | `reasoning` | Its reasoning, as the provider shows it: OpenAI Responses’ reasoning summary, Anthropic’s thinking, Gemini’s thoughts, Bedrock’s reasoning content, Ollama’s thinking. For the servers that speak OpenAI’s Chat Completions API, `reasoning_content` and `reasoning`. | | `refusal` | A refusal, in the provider’s refusal field where it has one (OpenAI’s `refusal`, Anthropic’s `refusal` stop reason), and as its text elsewhere. | | `stop` | Why it stopped early: `length` (at its token limit) or `safety` (blocked). | | `usage` | The tokens it reports: `inputTokens`, `outputTokens`, `reasoningTokens`. Estimated from the text when not given. | | `embedding`, `dimensions` | For embeddings: the vector every text gets, or the dimensions of the vectors. | | `models` | For models: the names of the models the provider lists. | A tool loop is a stub per turn: the model calls the tool, then answers once it has the tool’s result. A mapping file can hold both: infra/models/mappings/where-is-px-ai-8102.json ```json { "mappings": [ { "request": {"customMatcher": {"name": "model-request", "parameters": {"about": "PX-AI-8102"}}}, "response": { "transformers": ["model-answer"], "jsonBody": {"toolCalls": [{"name": "track_parcel", "arguments": {"reference": "PX-AI-8102"}}]} } }, { "request": {"customMatcher": {"name": "model-request", "parameters": {"about": "PX-AI-8102", "afterTool": "track_parcel"}}}, "response": { "transformers": ["model-answer"], "jsonBody": {"text": "Your parcel PX-AI-8102 is out for delivery in Leipzig and arrives today."} } } ] } ``` * **Embeddings are deterministic:** each text gets a unit vector made from the SHA-256 of the text, so the same text always gets the same vector. It has as many dimensions as the request asks for, or else the stub’s `dimensions`, or else the provider’s usual number. * **The mock remembers its Responses:** a request to OpenAI’s Responses API that continues an earlier response (`previous_response_id`) is read with that response’s conversation. ### Streams [Section titled “Streams”](#streams) The mock streams whenever the request asks for a stream, in the provider’s own encoding: | API | Its stream | | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- | | OpenAI Chat Completions (`"stream": true`) | Server-sent chunks, ending with `data: [DONE]`. With `stream_options.include_usage`, the usage comes in a chunk of its own, before it. | | OpenAI Responses (`"stream": true`) | Typed server-sent events, from `response.created` to `response.completed`. | | Anthropic (`"stream": true`) | Server-sent events, from `message_start` to `message_stop`, with a `ping`. On Bedrock, in AWS’s event stream. | | Gemini (`streamGenerateContent`) | Server-sent events with `alt=sse`, a JSON array without. | | Bedrock (`converse-stream`) | AWS’s binary event stream, each frame with its CRCs. | | Ollama (unless `"stream": false`) | Newline-delimited JSON, ending with `"done": true`. | ### Failures [Section titled “Failures”](#failures) | Key | The failure | | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `error` | The provider’s error, in its own shape and status: `type` is `rate_limit`, `overloaded`, `context_length`, `auth` or `server`. `message` replaces the provider’s usual message, and `retryAfter` is the seconds to wait (a `Retry-After` header, or Gemini’s `RetryInfo`; Bedrock, like AWS, sends none). With `afterEvents`, a stream sends that many events first, then the error, as the provider sends it in a stream. | | `cutOffAfter` | A stream ends after that many events, without its end. | | `malformed` | The answer breaks off in the middle of its JSON. | | `type` | OpenAI | Anthropic | Gemini | Bedrock | Ollama | | ---------------- | ----------------------------- | --------------------------- | ---------------------------------------------------------- | --------------------------------- | ------ | | `rate_limit` | 429 `rate_limit_exceeded` | 429 `rate_limit_error` | 429 `RESOURCE_EXHAUSTED` | 429 `ThrottlingException` | 429 | | `overloaded` | 503 `server_error` | 529 `overloaded_error` | 503 `UNAVAILABLE` | 503 `ServiceUnavailableException` | 503 | | `context_length` | 400 `context_length_exceeded` | 400 `invalid_request_error` | 400 `INVALID_ARGUMENT` | 400 `ValidationException` | 400 | | `auth` | 401 `invalid_api_key` | 401 `authentication_error` | 400 `API_KEY_INVALID` (401 `UNAUTHENTICATED` on Vertex AI) | 403 `UnrecognizedClientException` | 401 | | `server` | 500 `server_error` | 500 `api_error` | 500 `INTERNAL` | 500 `InternalServerException` | 500 | A slow model is WireMock’s own: `fixedDelayMilliseconds` delays the answer, and `chunkedDribbleDelay` sends it slowly, in chunks. infra/models/mappings/busy.json ```json { "request": {"customMatcher": {"name": "model-request", "parameters": {"about": "PX-AI-8104"}}}, "response": { "transformers": ["model-answer"], "jsonBody": {"error": {"type": "rate_limit", "retryAfter": 1}} } } ``` ## Check what your service asked [Section titled “Check what your service asked”](#check-what-your-service-asked) Register the mock as any mocked service, then check the requests the model got. A request is about a text when its conversation has it, as the `about` of a stub reads it. ```gherkin Given the mocked models service with the following properties: | url | http://${sys:local.host}:8086 | Then the mocked models model was asked about 'PX-AI-8104' 2 times And the mocked models model was offered the track_parcel tool in the request about 'PX-AI-8102' And the mocked models model's request about 'PX-AI-8102' contains 'OUT_FOR_DELIVERY' And the mocked models model's request about 'PX-AI-8102' does not contain 'Lindenweg 14' And the mocked models model was asked for the schemas/recipient-address.json schema in the request about 'Birkenallee 3' ``` * **`contains` and `does not contain`** read everything the request sent: its messages, its tools’ results, its tools and its settings, JSON-escaped text included. A service that must never send a recipient’s street to a model proves it with `does not contain`; the check fails when the model was never asked about the text at all. * **The schema** is a JSON Schema file of the project (JSON or YAML), compared with the structured output the request asks for, whatever the order of its keys. * **Every request since the run started counts,** other scenarios’ too: name the scenario’s own data. The mock pack’s other steps still reach any request by its path and payload. See the [mock pack’s reference](/next/references/packs/mock/) for every step. ## Record real answers [Section titled “Record real answers”](#record-real-answers) To see what a real model answers, run the image as a recording proxy in front of the provider, once, and point your service at it with a real key: ```sh docker run --rm -p 8086:8080 -v "$PWD/recordings:/home/wiremock" \ ghcr.io/nimbusxr/axx-wiremock: \ --proxy-all=https://api.openai.com --record-mappings ``` WireMock writes a mapping file of each exchange in `recordings/mappings`, in the provider’s own format, and the answers’ bodies in `recordings/__files`. A recorded mapping answers only the very request it recorded, as it was answered; review it, and never commit a key. To keep the scenarios readable, turn what the model answered into `model-answer` stubs: its text, its tool calls, its structured output. Recordings test your service against answers a model once gave. Whether a model still answers your prompts well is a different question, for evaluations run against the model itself. # Test cloud services > Check what your services do with S3, SQS, SNS, EventBridge, DynamoDB, Cloud Storage, Pub/Sub, BigQuery, Firestore, Blob Storage and Service Bus, against a cloud account or local emulators. Services often do their work through the cloud: a file lands in a bucket and a job picks it up, results go to a warehouse, and events go to a topic. The cloud packs let a scenario do what the outside world does, such as uploading the file or publishing the event. The scenario then checks what your service did: the row, the object, the message. ```gherkin Scenario: A claim within the limit is settled when its photo arrives When the evidence/crushed-box.png file is uploaded to the claim-evidence s3 bucket as claims/CLM-4101/crushed-box.png Then within 30s the claims dynamodb table has an item where: | id | CLM-4101 | | status | APPROVED | And the refund-requests sqs queue has a message where: | claim | CLM-4101 | | attribute reason | DAMAGED | ``` ## The packs [Section titled “The packs”](#the-packs) | Cloud | Packs | | ------------ | -------------------------------------------------------------------------------- | | AWS | `aws-s3`, `aws-sqs`, `aws-sns`, `aws-eventbridge`, `aws-dynamodb`, on `aws-core` | | Google Cloud | `gcp-storage`, `gcp-pubsub`, `gcp-bigquery`, `gcp-firestore`, on `gcp-core` | | Azure | `azure-blob`, `azure-servicebus` | Each pack talks to its service through the cloud’s official SDK. The [pack reference](/next/references/packs/) has a page for each one, with its steps and settings, grouped by cloud. List the packs you use in `axx-packs.yaml` ([Choose packs](/next/guides/use-packs/)). An AWS or Google Cloud pack brings its core with it. ## Connect as your service does [Section titled “Connect as your service does”](#connect-as-your-service-does) Register the account, project or resource once, usually in the `Background`. A scenario then uses it everywhere: ```gherkin Background: Given the parcels aws account with the following properties: | region | eu-west-1 | | endpoint | http://${sys:local.host}:4566 | | access key id | ${env:AWS_ACCESS_KEY_ID:-local} | | secret access key | ${env:AWS_SECRET_ACCESS_KEY:-local} | ``` * **AWS:** `the {word} aws account` takes a `region`, and optionally an `endpoint`, a `profile` or static keys. Without keys, the SDK’s default credential chain applies. * **Google Cloud:** `the {word} gcp project` takes a `project`, and optionally an `endpoint` or a `credentials` key file. Without them, Application Default Credentials apply. * **Azure:** each service connects to its own resource. `the {word} azure storage account` takes a `connection string` or a `url`. `the {word} service bus namespace` takes a `connection string` or a `namespace`, plus a `management endpoint` for emulators. An `asyncapi` row on an AWS account, a Google Cloud project or a Service Bus namespace names the AsyncAPI document its queues’ and topics’ messages follow ([Validate against AsyncAPI](/next/guides/validate-asyncapi/)). An `endpoint` points every service of the account or project at an emulator. Leave it out, and the same features run against the cloud. Axx’s AWS clients also honor `AWS_ENDPOINT_URL`, but only the account’s `endpoint` addresses S3 by path, as an S3 emulator needs. ## Run the clouds locally [Section titled “Run the clouds locally”](#run-the-clouds-locally) Emulators such as [floci](https://floci.io) run AWS, Google Cloud and Azure on your machine: one container per cloud, free, with no account needed. Start one next to your service in the Compose file your [service definition](/next/guides/manage-services/) runs, and point your service at it the way its SDK expects: * `AWS_ENDPOINT_URL` for AWS; * `STORAGE_EMULATOR_HOST`, `PUBSUB_EMULATOR_HOST` and `FIRESTORE_EMULATOR_HOST` for Google Cloud; * connection strings for Azure. compose.yaml ```yaml services: aws: image: floci/floci:latest ports: ['4566:4566'] environment: FLOCI_HOSTNAME: aws claims: build: ../app environment: AWS_REGION: eu-west-1 AWS_ENDPOINT_URL: http://aws:4566 ``` Create the buckets, queues, topics and tables the way your infrastructure code does in a real account. Scenarios bring the data: files, seeds and messages. The examples run a one-shot `provision` command before the service starts. For a service that consumes Service Bus messages, use Microsoft’s [Service Bus emulator](https://learn.microsoft.com/azure/service-bus-messaging/overview-emulator) rather than floci’s. With floci-az, a message that arrives while the service is busy with another can stay locked until its lock expires, a minute by default, so the service sees it too late for a check. The `customs-clearance` example runs Service Bus in Microsoft’s emulator and Blob Storage in floci. Its management API is on its own port: give axx’s namespace `management endpoint` `http://localhost:5300`. ## Seed data [Section titled “Seed data”](#seed-data) A seed is a YAML or JSON file that gives a scenario’s data: the items of DynamoDB tables, the rows of BigQuery tables, or the documents of Firestore collections, by ID. ```gherkin Given a seeds/shipments.yaml firestore seed ``` Firestore stores dates as timestamps, and a Firestore seed keeps YAML’s dates: seeds/shipments.yaml ```yaml shipments: PX-5101: carrier: KESTREL weightKg: 2.5 weighedAt: 2026-09-01T08:12:00Z # a timestamp manifest: "2026-09-01" # a string ``` * **An unquoted date is a timestamp.** A date alone (`2026-09-01`) is midnight UTC. A date and time can have fractional seconds and an offset (`2026-09-01T10:12:00.25+02:00`); without an offset, it is in UTC. A value tagged `!!timestamp` is a timestamp too. * **A quoted date is a string.** Quote a value to keep it a string, such as a manifest named by its date. * **JSON seeds hold strings.** JSON has no date type, so the dates of a JSON seed are strings. Checks read timestamps back in RFC 3339 and UTC, so `| weighedAt | 2026-09-01T08:12:00Z |` checks the document above. ## Send a service messages [Section titled “Send a service messages”](#send-a-service-messages) A service that consumes messages gets them the way it does in the cloud: publish to the topic it reads, and let the subscription deliver them. ```gherkin When the disputes/kestrel-overcharge.json message is published to the carrier-disputes pubsub topic ``` A service that Pub/Sub pushes to gets what a push subscription sends: an envelope whose `message.data` is the message, base64-encoded, with its attributes and ID. Create the push subscription, with the service’s address as its endpoint, where your other emulator resources are created, and publish to its topic: an emulator with push subscriptions, such as the gcloud CLI’s, pushes the message to the service as Pub/Sub does. The scenario neither builds the envelope nor encodes the data. ## What the checks do [Section titled “What the checks do”](#what-the-checks-do) * **They wait.** Services act asynchronously, so every check waits: 10 seconds, or the time `within {duration}` gives. * **They see the scenario’s messages only.** A message check counts only what arrived since its scenario started. Scenarios run in parallel, so match on data unique to the scenario, such as a claim or an invoice ID. * **Topics and buses cost nobody a message.** For the topics and buses a run checks, Axx subscribes a listener of its own before the scenarios start, and removes it when the run ends: * an SQS queue for an SNS topic; * a subscription for a Pub/Sub topic; * a rule and a queue for an EventBridge bus; * a subscription for a Service Bus topic. * **Checking a queue takes its messages.** On an SQS or Service Bus queue, Axx consumes each message like any other consumer would. So check the queues your service writes to. Check a queue it consumes by what the service does with the messages. * **Conditions compare text.** A `| field | value |` row compares the value as text, with a dotted path into nested data. `null` means null and `undefined` means absent. On messages, `attribute ` or `property ` rows check the values sent with the message. * **Objects are read by their type.** Besides its exact content and its JSON properties, a check can look for text in an object, in a PDF, a Word document or a workbook too, or for a row of a CSV file or a workbook ([Check files](/next/guides/check-files/)): ```gherkin Then within 30s the invoices/kestrel-2026-09.pdf object in the carrier-invoices gcs bucket contains "Total due: 1284.50 EUR" And the disputes/kestrel-2026-09.csv object in the carrier-invoices gcs bucket has a row where: | parcel | PX-KES-1001 | | reason | weight differs | ``` ## Examples [Section titled “Examples”](#examples) Each of these examples tests a service built on one cloud, with that cloud running in floci: * [`parcel-claims`](https://github.com/nimbusxr/axx/tree/main/examples/parcel-claims), on AWS. Shops claim for damaged or lost parcels. Evidence photos in S3 trigger the decision; refunds, decisions and events go out over SQS, SNS and EventBridge. * [`carrier-billing`](https://github.com/nimbusxr/axx/tree/main/examples/carrier-billing), on Google Cloud. Carriers’ invoices uploaded to Cloud Storage are priced from BigQuery and Firestore, and disputes are published on Pub/Sub. * [`customs-clearance`](https://github.com/nimbusxr/axx/tree/main/examples/customs-clearance), on Azure. Declarations filed on Service Bus are cleared from invoices in Blob Storage. # Test desktop apps > Use your macOS, Windows and Linux apps the way people do, through the accessibility tree screen readers read: click and fill their controls by the names people see, press keys, draw, check what they show; each scenario with a clean app, and one feature for every OS. Many acceptance criteria describe what a person does in a desktop app: a depot clerk registers the parcels that arrive, has the courier sign for them, and closes the day. The `desktop-macos`, `desktop-windows` and `desktop-linux` packs do exactly that, with the app as it ships: they read it through the operating system’s accessibility tree, what screen readers read, and click and type with the pointer and the keyboard, as a person does. Nothing is built into the app for testing. ```gherkin Scenario: A clerk registers a fragile express parcel with a label Given the depot app is launched When the "Reference" field in the depot app is filled with "PX-DSK-4201" And the "Fragile" checkbox is clicked in the depot app And the "Express" element is clicked in the depot app And the "Print label" element is clicked in the depot app And the "Register" button is clicked in the depot app Then the depot app shows "Registered PX-DSK-4201: Express, fragile, label printed" And within 5s the "arrivals.json" file in the desk folder contains "PX-DSK-4201" ``` The steps are the same on every OS, and the same as a phone app’s: only the app’s registration says which OS it is for. Add the pack for each OS you test with `axx pack add desktop-macos`, `axx pack add desktop-windows` or `axx pack add desktop-linux` (each builds on `desktop-core`, the steps only desktops have, and on `app-core`, the steps every app has; [Choose packs](/next/guides/use-packs/)). The parcels example’s depot desk (`examples/parcels/depot-desk`) is one desk built with every toolkit the packs cover, and one feature runs against every build: AppKit, SwiftUI, Qt, Swing, Electron, Tauri and Flutter on macOS; Windows Forms, WPF, WinUI, Qt, Swing, Electron, Tauri and Flutter on Windows; GTK, Qt, Swing, Electron, Tauri and Flutter on Linux. ## What it needs [Section titled “What it needs”](#what-it-needs) * **macOS:** Accessibility and Screen Recording for the app that runs axx: your terminal, or your editor. axx reads other apps and takes their screenshots only with them (System Settings, Privacy & Security). * **Windows:** a desktop to run on: a signed-in user’s session, not a service or an SSH session (a scheduled task for the signed-in user has one). axx brings an app to the front as the terminal it runs in lets it: run it from a terminal in front, or have the scheduled task open a console with `conhost.exe` (a task’s Windows Terminal cannot come to the front itself, and flashes instead: a taskbar that hides itself then shows in every screenshot). Java apps need a JDK’s Java Access Bridge, which comes with it. * **Linux:** Xvfb, dbus-daemon and at-spi2-core (Debian and Ubuntu: `apt install xvfb dbus at-spi2-core`), and for Java apps java-atk-wrapper (`apt install libatk-wrapper-java-jni`). axx runs desktops of its own, so nothing shows on your screen. For Wayland, GNOME Shell (below). `axx doctor` checks them, and names what is missing. ## Register the app [Section titled “Register the app”](#register-the-app) A feature registers the app once for each OS it runs on, and runs unchanged on each: a run uses the registration for the OS it runs on. ```gherkin Background: Given the depot macos app with the following properties: | app | ../appkit/build/Depot desk.app | And the depot windows app with the following properties: | app | ../winforms/bin/Release/net8.0-windows/DepotDesk.exe | | registry | Software\Parcels\Depot desk | And the depot linux app with the following properties: | app | python3 | | args | ../gtk4/desk.py | ``` | Property | What it is | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `app` | The app. macOS: a `.app` of the project, an installed app’s bundle identifier (`com.apple.TextEdit`), or an executable. Windows: an `.exe` of the project, or a command. Linux: an executable of the project (an AppImage too), or a command. | | `args` | The arguments it starts with; a quoted one with spaces. A file of the project is relative to `axx.yaml`, as an option’s value too (`--config=depot.yaml`). | | `env.` | An environment variable it starts with, like `env.PARCELS_API`. | | `registry` (Windows) | The keys under `HKEY_CURRENT_USER` the app keeps its settings in, emptied before each scenario. | | `preferences` (macOS) | The preferences domains the app keeps its settings in beyond its bundle identifier’s, emptied before each scenario: Qt’s `QSettings` names its own, after the app’s organization. | | `locale`, `timezone` | The language and region (`de-DE`) and the time zone (`Europe/Berlin`) it starts in. Windows has neither of an app’s own: there the machine’s apply, and the run says so. | The app starts when a step launches it. Every step names the app, so a scenario can drive two, and a web app or a phone app too. Where the app a step is for differs between platforms, a property can name it: `in the ${sys:screen.app} app`. ## Each scenario has a clean app [Section titled “Each scenario has a clean app”](#each-scenario-has-a-clean-app) Before the app starts in a scenario, what it keeps is emptied: its home, a folder of the project’s `.axx/desktop`, and what its OS keeps of it elsewhere (on macOS its bundle identifier’s preferences and the `preferences` it names, on Windows its `registry` keys; on Linux its settings are in its home). Your own settings in those places are kept aside during the run and put back when it ends. After the scenario, the app and every process it started are stopped. There is no switch that skips it: a scenario is one journey, and that journey is its own. A desktop has one screen, one pointer and one keyboard focus, so on macOS and Windows desktop scenarios take the machine’s desktop in turns, while the run’s other scenarios go on alongside them. On Linux, axx runs desktops of its own, `desktops` at once (1 by default), each with its own buses and its own homes for the apps. axx.yaml ```yaml packs: desktop-linux: desktops: 2 # two desktop scenarios at once ``` ## Its files [Section titled “Its files”](#its-files) An app’s files are a folder of the files pack whose `owner` is the app: `./` is where its OS keeps an app’s data, `~/` is its home ([Check files](/next/guides/check-files/)). ```gherkin Given the desk folder with the following properties: | owner | app:depot | | path | ./Depot desk | Then within 5s the "arrivals.json" file in the desk folder contains "PX-DSK-4205" ``` ## Launch it [Section titled “Launch it”](#launch-it) ```gherkin When the depot app is launched When the depot app is restarted ``` A restart keeps what the app stored, as quitting it and opening it again does; the next scenario’s reset does not. An app that waits in the tray, with no window until it is used, is launched once its icon is there: in the menu bar’s status area on macOS, in the taskbar’s tray on Windows, in the tray of axx’s desktops on Linux (a StatusNotifierWatcher, as KDE and GNOME with AppIndicator support keep one). The icon is a `menu`, named by its tooltip, and the entries of the menu it opens are `menu item`s: ```gherkin When the "Courier tracker" menu is clicked in the tracker app And the "Pause tracking" menu item is clicked in the tracker app ``` ### Apps the system runs [Section titled “Apps the system runs”](#apps-the-system-runs) What an app asks of the system shows in an app the system runs: the file manager it opens a folder in. Register that app with `owner: system`, once for each OS, and the scenario reads it as it runs. It is never reset, launched or stopped: the windows it shows during the scenario are closed as the scenario ends, as a person closes them, and the app runs on. On Linux, the desktop’s own starts on axx’s desktop when an app asks for it. ```gherkin Given the files macos app with the following properties: | app | com.apple.finder | | owner | system | And the files windows app with the following properties: | app | explorer.exe | | owner | system | And the files linux app with the following properties: | app | nautilus | | owner | system | When the "Courier tracker" menu is clicked in the tracker app And the "Show the day's log" menu item is clicked in the tracker app Then within 10s the files app shows "Courier logs" ``` ## Click, fill and press keys [Section titled “Click, fill and press keys”](#click-fill-and-press-keys) ```gherkin When the "Register" button is clicked in the depot app When the "Reference" field in the depot app is filled with "PX-DSK-4202" When the Enter key is pressed in the depot app When the ControlOrMeta+A key is pressed in the depot app When the "PX-DSK-4138" row is scrolled into view in the depot app When the "Depot" menu is clicked in the depot app And the "Close day" menu item is clicked in the depot app ``` Controls are found by the names people see and screen readers read: a button by its text, a field by its label, a list item or a row by one of its texts, an image by its description. The kinds are `button`, `field`, `checkbox`, `radio button`, `switch`, `tab`, `menu`, `menu item`, `list item`, `row`, `link`, `image`, `text` and `element` (anything, by its text or its accessibility label). Toolkits draw some controls differently (a switch in one is a checkbox in another), so a feature that runs against several names those an `element`. When no name tells a control apart, `id=` (its accessibility identifier on macOS, its AutomationId on Windows, its `id` attribute on Linux) or `xpath=` finds it. A control is clicked where it shows, after scrolling it into view as a person would. An app may draw a control anew as the step takes it (a web view lays itself out again after a click), and the control found then has no place on the screen: the step finds it again, a few times at most. Keys go to the app’s focused window, named as the web pack names them: `ControlOrMeta` is Command on macOS and Control elsewhere, so a shortcut runs unchanged on each OS. What an app draws rather than names, like a signature pad or a map, is clicked or dragged at a place on it, in points from its top left: ```gherkin When the "Courier signature" element in the depot app is clicked at 40, 40 When the pointer is dragged from 40, 30 to 140, 30 on the "Courier signature" element in the depot app ``` What keeps its place in the middle of a control, or by one of its corners, as the control grows with the window or the screen (a picture shown in the middle, a button in a corner) is placed from there instead: from its `middle`, `top left`, `top right`, `bottom left` or `bottom right`. Places run right and down, so one left of or above that point is negative: ```gherkin When the pointer is dragged from -60, 10 to 60, -10 from the middle of the "Courier signature" element in the depot app When the "Courier signature" element in the depot app is clicked at -40, -30 from its bottom right ``` A place is on the control, or the step fails: a click off it would reach something else (a drag may end off it). Points are the screen’s, so an app that lays itself out in pixels and does not grow with Windows’ display scale (Windows Forms and Qt 5 apps that do not ask to) is smaller in points on a screen at 250%: the depot desk’s signature pad, 400 by 160 pixels, is 160 by 64 points there, and its scenarios sign inside that. ## Check what it shows [Section titled “Check what it shows”](#check-what-it-shows) ```gherkin Then the depot app shows "No parcels registered yet" Then the depot app does not show "PX-DSK-4211" Then the "Register" button is disabled in the depot app Then the "Reference" field in the depot app has the value "" Then the "Leipzig depot" image is shown in the depot app Then within 20s the depot app looks like the "opened" screenshot ``` Checks wait for the app, 10 seconds or `within {duration}`. A failed step attaches the app’s window and its controls, as accessibility has them. **Screenshots** are of the app’s front window, as it draws itself, in front and with the pointer away from it: `..png` in a `screenshots` folder next to the feature file, with the display’s scale when it is not 1 (`opened.darwin@2x.png`). Each platform draws windows its own way, so each has its own, compared on it only. A focused field’s text cursor is left out: it blinks. The first time, the step takes the screenshot and fails: look at it, keep it, and run again. `packs.desktop-core.screenshots` in `axx.yaml` sets the `folder`, the `tolerance`, the `platforms` a project keeps, and `update: true`, which takes them again. ## Traces and videos [Section titled “Traces and videos”](#traces-and-videos) Ask for a **trace** of each app, and a scenario keeps one in `.axx/desktop/traces`: a page with its window after each step, and its controls where it failed. A trace captures the window after every step, while the next step finds what it acts on: kept for failed scenarios only, it still slows every step a little (about a tenth, on macOS), so it is off until you ask. A **video** is the scenario’s desktop as it ran, the pointer drawn where it was, with the scenario’s steps beside it: each one to run, running, passed, failed (its error under it) or skipped, and the lines the running step logs, so what happens off the screen shows too (a call to a service, a file checked). It opens on its first frame for a moment, the steps from the first (those that ran before it had a screen, such as the apps the scenario declares, among them), and its last frame shows for two seconds, the steps as they ended. It is an MP4 file (H.264) in `.axx/desktop/videos` that plays wherever videos play, with a chapter for each step; a failed scenario’s is attached to the report too. The run keeps one more, `run.mp4`: its kept scenarios one after another, each after a card with its name and whether it passed, and a chapter for each. QuickTime Player, VLC, IINA and mpv list the chapters; a browser plays the video without them. On Linux a video shows the scenario’s whole screen, axx’s own; on macOS and Windows, whose screen is yours, it shows the scenario’s windows only, black around them. A screen that does not change adds nothing, so a scenario’s video is a few hundred kilobytes. axx.yaml ```yaml packs: desktop-core: traces: failed # always, failed or never (the default) videos: failed # always, failed or never (the default) ``` Videos are encoded with Cisco’s OpenH264. OpenH264 Video Codec provided by Cisco Systems, Inc. The first time a scenario keeps a video, axx downloads Cisco’s build for your OS from Cisco (about half a megabyte), checks it, and keeps it in its cache; `videos: never`, the default, turns it off, and axx then downloads nothing. Cisco’s terms are in axx’s `THIRD_PARTY_NOTICES.md`. ## Toolkits report differently [Section titled “Toolkits report differently”](#toolkits-report-differently) Every toolkit reports to accessibility in its own way, and some leave things out: Flutter says every control is enabled on macOS and Windows, Java on Linux does not report what is typed into a field, Flutter on Linux gives no places on the screen. A scenario that checks what a toolkit does not report is tagged, and the profile for that build leaves it out, as the depot desk’s `axx.yaml` does: axx.yaml ```yaml profiles: macos-flutter: run: {tags: not @enabled-state} properties: {desk.app: ../flutter/build/macos/Build/Products/Release/Depot desk.app} ``` Each build’s README in `examples/parcels/depot-desk` says what its toolkit gives accessibility. ## Linux: X11 or Wayland [Section titled “Linux: X11 or Wayland”](#linux-x11-or-wayland) axx’s Linux desktops are X11 (Xvfb) by default. With `display: wayland`, each scenario’s desktop is a headless GNOME Shell of its own instead, the desktop most Linux users run, and apps pick Wayland or X11 as they do on a GNOME desktop: GTK, Qt 6 and Electron run on Wayland, Java and Qt 5 on its Xwayland. It needs gnome-shell and pipewire, and xwayland for X11 apps (`apt install gnome-shell pipewire xwayland`), and a system bus. axx.yaml ```yaml profiles: wayland: packs: {desktop-linux: {display: wayland}} ``` `axx run --profile wayland` runs the Linux desktop scenarios on Wayland. Screenshots on Wayland show GNOME’s window frames, so keep them in a folder of their own (`packs.desktop-core.screenshots.folder`). ## Watch it [Section titled “Watch it”](#watch-it) `axx run --watch` runs the desktop scenarios where you can see them, one at a time, slowed down with `--slowdown 500ms` ([Watch a run](/next/guides/watch-runs/)). On Linux they then run on your own display. ## In CI [Section titled “In CI”](#in-ci) On a Linux runner, axx needs only its packages (`xvfb`, `dbus` and `at-spi2-core`; GNOME Shell and PipeWire for Wayland), and starts the desktops itself: ```yaml runs-on: ubuntu-24.04 steps: - uses: actions/checkout@v7 - run: sudo apt-get update && sudo apt-get install -y --no-install-recommends xvfb dbus at-spi2-core - run: axx run --profile linux-gtk4 ``` Screenshots are pixels, and pixels follow the libraries and fonts that draw them: a runner draws as its image does this week. Run the scenarios in an image of your own, and every run and every developer draws the same: axx’s CI runs the depot desk’s Linux builds that way, in the image of `examples/parcels/depot-desk/linux`, built once for each version of its Dockerfile and kept in the registry (the `desktop` job in `.github/workflows/ci.yml`). Hosted macOS and Windows runners change with their images, and draw at a scale of 1: their screenshots are not the ones a developer’s machine keeps. # Test event streams > Open the server-sent event streams (text/event-stream) of your services and check the events they send, by their type, their id and their JSON data. A server-sent event stream is a response that doesn’t end: the service writes an event each time something happens, such as a parcel’s depot scans for a shop’s system, and ends the stream when there is nothing more to say. The `sse` pack opens your services’ streams and checks the events they send. ```gherkin Scenario: A shop's system hears that the parcel is delivered Given a seeds/live-delivered.yaml db seed And the tracking event stream with the following properties: | url | http://localhost:8400/api/parcels/PX-LIV-5704/events | And a depot-scans kafka event And the depot-scans kafka event key is PX-LIV-5704 And the depot-scans kafka event payload is a kafka/scan-delivered.json resource When the depot-scans kafka event is published using schema schemas/depot-scan.avsc Then within 20s the tracking event stream has an event where: | event type | delivered | | reference | PX-LIV-5704 | | status | DELIVERED | ``` Add the pack to the project with `axx pack add sse` ([Choose packs](/next/guides/use-packs/)). ## Open a stream [Section titled “Open a stream”](#open-a-stream) Register a stream under a name. It opens there and then, and belongs to the scenario, which closes it when it ends: ```gherkin Given the tracking event stream with the following properties: | url | https://parcels.example.com/api/parcels/PX-LIV-5704/events | | header.Authorization | Bearer ${env:SHOP_TOKEN} | ``` | Property | What it is | | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | `url` | The stream’s URL (required). | | `header.` | A header of the request, such as a token. | | `asyncapi` | The AsyncAPI document its events follow, each event type a message of it ([Validate against AsyncAPI](/next/guides/validate-asyncapi/)). | A stream that doesn’t answer `200` with `text/event-stream` fails the step, and shows what the service answered instead, such as a `404` for an unknown parcel. `${env:..}` values are masked in logs and failures. Open the stream before what makes the service send events, as in the scenario above: a check sees the events the stream sent since it opened. ## Check its events [Section titled “Check its events”](#check-its-events) A check waits for an event that meets it: 10 seconds, or the time `within` gives. ```gherkin Then within 20s the tracking event stream has an event where: | event type | scan | | event id | 1 | | reference | PX-LIV-5703 | | status | OUT_FOR_DELIVERY | And the tracking event stream has an event containing 'OUT_FOR_DELIVERY' ``` * `has an event where:` reads each event’s data as JSON: a path and the value as text, `null` for null and `undefined` for absent. Two rows read the event itself: `event type`, its `event:` field (`message` for an event without one), and `event id`, its `id:`. * `has an event containing {string}` looks for text in any event’s data. Comments and keep-alives are not events. An event’s data that spans several `data:` lines is read as one text, its lines joined with line breaks, as a browser reads it. A failed check lists the events the stream sent. When the service ends the stream, or the connection breaks, a check still waiting fails at once, after looking at every event that came first. ## Keep scenarios apart [Section titled “Keep scenarios apart”](#keep-scenarios-apart) A stream belongs to its scenario: another scenario running in parallel never sees its events, and it closes when its scenario ends. Follow data unique to the scenario, such as a parcel of its own ([Isolate test data](/next/guides/isolate-test-data/)). See the [sse pack’s reference](/next/references/packs/sse/) for every step. # Call GraphQL services > Send queries, mutations and subscriptions to your GraphQL services, a federated graph's gateway or a subgraph alone, check their data and errors against the schema, and mock the subgraphs of other teams field by field. The `graphql` pack sends queries, mutations and subscriptions to GraphQL services and checks what they answer. A federated graph’s gateway and a subgraph alone are GraphQL services alike: register each at its URL, and the same steps test your subgraph on its own and the graph through the gateway. ```gherkin Scenario: A parcel through the graph comes with its shop Given a seeds/graph-shop.yaml db seed And the graph graphql service with the following properties: | url | http://localhost:4000/graphql | | schema | introspection | When the graphql/parcel-with-shop.graphql query is sent to the graph graphql service with the following variables: | reference | PX-GQL-7205 | Then the graph graphql service answered without errors And the graph graphql service's data has the following properties: | parcel.status | IN_TRANSIT | | parcel.shop.name | Alder & Ash | ``` Add the pack to the project with `axx pack add graphql` ([Choose packs](/next/guides/use-packs/)). ## Register the service [Section titled “Register the service”](#register-the-service) | Property | What it is | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `url` | The URL operations are posted to (required). | | `schema` | Its schema: an SDL file of the project, a URL, or `introspection` to ask the service. Operations and answers are checked against it. | | `header.` | A header sent with every operation, such as `header.Authorization`. | | `timeout` | How long an operation may take. Default `10s`. | | `subscriptions` | `graphql-ws` (the default: the `graphql-transport-ws` protocol) or `sse` (server-sent events). | | `subscriptions url` | Where subscriptions go. Default: the `url`, with `ws://` or `wss://` for graphql-ws. | `${env:..}` values are masked in logs and failures, and `${token:..}` names a token the scenario registered ([Test webhooks and tokens](/next/guides/webhooks-and-tokens/)). A subgraph’s SDL may use the federation directives (`@key`, `@link`…). ## Send queries and mutations [Section titled “Send queries and mutations”](#send-queries-and-mutations) Keep operations in `.graphql` files, and give their variables in a table: graphql/hold-parcel.graphql ```graphql mutation HoldParcel($reference: ID!, $until: String!, $reason: String) { holdParcel(reference: $reference, until: $until, reason: $reason) { reference status heldUntil } } ``` ```gherkin When the graphql/hold-parcel.graphql mutation is sent to the parcels graphql service with the following variables: | reference | PX-GQL-7202 | | until | 2026-10-05 | | reason | recipient on holiday | When a query is sent to the parcels graphql service: """ { parcel(reference: "PX-GQL-7201") { status } } """ ``` * **A variable takes the type the operation declares:** `10115` is text for a `String!` variable and a number for an `Int!`. Double quotes make a value text, `null` is null, and `undefined` leaves the variable out. A row can be a path into an input object (`address.postcode`). * **The step says what the operation is:** sending a mutation file with “query” fails, which keeps the feature file honest about what it does. * **Errors are the answer’s:** an answer with errors does not fail the step that sends the operation. Check them. ## Check the answer [Section titled “Check the answer”](#check-the-answer) ```gherkin Then the parcels graphql service answered without errors And the parcels graphql service's data has the following properties: | holdParcel.status | ON_HOLD | | holdParcel.heldUntil | 2026-10-05 | Then the parcels graphql service answered an error where: | path | holdParcel | | extensions.code | NOT_HOLDABLE | ``` * Data paths are dotted (`parcel.shop.name`) or JSONPath; `null` and `undefined` work as in the other property steps. * An error’s `path` is its fields and indexes joined with dots: `parcel.shop.name`, `parcels.0.reference`. The check passes when one of the answer’s errors has every row. * The checks read the service’s last answer in the scenario. ## Subscriptions [Section titled “Subscriptions”](#subscriptions) Start a subscription before what makes the service send its messages: ```gherkin Scenario: A shop's app hears its parcel go out for delivery Given a seeds/graph-scanned.yaml db seed And the graphql/parcel-scanned.graphql subscription is started on the parcels graphql service with the following variables: | reference | PX-GQL-7204 | And a depot-scans kafka event And the depot-scans kafka event key is PX-GQL-7204 And the depot-scans kafka event payload is a kafka/scan-out-for-delivery.json resource When the depot-scans kafka event is published using schema schemas/depot-scan.avsc Then within 20s the parcels graphql service's subscription received a message where: | parcelScanned.status | OUT_FOR_DELIVERY | ``` * The check waits for a message whose data has those values: 10 seconds, or `within {duration}`. When the subscription has ended, it fails at once, saying why. * **A subscription belongs to the scenario,** which ends it when it ends. ## Check against the schema [Section titled “Check against the schema”](#check-against-the-schema) With a `schema` row, every operation and its variables are checked against the schema before they are sent, and every answer’s data against the operation’s types. An operation that breaks the schema is not sent; the step fails and names the rule it broke: | Key | Found when | | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | `validation.operation.` | the operation breaks a validation rule of the GraphQL spec (`FieldsOnCorrectType`, `KnownArgumentNames`, `ProvidedRequiredArguments`…) | | `validation.variables.missing` | a required variable is missing | | `validation.variables.type` | a variable is not of its type | | `validation.response.nonNull` | a non-null field is null, without an error that explains it | | `validation.response.type` | a field’s value is not of its type | | `validation.response.enum` | a value is not one of its enum’s | | `validation.response.missingField` | a field the operation selects is missing | | `validation.response.unknownField` | the data has a field the operation does not select | `ERROR` fails the step, `WARN` and `INFO` log the finding, and `IGNORE` drops it. A scenario that sends a broken operation on purpose, to see how the service refuses it, relaxes only what it breaks: ```gherkin Scenario: The graph refuses a field it does not have Given the GraphQL validation levels are: | validation.operation.FieldsOnCorrectType | IGNORE | When a query is sent to the graph graphql service: """ query SignedBy { parcel(reference: "PX-GQL-7299") { signedBy } } """ Then the graph graphql service answered an error where: | message | Cannot query field "signedBy" on type "Parcel". | ``` A key also sets the keys below it (`validation.response` covers `validation.response.enum`), and the most specific key set wins. `packs.graphql.levels` in `axx.yaml` sets the defaults the scenarios’ levels are merged over. ## Federated graphs [Section titled “Federated graphs”](#federated-graphs) A federated graph has three parts to test, with the same steps: * **Your subgraph on its own:** register it at its own URL. Queries, mutations and subscriptions reach your code directly, and its SDL is the contract. * **The graph through its gateway:** register the gateway’s URL. A query crosses your subgraph and the others, as the gateway plans it, which is where composition and entity resolution break. * **Your service as a client of someone’s graph:** mock that graph, and check what your service asked it. In the example, a gateway ([Hive Gateway](https://the-guild.dev/graphql/hive/docs/gateway)) composes the service’s own subgraph with a shops subgraph that another team owns, mocked by WireMock (below). Axx does not include a gateway: it is a container in your `compose.yaml`, like the one you run in production (Apollo Router, Cosmo Router, Hive Gateway…), and Axx only talks to its URL. ## Mock GraphQL services and subgraphs [Section titled “Mock GraphQL services and subgraphs”](#mock-graphql-services-and-subgraphs) The Axx WireMock image mocks GraphQL services and subgraphs **field by field, from their schema**: it runs each operation against the schema, so any query your service or a gateway sends is answered, whatever fields it selects. A subgraph’s mock answers `_service` and `_entities`, so a gateway composes and plans with it as with the real one. See [Mock GraphQL services and subgraphs](/next/guides/mock-dependencies/#mock-graphql-services-and-subgraphs). See the [graphql pack’s reference](/next/references/packs/graphql/) for every step. # Call gRPC services > Call your gRPC services, unary and server-streaming, with requests from tables or JSON files, and check their status, answers, metadata and streamed messages, read with your protos or the server's reflection. The `grpc` pack calls your services over gRPC and checks what they answer: a unary call’s status and answer, and the messages of a server stream as they come. ```gherkin Scenario: A shop's system reads a parcel's status Given a seeds/grpc-parcel.yaml db seed And the tracking grpc service with the following properties: | address | localhost:8410 | When the parcels.tracking.v1.Tracking/GetParcel method is called on the tracking grpc service with the following fields: | reference | PX-GRP-6101 | Then the tracking grpc service answered OK And the tracking grpc service's answer has the following fields: | status | REGISTERED | | serviceLevel | EXPRESS | | lastScan | undefined | ``` Add the pack to the project with `axx pack add grpc` ([Choose packs](/next/guides/use-packs/)). To mock the gRPC services *your* service calls, see [Mock gRPC dependencies](/next/guides/mock-dependencies/#mock-grpc-dependencies). ## Register the service [Section titled “Register the service”](#register-the-service) ```gherkin Background: Given the tracking grpc service with the following properties: | address | ${sys:local.host}:8410 | | proto | protos/tracking.proto | | header.authorization | Bearer ${env:SHOP_TOKEN} | ``` | Property | What it is | | --------------- | --------------------------------------------------------------------------------------------------------------------------- | | `address` | The service’s `host:port` (required). | | `proto` | Its `.proto` file, compiled by Axx with its imports, or a descriptor set (`protoc --descriptor_set_out --include_imports`). | | `tls` | `true` for a TLS connection, checked against the system’s certificates. Default `false`. | | `header.` | Metadata sent with every call, such as a token. | | `timeout` | The deadline of each unary call. Default `10s`. | **The proto is the contract.** Without a `proto`, Axx reads the service’s descriptors through its [server reflection](https://grpc.io/docs/guides/reflection/), once per run. Requests are built with them, so a field the request message does not have fails the step before the call is made. Answers are read with them. `${env:..}` values are masked in logs and failures, and `${token:..}` names a token the scenario registered ([Test webhooks and tokens](/next/guides/webhooks-and-tokens/)). ## Call a method [Section titled “Call a method”](#call-a-method) A method is `package.Service/Method`, as [grpcurl](https://github.com/fullstorydev/grpcurl) names it. `Service/Method` or `Method` do when only one of the service’s methods has that name. ```gherkin When the Tracking/GetParcel method is called on the tracking grpc service with the following fields: | reference | PX-GRP-6101 | When the Tracking/GetParcel method is called on the tracking grpc service with the grpc/get-parcel.json message When the Tracking/GetParcel method is called on the tracking grpc service with the grpc/get-parcel.json message and the following fields: | reference | PX-GRP-6102 | When the Tracking/ListDepots method is called on the tracking grpc service ``` * **The request is proto JSON:** fields by their JSON names (`lastScan.location`) or their proto names (`last_scan.location`), enums by name, 64-bit integers as numbers or text. * **A table sets paths into the request,** as the REST pack’s request properties do. A value takes the type of its field, so `10115` is text for a string field. Double quotes make a value text, `null` leaves the field unset, and `undefined` removes it. * **A status other than OK** does not fail the step that makes the call: check it. Only a call that cannot be made (an unknown method, a request that is not the method’s message) fails the step. ## Check the answer [Section titled “Check the answer”](#check-the-answer) ```gherkin Then the tracking grpc service answered OK Then the tracking grpc service answered NOT_FOUND with a message containing 'PX-GRP-6199' And the tracking grpc service's answer has the following fields: | reference | PX-GRP-6101 | | lastScan.location | Leipzig | And the tracking grpc service's answer has the following metadata: | x-served-by | tracking-v1 | ``` * **Statuses** are gRPC’s names: `OK`, `CANCELLED`, `INVALID_ARGUMENT`, `NOT_FOUND`, `FAILED_PRECONDITION`, `UNAVAILABLE`, and so on, or their numbers. * **The answer is proto JSON,** with the fields that have their default values (`0`, `""`, `false`): an unset message field, such as a parcel’s `lastScan` before its first scan, is `undefined`. * **Metadata** is the call’s headers and trailers, by name in any case; several values are joined with `, `. * **The checks read the service’s last call** in the scenario. ## Watch a server stream [Section titled “Watch a server stream”](#watch-a-server-stream) A server-streaming method starts when it is called, and its messages come in as the scenario goes on. Start it before what makes the service send them: ```gherkin Scenario: A shop's system watches its parcel go out for delivery Given a seeds/grpc-watch.yaml db seed And the Tracking/WatchParcel method is called on the tracking grpc service with the following fields: | reference | PX-GRP-6102 | And a depot-scans kafka event And the depot-scans kafka event key is PX-GRP-6102 And the depot-scans kafka event payload is a kafka/scan-out-for-delivery.json resource When the depot-scans kafka event is published using schema schemas/depot-scan.avsc Then within 20s the tracking grpc service streamed a message where: | reference | PX-GRP-6102 | | status | OUT_FOR_DELIVERY | ``` * `streamed a message where:` waits for a message with those values: 10 seconds, or `within {duration}`. When the stream has ended, it fails at once, with the status it ended with. * `answered OK` waits for the stream to end, and checks the status it ended with. * **The stream belongs to the scenario,** which closes it when it ends. Client-streaming and bidirectional methods are not supported. ## Keep scenarios apart [Section titled “Keep scenarios apart”](#keep-scenarios-apart) Scenarios run in parallel against the same service, so each calls with data of its own, such as its parcel’s reference, and its streams see only what their calls asked for. See the [grpc pack’s reference](/next/references/packs/grpc/) for every step. # Call JSON-RPC services > Call your JSON-RPC 2.0 services with params from tables or doc strings, check their results and errors, and check every call against the service's OpenRPC document, with levels to relax a rule. The `jsonrpc` pack calls your JSON-RPC 2.0 services over HTTP, and checks the results and errors they answer. With the service’s [OpenRPC](https://open-rpc.org/) document, every call is also checked against its contract, as the REST pack checks requests against OpenAPI. ```gherkin Scenario: A depot holds a parcel until the day the recipient chose Given a seeds/depot-hold.yaml db seed And the depots jsonrpc service with the following properties: | url | http://localhost:8400/rpc | | openrpc | rpc.discover | When the parcel.hold method is called on the depots jsonrpc service with the following params: | reference | PX-RPC-7102 | | until | 2026-10-05 | | reason | recipient on holiday | Then the depots jsonrpc service's result has the following properties: | status | ON_HOLD | | heldUntil | 2026-10-05 | ``` Add the pack to the project with `axx pack add jsonrpc` ([Choose packs](/next/guides/use-packs/)). ## Register the service [Section titled “Register the service”](#register-the-service) | Property | What it is | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | `url` | The URL calls are posted to (required). | | `header.` | A header sent with every call, such as `header.Authorization`. | | `timeout` | How long a call may take. Default `10s`. | | `openrpc` | The service’s OpenRPC document: a file of the project, a URL, or `rpc.discover` to ask the service for it with the method OpenRPC defines. | `${env:..}` values are masked in logs and failures, and `${token:..}` names a token the scenario registered. ## Call a method [Section titled “Call a method”](#call-a-method) ```gherkin When the parcel.get method is called on the depots jsonrpc service with the params: """ ["PX-RPC-7101"] """ When the parcel.hold method is called on the depots jsonrpc service with the following params: | reference | PX-RPC-7102 | | until | 2026-10-05 | When the depot.list method is called on the depots jsonrpc service ``` * **A table gives params by name,** set as the REST pack’s request properties are: a row can be a path into a param (`address.postcode`), numbers and booleans are JSON’s, double quotes make a value text, `null` is null and `undefined` leaves it out. With an OpenRPC document, a param whose schema says `string` takes the cell as text. * **A doc string** gives an object (by name) or an array (by position). * **An error** does not fail the step that makes the call: check it. Only a service that does not answer JSON-RPC 2.0, such as an HTML error page, fails it. ## Check the result or the error [Section titled “Check the result or the error”](#check-the-result-or-the-error) ```gherkin Then the depots jsonrpc service's result has the following properties: | reference | PX-RPC-7101 | | heldUntil | undefined | Then the depots jsonrpc service's result is 'true' Then the depots jsonrpc service answered the error 4009 with a message containing 'can no longer be held' And the depots jsonrpc service's error has the following properties: | data.status | OUT_FOR_DELIVERY | ``` * `result is` compares a text result’s text, or any other result’s JSON. * The error’s properties are its `code`, its `message` and its `data`. * The checks read the service’s last call in the scenario. ## Check calls against OpenRPC [Section titled “Check calls against OpenRPC”](#check-calls-against-openrpc) With an `openrpc` row, every call’s params are checked against its method before it is sent, and every result and error when it comes back. A call that breaks the document fails the step that makes it, and names the rule it broke: | Key | Found when | | ------------------------------------ | ---------------------------------------------------------------------------------------------------------------- | | `validation.method.unknown` | the document has no such method | | `validation.params.structure` | the params are by name for a method that takes them by position (`paramStructure`), or the other way round | | `validation.params.missing` | a required param is missing | | `validation.params.unknown` | a param the method does not have | | `validation.params.schema.` | a param breaks a keyword of its schema (`required`, `type`, `pattern`…) | | `validation.result.schema.` | the result breaks a keyword of its schema | | `validation.error.unknown` | the error is not one the method declares, nor one of JSON-RPC’s own (-32700, -32600 to -32603, -32000 to -32099) | The document’s schemas are JSON Schema, draft 7 unless they say otherwise, and its `$ref`s reach its components and other files. ### Relax a rule [Section titled “Relax a rule”](#relax-a-rule) `ERROR` fails the step, `WARN` and `INFO` log the finding, and `IGNORE` drops it. Everything is `ERROR` by default. A scenario that sends a broken call on purpose, to see what the service answers, relaxes only what it breaks: ```gherkin Scenario: A hold without its day is refused Given the OpenRPC validation levels are: | validation.params.missing | IGNORE | When the parcel.hold method is called on the depots jsonrpc service with the following params: | reference | PX-RPC-7104 | Then the depots jsonrpc service answered the error -32602 with a message containing 'until' ``` A key also sets the keys below it (`validation.params` covers `validation.params.schema.pattern`), and the most specific key set wins. `packs.jsonrpc.openrpc.levels` in `axx.yaml` sets the defaults the scenarios’ levels are merged over: axx.yaml ```yaml packs: jsonrpc: openrpc: levels: validation.error.unknown: WARN ``` See the [jsonrpc pack’s reference](/next/references/packs/jsonrpc/) for every step. # Test Kafka with Avro > Configure Kafka topic clients, build and publish Avro events with a Schema Registry, verify consumed events, and address several events or clusters in one scenario. The Kafka steps publish the events your service consumes and check the events it produces, with Avro and a Schema Registry or as plain text. ## Register the cluster and a topic client [Section titled “Register the cluster and a topic client”](#register-the-cluster-and-a-topic-client) ```gherkin Background: Given the events kafka service with the following properties: | brokers | ${sys:local.host}:9092 | And a depot-scans kafka topic client with the following properties: | producer.value.serializer | io.confluent.kafka.serializers.KafkaAvroSerializer | | producer.schema.registry.url | http://${sys:local.host}:9081 | And a parcel-events kafka topic client with the following properties: | consumer.value.deserializer | io.confluent.kafka.serializers.KafkaAvroDeserializer | | consumer.schema.registry.url | http://${sys:local.host}:9081 | ``` The service consumes depot scans, so the suite publishes to `depot-scans`; it announces registered parcels on `parcel-events`, so the suite reads that topic. A topic the suite both publishes to and reads takes both sets of properties. An `asyncapi` row on the service names the AsyncAPI document its topics’ events follow: every event a scenario publishes, and every event a check consumes, is checked against it, Avro events in Avro’s JSON encoding ([Validate against AsyncAPI](/next/guides/validate-asyncapi/)). The topic client’s properties use the Kafka client property names, prefixed with `producer.` or `consumer.`. Axx translates them to its own Go client, so you do not need a JVM; the [Kafka pack reference](/next/references/packs/kafka/) lists every property it understands. Consumer group settings (`group.id`, `enable.auto.commit`) have no effect, because Axx reads topics without a group. For a TLS-secured cluster: ```gherkin Given a secure-events kafka topic client with the following properties: | producer.security.protocol | SASL_SSL | | producer.sasl.mechanism | SCRAM-SHA-512 | | producer.sasl.jaas.config | org.apache.kafka.common.security.scram.ScramLoginModule required username="app" password="${env:KAFKA_PASSWORD}"; | | producer.ssl.truststore.location | certs/truststore.p12 | | producer.ssl.truststore.password | ${env:TRUSTSTORE_PASSWORD} | ``` The consumer takes the same settings with the `consumer.` prefix. For a topic on a different cluster, register that cluster and the client on it explicitly: ```gherkin Given the depots kafka service with the following properties: | brokers | depots-kafka:9092 | And a depot-scans kafka topic client on the depots kafka service with the following properties: | producer.value.serializer | io.confluent.kafka.serializers.KafkaAvroSerializer | | producer.schema.registry.url | http://depots-registry:8081 | ``` ## Build an event [Section titled “Build an event”](#build-an-event) ```gherkin Given a depot-scans kafka event And the depot-scans kafka event key is PX-TRK-3002 And the depot-scans kafka event payload is a kafka/scan-delivered.json resource And the depot-scans kafka event payload properties are: | $.scanId | SC-3002-2 | | $.parcelRef | PX-TRK-3002 | And the depot-scans kafka event headers are: | X-Scanner-Id | leipzig-dock-4 | ``` * The payload starts from a file in Avro’s JSON encoding. Union values are written as `{"string": "..."}` or `null`; set `packs.kafka.lenientUnions: true` in `axx.yaml` to accept bare values when only one branch fits. * `the depot-scans kafka event payload properties are:` sets JSONPath properties of that topic’s first event; `the kafka event payload properties are:` uses the first topic client’s. Values are always set as **strings** (an empty cell sets null), and each property must already exist in the payload, so give numbers and objects their values in the file. * `the depot-scans kafka event payload property is null` sets a property to JSON null (for an Avro union with `null`). * Keep payload files schema-valid with [fixture factories](/next/guides/fixture-factories/) (the `avro` family); `kafka/scan-delivered.json` is generated that way. ## Publish [Section titled “Publish”](#publish) ```gherkin When the depot-scans kafka event is published using schema schemas/depot-scan.avsc ``` The payload is read with the Avro schema (`.avsc`, resolved against `resources`), the schema is registered under `-value` (or looked up, with `producer.auto.register.schemas=false`), and the record is sent in the Confluent wire format with the key and headers you set. A payload that does not fit the schema fails with the path of the mismatch, such as `$.location: expected string, got number 5`. To publish without Avro (plain text or JSON), a topic client with the default configuration is enough: its `StringSerializer` sends the payload as text. Every event needs a topic client for its topic: ```gherkin Given the delivery-notifications kafka topic client And a delivery-notifications kafka event And the delivery-notifications kafka event payload is a kafka/delivery-notification.json resource When the delivery-notifications kafka event is published ``` ## Verify consumed events [Section titled “Verify consumed events”](#verify-consumed-events) Check the events your service publishes. After a scenario registers parcel `PX-EVT-4001` through the API: ```gherkin Then the parcel-events kafka event named registered key is PX-EVT-4001 And the parcel-events kafka event named registered payload properties are: | $.reference | PX-EVT-4001 | | $.weightGrams | 1200 | | $.zone | DE-1 | | $.source | api | And the parcel-events kafka event named registered headers are: | X-Event-Type | ParcelRegistered | And the parcel-events kafka event named registered headers match: | X-Event-Type | ^Parcel.*$ | ``` `named registered` names an expectation. Each step adds its checks to the name and passes when one record of the topic meets all of them at once, so the four steps above check that a single record has that key, those properties and those headers. Records are read from the start of the topic (with `consumer.auto.offset.reset=latest`, only those produced after the step starts), and a step waits up to 30 seconds (`packs.kafka.timeout`). Expected payload values are typed, so `1200` does not equal `1200.0` ([the rules](/next/references/packs/kafka/#kafkaconsumedproperties)). `headers are` needs each header exactly once with that value; `headers match` takes regular expressions. When a step fails, the report lists the name’s checks and the latest records of the topic with the reason each one did not match. ## Test what your service does with an event [Section titled “Test what your service does with an event”](#test-what-your-service-does-with-an-event) Publishing and then consuming on the same topic only checks the wiring. An acceptance test publishes the event your service consumes and checks what the service does: a row in its database, a document, a response from its API, or an event on another topic. The parcels service keeps a tracking summary in MongoDB for every parcel the depots scan: ```gherkin Scenario: A delivery scan marks the parcel delivered Given a depot-scans kafka event And the depot-scans kafka event key is PX-TRK-3003 And the depot-scans kafka event payload is a kafka/scan-delivered.json resource And the depot-scans kafka event payload properties are: | $.scanId | SC-3003-1 | | $.parcelRef | PX-TRK-3003 | When the depot-scans kafka event is published using schema schemas/depot-scan.avsc Then within 20s a selection of at least 1 document is retrieved from the tracking collection where: | _id | PX-TRK-3003 | | delivered | true | And the 1st document for the selection properties are: | status | DELIVERED | | lastLocation | Leipzig | ``` The `Background` registers the Kafka service, `the depot-scans kafka topic client` and the MongoDB database. The service handles the event in the background, so the `Then` step polls until the summary appears instead of sleeping. ## Several events in one scenario [Section titled “Several events in one scenario”](#several-events-in-one-scenario) Address events by ordinal when a scenario builds more than one. `a 2nd ordered ... kafka event` must follow the 1st; steps without an ordinal use the first event: ```gherkin Given a 1st ordered depot-scans kafka event And the 1st ordered depot-scans kafka event key is PX-TRK-3002 And the 1st ordered depot-scans kafka event payload is a kafka/scan-out-for-delivery.json resource And the 1st ordered depot-scans kafka event payload properties are: | $.scanId | SC-3002-1 | | $.parcelRef | PX-TRK-3002 | And a 2nd ordered depot-scans kafka event And the 2nd ordered depot-scans kafka event key is PX-TRK-3002 And the 2nd ordered depot-scans kafka event payload is a kafka/scan-delivered.json resource And the 2nd ordered depot-scans kafka event payload properties are: | $.scanId | SC-3002-2 | | $.parcelRef | PX-TRK-3002 | When the 1st ordered depot-scans kafka event is published using schema schemas/depot-scan.avsc And the 2nd ordered depot-scans kafka event is published using schema schemas/depot-scan.avsc ``` ## Several clusters [Section titled “Several clusters”](#several-clusters) Every step has a form that names the Kafka service, such as `the depot-scans kafka event key is PX-TRK-3002 on the depots kafka service`. See the [step index](/next/references/step-index/) for the full list. ## Keep events apart [Section titled “Keep events apart”](#keep-events-apart) Topics keep their events across runs, and scenarios run in parallel. Use a key and identifiers that belong to one scenario (a parcel reference, a scan id), and assert on them. For fixture files, a generated `axx lint` rule can enforce that every event file owns a unique id, such as the `scanId` of every depot scan ([Isolate test data](/next/guides/isolate-test-data/)). # Test MCP servers > Call the tools of your MCP server as an AI assistant calls them, over stdio or HTTP, with the tools' own schemas as the contract; read its resources and prompts; and mock the MCP servers your service calls. An MCP server (Model Context Protocol) offers AI assistants tools, like `track_parcel`: the assistant reads each tool’s name, description and argument schema, calls tools as it works, and gets their results back. The `mcp` pack tests your server by doing what an assistant does: it calls the tools and checks what they answer. No model is involved, so the scenarios are deterministic. ```gherkin Scenario: An assistant tracks a parcel Given a seeds/mcp-track.yaml db seed And the parcels mcp server with the following properties: | url | http://localhost:8400/mcp | When the track_parcel tool is called on the parcels mcp server with the following arguments: | reference | PX-MCP-9101 | Then the track_parcel tool's result is not an error And the track_parcel tool's result has the following properties: | status | OUT_FOR_DELIVERY | | serviceLevel | EXPRESS | ``` Add the pack to the project with `axx pack add mcp` ([Choose packs](/next/guides/use-packs/)). ## Register the server [Section titled “Register the server”](#register-the-server) A server is reached over streamable HTTP (a `url`), or run over stdio (a `command`), as assistants run local servers. Which one follows how the server’s clients use it: * **A server its clients start, over stdio.** Claude Code, Cursor and the other assistants run a local server as their own child process and talk to it through its stdin and stdout; nothing else can reach it. In a scenario, the scenario is the client, so the step runs it with `command`, the way the cli pack runs a command-line tool. It is not a service: nothing goes under `services:` in `axx.yaml`. Each scenario runs its own server, so a server that keeps data on disk can keep each scenario’s apart (point its data folder, through `env.`, at a folder of the scenario’s own). * **A server that runs on its own, over HTTP,** deployed where clients connect to it. axx starts it as a service in `axx.yaml` ([Manage the services under test](/next/guides/manage-services/)), and the step connects with `url`. Test a server the way its clients reach it: a stdio server switched to HTTP only for its tests is tested over a transport its users never use. ```gherkin Given the desk mcp server with the following properties: | command | docker compose exec -T app parcels mcp | | dir | ../infra | ``` | Property | What it is | | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | `url` | The server’s streamable HTTP endpoint. | | `command` | The command that runs the server over stdio, split into words without a shell. The scenario runs it, and ends it, and whatever it started, when it ends. | | `dir` | The folder the command runs in, relative to the project. Default: a folder of the scenario’s own. | | `env.` | An environment variable of the command. | | `header.` | A header sent with every request to the url, such as `header.Authorization`. | | `timeout` | How long connecting, or a call, may take. Default `30s`. | | `protocol version` | The MCP version to speak, like `2025-06-18`, to check that the server still answers an earlier one. Default: the latest the server speaks. | The session agrees on the protocol version with the server: MCP 2026-07-28, which has no sessions, or an earlier one. `${env:..}` values are masked in logs and failures, and `${token:..}` names a token the scenario registered ([Test webhooks and tokens](/next/guides/webhooks-and-tokens/)). ## What an assistant sees [Section titled “What an assistant sees”](#what-an-assistant-sees) Before an assistant calls a tool, it reads the server’s list of tools: their titles, descriptions and hints. Check them, and check that a tool the server must not offer is not there: ```gherkin Then the parcels mcp server has the track_parcel tool with the following properties: | title | Track a parcel | | annotations.readOnlyHint | true | And the parcels mcp server does not have the cancel_parcel tool ``` ## Call tools [Section titled “Call tools”](#call-tools) ```gherkin When the hold_parcel tool is called on the parcels mcp server with the following arguments: | reference | PX-MCP-9102 | | until | 2026-10-05 | When the hold_parcel tool is called on the parcels mcp server with the mcp/hold-parcel.json arguments ``` * **An argument takes the type the tool’s input schema gives it:** `01067` is text for a string, and a number for an integer. Double quotes make a value text, `null` is null, and `undefined` leaves it out. A row can be a path into an argument (`address.postcode`, `lines[0].reference`). * **A file’s arguments** are a JSON or YAML object. * **A result that is an error does not fail the call,** nor does a call the server refuses: check them. ## Check results [Section titled “Check results”](#check-results) ```gherkin Then the hold_parcel tool's result has the following properties: | status | ON_HOLD | | heldUntil | 2026-10-05 | Then the hold_parcel tool's result is an error And the hold_parcel tool's result contains 'it can no longer be held' Then the cancel_parcel tool's call failed with the error code -32602 ``` A tool fails in two ways, which the checks tell apart: * **The tool ran, and says what went wrong:** its result is an error (`is an error`), with text the assistant reads, to tell the user or try again. A parcel that can no longer be held is one. * **The server refused the call:** a JSON-RPC error, with its code (`failed with the error code`): `-32602` for a tool it does not have or invalid parameters. `has the following properties` reads the tool’s structured result, or the JSON of its text for a tool that answers JSON as text. `contains` reads its text and its structured result. The checks read the tool’s latest result in the scenario; when two servers of the scenario have the same tool, name the server: `the track_parcel tool's result on the desk mcp server ...`. ## The schemas are the contract [Section titled “The schemas are the contract”](#the-schemas-are-the-contract) A tool’s arguments are checked against its input schema before they are sent, and its structured result against its output schema. A finding fails the step: | Key | Found when | | --------------------------------------- | ------------------------------------------------------------------------- | | `validation.arguments.schema.` | the arguments break the tool’s input schema (`required`, `type`, `enum`…) | | `validation.result.schema.` | the structured result breaks the tool’s output schema | | `validation.result.missing` | a tool with an output schema answered no structured result | A scenario that sends broken arguments on purpose, to see how the server refuses them, relaxes only what it breaks: ```gherkin Scenario: A hold without its day is refused by the server too Given the MCP validation levels are: | validation.arguments.schema.required | IGNORE | When the hold_parcel tool is called on the parcels mcp server with the following arguments: | reference | PX-MCP-9102 | Then the hold_parcel tool's result is an error ``` `ERROR` (or `FAIL`) fails the step, `WARN` and `INFO` log the finding, and `IGNORE` drops it. A key also sets the keys below it, and `packs.mcp.levels` in `axx.yaml` sets the defaults the scenarios’ levels are merged over. ## Resources and prompts [Section titled “Resources and prompts”](#resources-and-prompts) ```gherkin When the parcels://PX-MCP-9103/label resource is read from the parcels mcp server Then the parcels://PX-MCP-9103/label resource has the following properties: | reference | PX-MCP-9103 | When the delivery_update prompt is requested from the parcels mcp server with the following arguments: | reference | PX-MCP-9103 | Then the delivery_update prompt contains 'parcel PX-MCP-9103' ``` A resource is read by its URI; `has the following properties` reads its text as JSON. A prompt’s check reads the text of its messages. ## Mock the MCP servers your service calls [Section titled “Mock the MCP servers your service calls”](#mock-the-mcp-servers-your-service-calls) When your service calls another team’s MCP server, the Axx WireMock image mocks it from a description of its tools, with a mapping file per answer, and the mock pack checks what your service called. See [Mock MCP servers and A2A agents](/next/guides/mock-dependencies/#mock-mcp-servers-and-a2a-agents). See the [mcp pack’s reference](/next/references/packs/mcp/) for every step. # Test message brokers > Send and publish messages to AMQP queues and exchanges (RabbitMQ, ActiveMQ Artemis), MQTT topics and NATS subjects, and check the messages your services send there, JetStream streams included. Services talk through brokers: label printers report over AMQP, a depot’s handheld scanners publish over MQTT, and couriers’ confirmations and tracking updates fan out over NATS. The `amqp`, `mqtt` and `nats` packs let a scenario do what the other side does, such as publishing the printer’s report. The scenario then checks what your service did: the row it wrote, or the message it sent on. ```gherkin Scenario: A report of a parcel the service does not know is set aside When the amqp/label-printed-unknown.json message is published to the printers amqp exchange with the following properties: | routing key | printed.LEJ | | header printer | LEJ-3 | Then within 10s the parcels.label-printed.rejected amqp queue has a message where: | reference | PX-AMQ-8199 | | header reason | unknown parcel | | header printer | LEJ-3 | ``` ## The packs [Section titled “The packs”](#the-packs) | Pack | Talks to | A step names | | ------ | ----------------------------------------------------------------------------------- | ------------------------------------- | | `amqp` | RabbitMQ over AMQP 0-9-1, and brokers that speak AMQP 1.0, such as ActiveMQ Artemis | an `amqp queue` or an `amqp exchange` | | `mqtt` | MQTT 5 brokers: Mosquitto, EMQX, HiveMQ, RabbitMQ’s MQTT plugin | an `mqtt topic` | | `nats` | NATS, with JetStream | a `nats subject` or a `nats stream` | The three speak the same grammar as the cloud packs’ queues and topics ([Test cloud services](/next/guides/test-cloud-services/)). Add the ones you use with `axx pack add amqp mqtt nats` ([Choose packs](/next/guides/use-packs/)). ## Connect to the broker [Section titled “Connect to the broker”](#connect-to-the-broker) Register the broker once, usually in the `Background`. Every step of that pack in the scenario uses it: ```gherkin Background: Given the depot amqp broker with the following properties: | url | amqp://parcels:${env:RABBITMQ_PASSWORD}@${sys:local.host}:5672/ | And the depots mqtt broker with the following properties: | url | mqtt://${sys:local.host}:1883 | | username | depot-scanners | | password | ${env:MQTT_PASSWORD} | And the tracking nats server with the following properties: | url | nats://${sys:local.host}:4222 | | token | ${env:NATS_TOKEN} | ``` * **AMQP:** a `url` with the user, password and virtual host, and a `protocol`: `0-9-1` (the default) or `1.0`. * **MQTT:** a `url` (`mqtt://`, `mqtts://`, or `ws://` and `wss://` for MQTT over WebSockets), and a `username` and `password`. * **NATS:** a `url`, or several separated by commas, and one way to sign in: a `token`, a `username` and `password`, or a `creds` file. `${env:..}` values, such as passwords, are masked in logs and failures, and so is a password written in a URL. An `asyncapi` row names the AsyncAPI document the broker’s messages follow: every message a scenario sends there, and every message a check finds, is checked against it ([Validate against AsyncAPI](/next/guides/validate-asyncapi/)). With AMQP 1.0, a queue is an anycast address and an exchange a multicast one, as ActiveMQ Artemis and Qpid have them, and the routing key is the message’s subject. RabbitMQ routes the same messages over 0-9-1 whatever protocol your services use, so keep the default for it. ## Send and publish [Section titled “Send and publish”](#send-and-publish) A message’s body is the doc string, or a file of the project: ```gherkin When a message is published to the printers amqp exchange with the routing key 'printed.LEJ': """ {"reference": "PX-AMQ-8102", "printer": "LEJ-3", "printedAt": "2026-09-28T08:15:00Z"} """ When a message is sent to the parcels.label-printed amqp queue: """ {"printer": "LEJ-3", "job": "JOB-AMQ-8105"} """ When a message is published to the depots/LEJ/scans mqtt topic: """ {"scanId": "SC-8201-1", "parcelRef": "PX-MQT-8201", "status": "OUT_FOR_DELIVERY"} """ When the nats/PX-NAT-8303-delivered.json message is published to the deliveries.confirmed nats subject with the following headers: | courier | CR-LEJ-12 | ``` A file’s message takes a table of what goes with it: | Pack | Rows | | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | | `amqp` | `routing key` (exchanges), `header `, `content type`, `correlation id`, `message id`, `reply to` | | `mqtt` | `qos` (`1` unless it says otherwise), `retain` (`false`), `property ` (user properties), `content type`, `response topic`, `correlation data` | | `nats` | a header on each row: its name and its value | A few more things happen when a message is sent: * **AMQP messages are persistent,** and a message whose body is JSON is sent as `application/json` unless a `content type` row says otherwise. * **A message that goes nowhere fails the step:** the broker returns an AMQP message that no queue is bound for, and an MQTT broker may say nothing subscribes to a topic. The step says so, rather than letting the message vanish. * **A NATS message waits for its stream:** when a JetStream stream captures the subject, the message is published through JetStream, and the step waits for the stream to store it. ## Check what your services sent [Section titled “Check what your services sent”](#check-what-your-services-sent) A check waits for a message that meets it: 10 seconds, or the time `within` gives. It looks only at the messages received since its scenario started. ```gherkin Then within 10s the labels amqp exchange has a message where: | routing key | print.express | | reference | PX-AMQ-8101 | | header sender | alder-and-ash | Then within 10s the depots/+/alerts mqtt topic has a message where: | topic | depots/LEJ/alerts | | parcelRef | PX-MQT-8299 | | property scanner | LEJ-HANDHELD-7 | Then within 20s the TRACKING nats stream has a message where: | subject | tracking.PX-NAT-8303 | | header courier | CR-LEJ-12 | ``` Each row is a path into the message’s JSON body and the value it has, as text: `null` for null and `undefined` for absent. A few rows are on the message itself instead: | Pack | Rows on the message | | ------ | ------------------------------------------------------------------------------------------ | | `amqp` | `header `, `routing key`, `content type`, `correlation id`, `message id`, `reply to` | | `mqtt` | `property `, `topic`, `content type`, `response topic`, `correlation data` | | `nats` | `header `, `subject` | A check can name many topics or subjects at once: MQTT’s `+` and `#` (`the depots/+/alerts mqtt topic`) and NATS’s `*` and `>` (`the tracking.> nats subject`). A `topic` or `subject` row then checks the one a message came on. ## How the checks listen [Section titled “How the checks listen”](#how-the-checks-listen) For the targets a run’s checks name, axx starts listening once the services are up, before the first scenario, so no message is missed: * **An AMQP queue** is read, and each message acknowledged, as any consumer would. Check the queues your services write to and nothing else reads. * **An AMQP exchange** takes nothing from anyone. Axx declares a queue of its own for the run (`axx-.`, exclusive, deleted with the run), and binds it with `#` and with the routing keys the checks name. That covers topic, fanout, headers and direct exchanges. * **An MQTT topic** gets a subscription of its own, on a connection of its own. It doesn’t receive the retained messages the broker keeps for new subscribers, which earlier runs may have left. * **A NATS subject** gets a subscription of its own. * **A JetStream stream** is read with an ordered consumer of its own, from the messages stored after the services are up, so what earlier runs stored never counts. It takes nothing from the stream’s other consumers. A work-queue stream allows no other consumer, so check its subject instead. ## Keep scenarios apart [Section titled “Keep scenarios apart”](#keep-scenarios-apart) Scenarios run in parallel on the same brokers, and every listener sees every scenario’s messages. So a check tells a scenario’s message apart by data unique to it, such as its own parcel reference ([Isolate test data](/next/guides/isolate-test-data/)). A queue the check reads is shared by the whole run: a message another scenario sent is still there for that scenario’s check. ## Run the brokers locally [Section titled “Run the brokers locally”](#run-the-brokers-locally) Start the brokers next to your service in the Compose file your [service definition](/next/guides/manage-services/) runs: compose.yaml ```yaml services: rabbitmq: image: rabbitmq:4.2.9-alpine ports: ['5672:5672'] environment: { RABBITMQ_DEFAULT_USER: parcels, RABBITMQ_DEFAULT_PASS: parcels } mosquitto: image: eclipse-mosquitto:2.0.22 command: ['mosquitto', '-c', '/mosquitto-no-auth.conf'] ports: ['1883:1883'] nats: image: nats:2.15.0-alpine command: ['nats-server', '-js'] ports: ['4222:4222'] ``` RabbitMQ’s default `guest` user can only connect from the broker’s own host, so give it a user of your own. Mosquitto only listens to its own host without a configuration of its own, and `/mosquitto-no-auth.conf` opens it up. NATS needs `-js` for JetStream. See the [amqp](/next/references/packs/amqp/), [mqtt](/next/references/packs/mqtt/) and [nats](/next/references/packs/nats/) packs’ references for every step. # Test mobile apps > Use your Android and iOS apps the way people do, on emulators and simulators axx runs: tap and fill their controls by the names people see, open them with deep links, check what they show; each scenario with a device to itself and a clean app. Many acceptance criteria describe what a person does in an app: a courier signs in, taps a parcel and marks it delivered. The `mobile-android` and `mobile-ios` packs do exactly that, on Android emulators and iOS simulators, driven by [Appium](https://appium.io), and a scenario can use every other pack too: it checks what your service did with the delivery, and what the parcel’s tracking page shows. ```gherkin Scenario: A courier delivers a parcel, and its recipient sees it delivered Given a seeds/courier-app-deliver.yaml db seed When the courier app is opened with the "parcels-courier://deliveries/PX-MOB-9411" link And the "Courier ID" field in the courier app is filled with "CR-LEJ-14" And the "PIN" field in the courier app is filled with "${env:COURIER_PIN}" And the "Sign in" button is tapped in the courier app Then the courier app shows "Eisenbahnstr. 41, 04315 Leipzig" When the "Signed by" field in the courier app is filled with "C. Busch" And the "Mark delivered" button is tapped in the courier app And the "Confirm" button is tapped in the courier app Then the courier app shows "Delivered" And the courier app shows a notification "PX-MOB-9411 delivered" When the "/track/PX-MOB-9411" page is opened Then within 20s the page shows "Delivered on" ``` The steps are the same on Android and on iOS: only the app’s registration says which it is. Add the pack for each platform you test with `axx pack add mobile-android` or `axx pack add mobile-ios` (both build on `mobile-core`, the steps of phones, and on `app-core`, the steps every app has, on a phone or a desktop; [Choose packs](/next/guides/use-packs/)). ## What it needs [Section titled “What it needs”](#what-it-needs) **For Android:** * **The Android SDK**, with `ANDROID_HOME` pointing at it: its emulator, and adb. Android Studio installs it, or its command-line tools. An adb server axx starts has its mDNS discovery off (`ADB_MDNS=0`, unless you set `ADB_MDNS`): axx needs none, and on some networks it makes adb’s server abort. * **An emulator’s device (AVD)**, like the one Android Studio’s Device Manager makes, or one `avdmanager` makes: ```sh sdkmanager --install "system-images;android-35;google_apis;arm64-v8a" # x86_64 on Intel and AMD avdmanager create avd --name parcels-pixel --package "system-images;android-35;google_apis;arm64-v8a" --device pixel_8 ``` * **For an app’s ARCore** ([below](#what-the-camera-sees-and-ar)): the SDK’s NDK and build-tools, and a JDK, which axx builds its agent with, once. **For iOS:** * **A Mac with Xcode**, and an iOS runtime for its simulators (Xcode’s Settings, Components). axx makes the simulators itself. * **The app built for the simulator**: the `.app` that `xcodebuild -sdk iphonesimulator` builds, or it zipped. `axx doctor` checks them: the Android SDK and its devices (and KVM on Linux), Xcode and its iOS runtimes. **Nothing else:** axx downloads what drives the apps the first time a run needs it, pinned, as it downloads a browser driver for web apps. For Android, that is the UiAutomator2 server, which axx installs on each device and starts itself, and keeps running while the device is in use: a scenario’s session starts in the time its app takes to reset. For iOS, it is WebDriverAgent, the app on the simulator that drives yours, as the [Appium](https://appium.io) project builds it, which axx starts on each simulator itself: nothing is built with Xcode, and nothing needs signing. ## Register the app [Section titled “Register the app”](#register-the-app) ```gherkin Given the courier android app with the following properties: | apk | ../courier/android/app/build/outputs/apk/debug/app-debug.apk | | device | parcels-pixel | | permissions | POST_NOTIFICATIONS | | timezone | Europe/Berlin | | host ports | 8400 | ``` ```gherkin Given the courier ios app with the following properties: | app | ../courier/ios/build/Debug-iphonesimulator/Courier.app | | device | iPhone 16 | | timezone | Europe/Berlin | ``` | Property | What it is | | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `apk` (Android), `app` (iOS) | The app, a file of the project: it is the app the scenario tests. On iOS, a simulator build (`.app`), or it zipped. | | `package`, `activity` (Android), `bundle id` (iOS) | The app’s identity: with no `apk` or `app`, an app the device has. On iOS, the `app`’s own bundle identifier by default. | | `device` | Android: an emulator’s device (AVD) axx starts, or a device adb lists, by its serial. iOS: a device type axx makes simulators of, like `iPhone 16` or `iPhone 16, iOS 18.1` (the newest iOS Xcode has, by default); a simulator you set up, by its name, which axx clones; or a simulator’s UDID. | | `permissions` | What the app may use from the start, and nothing else. Android: permissions like `POST_NOTIFICATIONS`. iOS: services like `location, photos` (`xcrun simctl help privacy` lists them). | | `locale`, `timezone`, `location` | The language and region (`de-DE`), the time zone (`Europe/Berlin`), and where the device says it is (`51.3397, 12.3731`). On Android the language is the app’s own (Android 13 and later), and the device’s stays as it is. | | `photos`, `documents` | Files the phone has before the app starts, for the app to pick in the system’s pickers, like `photos/begin.jpg` or `documents/safe-place-authorization.pdf`: found as a step finds a file, next to the feature file first. Photos go in the photo library; documents where the file picker shows them (see [Pick a document](#pick-a-document)). | | `host ports` (Android) | Ports of the machine axx runs on that the app reaches as `localhost` on the device, like `8400, 5500`: the app calls `http://localhost:8400` as on a developer’s device, and gets the service there. Each scenario has only its own registration’s. | | `appium`, `capability.` | An Appium server of your own or a device farm’s, which runs the device, and the capabilities it takes. On Android without `appium`, the capabilities go to the UiAutomator2 server’s session. | The app starts when a step launches it or opens a link in it. Every step names the app, so a scenario can drive two, and a web app too. ## Each scenario has a device, and a clean app [Section titled “Each scenario has a device, and a clean app”](#each-scenario-has-a-device-and-a-clean-app) A scenario leases a device for its whole run: no other scenario uses it meanwhile. The packs run devices as scenarios need them, up to `devices` of each at once (1 by default: those scenarios then take turns while the others run). axx.yaml ```yaml packs: mobile-android: devices: 2 # two emulators: two Android scenarios at once mobile-ios: devices: 2 # two simulators: two iOS scenarios at once ``` * **Android** starts emulators of the device read-only, so that several run at once and nothing a scenario changes outlives it. A device with a boot snapshot (boot it once, and close it) starts in seconds; without one, each emulator boots cold. * **iOS** gives each device it runs a clone, deleted when the run ends: of the simulator you set up, or, for a device type, of axx’s own simulator of it. axx makes that one and boots it once to set it up, the same on every Mac, and keeps it in its cache (`~/Library/Caches/axx/mobile/simulators`, apart from Xcode’s). No scenario ever runs on it, and its clones boot in seconds. WebDriverAgent, the app that drives yours, starts once on each simulator, for all the scenarios that run on it. Before its app starts, a scenario’s app is reset, and what it sees of the device is set: * **Android:** its data cleared (and with it its sign-in, its permissions and its notifications), the permissions its registration names granted, the notification shade and any dialog of the system’s closed, the ports it reaches on this machine set, the photos and documents a scenario before put on the device taken out (and the registration’s put in), the system’s file picker started afresh, and the device’s language, time zone and location set: an emulator goes back to where it starts when the registration names none. On the emulators axx starts, other apps that hang or crash show no dialog over yours, and Chrome opens a page at once, without its first-run screens. * **iOS:** the app installed afresh (its data and notifications go with the old one), the simulator’s keychain reset (a sign-in kept there outlives the app), its permissions reset and those its registration names granted, the documents a scenario before put On My iPhone taken out (and the registration’s put in), and the location set. It starts in the registration’s language, region and time zone. There is no switch that skips it: a scenario is one journey, and that journey is its own. Its data stays unique as everywhere in axx: each scenario above has a courier and parcels of its own. ### Keep simulators warm [Section titled “Keep simulators warm”](#keep-simulators-warm) A run makes its iOS simulators and deletes them at its end. With `keep`, it keeps them booted for the next run instead, which then starts at once: in the parcels example, its three iOS scenarios ran in 41 seconds instead of 68. axx.local.yaml ```yaml packs: mobile-ios: keep: true ``` The kept simulators are in Xcode’s own set, named like `axx iPhone 17, iOS 27.0 (1)`, where Device Hub lists them. A run uses one only while no other run does, and each scenario still has its simulator to itself, its app reset. A watched run keeps them. To get rid of them, delete them in Device Hub, or with `xcrun simctl delete`. ## Run one platform [Section titled “Run one platform”](#run-one-platform) A profile per platform runs the scenarios of one app, with no tags on the features: `run.uses` picks the scenarios by the packs their steps use ([Select by what scenarios use](/next/guides/tags-and-filtering/#select-by-what-scenarios-use)). axx.yaml ```yaml profiles: ios: {run: {uses: [mobile-ios]}} android: {run: {uses: [mobile-android]}} ``` `axx run --profile ios` runs the iOS app’s scenarios, and `--profile ios,watch` runs them watched. ## Watch it [Section titled “Watch it”](#watch-it) `axx run --watch` shows what the scenarios do as they do it: an iOS simulator in Device Hub (the Simulator app before Xcode 27), an Android emulator in its window, a browser in its own, one scenario at a time, slowed down with `--slowdown 500ms` ([Watch a run](/next/guides/watch-runs/)). Without `--watch`, the devices run without a window, and a Device Hub you have open stays open. ## Launch it, or open it with a link [Section titled “Launch it, or open it with a link”](#launch-it-or-open-it-with-a-link) ```gherkin When the courier app is launched When the courier app is opened with the "parcels-courier://deliveries/PX-MOB-9411" link When the courier app is sent to the background When the courier app is brought back When the courier app is restarted ``` A deep link jumps straight to the screen a scenario tests, instead of tapping through the ones before it. A restart keeps what the app stored, like a sign-in; the next scenario’s reset does not. ## An app’s files [Section titled “An app’s files”](#an-apps-files) An app’s files are a folder of the files pack whose `owner` is the app: `./` is its data on the device (on iOS its data container, on Android its data folder, which Android opens to a debuggable build only) ([Check files](/next/guides/check-files/)). A scenario reads what the app saved there, and puts files in place before the app first reads them: the folder starts the app’s device, installs the app and resets it, as launching does, without launching it. A scenario about a returning user starts so, with the app’s settings as that user’s app has them: ```gherkin Given the settings folder with the following properties: | owner | app:courier | | path | Library/Preferences | And the example.parcels.courier.plist file in the settings folder is a copy of the fixtures/signed-in-before.plist file When the courier app is launched ``` ## Tap and fill [Section titled “Tap and fill”](#tap-and-fill) ```gherkin When the "Sign in" button is tapped in the courier app When the "PX-MOB-9401" list item is tapped in the courier app When the "PIN" field in the courier app is filled with "${env:COURIER_PIN}" When the courier app is swiped down When the "PX-MOB-9412" list item is scrolled into view in the courier app When the courier app's back button is pressed ``` Controls are found by the names people see: a button by its text, a field by its label, a list item by one of its texts (its reference, say), an image by its description. The kinds are `button`, `field`, `checkbox`, `switch`, `tab`, `list item`, `image`, `text` and `element` (anything). Jetpack Compose and Android’s views both work, and SwiftUI and UIKit on iOS. When no name tells a control apart, `id=` (its resource ID on Android, its accessibility identifier on iOS) or `xpath=` finds it. `${env:..}` values are secrets: masked in logs and failures. `swiped down` pulls a list to refresh; `up`, `left` and `right` swipe too. The back button is Android’s; on iOS, tap the back button the screen shows, by its text. `scrolled into view` scrolls the list or text that holds the control until it shows, as a person does. A control far down a long text, like the end of a terms of service, is reached in one move on iOS, which grabs the scroll bar and drags it there, and in a few flings on Android. ## Check what it shows [Section titled “Check what it shows”](#check-what-it-shows) ```gherkin Then the courier app shows "Hello, Hanna Wolf" Then the courier app does not show "PX-MOB-9401" Then the "Mark delivered" button is disabled in the courier app Then the "Courier ID" field in the courier app has the value "CR-LEJ-12" Then the courier app shows a notification "PX-MOB-9411 delivered" Then the courier app looks like the "deliveries" screenshot ``` Checks wait for the app, 10 seconds or `within {duration}`, as apps take their time. A failed step attaches what the app showed, and the scenario’s failure says which device and which resets it had. A notification is looked for where people look: Android’s notification shade, iOS’s Notification Center. **Screenshots** are `..png` in a `screenshots` folder next to the feature file, like `deliveries.android-parcels-pixel.png` or `deliveries.ios-iPhone-16-iOS-18-1.png`: each device has its own. The device’s status and navigation bars, its clock among them, are left out. The first time, the step takes the screenshot and fails: look at it, keep it, and run again. `packs.mobile-core.screenshots` in `axx.yaml` sets the `folder`, the `tolerance` and `update: true`, which takes them again. ## Tap a notification [Section titled “Tap a notification”](#tap-a-notification) ```gherkin When the "PX-MOB-9402 delivered" notification is tapped in the courier app When the "Next delivery" action of the "PX-MOB-9401 delivered" notification is tapped in the courier app ``` A notification is tapped where people find it: the step opens Android’s notification shade or iOS’s Notification Center, opens the group it is folded into (on iOS, one already seen waits below the screen, and is swiped into view), and taps it, and the app opens where the notification leads. An action is one of the buttons a notification offers: Android shows them under it, and iOS when a person holds the notification. On a simulator, iOS answers a tap on a notification as on an iPhone without Face ID: it slides the notification aside to show Open, which the step taps too. And it shows a notification’s actions in Notification Center only on a phone it can tell is in a hand, one with a proximity sensor (or Face ID unlock, which takes a passcode a simulator cannot have). So axx gives the simulator’s SpringBoard the proximity sensor an iPhone has: a small library axx builds with Xcode the first time, and loads into the simulators it boots. In your app, it does only what [background tasks](#run-a-background-task-now) need. A simulator that runs already (one of yours, or one kept warm) gets it as SpringBoard starts again, before the scenario. ## Run a background task now [Section titled “Run a background task now”](#run-a-background-task-now) ```gherkin When the "round-reminder" background task is run in the courier app Then within 30s the courier app shows a notification "2 deliveries left today" ``` An app schedules work for the phone to run later: a sync, a reminder, a check. The phone chooses when, every 15 minutes at the soonest on Android, and whenever it suits it on iOS. The step has the phone run one of the app’s tasks now, as it does when the task’s time comes. It is a nudge, not a stand-in: the app’s own code runs, in the app, and does what it does then. Only the moment moves. The app must have scheduled the task. The step waits for it to be done, so a check after it finds what it did: on iOS, until the app says it is done (`setTaskCompleted`), in the 30 seconds iOS gives a task; on Android, until the work ends, as WorkManager’s database has it. The task is named as the app names it: * **Android:** its WorkManager work’s unique name (`enqueueUniquePeriodicWork("round-reminder", ...)`), or one of its tags. WorkManager tags each piece of work with its worker’s class, so `RoundReminderWorker` names it too. axx asks Android’s job scheduler to run the work’s job now, and Android runs it only if the conditions the app set for it, a network say, are met. WorkManager runs periodic work no sooner than its period allows, so axx first tells it the period has passed, in its own database on the phone. That takes the phone’s `sqlite3`, which emulators with a Google APIs image have, and a debuggable build, as an app’s files do. * **iOS:** its identifier, one of `BGTaskSchedulerPermittedIdentifiers` in its `Info.plist`. iOS runs the task as Xcode’s debugger command does on an iPhone: the handler the app registered gets the task the app scheduled, once for each time it scheduled it. A simulator refuses to schedule any task, so axx’s library, which the simulators it boots load, keeps the app’s requests as an iPhone does. An app in the background is suspended, and SpringBoard wakes it for the task, as iOS does, and holds it awake until the task is done. A name the app does not have fails, and the failure lists the tasks it has. ## Pick a document [Section titled “Pick a document”](#pick-a-document) An app that takes a file from the phone, like a courier’s app attaching a recipient’s signed authorization to a delivery, opens the system’s file picker. The `documents` property puts the files on the phone before the app starts, and the scenario picks one as a person does, with the tap steps: ```gherkin Given the courier android app with the following properties: | apk | ../courier/android/app/build/outputs/apk/debug/app-debug.apk | | device | Pixel_9 | | documents | documents/safe-place-PX-MOB-9481.pdf | And the courier app is opened with the "parcels-courier://deliveries/PX-MOB-9481" link When the "Attach document" button is tapped in the courier app And the "Show roots" button is tapped in the courier app And the "Downloads" text is tapped in the courier app And the "safe-place-PX-MOB-9481.pdf" text is tapped in the courier app Then the courier app shows "safe-place-PX-MOB-9481.pdf attached" ``` * **Android** puts them in the device’s Downloads, indexed as a download is. The picker opens at Recent: *Show roots*, then *Downloads*, then the file by its name. On an emulator axx starts, each scenario’s reset empties Downloads (what an app saved there too) and starts the picker afresh, as it keeps where each app last picked a file; on a device of yours, it takes out only the documents axx put there. * **iOS** puts them On My iPhone, in the simulator’s Files. In the picker, *Browse*, then *On My iPhone*, then the file as a list item, by its name without its extension, as Files shows it (below). On a simulator axx makes, each scenario’s reset empties On My iPhone (what an app saved there too); on a simulator of yours, it takes out only the documents axx put there. ```gherkin When the "Attach document" button is tapped in the courier app And the "Browse" button is tapped in the courier app And the "On My iPhone" text is tapped in the courier app And the "safe-place-PX-IOS-9591" list item is tapped in the courier app ``` The phone has its documents in one folder, by their names: two files of one name are refused. A device of an Appium server’s, like a device farm’s, takes files its own way. ## Dialogs the system shows [Section titled “Dialogs the system shows”](#dialogs-the-system-shows) ```gherkin Then the courier app's dialog shows "send you notifications" When the courier app's dialog is dismissed ``` A dialog the system shows over the app, like a request for a permission it asked for, is accepted (allowed) or dismissed. iOS lets no one grant notifications ahead: an app that uses them asks, and the scenario answers. Accepted, a request with a choice of how to allow allows for good: *While using the app* for a location (not *Only this time*), *Allow Full Access* for photos. A dialog of the app’s own is part of its screen: tap its buttons, like `the "Confirm" button is tapped in the courier app`. ## Move the phone along a route [Section titled “Move the phone along a route”](#move-the-phone-along-a-route) The `location` property puts the phone in one place. An app that follows the phone as it moves, like a courier’s app sharing where the van is, needs it to travel, with its GPS reporting where it is, how fast it goes and which way: ```gherkin When the courier app's phone travels at 30 km/h through: | 51.3397 | 12.3731 | | 51.3397 | 12.3760 | ``` The table’s rows are the points the route goes through, each a latitude and a longitude; the phone keeps one speed (`mph`, `km/h` or `m/s`) from the first to the last, and stays at the last. A route recorded on the road, or drawn in a mapping tool, is a GPX file: its track, else its route, else its waypoints. The phone keeps to the file’s own times, stops included, or goes faster, or keeps one speed throughout: ```gherkin When the courier app's phone travels the routes/grimmaische-strasse.gpx route When the courier app's phone travels the routes/grimmaische-strasse.gpx route at 4x When the courier app's phone travels the routes/grimmaische-strasse.gpx route at 30 km/h ``` `travels` waits for the phone to arrive. `starts traveling` lets the scenario go on while the phone moves, to check what the app does on the way, and a later step waits for it to get there: ```gherkin When the courier app's phone starts traveling at 30 km/h through: | 51.3397 | 12.3731 | | 51.3397 | 12.3760 | Then within 20s the tracking event stream has an event where: | event type | courier | | speedKmh | 30 | And the courier app's phone reaches the end of its route ``` The phone moves in real time, as a phone on the road does: keep routes short, or play a long one faster. On iOS, the simulator reports the route as CoreLocation reports a moving phone (`simctl location`): its speed, and its course where the simulator gives one (older simulators report none). On Android, the emulators axx starts report it every second, as NMEA sentences that carry the speed and course a plain location fix lacks; a device of your own reports where it really is. ## What the camera sees, and AR [Section titled “What the camera sees, and AR”](#what-the-camera-sees-and-ar) An iOS simulator has no camera, and an Android emulator’s sees a room of its own. An app that scans, reads or photographs through it (a QR code, a printed reference, a photo at the door) has nothing to see, and an app that measures or places things with ARKit (RealityKit’s `ARView`) or ARCore has no AR either: the simulator’s ARKit says world tracking is not supported, and an emulator has no Google services for AR. axx plays the camera from what a scenario says it sees, `the courier app's camera sees `, on iOS simulators and on the Android emulators it starts: * **an image** (PNG or JPEG, and HEIC on iOS), as the phone held upright sees it: a parcel’s label, say; * **a video** (MP4 or MOV; H.264 on Android), which plays in a loop: a parcel lifted into view; * **a 3D scene**, a glTF file (`.glb`, or `.gltf` with its buffers and images), seen from where the phone is in it: what the camera sees changes as the phone moves (below), and an app’s AR finds the scene’s surfaces and objects where they are. ```gherkin Given the courier app's camera sees labels/label-PX-IOS-9621.png And the courier app is launched And the "Courier ID" field in the courier app is filled with "CR-LEJ-85" And the "PIN" field in the courier app is filled with "${sys:couriers.pin}" And the "Sign in" button is tapped in the courier app When the "Scan label" button is tapped in the courier app Then the courier app shows "Ida Lorenz" ``` On a simulator, the app’s capture session takes the camera as its input, as on a phone (`AVCaptureDevice.default(.builtInWideAngleCamera, for: .video, position: .back)`), and gets its frames, 30 a second, 1920 by 1080, the sensor’s way up as an iPhone’s: * a **video data output** gets them in the pixel format it asks for (BGRA, or YCbCr 4:2:0), turned as its connection says, for Vision and the like; * a **metadata output** gets the codes in them, QR codes and barcodes, read with Vision; * a **photo output** takes a photo of the frame, a JPEG (or HEIC) marked to be turned upright as an iPhone’s is; * a **preview layer** shows them, and its conversions (a code’s place on the screen, a point of interest) go by them; * the camera answers what an app sets (focus, exposure, zoom, torch, HDR) as a phone’s does, without changing what it sees. The app asks for the camera as it would on a phone; `permissions | camera` (`CAMERA` on Android) grants it from the start. A scenario taps what an AR app’s camera shows by its name in the scene: ```gherkin Given the courier app's camera sees scenes/parcel-on-porch.glb And the courier app is opened with the "parcels-courier://deliveries/PX-IOS-9581" link And the "Measure parcel" button is tapped in the courier app When the "front left corner" in the courier app's camera view is tapped And the "front right corner" in the courier app's camera view is tapped Then the courier app shows "60 cm" ``` The scene is where the phone is and what it sees: * **The phone** is the scene’s camera, or the camera named `phone` when it has several: where it stands, held upright, looking along its −Z, its +Y the top of the phone (a glTF camera’s own axes). The phone’s camera is an iPhone’s wide camera (about 62° across, the picture filling the view), or the Android emulator’s (its sensor 1280 by 960, about 53° across held upright); the scene’s camera gives only where it is and which way it looks. * **Objects** a scenario names are the scene’s top-level nodes with a name, but for its cameras and its furniture: nodes named with a leading `_`, like `_road` or `_porch`. * **A tap on an object** lands on its middle, or on its `_tap` child, a node that marks where a tap on it lands: a corner guard’s `_tap` is the parcel’s corner it guards. The app’s AR finds the scene’s surfaces where they are, and a raycast that hits an object lands where a tap on it lands, so what an app measures between two objects is their distance in the scene, whatever the phone’s place. * **What is drawn** is the scene’s triangle meshes, each node’s transform (its matrix, or its translation, rotation and scale), and its materials’ base color, a color and an image (PNG or JPEG), lit by a soft sky and the sun. Points and lines are left out, as are skins and morph targets; an animation is a path the phone can travel (below), and the scene’s objects stay still. A tool like Blender exports a scene as glTF; a small program writes one too, as the parcels example’s [`make-scenes.go`](https://github.com/nimbusxr/axx/blob/main/examples/parcels/acceptance/scenes/make-scenes.go) does. The step reads the scene, and fails on one it cannot play: no camera, two cameras none named `phone`, two objects of one name. A running app starts again, to see it; the step before the app is launched gives the scene from the start. ### Moving the phone [Section titled “Moving the phone”](#moving-the-phone) The phone moves through the scene as the person holding it does, and the app’s camera, and its AR, see the scene from wherever it is: ```gherkin Given the courier app's camera sees scenes/parcel-on-porch.glb And the courier app's phone moves to the scene's "by the van" camera And the courier app is opened with the "parcels-courier://deliveries/PX-IOS-9582" link And the "Measure parcel" button is tapped in the courier app When the courier app's phone travels the scene's "up the garden path" path And the courier app's phone moves 1 m forward And the courier app's phone turns 40 degrees down And the "front left corner" in the courier app's camera view is tapped ``` * **A place the scene names** is one of its cameras: `moves to the scene's "by the van" camera` puts the phone where that camera is, looking the way it looks, at once. * **A path** is one of the scene’s animations that moves a camera, the phone’s or another one it follows: `travels` carries the phone along it as it was recorded, and waits for it to arrive; `starts traveling` lets the scenario go on while it moves, and `the courier app's phone reaches the end of its route` waits for it. `at 2x` goes twice as fast; `at 1.4 m/s` keeps one speed along the way. * **A step** moves the phone at a walking pace, still facing the same way: `forward` and `back` along the ground the way it looks (where its top points, when it looks straight down), `left` and `right` across it, `up` and `down`. * **A turn** turns it where it stands, as the person holding it turns: `left` and `right` about the upright, keeping its tilt; `up` and `down` tilt it. The phone stays where a step leaves it, the app started again or not: a step before the app is launched puts the phone where the app starts. On a simulator, axx loads a library of its own into the app when it starts, built once with Xcode into axx’s cache: it plays the camera for the app’s capture sessions; from a scene, it also makes world tracking supported, runs the session’s frames from the scene, draws the camera’s picture behind the `ARView`, and answers RealityKit’s raycasts. The app is the app you ship, unchanged; the library plays the camera and ARKit, which a simulator has not got. It plays apps that show AR with RealityKit’s `ARView`; an `ARSCNView` (SceneKit) app is not played yet, nor the system’s own camera screen (`UIImagePickerController` with the `.camera` source), which an app that checks `isSourceTypeAvailable(.camera)` does not offer on a simulator. It leans on ARKit’s internals, so each Xcode is checked against it: it is checked with Xcode 27.0, on the iOS 27.0 and 18.1 simulators. A real iPhone’s camera, through an Appium server, sees what is in front of it: the step is for simulators. On an Android emulator, its own back camera shows what the camera sees, to any app (CameraX, Camera2, the system’s camera app): axx starts its emulators with the camera that renders the AVD’s environment (`-camera-back environment`, from the AVD’s `environment.ini`, which axx writes before each one starts), and sets it for each scenario through the emulator’s control service: a picture, a video, or the scene, seen from where axx puts the phone in it. The emulator shows a picture off its middle, so axx lays out a picture, and a video frame by frame, so that the camera sees all of it, aspect-filled into its 4:3 sensor held upright; it re-encodes an H.264 video with Cisco’s OpenH264, which it downloads the first time, as it does for desktop videos (OpenH264 Video Codec provided by Cisco Systems, Inc.). An emulator started from a quick-boot snapshot keeps the camera the snapshot was saved with: axx starts it afresh, cold, the first time a scenario’s camera sees something, unless the snapshot was saved with `hw.camera.back=environment` in the AVD’s `config.ini`, as the parcels example’s [`create-emulator.sh`](https://github.com/nimbusxr/axx/blob/main/examples/parcels/courier/android/create-emulator.sh) saves it. An Android app’s AR is ARCore’s. In a debuggable app that has ARCore, axx attaches an agent of its own as the app starts, which answers the app’s ARCore calls from the scene: a session that runs, its camera at the phone’s pose, the emulator’s camera’s picture in the texture the app gave ARCore, filling the view as ARCore fills it, and hit tests that land on the scene’s surfaces, and on a named object where a tap on it lands. It answers the session, its frames and camera, hit results, planes and points, anchors and light estimates; depth, cloud anchors, geospatial and recording are not played. axx builds the agent once into its cache, with the SDK’s NDK and build-tools and a JDK, as the models-android pack builds its own, and attaches both when a scenario needs both. An app that is not debuggable runs its ARCore as it is. A real phone’s camera, through an Appium server, sees what is in front of it: the step is for emulators. ## Traces and videos [Section titled “Traces and videos”](#traces-and-videos) Ask for a **trace** of each app, and a scenario keeps one in `.axx/mobile/traces`: a page with its screen after each step, and its controls where it failed. A trace waits, after every step, for the screen’s next frame (about a tenth of a second): kept for failed scenarios only, it still slows every step a little, so it is off until you ask. A **video** is the scenario’s phones as they ran, each tap and swipe drawn where the finger was, with the scenario’s steps beside them: each one to run, running, passed, failed (its error under it) or skipped, and the lines the running step logs, so what happens off the screen shows too (a call to a service, a row in a database). It opens on its first frame for a moment, the steps from the first (those that ran before the app had a screen, such as its registration, among them), and its last frame shows for two seconds, the steps as they ended. It is an MP4 file (H.264) in `.axx/mobile/videos` that plays wherever videos play, with a chapter for each step; a failed scenario’s is attached to the report too. The run keeps one more, `run.mp4`: its kept scenarios one after another, each after a card with its name and whether it passed, and a chapter for each. Every phone has the same place in a video, so an iOS scenario and an Android one are in the same `run.mp4`; a scenario that drives two apps shows both phones, side by side. QuickTime Player, VLC, IINA and mpv list the chapters; a browser plays the video without them. The screens come from the drivers’ streams of them: WebDriverAgent’s on iOS, the UiAutomator2 server’s on Android, ten frames a second at half the screen’s size. A farm’s device, driven through `appium`, streams none: its scenarios keep neither. axx.yaml ```yaml packs: mobile-core: traces: failed # always, failed or never (the default) videos: failed # always, failed or never (the default) ``` Videos are encoded with Cisco’s OpenH264. OpenH264 Video Codec provided by Cisco Systems, Inc. The first time a scenario keeps a video, axx downloads Cisco’s build for your OS from Cisco (about half a megabyte), checks it, and keeps it in its cache; `videos: never`, the default, turns it off, and axx then downloads nothing. Cisco’s terms are in axx’s `THIRD_PARTY_NOTICES.md`. ## In CI [Section titled “In CI”](#in-ci) **Android:** Linux runners start emulators with KVM, and draw them without a GPU. The parcels example’s workflow does it this way, with the Android SDK the runner has: ```yaml - name: KVM, for the emulator run: | echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules sudo udevadm control --reload-rules sudo udevadm trigger --name-match=kvm - run: ./create-emulator.sh # sdkmanager and avdmanager: the device the features name - run: axx run --tags @android ``` `create-emulator.sh` saves the device’s boot snapshot with the camera axx plays (`hw.camera.back=environment` in its `config.ini`), so that the emulators axx starts from it are ready for what a scenario’s camera sees. It gives the device a front camera too (`hw.camera.front=emulated`): the system image says its phone has one, and on an AVD without, CameraX looks for it for some 6 seconds before an app’s camera shows. An emulator draws its camera without a GPU too, slower than a Mac’s: it takes seconds to show its first frame, which a scenario that reads what the camera sees gives time (`within 30s`). `packs.mobile-android.emulatorArgs` passes more arguments to the emulators, and `bootTimeout` gives a slow machine longer than 3 minutes to boot one. A feature too long for one runner, whose emulator runs a scenario at a time, [shares a run](/next/guides/run-in-ci/#share-a-run-between-runners) with others: `axx run --shard 1/2` on one, `--shard 2/2` on another. **iOS:** macOS runners have Xcode and its simulators. A new machine’s first simulator boot builds the iOS runtime’s shared cache, which takes over ten minutes on GitHub’s runners, and every hosted run is a new machine: keep that cache between runs, with what axx downloads, keyed by the macOS and runtime builds it belongs to. Never keep a simulator: its keychain belongs to the machine it was made on. ```yaml runs-on: macos-latest steps: - uses: actions/checkout@v7 - id: ios run: | runtime=$(xcrun simctl list runtimes -j | jq -r '[.runtimes[] | select(.platform == "iOS" and .isAvailable)] | sort_by(.version | split(".") | map(tonumber)) | last | .buildversion') echo "key=ios-$(sw_vers -buildVersion)-$runtime" >> "$GITHUB_OUTPUT" - uses: actions/cache@v6 with: path: | ~/Library/Developer/CoreSimulator/Caches/dyld ~/Library/Caches/axx/mobile !~/Library/Caches/axx/mobile/simulators key: ${{ steps.ios.outputs.key }} - run: xcodebuild -project Courier.xcodeproj -target Courier -configuration Debug -sdk iphonesimulator ONLY_ACTIVE_ARCH=NO SYMROOT=build - run: axx run --tags @ios ``` With the cache, a run sets its simulators up in a minute or two; without it, the first scenario’s steps need over ten minutes (`run.timeouts: {step: 20m}` in `axx.yaml` gives them that; `packs.mobile-ios.bootTimeout`, 20 minutes by default, bounds a simulator’s boot). A simulator on a hosted runner stays slow for a few minutes after it boots: give checks there `within` to spare. GitHub’s macOS runners have no Docker: when the services your scenarios talk to run in containers, run them elsewhere, or run the iOS scenarios on a Mac that has them. See the [app-core](/next/references/packs/app-core/), [mobile-core](/next/references/packs/mobile-core/), [mobile-android](/next/references/packs/mobile-android/) and [mobile-ios](/next/references/packs/mobile-ios/) references for every step. # Seed and check Redis > Seed keys into Redis or Valkey, Dragonfly, KeyDB and Garnet, and check the keys your services write there, by their value, their JSON properties or hash fields, or that they are gone. Services keep what they are asked for often in Redis: a cached view, an idempotency key, a session. The `redis` pack seeds keys before a scenario acts, and checks what your service wrote, or that it dropped a key it had to drop. It works with every server that speaks Redis’s protocol: Redis itself, Valkey, Dragonfly, KeyDB and Garnet. ```gherkin Scenario: A scan drops the parcel's cached tracking view Given a seeds/cache-dropped.yaml db seed And a redis/tracking-PX-RDS-9603.yaml redis seed When a message is published to the depots/LEJ/scans mqtt topic: """ {"scanId": "SC-9603-1", "parcelRef": "PX-RDS-9603", "status": "OUT_FOR_DELIVERY"} """ Then within 20s the tracking:PX-RDS-9603 redis key does not exist ``` Add the pack to the project with `axx pack add redis` ([Choose packs](/next/guides/use-packs/)). ## Register the server [Section titled “Register the server”](#register-the-server) Register the server once, usually in the `Background`: ```gherkin Background: Given the cache redis server with the following properties: | url | redis://${sys:local.host}:6379/0 | ``` The `url` is `redis://user:password@host:6379/0`, or `rediss://` for TLS. A `database` row picks another database than the URL’s. A password, in the URL or from `${env:..}`, is masked in logs and failures. ## Seed keys [Section titled “Seed keys”](#seed-keys) A seed file maps each key to what it holds, and optionally how long it lasts: redis/tracking-PX-RDS-9602.yaml ```yaml "tracking:PX-RDS-9602": value: {"parcelRef": "PX-RDS-9602", "status": "OUT_FOR_DELIVERY", "lastLocation": "Leipzig"} ttl: 10m "shop:hawthorn-home": hash: {name: Hawthorn Home, tier: gold} "depot:LEJ:sorting": list: [PX-RDS-9601, PX-RDS-9602] "parcels:express": set: [PX-RDS-9611, PX-RDS-9612] "couriers:deliveries": sorted set: {CR-LEJ-12: 42, CR-LEJ-7: 17} ``` ```gherkin Given a redis/tracking-PX-RDS-9602.yaml redis seed ``` * **What a key holds:** `value` is text, or JSON for an object or an array. The other kinds are `hash`, `list`, `set` and `sorted set`, whose members map to their scores. * **Replacing:** a seed replaces any key of the same name, so a scenario starts from what its seed says. * **`ttl`:** a duration like `10m`, after which the key expires. ## Check keys [Section titled “Check keys”](#check-keys) ```gherkin Then the idempotency:rowanberry-crafts:order-8812 redis key has the value 'PX-RDS-9604' And the tracking:PX-RDS-9601 redis key has the following properties: | parcelRef | PX-RDS-9601 | | status | REGISTERED | And within 20s the tracking:PX-RDS-9603 redis key does not exist ``` * `has the value` compares a string key’s text, all of it. * `has the following properties:` reads the key as JSON, whatever its type: * a string holding JSON is that JSON; * a hash is an object of its fields; * a list is an array; * a set is a sorted array; * a sorted set is an object of its members and their scores. Each row is a path into it and the value it has, as in the other JSON property steps: `null` for null and `undefined` for absent. * `does not exist` checks that a key is gone, for a service that deletes it or lets it expire. Every check waits for the key to be as it says: 10 seconds, or the time `within` gives. ## Keep scenarios apart [Section titled “Keep scenarios apart”](#keep-scenarios-apart) Scenarios run in parallel against the same server, so each seeds and checks keys of its own, named after data unique to it, such as a parcel reference. An `axx lint` rule can keep seeded key names unique across files: axx.yaml ```yaml lint: rules: - name: Redis keys in seeds filePatterns: ["redis/*.yaml"] regex: '^"([^"]+)":' validation: cross-file-unique ``` See the [redis pack’s reference](/next/references/packs/redis/) for every step. # Test web apps > Use your web app in real browsers the way people do (Chromium, Firefox, WebKit, Chrome or Edge, on a desktop or a phone), driven by Playwright on your machine, and check in the same scenario what your service did. Many acceptance criteria describe what a person does on a page: a shop fills in a form, clicks a button and sees its parcel registered. The `web-core` pack does exactly that in real browsers, driven by [Playwright](https://playwright.dev). A scenario can use every other pack too, so it also checks what your service did with the form: the row it stored, the event it published. ```gherkin Scenario: A parcel registered in the portal is stored and announced When the "/parcels/new" page is opened And the "Shop account" field is filled with "fjord-outdoor" And the "Parcel reference" field is filled with "PX-WEB-5001" And the "Weight (grams)" field is filled with "1200" And the "Recipient name" field is filled with "Anna Weber" And the "Postcode" field is filled with "10115" And "Germany" is chosen in the "Destination country" field And the "Leave with a neighbour" checkbox is unchecked And the "Register the parcel" button is clicked Then the "/parcels/PX-WEB-5001" page is shown And the page shows "Parcel PX-WEB-5001 registered" And a selection of rows is retrieved from the parcels.parcels table where: | reference | PX-WEB-5001 | And the selection has 1 row And the parcel-events kafka event named registered key is PX-WEB-5001 ``` Add the pack to the project with `axx pack add web-core` ([Choose packs](/next/guides/use-packs/)). This guide sets it up; the others go further: * [Work with pages](/next/guides/web-pages/): forms, clicks and gestures, keys, dialogs, browser tabs, frames, downloads, the browser’s clock, its connection and devices. * [Check what pages show](/next/guides/web-checks/): text, elements and their state, tables, counts, files, script errors and requests. * [Sign in to web apps](/next/guides/web-sign-in/): the sign-in flow, or a session to start with, and secrets. * [Compare screenshots](/next/guides/web-screenshots/): pages that look as designed, each platform with its own screenshots (the `web-screenshots` pack). * [Check accessibility](/next/guides/web-accessibility/): WCAG audits with axe-core, and what a screen reader reads (the `web-a11y` pack). * [Control what pages fetch](/next/guides/web-network/): requests that fail, answer with a file or take their time, a slow connection, recordings, and websocket messages (the `web-network` pack). * [Audit pages with Lighthouse](/next/guides/web-lighthouse/): performance, accessibility, best practices and SEO scores, and the Core Web Vitals (the `web-lighthouse` pack). * [Measure JavaScript coverage](/next/guides/web-coverage/): how much of the web apps’ JavaScript the scenarios run, as lcov and Istanbul reports (the `web-coverage` pack). * [Watch and debug browsers](/next/guides/watch-web-browsers/): see the browsers as they go, pause a scenario in Playwright’s Inspector, pick elements and record steps, traces and videos. * [How the web-core pack works](/next/explanations/web-browsers/): the browsers, isolation, waiting, names and selectors. ## The browsers [Section titled “The browsers”](#the-browsers) The browsers run on the machine that runs Axx, as in any Playwright project: Chromium, Firefox and WebKit, the builds of Playwright 1.62.1, the version the pack drives. The pack downloads each one the first time a scenario uses it, and keeps it where Playwright keeps its browsers, for the next runs and for other Playwright projects of that version: `~/Library/Caches/ms-playwright` on macOS, `~/.cache/ms-playwright` on Linux, `%LOCALAPPDATA%\ms-playwright` on Windows (`PLAYWRIGHT_BROWSERS_PATH` moves it). The engines `chrome` and `msedge` are the Google Chrome and Microsoft Edge installed on the machine. Chromium is the whole browser, with a window or without one, so that a run behaves the same whether you watch it or not. The browsers use your web apps at the addresses you would: `localhost` and the port their service listens on, whether it runs in Compose, publishing its port, or on your machine, from your IDE or with `axx run --debug` ([Manage the services under test](/next/guides/manage-services/)). ### In CI [Section titled “In CI”](#in-ci) On Linux, the browsers need system libraries that a bare machine or a slim image lacks. Install them before the run, with Playwright’s command for the engines you use: .github/workflows/acceptance.yml ```yaml - run: sudo npx -y playwright@1.62.1 install-deps chromium ``` Or run the job in Playwright’s image, `mcr.microsoft.com/playwright:v1.62.1-noble`, which has them. Keep Playwright’s folder of browsers in the CI cache to save the download. When a library is missing, the step that opens the first page says which command installs them. ## Register the web app [Section titled “Register the web app”](#register-the-web-app) Register the app with its address: ```gherkin Background: Given the parcels web app with the following properties: | url | http://localhost:8400/portal | ``` | Property | | | -------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | `url` | where the browser finds the app: page paths are below it (required) | | `engine` | `chromium` (the default), `firefox`, `webkit`, or `chrome` and `msedge`: the Google Chrome and Microsoft Edge installed on the machine | | `device` | a device Playwright knows, like `iPhone 15`, `Pixel 7` or `iPad Pro 11`: its screen size, user agent and touch screen | | `viewport` | the size of the page, like `1280x720` (the default) | | `locale` | the browser’s language, like `de-DE` (`en-US` by default, whatever the machine’s) | | `timezone` | the browser’s time zone, like `Europe/Berlin` (`UTC` by default, whatever the machine’s) | | `color scheme` | `light`, `dark` or `no-preference`: what `prefers-color-scheme` says | | `reduced motion` | `reduce` or `no-preference`: what `prefers-reduced-motion` says | | `media` | `screen` (the default) or `print`: the pages as they print, with their print styles | | `location` | where the browser says it is: a latitude and a longitude, like `52.5200, 13.4050` | | `permissions` | permissions the app has, like `notifications, clipboard-read` | | `user agent` | the user agent the browser sends | | `tls.verify` | `false` accepts the app’s certificate whatever it is, for test environments with their own | | `cookie.`, `header.`, `local storage.`, `session storage.`, `username`, `password` | a session to start with ([Sign in to web apps](/next/guides/web-sign-in/#start-with-a-session)) | To run the same features in another engine, make it a property: `| engine | ${sys:browser.engine:-chromium} |`, then `axx run -D browser.engine=webkit`. A phone is another web app of the same address: ```gherkin Scenario: Shops register parcels from their phones too Given the phone web app with the following properties: | url | http://localhost:8400/portal | | engine | webkit | | device | iPhone 15 | When the "/quote" page of the phone web app is opened And the "Menu" button is tapped And the "Register a parcel" link is tapped Then the "/parcels/new" page is shown ``` `the "/quote" page is opened` opens a page of the app, and the steps after it act on that page. With several web apps registered, `the "/quote" page of the parcels web app is opened` names the one. ## Find things the way people do [Section titled “Find things the way people do”](#find-things-the-way-people-do) Steps name what people see on the page: * **Fields** by their label: `the "Weight (grams)" field`. A field without a label is found by its placeholder. * **Drop-downs** are fields too; `"Germany" is chosen in the "Destination country" field` picks an option by its text. * **Radio buttons** by their label: `the "Express" option is chosen`. **Checkboxes** the same way: `the "Insure this parcel" checkbox is checked`. * **Buttons** by their name (their text, or their label), **links** by their text, **tabs** in a row of tabs on the page by their name, and **menu items** in an open menu. * **Elements**, anything else: by their text, their label, an image’s alternative text or a title: `the "Ella Brandt" element is double-clicked`. * **Text** anywhere on the page: `the page shows "6.90 EUR"`, part of what the page shows, where case matters. Names match exactly. When nothing on the page has the name, the step says what the page has instead: ```text No field named "Weight" on the page; 3 fields: "Weight (grams)" "Destination country" "Postcode" ``` A field that people can use but a step cannot find usually has no label. Give it one: the page becomes more accessible too. Steps look inside frames as well, such as an embedded widget or a payment form: what people see on the page, the steps find, whichever frame it is in. ### Selectors, when names are not enough [Section titled “Selectors, when names are not enough”](#selectors-when-names-are-not-enough) Where a name is not enough, or not there, a step takes a selector instead, in the same place: a CSS selector, an XPath, or a test id, the `data-testid` attribute of your markup. ```gherkin Then the "testid=parcel-count" element shows "25 parcels" And the page shows 20 "testid=parcel-row" elements And the "xpath=//tbody/tr[20]/td[1]" element shows "PX-WEB-5220" And the "css=.quote-result" element shows "6.90 EUR" ``` `css=`, `xpath=` and `testid=` start a selector; anything else is a name. `packs.web-core.testIdAttribute` in `axx.yaml` changes the attribute test ids are in. Prefer names where they work: they read like the page, they keep working when the markup changes, and a page people can find things on is one they can use. ## Waiting [Section titled “Waiting”](#waiting) Pages change after the fact: a price arrives from a pricing service, a message goes away. Every step waits for what it needs, so there is never a reason to sleep: * An action waits for its element to be there and ready: visible, enabled, not moving. * A check waits for the page to show what it expects: 10 seconds, or the time `within {duration}` gives. ```gherkin Scenario: A quote takes a moment When the "/quote" page is opened And the "Weight (grams)" field is filled with "1200" And "Germany" is chosen in the "Destination country" field And the "Postcode" field is filled with "10115" And the "Get a quote" button is clicked Then within 5s the page shows "Price: 6.90 EUR" And the page does not show "Calculating" ``` ## Scenarios run side by side [Section titled “Scenarios run side by side”](#scenarios-run-side-by-side) Every scenario has a browser context of its own, and one for each web app it registers: its own cookies, storage and pages. Scenarios run in parallel like any others, and a shop signed in in one scenario is never signed in in another. Like all parallel scenarios, give each one its own data, such as a parcel reference ([Isolate test data](/next/guides/isolate-test-data/)). The [`parcels` example](https://github.com/nimbusxr/axx/tree/main/examples/parcels) has a shop portal and its features: `shop-portal.feature`, `portal-parcels.feature`, `portal-planning.feature` and `portal-tracking.feature`. # Test WebSockets > Open WebSocket connections to your services, send messages on them, and check the messages they receive and the code they close with. A WebSocket keeps a connection open both ways: a page follows a parcel’s scans as they happen, a courier’s app gets its next stop. The `websocket` pack opens connections to your services, sends messages on them and checks what comes back, whatever made the service send it: a Kafka event, a row, a request. ```gherkin Scenario: The tracking page hears that its parcel is out for delivery Given a seeds/live-websocket.yaml db seed And the tracking websocket with the following properties: | url | ws://localhost:8400/portal/track/PX-LIV-5701/live | And a message is sent to the tracking websocket: """ {"follow": "PX-LIV-5701"} """ And a depot-scans kafka event And the depot-scans kafka event key is PX-LIV-5701 And the depot-scans kafka event payload is a kafka/scan-out-for-delivery.json resource When the depot-scans kafka event is published using schema schemas/depot-scan.avsc Then within 20s the tracking websocket received a message where: | reference | PX-LIV-5701 | | status | OUT_FOR_DELIVERY | ``` Add the pack to the project with `axx pack add websocket` ([Choose packs](/next/guides/use-packs/)). To check what a page in a browser sends and receives on its WebSockets, use the web packs instead ([Web network](/next/guides/web-network/)). ## Open a connection [Section titled “Open a connection”](#open-a-connection) Register a connection under a name. It opens there and then, and belongs to the scenario, which closes it when it ends: ```gherkin Given the tracking websocket with the following properties: | url | wss://parcels.example.com/portal/track/PX-LIV-5701/live | | header.Authorization | Bearer ${env:SHOP_TOKEN} | | subprotocol | parcels.v1 | ``` | Property | What it is | | --------------- | ------------------------------------------------------------------------------------------------------------------- | | `url` | The `ws://` or `wss://` URL (required). | | `header.` | A header of the opening request, such as a token. | | `subprotocol` | The subprotocol to ask for. A server that picks none is still accepted. | | `asyncapi` | The AsyncAPI document the messages both ways follow ([Validate against AsyncAPI](/next/guides/validate-asyncapi/)). | A connection the server refuses fails the step, with what the server answered. `${env:..}` values, such as a token, are masked in logs and failures. ## Send messages [Section titled “Send messages”](#send-messages) Send a text message from the doc string, or from a file of the project: ```gherkin When a message is sent to the tracking websocket: """ {"follow": "PX-LIV-5701"} """ And the tracking/follow.json message is sent to the tracking websocket ``` ## Check what it received [Section titled “Check what it received”](#check-what-it-received) A check looks at every message the connection received since it opened, and waits for one that meets it: 10 seconds, or the time `within` gives. ```gherkin Then within 20s the tracking websocket received a message where: | reference | PX-LIV-5701 | | status | OUT_FOR_DELIVERY | | location | Leipzig | And the tracking websocket received a message containing 'PX-LIV-5701' ``` * `received a message where:` reads each message as JSON: a path and the value as text, `null` for null and `undefined` for absent, like the message checks of queues and topics. * `received a message containing {string}` looks for text in any message. A failed check lists the messages the connection received. A binary message shows as its size. ## Check how it closed [Section titled “Check how it closed”](#check-how-it-closed) When a server closes a connection, a check still waiting for a message fails at once, and says why. Check the code the server closed it with: ```gherkin Scenario: The tracking websocket closes a connection that does not say which parcel it follows Given the tracking websocket with the following properties: | url | ws://localhost:8400/portal/track/PX-LIV-5702/live | When a message is sent to the tracking websocket: """ {"parcel": "PX-LIV-5702"} """ Then the tracking websocket was closed with code 1008 ``` The check waits for the connection to close, like the others. ## Keep scenarios apart [Section titled “Keep scenarios apart”](#keep-scenarios-apart) A connection belongs to its scenario: another scenario running in parallel never sees it or its messages, and it closes when its scenario ends. What a service sends on a connection often depends on shared data, so follow data unique to the scenario, such as a parcel of its own ([Isolate test data](/next/guides/isolate-test-data/)). See the [websocket pack’s reference](/next/references/packs/websocket/) for every step. # Choose packs > Pick the packs a project uses with axx-packs.yaml and axx pack add, remove, list and update. A **pack** is a set of steps. Axx publishes `rest`, `mock` (mocked dependencies, and mocked AI models: [Test AI features](/next/guides/test-ai-features/)), `grpc` ([Call gRPC services](/next/guides/test-grpc/)), `jsonrpc` ([Call JSON-RPC services](/next/guides/test-jsonrpc/)), `graphql` ([Call GraphQL services](/next/guides/test-graphql/)), `mcp` ([Test MCP servers](/next/guides/test-mcp/)), `a2a` ([Test A2A agents](/next/guides/test-a2a/)), `sql`, `mongo`, `redis` ([Seed and check Redis](/next/guides/test-redis/)), `kafka`, `amqp`, `mqtt`, `nats` ([Test message brokers](/next/guides/test-message-brokers/)), `websocket` ([Test WebSockets](/next/guides/test-websockets/)), `sse` ([Test event streams](/next/guides/test-event-streams/)), `asyncapi` (their messages checked against your AsyncAPI document: [Validate against AsyncAPI](/next/guides/validate-asyncapi/)), `logs`, `files` (the files your services write: [Check files](/next/guides/check-files/)), `mail` ([Check emails](/next/guides/check-emails/)), `cli` (commands: [Run commands](/next/guides/run-commands/)) and `web-core` (web apps in real browsers: [Test web apps](/next/guides/test-web-apps/)), with the packs that build on it: `web-screenshots`, `web-a11y`, `web-network`, `web-lighthouse` and `web-coverage`, `mobile-android` and `mobile-ios` (Android apps on emulators, iOS apps on simulators: [Test mobile apps](/next/guides/test-mobile-apps/)), which build on `mobile-core`, and `desktop-macos`, `desktop-windows` and `desktop-linux` (desktop apps, through the operating system’s accessibility tree), which build on `desktop-core`. It also publishes packs for cloud services, one per service, named after its cloud ([Test cloud services](/next/guides/test-cloud-services/)): * **AWS:** `aws-s3`, `aws-sqs`, `aws-sns`, `aws-eventbridge` and `aws-dynamodb`, which build on `aws-core`. * **Google Cloud:** `gcp-storage`, `gcp-pubsub`, `gcp-bigquery` and `gcp-firestore`, which build on `gcp-core`. * **Azure:** `azure-blob` and `azure-servicebus`. Each pack’s steps, settings and tools are on its page of the [pack reference](/next/references/packs/). A project can write packs of its own, or use packs other teams publish. Every pack, Axx’s or anyone else’s, is used the same way: the project lists it in `axx-packs.yaml`, next to `axx.yaml`. ## Choose the packs a project uses [Section titled “Choose the packs a project uses”](#choose-the-packs-a-project-uses) ```console $ axx pack add rest sql created axx-packs.yaml added rest, sql $ axx pack list rest axx 46 steps sql axx 17 steps mock axx not used mongo axx not used ... ``` axx-packs.yaml ```yaml packs: - rest - sql ``` Only the listed packs are loaded, so their steps are the only ones `axx validate`, `axx steps` and the skills know about. Listing a pack that builds on another (a web pack such as `web-a11y`, `mobile-android`, `desktop-macos`, or an AWS or Google Cloud pack) loads that one too: `web-core`, `mobile-core` or `desktop-core` (and `app-core`, which they build on), `aws-core` or `gcp-core`. `axx pack remove` takes packs off the list. `axx init` starts the file with `rest`, which its first feature uses. Commit `axx-packs.yaml`. The first `axx` command that needs the project’s steps prepares Axx with its packs: ```console $ axx validate axx: preparing rest, sql (once; cached for later runs) axx: ready in 31s 2 files, 6 scenarios, 41 steps: ok ``` This happens once for each list of packs. Later runs start immediately. The first time, Axx downloads what it needs, so it takes longer. In CI, keep Axx’s cache between runs ([Run in CI](/next/guides/run-in-ci/#keep-axxs-cache)). ## Add your own packs [Section titled “Add your own packs”](#add-your-own-packs) Your own packs are listed the same way, by path or by Go module: ```console $ axx pack new ./steps # create a pack in this repository $ axx pack add github.com/team/axx-grpc@v1.2.0 # use a pack published as a Go module ``` [Write custom steps](/next/guides/write-custom-steps/) shows what goes in a pack. When you change a pack in the repository, the next `axx` command prepares Axx with the new version. Versions of module packs are pinned in `axx-packs.lock` (commit it). `axx pack update` moves them to the latest version, or to the version written in `axx-packs.yaml`, and prepares Axx again. # Validate against AsyncAPI > Check every message a scenario sends and every message a check finds against your AsyncAPI document, over Kafka, AMQP, MQTT, NATS, cloud queues and topics, WebSockets and event streams, and relax individual rules for negative tests. Give a broker’s registration an `asyncapi` property and Axx checks messages against that document: * **Every message a scenario sends,** before it goes. This catches test data that drifted from the contract. * **Every message a check finds,** before the check passes. This catches a service that broke the contract. A message that breaks the document fails the step and names the rule it broke. Your AsyncAPI document stays the contract for your events, as your OpenAPI document is for your requests ([Validate against OpenAPI](/next/guides/validate-openapi/)). ```gherkin Scenario: A scanner is alerted when it scans a parcel the service does not know Given the depots mqtt broker with the following properties: | url | mqtt://localhost:1883 | | asyncapi | http://localhost:8400/asyncapi.yaml | When the mqtt/scan-unknown.json message is published to the depots/LEJ/scans mqtt topic Then within 10s the depots/+/alerts mqtt topic has a message where: | parcelRef | PX-MQT-8299 | | problem | unknown parcel | ``` Add the pack to the project with `axx pack add asyncapi` ([Choose packs](/next/guides/use-packs/)). An `asyncapi` row without it fails the registration and says so. ## Turn it on [Section titled “Turn it on”](#turn-it-on) `asyncapi` is a URL or a file path (resolved against `resources`), in YAML or JSON. AsyncAPI 3.x and 2.x (2.6 and earlier) are supported, with `$ref`s to other files and URLs, relative to the document that holds them. Point it at the document your service serves, or at the file in your repository. The row works on every registration that messages go through: | Registration | Its messages | | ---------------------------------- | ----------------------------- | | `the {word} kafka service` | the events of its topics | | `the {word} amqp broker` | its queues and exchanges | | `the {word} mqtt broker` | its topics | | `the {word} nats server` | its subjects and streams | | `the {word} aws account` | its SQS queues and SNS topics | | `the {word} gcp project` | its Pub/Sub topics | | `the {word} service bus namespace` | its queues and topics | | `the {word} websocket` | the messages both ways | | `the {word} event stream` | its events | ## How a message finds its place in the document [Section titled “How a message finds its place in the document”](#how-a-message-finds-its-place-in-the-document) A message is checked against a **channel**, then one of the channel’s **messages**: * **Channels, by address.** The channel whose address is where the message goes: a topic, a queue, a subject, or an exchange then its routing key. For a WebSocket or an event stream, it is the path of the URL; a server’s `pathname` may come before it. * A `{parameter}` stands for any text without a slash: `depots/{depot}/scans` is the channel of `depots/LEJ/scans`, and `tracking.{reference}` of `tracking.PX-NAT-8301`. * A check on a filter or a stream (`the depots/+/alerts mqtt topic`, `the TRACKING nats stream`) looks for the channel of the topic or subject each message came on. * When channels name their servers, only the servers of the registration’s protocol count, so one document can describe the same address on Kafka and on MQTT. * **Messages, by their schemas.** The message’s payload and headers must follow the schemas of one of the channel’s messages. An event stream’s event type picks the message of that name (`scan`, `delivered`). * When none fits, the failure names the message it comes closest to: the one that declares the most of its properties. * **Schemas.** * JSON Schema is draft 7 unless the schema says otherwise, and AsyncAPI’s own schema format is read as JSON Schema. * Avro payloads (`schemaFormat: application/vnd.apache.avro;version=1.9.0`) are in Avro’s JSON encoding, as the Kafka pack publishes them. * Payloads are JSON unless their content type says YAML or text. * A message’s `contentType` (or the document’s `defaultContentType`) is checked against the content type the message has, when it has one. ## Validation levels [Section titled “Validation levels”](#validation-levels) Each rule has a key and a level: | Key | Found when | | --------------------------------------------- | ------------------------------------------------------------------------------------------------- | | `validation.channel.unknown` | the document has no channel for the message | | `validation.message.contentType` | the message’s content type is not its message’s | | `validation.message.payload.format` | the payload is not JSON (or YAML) | | `validation.message.payload.schema.` | the payload breaks a JSON Schema keyword (`required`, `type`, `enum`…), or `avro` its Avro schema | | `validation.message.headers.schema.` | the headers break a keyword of their schema | `ERROR` fails the step, `WARN` and `INFO` log the finding, and `IGNORE` drops it. Everything is `ERROR` by default; `FAIL` is an alias of `ERROR`. A key also sets the keys below it, and the most specific key set wins: `validation.message.payload` relaxes `validation.message.payload.schema.required` too. A key must be one Axx reports, or a prefix of such keys: a misspelled key fails the step (or the run, in `axx.yaml`), and the message names the closest keys. ### Relax a rule in a scenario [Section titled “Relax a rule in a scenario”](#relax-a-rule-in-a-scenario) A scenario that sends a broken message on purpose, to check what the service does with it, relaxes only what it breaks: ```gherkin Scenario: A report without a parcel reference is set aside Given the AsyncAPI validation levels are: | validation.message.payload | IGNORE | When a message is sent to the parcels.label-printed amqp queue: """ {"printer": "LEJ-3", "job": "JOB-AMQ-8105"} """ Then within 10s the parcels.label-printed.rejected amqp queue has a message where: | job | JOB-AMQ-8105 | | header reason | not a label report | ``` `WARN` keeps a known gap visible in the logs without failing: the example’s older printers name themselves in a header, not in the body the contract describes. ```gherkin Scenario: An older printer reports straight to the service's queue Given a seeds/printing-queue.yaml db seed And the AsyncAPI validation levels are: | validation.message.payload.schema.required | WARN | When the amqp/label-printed-PX-AMQ-8103.json message is sent to the parcels.label-printed amqp queue with the following properties: | header printer | DRS-1 | ``` The levels apply to every contract the scenario checks messages against, until it ends. ### Defaults for the whole suite [Section titled “Defaults for the whole suite”](#defaults-for-the-whole-suite) `packs.asyncapi.levels` in `axx.yaml` sets the defaults the scenarios’ levels are merged over. Use it for a known gap in a document you do not control: axx.yaml ```yaml packs: asyncapi: levels: validation.message.contentType: WARN ``` ## Build messages from the document [Section titled “Build messages from the document”](#build-messages-from-the-document) A fixture factory builds messages from the payload of an AsyncAPI message, the way it builds request payloads from OpenAPI components ([Fixture factories](/next/guides/fixture-factories/)): mqtt/scans.factory.yaml ```yaml factory: family: json schema: ../../app/asyncapi.yaml#/components/messages/scan prototype: location: Leipzig identity: - path: scanId derive: authored fixtures: scan-unknown: scanId: SC-8299-1 parcelRef: PX-MQT-8299 status: IN_TRANSIT ``` The payload’s `$ref`s into the document’s components resolve, and so do those to other files. An Avro payload is built with the `avro` family, from its `.avsc` file. See the [asyncapi pack’s reference](/next/references/packs/asyncapi/) for the step and the keys. # Validate against OpenAPI > Check every request and response against your OpenAPI document, build payloads from its examples, and relax individual rules for negative tests. Give a REST service an `openapi` property and Axx validates every request it sends and every response it receives against that document. A violation fails the `When the request is executed` step and names the rule that broke. Your OpenAPI document stays the contract, and the acceptance suite proves the service honors it. ## Turn it on [Section titled “Turn it on”](#turn-it-on) ```gherkin Background: Given the parcels service with the following properties: | url | http://localhost:8400 | | openapi | http://localhost:8400/openapi.json | ``` `openapi` is a URL or a file path (resolved against `resources`), in JSON or YAML. OpenAPI 3.0 and 3.1 are supported. Point it at the document your service serves (`/openapi.json`, `/v3/api-docs`) or at the file in your repository. Do not point it at a third party’s live URL: if it is unreachable, your tests fail for reasons that have nothing to do with your service. Mock third parties instead: the WireMock image that mocks them checks their contracts, with settings of their own ([Mock dependencies](/next/guides/mock-dependencies/#check-the-dependencys-contract)). ## Build payloads from examples [Section titled “Build payloads from examples”](#build-payloads-from-examples) Payloads in Gherkin tables get long. Start from an example in the OpenAPI document and change only what the scenario is about: openapi.yaml (excerpt) ```yaml paths: /api/parcels: post: requestBody: content: application/json: examples: Standard parcel: value: reference: PX-EXAMPLE-1 sender: shop-example weightGrams: 1200 serviceLevel: STANDARD recipient: name: Ada Lovelace street: Invalidenstrasse 116 city: Berlin postcode: "10115" country: DE ``` ```gherkin Scenario: A shop registers a parcel Given a POST request to /api/parcels And a request payload using an application/json content example named 'Standard parcel' And the request payload properties are: | reference | PX-REG-1001 | | weightGrams | 2500 | When the request is executed Then the response status code is 201 ``` * `a request payload using an application/json content example` takes the operation’s first example; add `named ''` to pick one. * `a request payload using an application/json empty content template` starts from an empty body instead. * Examples can use `externalValue` to keep large payloads in their own files. ### Setting values in a payload [Section titled “Setting values in a payload”](#setting-values-in-a-payload) Keys are JSONPath expressions (`reference`, `recipient.postcode`, `items[0].sku`). A value keeps the type of the property it replaces, double quotes make it a string, `null` sets JSON null and `undefined` removes the property. The [rest pack reference](/next/references/packs/rest/#restrequestproperties) has the exact rules. Double-quote values that look like numbers but are strings in the schema, such as postcodes (`"50667"`) or long numeric identifiers, so they stay exact strings. ## Validation levels [Section titled “Validation levels”](#validation-levels) Each rule has a level: `ERROR` fails the step, while `WARN`, `INFO` and `IGNORE` let it pass. Everything is `ERROR` by default. `FAIL` is accepted as an alias of `ERROR`. For a negative test, where you send an invalid request on purpose to check that the service rejects it, relax only the rule you are breaking, only in that scenario: ```gherkin Scenario: Parcels over 30 kg are refused Given the OpenAPI validation levels are: | validation.request.body.schema.maximum | IGNORE | And a POST request to /api/parcels And a request payload using an application/json content example And the request payload properties are: | reference | PX-REG-1005 | | weightGrams | 31000 | When the request is executed Then the response status code is 400 And the response payload property detail is 'weightGrams must be at most 30000' ``` The document says `weightGrams` is at most 30000, so without the relaxed level the request step would fail on the contract violation, and the scenario could never check how the service answers. Only the `maximum` rule is relaxed; the rest of the body is still checked. With several REST services, name the one to relax: ```gherkin Given the OpenAPI validation levels on parcels are: | validation.request.body.schema.maximum | IGNORE | ``` Levels apply when the request is executed, so a scenario can relax a rule for one request and restore it for the next by setting `ERROR` again. Common keys: | Key | Checks | | ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `validation.request.body` | the request body against the schema | | `validation.request.body.schema.` | one schema keyword in the request body (`maximum`, `pattern`, `required`, …) | | `validation.request.parameter.missing` | required path parameters (query and header parameters have `validation.request.parameter.query.missing` and `validation.request.parameter.header.missing`) | | `validation.request.security.missing` | required security schemes | | `validation.response.body` | the response body against the schema | | `validation.response.body.schema.required` | required properties in the response | | `validation.response.body.schema.type` | property types in the response | The failure message of a violated rule includes its key, so you can copy it into the table. A key must be one Axx reports, or a prefix of such keys, like `validation.request.body`: a misspelled key fails the step (or the run, in `axx.yaml`), and the message names the closest keys. ### Defaults for the whole suite [Section titled “Defaults for the whole suite”](#defaults-for-the-whole-suite) `openapi.levels` in `axx.yaml` sets suite-wide defaults. Use it for a known gap in a document you do not control, not to silence your own contract: axx.yaml ```yaml openapi: levels: validation.request.security.missing: IGNORE ``` ## OpenAPI 3.1 and `nullable` [Section titled “OpenAPI 3.1 and nullable”](#openapi-31-and-nullable) OpenAPI 3.1 replaced `nullable: true` with JSON Schema type unions. In a document that declares `openapi: 3.1.0`, a schema that uses `nullable` does not compile, so every request or response validated against it fails with a `...schema.processingError`, whatever its values: ```yaml # openapi: 3.1.0 nickname: type: [string, "null"] # not: type: string + nullable: true ``` `nullable` is still correct in `openapi: 3.0.x` documents. ## The other side: your dependencies’ contracts [Section titled “The other side: your dependencies’ contracts”](#the-other-side-your-dependencies-contracts) This page covers your service’s own contract. The services your service calls have contracts too: run their WireMock mocks with the OpenAPI validation extension, and every call your service makes and every stubbed answer is checked against the dependency’s document. Those checks have levels of their own; `the OpenAPI validation levels are:` never relaxes them. See [Mock dependencies](/next/guides/mock-dependencies/#check-the-dependencys-contract). # Watch a run > See what scenarios do as they do it, web and mobile alike, with browsers and devices in windows on your screen, one scenario at a time, slowed down to follow. A run you watch shows what its scenarios do as they do it. Browsers open in windows on your screen, an iOS simulator in Device Hub, and an Android emulator in its own window. One switch covers every kind of app a scenario uses: ```sh axx run --watch # one scenario at a time, on screen axx run --watch --slowdown 500ms # with a pause after every action, to follow it ``` * `--watch` runs one scenario at a time, unless `--workers` (or `run.workers`) says otherwise. * `--slowdown` pauses after every action: a click, a tap, a typed field, a launch. * Watching needs a screen. On a machine without one, like a CI runner, run without it. The same settings work as `run.watch` and `run.slowdown` in `axx.yaml`. A profile keeps them for when you want them: axx.yaml ```yaml profiles: watch: run: watch: true slowdown: 300ms ``` `axx run --profile watch` runs with it, and with another profile after a comma, like `--profile ios,watch`, it watches that profile’s run ([Select by what scenarios use](/next/guides/tags-and-filtering/#select-by-what-scenarios-use)). `axx.local.yaml` can hold the settings for you alone. In your editor, *Watch* (next to *Run* and *Debug*) runs a scenario this way ([Set up your editor](/next/guides/set-up-your-editor/#watch-a-run)). ## What you see [Section titled “What you see”](#what-you-see) | App | While you watch | | ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Web ([`web-core`](/next/guides/test-web-apps/)) | Each browser in a window you can use as your own: click in it, and open DevTools on its page, console and network. `packs.web-core.watch` shows the browsers alone ([Watch and debug browsers](/next/guides/watch-web-browsers/)). | | iOS ([`mobile-ios`](/next/guides/test-mobile-apps/)) | The simulator in Device Hub (the Simulator app before Xcode 27), opened on it before the scenario starts. The simulator is in Xcode’s own set, named like `axx iPhone 17, iOS 27.0 (1)`, and stays booted after the run, so the next watched run starts at once ([keep](/next/guides/test-mobile-apps/#keep-simulators-warm)). | | Android ([`mobile-android`](/next/guides/test-mobile-apps/)) | The emulator in its window. | Each scenario still has its device to itself, and its app reset. Watching changes what you see, not where a scenario starts from. ## Stop where it matters [Section titled “Stop where it matters”](#stop-where-it-matters) Watching doesn’t pause. To stop a web scenario before a step, or where it fails, pause it in Playwright’s Inspector ([Watch and debug browsers](/next/guides/watch-web-browsers/#pause-a-scenario)). What a failed scenario keeps (screenshots, traces, videos, and the page or screen it was on) is in the [report](/next/guides/reports/). # Watch and debug browsers > Watch the browsers of a run as they go, pause a scenario in Playwright's Inspector before a step or where it failed, and read the trace, video and screenshot a failed scenario keeps. The browsers run on your machine, and you can see them: live while a run goes, and afterwards, in what a failed scenario keeps. ## Watch a run [Section titled “Watch a run”](#watch-a-run) `axx run --watch` shows what the scenarios do: their browsers open in windows on your screen as they go, one scenario at a time, and their devices too ([Watch a run](/next/guides/watch-runs/)). The windows are yours to use: click in them, and open DevTools to look at the page, its console and its network. ```sh axx run --watch --slowdown 300ms # wait after every action, to follow what happens ``` * `packs.web-core.watch: true` shows the browsers alone, and `packs.web-core.slowdown` sets their own pace. * Watching needs a screen: on a Linux machine without one, such as a CI runner, run without it. In the IntelliJ plugin and the VS Code extension, *Watch* runs scenarios this way, one at a time, with a slowdown of 300ms unless you set another ([Set up your editor](/next/guides/set-up-your-editor/#watch-a-run)). Watching only shows the browsers: to stop at a step, or where a scenario fails, pause it. ## Pause a scenario [Section titled “Pause a scenario”](#pause-a-scenario) `--pause-at` pauses a run before a step, the line of a feature file; in your editor, a breakpoint on the step’s line and *Debug* do: ```sh axx run features/shop-portal.feature --pause-at features/shop-portal.feature:24 ``` The browser shows, and Playwright’s Inspector opens next to it, with the feature file at the step: * **Resume**, or **step over** the browser’s actions one at a time. * **Pick** an element on the page: the Inspector names it as steps do, `the "Weight (grams)" field`, `the "Get a quote" button`. Type a name in its locator box to see what it finds on the page. * **Record**: use the page, and the Inspector writes the steps you take, ready to copy into the scenario: ```gherkin When the "Weight (grams)" field is filled with "1200" And "Germany" is chosen in the "Destination country" field And the "Get a quote" button is clicked Then the "testid=quote" element shows "6.90 EUR" ``` Where people see no name for an element, no step can find it: the recording says so, and a label, a name or a test id fixes it. * The browser is yours meanwhile: click in it, open DevTools. A run that pauses has no timeouts, however long you look. With `packs.web-core.pauseOnFailure: true` (`axx run --set packs.web-core.pauseOnFailure=true` for one run), a failed scenario pauses the same way, at the step that failed: look at the page as the failure left it. The run says which scenario waits for you. Your editor’s *Debug* does both: it pauses at your breakpoints and where a scenario fails. The Inspector is a Chromium window, whatever the engine: the pack downloads Playwright’s Chromium for it when a run in another engine pauses. ## What a failed scenario keeps [Section titled “What a failed scenario keeps”](#what-a-failed-scenario-keeps) * The failed step attaches a screenshot of the page; the [HTML report](/next/guides/reports/) shows it with the step. * The failure report says which page each web app was on, and the dialog it shows (the `context` of `axx run --json`), and lists the pages’ script errors. * A Playwright **trace** of each web app, in `.axx/web/traces/` and attached to the report: the scenario step by step, each step’s browser actions under it with a snapshot of the page before and after them, the console and the network. The trace keeps the feature file: its Source tab shows the step. Your editor opens it in Playwright’s trace viewer, where elements have the names steps give them; `npx playwright show-trace ` and [trace.playwright.dev](https://trace.playwright.dev) open it too, with Playwright’s names. * A **video** of each browser tab, when `videos` says so, in `.axx/web/videos/`, and in the report, where it plays. * The files the browser **downloaded** that its steps checked, in `.axx/web/downloads/`. Secrets stay out of all of them but the screenshot and the videos, which show the page as it is ([Keep secrets out of reports](/next/guides/web-sign-in/#keep-secrets-out-of-reports)). ## Settings [Section titled “Settings”](#settings) What scenarios keep, and how browsers run, is up to `packs.web-core` in `axx.yaml`: | Setting | | | ----------------- | ---------------------------------------------------------------------------------------------------------------------------- | | `traces` | which scenarios keep a trace: `failed` (the default), `always` or `never` | | `videos` | which scenarios keep a video of each browser tab: `never` (the default), `failed` or `always` | | `watch` | `true` shows the browsers in windows on your screen instead of running them unseen | | `slowdown` | a wait after every action, like `300ms` | | `pauseOnFailure` | pause a failed scenario where it failed, in Playwright’s Inspector | | `scriptErrors` | `report` (the default) lists the pages’ script errors with a failed scenario; `fail` fails any scenario whose pages had them | | `testIdAttribute` | the attribute `testid=` selectors look at (`data-testid` by default) | # Check accessibility > Audit pages against the WCAG rules with axe-core, and check what assistive technology reads on them, in the scenarios that use them. The `web-a11y` pack checks that pages can be used whatever the abilities of the people who use them: that a screen reader can name every field, that text stands out from its background, that images say what they show. It builds on the `web-core` pack, which comes with it: `axx pack add web-a11y` adds both. ## Audit a page [Section titled “Audit a page”](#audit-a-page) `the page has no accessibility violations` audits the page with [axe-core](https://github.com/dequelabs/axe-core), the accessibility engine of Deque that Lighthouse runs too, against the WCAG 2.1 AA rules: ```gherkin Scenario: Shops can get a quote whatever their abilities When the "/quote" page is opened Then the page has no accessibility violations ``` A violation fails the step, which says which rule, how serious it is, and which elements break it, with a link to the rule: ```text The page has 2 accessibility violations (wcag21aa): label (critical): Form elements must have labels; 1 element: #tracking https://dequeuniversity.com/rules/axe/4.13/label?application=axx color-contrast (serious): Elements must meet minimum color contrast ratio thresholds; 1 element: #note https://dequeuniversity.com/rules/axe/4.13/color-contrast?application=axx ``` The audit looks in every frame of the page, and the report has the violations axe-core found attached, in full. `the "css=form" element has no accessibility violations` audits one part of the page, by its name or a selector. Audit the page in the state the scenario is about: after the form showed its errors, with the menu open. The pack downloads axe-core 4.13.0 the first time, as the web-core pack downloads its browsers. ## Check what a screen reader reads [Section titled “Check what a screen reader reads”](#check-what-a-screen-reader-reads) `the page's accessible structure is:` checks what assistive technology reads on the page, in Playwright’s [ARIA snapshot](https://playwright.dev/docs/aria-snapshots) format: a role and a name a line, nested for what is inside: ```gherkin Scenario: The registration form says what each field is When the "/parcels/new" page is opened Then the "css=form" element's accessible structure is: """ - paragraph: - textbox "Shop account" - paragraph: - textbox "Parcel reference" - group "Service": - radio "Standard" [checked] - radio "Express" - group "Delivery": - paragraph: - checkbox "Leave with a neighbour" [checked] - button "Register the parcel" [disabled] """ ``` Only what is listed is checked, in its order, as it is nested on the page: here each field is in a paragraph, and the options and the checkbox in their groups. The rest may be anything. A name between slashes is a regular expression, `- paragraph: "/Price: \\d+\\.\\d+ EUR/"` (in quotes, because of the `: `inside). The check waits for the page to match, 10 seconds or `within {duration}`; when it doesn’t, the step says what the page reads instead. The Inspector of a [paused scenario](/next/guides/watch-web-browsers/#pause-a-scenario) records the structure of what you pick, ready to copy. ## Settings [Section titled “Settings”](#settings) | Setting | | | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | `standard` | the WCAG level pages are audited against: `wcag2a`, `wcag2aa`, `wcag21a`, `wcag21aa` (the default) or `wcag22aa`; a level includes those below it | | `ignore` | rules the audits leave out, by their axe-core id, like `color-contrast` | axe-core is [MPL-2.0](https://github.com/dequelabs/axe-core/blob/develop/LICENSE), by Deque Systems: Axx downloads it for you, and does not ship it. # Check what pages show > Check a web app's pages - the text they show, their elements and their state, tables and counts, the page and its title, downloaded files, script errors and the requests the page sent. Every check waits for what it expects: 10 seconds, or the time `within {duration}` gives ([Waiting](/next/guides/test-web-apps/#waiting)). When it does not come, the check says what the page shows instead. ## Text [Section titled “Text”](#text) `the page shows "Price: 6.90 EUR"` looks for the text anywhere on the page, in every frame, as part of what the page shows; case matters. `the page does not show "Calculating"` waits for it to go. When the text is not there, the failure shows the lines of the page most like it: ```text The page does not show "Price: 99.00 EUR" expected: "Price: 99.00 EUR" actual: "Price: 6.90 EUR · delivered in 2 days" ``` ## Elements [Section titled “Elements”](#elements) Elements are named as people see them, or by a selector ([Find things the way people do](/next/guides/test-web-apps/#find-things-the-way-people-do)): ```gherkin Scenario: A parcel on its way can no longer be cancelled Given a seeds/portal-track.yaml db seed And the "/parcels/PX-WEB-5112" page is opened When the "Actions" button is clicked Then the "Download the label" menu item is shown And the "Cancel the parcel" menu item is not shown And the "Actions" button has the aria-expanded attribute "true" And the "Track this parcel" link has the href attribute "/portal/track/PX-WEB-5112" ``` * `the "…" button is shown` and `… is not shown`, for any kind of element: a `button`, `field`, `checkbox`, `option`, `link`, `tab`, `menu item` or `element`. * `the "testid=quote" element shows "6.90 EUR"`: the text of one element. * `the "Actions" button has the aria-expanded attribute "true"`: one of its attributes. * `the "Weight (grams)" field has the focus`: where the keyboard types. * `the "Weight (grams)" field has the value "1200"`: what a field holds, or for a drop-down, the text of the option chosen. ## State [Section titled “State”](#state) * `the "Leave with a neighbour" checkbox is ticked` and `… is not ticked`. * `the "Standard" option is selected` and `… is not selected`, for radio buttons. * `the "Register the parcel" button is disabled` and `… is enabled`; the same for fields: `the "Shop account" field is disabled`. ## Counts and tables [Section titled “Counts and tables”](#counts-and-tables) ```gherkin Scenario: A shop sees its parcels Given a seeds/portal-busy.yaml db seed When the "/parcels?shop=harbour-hardware" page is opened Then the page shows 20 "testid=parcel-row" elements And the page shows a table row where: | Parcel | PX-WEB-5205 | | Status | IN_TRANSIT | And the page shows 3 table rows where: | Status | IN_TRANSIT | ``` * `the page shows 3 "css=.parcel-card" elements` counts what the page shows, by name or selector. * `the page shows a table row where:` finds a row by the table’s column headers, and `the page shows 3 table rows where:` counts them. A table’s headers are its ``, or a first row of header cells. ## The page [Section titled “The page”](#the-page) * `the "/parcels/PX-WEB-5001" page is shown` is where the browser is: the path below the app’s `url`, with the query only when the step gives one, or a whole URL. * `the page title is "Get a quote · Parcels"` is the title the browser tab shows. ## Downloaded files [Section titled “Downloaded files”](#downloaded-files) ```gherkin When the "Export as CSV" link is clicked Then the "parcels-birch-and-bloom.csv" file is downloaded And the downloaded "parcels-birch-and-bloom.csv" file contains "PX-WEB-5101,Lena Hoffmann,EXPRESS,IN_TRANSIT" And the downloaded "parcels-birch-and-bloom.csv" file has a row where: | Parcel | PX-WEB-5101 | | Status | IN_TRANSIT | ``` `the "…" file is downloaded` waits for the browser to download a file with that name, and `the downloaded "…" file is identical to the labels/PX-WEB-5111.zpl file` compares it with a file of the project. `the downloaded "…" file contains "…"` checks part of its text, as its type has it: the text of a PDF’s pages, the paragraphs and tables of a Word document (`.docx`), the cells of an Excel workbook (`.xlsx`), or the text itself. `the downloaded "…" file has a row where:` finds a row of a CSV or TSV file, or of an Excel workbook’s first sheet, by the columns its first row names. ## Script errors [Section titled “Script errors”](#script-errors) `the page has no script errors` checks that the web app’s pages had no errors in their scripts since the scenario opened them: none thrown and not caught, and none logged to the console. A failed scenario lists its pages’ script errors either way. With `packs.web-core.scriptErrors: fail` in `axx.yaml`, any script error fails its scenario, without the step. The browser logs to the console what it fails to load too, like the site’s icon: a browser asks for `/favicon.ico` unless the page names its icon (``), and a site without one has `Failed to load resource: the server responded with a status of 404 (Not Found)` in every page’s console. Give the app its icon. ## Requests the page sent [Section titled “Requests the page sent”](#requests-the-page-sent) `the browser sent a POST request to "/quote"` checks that the page sent a request with that method to that address: a path below the web app’s `url` (the query counts only when the step gives one), or a whole URL. What a service did with a request is best checked on the service (its database, its events, its mocks); this step is for what only the browser knows about, such as a call to a third party. # Measure JavaScript coverage > Find out how much of the web apps' JavaScript the scenarios run, as lcov and Istanbul reports for editors, CI and coverage services. The `web-coverage` pack collects the coverage of the scripts the web apps’ pages load while the scenarios run, and writes it when the run ends. It has no steps: add it (`axx pack add web-coverage`, which brings the `web-core` pack it builds on), and every scenario’s pages count. ```text the web apps' JavaScript coverage: lines 90.32%, statements 90.32%, functions 100%, branches 89.47%, in .axx/web/coverage/lcov.info ``` ## What counts [Section titled “What counts”](#what-counts) The scripts the pages load from their web app’s origin, named by their URL path, like `portal/js/tracking.js`. A script with a source map counts as its original files instead: the TypeScript, not the bundle. Scripts from other origins (a chat widget, analytics), inline scripts and event handlers in the HTML do not count. ## Name the project’s files [Section titled “Name the project’s files”](#name-the-projects-files) `sources` says where the scripts are in the project, by the start of their URL path, so that the reports name the files editors and coverage services know: axx.yaml ```yaml packs: web-coverage: sources: /portal/js/: ../app/web/js # the scripts the portal serves under /portal/js/ /portal/src/: ../app/web/src # the original files of their source maps ``` With `sources`, only the scripts under those paths count, named by the folder and the rest of the path, relative to the project: `../app/web/js/tracking.js`. When several paths match, the longest wins. A source map that lacks its original files’ content finds them there too. ## The reports [Section titled “The reports”](#the-reports) When the run ends, the pack writes to `.axx/web/coverage` (`folder`): | File | For | | ----------------------- | ------------------------------------------------------------------------------------------- | | `lcov.info` | editors’ coverage gutters, `genhtml`, and coverage services (Codecov, Coveralls, SonarQube) | | `coverage-final.json` | Istanbul’s own format: `npx nyc report --reporter=html` makes an HTML report of it | | `coverage-summary.json` | the totals, for CI thresholds and badges | The run prints the totals at the end. lcov.info, the start ```text SF:../app/web/js/tracking.js FN:3,$ FN:5,day FNDA:16,$ FNDA:6,day DA:11,6 DA:12,0 ``` A line that ran 0 times is one no scenario reached: in the parcels example, the tracking page’s message for an estimate it cannot fetch. ## How it counts [Section titled “How it counts”](#how-it-counts) The pack uses Chromium’s own coverage: the lines, functions and branches that ran, and how many times. It takes it after each step, and before a page leaves its document for another, and adds up every page of every scenario. The reports are those Node’s tools (c8) write from the same coverage, byte for byte. It works in Chromium, Chrome and Edge. A web app in Firefox or WebKit runs without coverage, and the run says so, once for each web app. Not counted: web workers, frames from other origins, and what runs while a tab a page opened is still loading. Pages run a little slower while the pack collects. ## Settings [Section titled “Settings”](#settings) | Setting | | | --------- | -------------------------------------------------------------------- | | `folder` | where the reports go, in the project (default `.axx/web/coverage`) | | `sources` | where the scripts are in the project, by the start of their URL path | # Audit pages with Lighthouse > Check a page's Lighthouse scores for performance, accessibility, best practices and SEO, and how fast it loads by the Core Web Vitals. The `web-lighthouse` pack audits a web app’s pages with [Lighthouse](https://developer.chrome.com/docs/lighthouse), Google’s tool for the quality of web pages, in the browser the scenario already uses. It builds on the `web-core` pack, which comes with it: `axx pack add web-lighthouse` adds both. ```gherkin Scenario: The quote page is quick and sound on a phone When the "/quote" page is opened Then the "/quote" page scores at least: | performance | 90 | | accessibility | 100 | | best practices | 100 | | seo | 100 | And the "/quote" page loads within: | largest contentful paint | 2.5s | | total blocking time | 200ms | | cumulative layout shift | 0.1 | ``` The page is a path below the web app’s `url`, or a whole URL. A page of the web app must be open first: the audit runs beside it. ## Scores [Section titled “Scores”](#scores) `the {string} page scores at least:` checks the page’s Lighthouse scores, from 0 to 100: `performance`, `accessibility`, `best practices` and `seo`, a row each, with the lowest score each may have. Lighthouse audits the categories listed only. A score below its minimum fails the step, which lists the audits that cost the page the most: ```text The "/parcels/new" page scores below what it should: performance: 72, at least 90 Total Blocking Time: 1,150 ms Cumulative Layout Shift: 0.135 Use efficient cache lifetimes: Est savings of 97 KiB Improve image delivery: Est savings of 80 KiB Layout shift culprits and 2 more in Lighthouse's report (accessibility: 100, best practices: 100, seo: 100) ``` ## Loading [Section titled “Loading”](#loading) `the {string} page loads within:` checks how fast the page loads, a metric a row: the Core Web Vitals `largest contentful paint` and `cumulative layout shift`, and `first contentful paint`, `total blocking time` and `speed index`. A limit is a duration (`2.5s`, `200ms`), except for the layout shift, which has no unit (`0.1`). A metric over its limit fails the step, with what would save the most on it. Google’s limits for a good page are 2.5 s for the largest contentful paint, 200 ms of blocking time and a shift of 0.1. ## How it audits [Section titled “How it audits”](#how-it-audits) * **Beside the scenario.** Lighthouse loads the page afresh in a tab of its own, with the web app’s cookies and storage: a signed-in page is audited signed in. The scenario’s steps go on in their tab. * **As a first visit.** Lighthouse clears the browser’s cache and the site’s service workers first, for the scenario’s tabs too. * **On a phone, or a desktop.** It measures the page as a mid-range phone on a slow 4G connection loads it (the `device` setting: `desktop` for Lighthouse’s desktop preset), simulating the device and the connection rather than slowing the browser down. An audit takes about 5 seconds, and audits take turns in a run, so that scenarios running beside one another do not slow each other’s pages. * **With a report.** Each audit attaches Lighthouse’s HTML report, which the pack also keeps in `.axx/web/lighthouse`. Scores and timings vary a little from run to run, and from machine to machine: leave room in the limits. The blocking time is time the browser spends on the CPU, which other scenarios’ browsers share: on a busy machine, such as a CI runner, run the scenarios that time pages alone, after the others, by tagging them and listing the tag in `run.exclusive` ([Run in parallel](/next/guides/parallel-runs/)). Lighthouse audits pages in Chromium, Chrome and Edge only; for a web app in Firefox or WebKit the step fails. ## Settings [Section titled “Settings”](#settings) | Setting | | | -------- | ------------------------------------------------------------------------------------------------------------ | | `device` | `mobile` (the default): a mid-range phone on a slow 4G connection; or `desktop`: Lighthouse’s desktop preset | Lighthouse is [Apache-2.0](https://github.com/GoogleChrome/lighthouse/blob/main/LICENSE), by Google: Axx downloads Lighthouse 13.5.0 the first time (about 18 MB, from npm’s registry, or the mirror `npm_config_registry` names) and runs it with the web-core pack’s Node.js. It does not ship it. # Control what pages fetch > Make a page's requests fail, answer with a status or a file, or take their time, slow the browser's connection, answer from a recording, and check websocket messages. Pages fetch from the browser: an estimate, a price, a map, live updates over a websocket. The `web-network` pack sees and changes those requests, to check what a page does when an answer is late, wrong or missing. It builds on the `web-core` pack, which comes with it: `axx pack add web-network` adds both. These are the pages’ own requests. The services your app calls from its server are mocked on the server’s side instead ([Mock dependencies](/next/guides/mock-dependencies/)). ## Name a request [Section titled “Name a request”](#name-a-request) A step names requests by their address: * a path below the web app’s `url`: with the `url` `http://localhost:8400/portal`, `"/pickups"` is `http://localhost:8400/portal/pickups`; * or a whole URL, for another site: `"https://maps.example.com/tiles/**"`. `*` stands for any part of a path (`"/track/*/estimate"`), and `**` for any parts. The query doesn’t count, unless the address has one: `"/pickups"` names `/pickups?day=Friday` too, and `"/pickups?day=Friday"` only that one. What a step sets holds for every web app of the scenario, from their next request on. Set it before the page opens, so that the page’s first requests get it too. When two steps name the same request, the later one answers it. A step sees the requests a page starts, not where a redirect takes them: when an answer redirects, the browser follows it within the same request, and a step that names the address it leads to does not apply. So a sign-in or a payment that your server redirects to a provider, like `https://checkout.stripe.com/...`, reaches the real provider, whatever a step says about that address. Keep such a flow local on the server’s side: point the app at a mock of the provider that hands out an address of your app ([Mock dependencies](/next/guides/mock-dependencies/)), and check that the browser gets there. ## Requests that fail or answer [Section titled “Requests that fail or answer”](#requests-that-fail-or-answer) ```gherkin Scenario: A shop is told when a pickup cannot be planned for want of a connection Given a seeds/portal-pickup-offline.yaml db seed And the page's requests to "/pickups" fail And the "/parcels?shop=fern-and-flax" page is opened When the "Pickup PX-WEB-5321" element is dragged onto the "Tomorrow" element Then the page shows "The pickup could not be planned: check your connection and try again" ``` * `the page's requests to "/pickups" fail`: the request fails as when the service is down or out of reach. The page’s `fetch` throws. * `the page's requests to "/pickups" answer with status 503`: the request gets that status and nothing else. * `the page's requests to "/track/*/estimate" answer with the network/estimate-tuesday.json file`: the request gets a file of the project, with the type of its extension. This is for an answer the page only shows, such as one that depends on the day. * `the page's requests to "/track/*/estimate" take 5s`: the request goes out that much later, as to a busy service. The page shows what it shows while it waits. ```gherkin Scenario: The tracking page says it is still working out when the parcel arrives Given a seeds/portal-tracking-waiting.yaml db seed And the page's requests to "/track/*/estimate" take 5s When the "/track/PX-WEB-5602" page is opened Then the page shows "Working out when your parcel arrives" ``` To cut the pages off from everything, use `the browser is offline` ([The connection](/next/guides/web-pages/#the-connection)). ## A slow connection [Section titled “A slow connection”](#a-slow-connection) `the browser's connection is slow` makes the browser’s connection that of a phone with a poor signal: 400 ms more for every request, and 500 kbit/s each way. It works in Chromium, Chrome and Edge only: with another engine, the step fails, or the one that opens the page. ```gherkin Scenario: The tracking page works on a phone with a poor signal Given a seeds/portal-tracking-slow.yaml db seed And the browser's connection is slow When the "/track/PX-WEB-5604" page is opened Then the page shows "Arrives on Monday, September 28, between 09:00 and 18:00" ``` The checks wait for the page as usual, 10 seconds or `within {duration}`. ## Answer from a recording [Section titled “Answer from a recording”](#answer-from-a-recording) With `record: true`, each scenario’s pages record what they fetch, in a HAR file for each web app in `.axx/web/recordings`: ```sh axx run --set packs.web-network.record=true features/portal-tracking.feature:52 ``` Copy a recording into the project and keep the requests you want answered. `the page's requests are answered from the network/tracking-delivered.har recording` then answers the requests the recording has, as they were recorded. The requests it lacks go out, and so do websockets. ```gherkin Scenario: A delivered parcel's tracking page says when it was delivered # Recorded from the service, with the parcel's delivery scan in the tracking store. Given a seeds/portal-tracking-delivered.yaml db seed And the page's requests are answered from the network/tracking-delivered.har recording When the "/track/PX-WEB-5605" page is opened Then the page shows "Delivered on Thursday, September 24, at 10:12" ``` A request matches a recorded one by its URL and method, and its body for a `POST`. So record with the web app’s `url` the scenarios use. ## Websocket messages [Section titled “Websocket messages”](#websocket-messages) The pack listens to the websockets of the scenario’s pages from the start: * `the page sent a websocket message containing "PX-WEB-5607"` checks what a page sent; * `the page received a websocket message containing "OUT_FOR_DELIVERY"` checks what it received. Both wait for the message, 10 seconds or `within {duration}`. When it doesn’t come, the step lists the messages that did. ```gherkin Scenario: The tracking page shows a depot scan as it happens Given a seeds/portal-tracking-live.yaml db seed And the "/track/PX-WEB-5606" page is opened And a depot-scans kafka event And the depot-scans kafka event key is PX-WEB-5606 And the depot-scans kafka event payload is a kafka/scan-out-for-delivery.json resource And the depot-scans kafka event payload properties are: | $.scanId | SC-5606-1 | | $.parcelRef | PX-WEB-5606 | When the depot-scans kafka event is published using schema schemas/depot-scan.avsc Then the page received a websocket message containing "OUT_FOR_DELIVERY" And the page shows "Out for delivery from Leipzig" ``` ## Settings [Section titled “Settings”](#settings) | Setting | | | -------- | --------------------------------------------------------------------------------------------------------------- | | `record` | record what each scenario’s pages fetch, a HAR file for each web app in `.axx/web/recordings` (default `false`) | In Chromium, a page that a step answers (from a file or a recording) is not one the browser got from the network. So the browser would treat it as a page of the internet, and keep it from reaching the machine’s own addresses, where the app’s websockets and services are. The pack lets such pages reach them. # Work with pages > Do on a web app's pages what people do - fill in forms, click, tap, drag and scroll, press keys, answer dialogs, follow new browser tabs, download files - and set the browser's clock, connection and device. The steps below act on the current page of the scenario’s web app ([Test web apps](/next/guides/test-web-apps/) sets it up). Each one finds its element by the name people see, or by a selector ([Find things the way people do](/next/guides/test-web-apps/#find-things-the-way-people-do)), waits for it to be ready, and does what a person does. ## Forms [Section titled “Forms”](#forms) ```gherkin Scenario: A shop sets its pickup days, and the logo on its labels When the "/settings?shop=willow-and-wax" page is opened And the "Pickup address" field is filled with "Weserstr. 40, 12045 Berlin" And the following options are chosen in the "Pickup days" field: | Monday | | Thursday | And the "Email me when a parcel is delivered" checkbox is checked And the logos/willow-and-wax.png file is uploaded in the "Upload your logo" field And the "Save the settings" button is clicked Then the page shows "Settings saved" ``` * `the "…" field is filled with "…"` replaces what a field holds. Date fields take dates as `2026-10-01`. * `"Germany" is chosen in the "Destination country" field` picks one option of a drop-down; `the following options are chosen in the "…" field:` picks several, one a row, in a drop-down that takes more than one. * `the "Express" option is chosen` picks a radio button; `the "…" checkbox is checked` and `… is unchecked` tick and clear a checkbox, and leave it as it is when it already is. * `the invoices/INV-2041.pdf file is uploaded in the "Customs invoice" field` picks a file of the project, as the file dialog would: for a file field with that label, or for a button with that name that opens the file dialog. ## Clicks, taps and the pointer [Section titled “Clicks, taps and the pointer”](#clicks-taps-and-the-pointer) * `the "Get a quote" button is clicked`, `the "Your parcels" link is clicked`, `the "History" tab is clicked` (a tab in a row of tabs on the page), `the "Cancel the parcel" menu item is clicked` (in an open menu), `the "PX-4101" element is clicked` (anything else). * `the "Ella Brandt" element is double-clicked` and `the "Nora Fischer" element is right-clicked`, which opens the page’s own menu when it has one. * `the "Menu" button is tapped`, on a touch screen: the web app’s `device` must have one, like `iPhone 15`. * `the pointer is moved over "About Express"` moves the pointer over a button, link, tab or menu item with that name, or else over that text, so that what opens on hover opens: a menu, a tip. * `the "Pickup PX-WEB-5302" element is dragged onto the "Friday" element` drags, as with a mouse. ```gherkin Scenario: A shop tracks a parcel from its row's menu Given a seeds/portal-row-menu.yaml db seed And the "/parcels?shop=thistle-and-thread" page is opened When the "Nora Fischer" element is right-clicked And the "Track this parcel" menu item is clicked Then the "/track/PX-WEB-5511" page is shown ``` ## Keys [Section titled “Keys”](#keys) Keys go to a field, or to whatever has the focus: `the Enter key is pressed in the "Find a parcel" field`, `the Escape key is pressed`. Key names are Playwright’s: `Enter`, `Escape`, `Tab`, `ArrowDown`, `Backspace`, a letter, and combinations like `Control+A`. ## Scrolling [Section titled “Scrolling”](#scrolling) Actions scroll to their element on their own. A page that shows more as it scrolls needs more: `the page is scrolled to the bottom` scrolls as far as the page goes, and `the "Pickups" element is scrolled into view` scrolls until it shows (in the page, or in the list it is in). `the page is scrolled to the top` scrolls back. ## Dialogs [Section titled “Dialogs”](#dialogs) A dialog a page opens (a message, a confirmation such as “Cancel parcel PX-WEB-5121?”, or a question) waits for an answer, as it does for people, and so does the page: until the dialog is answered, the other steps fail and say that a dialog is open. ```gherkin Scenario: The portal asks before it cancels a parcel Given a seeds/portal-ask.yaml db seed And the "/parcels/PX-WEB-5124" page is opened And the "Actions" button is clicked When the "Cancel the parcel" menu item is clicked Then the dialog shows "Cancel parcel PX-WEB-5124?" Scenario: A shop cancels a parcel before it is picked up Given a seeds/portal-cancel.yaml db seed And the "/parcels/PX-WEB-5121" page is opened And the "Actions" button is clicked And the "Cancel the parcel" menu item is clicked When the dialog is accepted Then the page shows "Parcel PX-WEB-5121 cancelled" ``` `the dialog is dismissed` answers Cancel, and `the dialog is answered with "The recipient moved"` types the answer to a question and accepts it. ## Browser tabs, going back, reloading [Section titled “Browser tabs, going back, reloading”](#browser-tabs-going-back-reloading) A link that opens a new browser tab, such as a tracking page for the recipient, makes it the current tab, as a browser shows it; the steps then act on it. `the browser tab is closed` closes it, and the tab it was opened from is the current tab again. ```gherkin Scenario: A shop shares a parcel's tracking page with its recipient Given a seeds/portal-share.yaml db seed And the "/parcels/PX-WEB-5117" page is opened When the "Track this parcel" link is clicked Then the "/track/PX-WEB-5117" page is shown And the page shows "Your parcel from maple-crafts is in transit." Scenario: The tracking page opens in a browser tab of its own Given a seeds/portal-share-closed.yaml db seed And the "/parcels/PX-WEB-5118" page is opened And the "Track this parcel" link is clicked When the browser tab is closed Then the "/parcels/PX-WEB-5118" page is shown ``` `the browser's back button is clicked` goes back a page, and `the page is reloaded` reloads it. ## Frames [Section titled “Frames”](#frames) Steps look inside frames: an embedded map, a pickup point chooser, a payment form from another address. Nothing to do: what people see on the page, the steps find, whichever frame it is in. ## Downloads [Section titled “Downloads”](#downloads) A file the page downloads, by a link or a button, is checked by the name the browser gives it: ```gherkin When the "Download the label" menu item is clicked Then the "PX-WEB-5111-label.zpl" file is downloaded And the downloaded "PX-WEB-5111-label.zpl" file is identical to the labels/PX-WEB-5111.zpl file ``` `the downloaded "parcels.csv" file contains "PX-WEB-5101,Lena Hoffmann"` checks part of it ([Check what pages show](/next/guides/web-checks/#downloaded-files)). A failed scenario keeps the downloads its steps checked in `.axx/web/downloads/`. WebKit opens some files in the page instead of downloading them, text files among them, unless their link has the `download` attribute, as a download link should. ## The browser’s clock [Section titled “The browser’s clock”](#the-browsers-clock) Pages that show the time, count down, or end a session after a while need a clock the scenario sets: ```gherkin Scenario: The portal ends a session after 30 minutes without activity Given the browser's clock is set to "2026-09-25T09:00:00+02:00" And the "/quote" page is opened When the browser's clock is moved forward by 31m Then the page shows "Your session ended after 30 minutes without activity" ``` `the browser's clock is set to "…"` sets the time the pages see, a time like `2026-09-25T14:00:00+02:00` (or `2026-09-25 14:00`, in UTC). Time goes on from there, so the pages’ timers run. `the browser's clock is moved forward by 31m` jumps ahead, as if the computer slept: the timers due in between fire once, and what expires expires. Set the clock before opening the pages whose timers it moves; it holds for every web app of the scenario. ## The connection [Section titled “The connection”](#the-connection) `the browser is offline` cuts the web apps’ pages off from everything, and `the browser is online` brings them back: for pages that say they are offline, keep a draft, or sync later. ```gherkin Scenario: The portal says when it cannot save anything Given the "/quote" page is opened When the browser is offline Then the page shows "You are offline: nothing is saved until you are back online" Scenario: The portal says when it can save again Given the "/quote" page is opened And the browser is offline When the browser is online Then the page does not show "You are offline" ``` ## Devices, languages and more [Section titled “Devices, languages and more”](#devices-languages-and-more) How the browser presents itself is up to the web app’s properties ([Register the web app](/next/guides/test-web-apps/#register-the-web-app)): a `device` (a phone, a tablet), a `viewport`, a `locale` and a `timezone`, a `color scheme` (dark mode), `reduced motion`, the `media` (a page as it prints), a `location`, `permissions` and a `user agent`. A web app that is the same address with other properties is another browser for the same app: a phone next to a desktop, a shop in Berlin next to one in New York. ```gherkin Scenario: A shop in Germany sees times in its own time zone Given the berlin web app with the following properties: | url | http://localhost:8400/portal | | locale | de-DE | | timezone | Europe/Berlin | And a seeds/portal-times.yaml db seed When the "/parcels/PX-WEB-5131" page of the berlin web app is opened Then the page shows "Registered: 25.09.2026, 14:03" ``` Pages learn the browser’s `location` on secure addresses only (`https://`, or `localhost`), as browsers allow. # Compare screenshots > Check that pages and elements look as designed, pixel by pixel as Playwright compares them, with screenshots kept per platform, each compared on its own. The `web-screenshots` pack checks that pages look as designed: the page, or one element of it, compared with a screenshot taken before, pixel by pixel, as Playwright does. Small differences of color and anti-aliased edges don’t count. It builds on the `web-core` pack, which comes with it: `axx pack add web-screenshots` adds both. ```gherkin Scenario: The quote page looks as designed Given the desktop web app with the following properties: | url | http://localhost:8400/portal | | viewport | 1280x800 | And the browser's clock is set to "2026-09-25T09:00:00Z" When the "/quote" page of the desktop web app is opened Then the page looks like the "quote" screenshot ``` A viewport keeps the size of the page the same wherever it runs, and the browser’s clock keeps the time it shows. ## Each platform compares its own [Section titled “Each platform compares its own”](#each-platform-compares-its-own) Browsers draw the same page differently on each operating system, fonts first, so a screenshot is compared only on the platform it was taken on. Each engine and platform has its own: `screenshots/quote.chromium-linux.png`, `quote.chromium-darwin.png`. Keep the screenshots of every platform you test on, your Mac and your CI’s Linux say, and each of them compares, locally as in CI. `platforms` names them: axx.yaml ```yaml packs: web-screenshots: platforms: [linux, darwin] ``` On a platform that is not in the list, the steps pass without comparing and say so; the rest of the scenario runs as usual. Keep only your CI’s platform (`platforms: [linux]`) to compare in CI alone. Without `platforms`, every platform compares, and one without its screenshots takes them. ## Taking and updating screenshots [Section titled “Taking and updating screenshots”](#taking-and-updating-screenshots) * The first time, there is no screenshot: the step takes it and fails. Look at it, commit it, and the next runs compare with it. * When the page looks different, the step attaches the screenshot, the page and their difference to the report (red: what differs; yellow: anti-aliasing), and keeps the page in `.axx/web/screenshots/`. When the change is right, copy that over the screenshot, or delete the screenshot and run again. * `packs.web-screenshots.update: true` takes every screenshot again, as the pages look now, for the platform it runs on: `axx run --set packs.web-screenshots.update=true` on your Mac, and on Linux in the CI job that compares them, keeping them as an artifact. Take a platform’s screenshots where they are compared: another Linux draws text with other fonts and smoothing, and Playwright’s image (`mcr.microsoft.com/playwright`) does not draw like GitHub’s Ubuntu runners. * The check waits for the page to settle, such as a price that arrives, before it fails. ## One element [Section titled “One element”](#one-element) `the "testid=quote" element looks like the "express quote" screenshot` compares one element, by its name or a selector: a card, a chart, a label, without the rest of the page around it. ## What changes from run to run [Section titled “What changes from run to run”](#what-changes-from-run-to-run) A page that shows the time, a generated number or today’s date looks different every run. Paint it over, in both the screenshot and the page, one element a row, by its text, its label or a selector: ```gherkin Scenario: A parcel's page looks as designed, whatever the time it was registered Given the desktop web app with the following properties: | url | http://localhost:8400/portal | | viewport | 1280x800 | And a seeds/portal-look.yaml db seed When the "/parcels/PX-WEB-5181" page of the desktop web app is opened Then the page looks like the "parcel" screenshot, apart from: | css=time | ``` What is painted over in either screenshot is left out of the comparison, so an element may change size from run to run, like a time whose digits differ. Better still, give pages fixed data: a seeded parcel, and the browser’s clock set to a time ([The browser’s clock](/next/guides/web-pages/#the-browsers-clock)). ## Settings [Section titled “Settings”](#settings) | Setting | | | ----------- | --------------------------------------------------------------------------------------------------------------------------- | | `platforms` | the platforms whose screenshots you keep, and compare on: `linux`, `darwin`, `windows` (every platform by default) | | `folder` | the folder of the screenshots, in the project; unset, each feature keeps its own in a `screenshots` folder next to its file | | `tolerance` | the share of a screenshot’s pixels that may differ, like `0.01` (one pixel in a hundred); keep it at 0 when you can | | `update` | take every screenshot again instead of comparing | # Sign in to web apps > Sign in the way people do, through every page of your identity provider, or start scenarios with a session - cookies, a token, storage or HTTP credentials - and keep passwords and tokens out of reports and traces. Every scenario has a browser context of its own, a fresh browser profile, so each one signs in: nothing a scenario signs in to is shared with another, which is what lets them run side by side ([How the web-core pack works](/next/explanations/web-browsers/#isolation)). There are two ways. ## Sign in as people do [Section titled “Sign in as people do”](#sign-in-as-people-do) Signing in is pages, fields and buttons like any others, however many pages it takes: the browser follows your app to its identity provider and back, as it does for people. A `Background` does it for all the scenarios of a feature: ```gherkin Background: Given the parcels web app with the following properties: | url | http://localhost:8400/portal | And the "/account" page is opened And the "Sign in" link is clicked And the "Log in with Email" button is clicked And the "Email Address" field is filled with "orders@fjord-outdoor.example" And the "Password" field is filled with "${env:FJORD_OUTDOOR_PASSWORD}" And the "Login" button is clicked And the "Grant Access" button is clicked And the page shows "Signed in as Fjord Outdoor" ``` This is the sign-in your shops go through, so it is checked in every scenario, and against a provider in your Compose file it takes about a second. ## Start with a session [Section titled “Start with a session”](#start-with-a-session) When a scenario is about what a signed-in shop does rather than about signing in, start its browser signed in, with what your app keeps its session in. Each is a property of the web app, and each goes to the app’s own address only: | Property | | | ----------------------- | ------------------------------------------------------------------------------------------------------------- | | `cookie.` | a cookie, such as a session cookie: `cookie.session` | | `header.` | a header on every request to the app, such as a token: `header.Authorization` with `Bearer ${env:SHOP_TOKEN}` | | `local storage.` | an item of the page’s local storage, where single-page apps often keep their tokens | | `session storage.` | an item of each tab’s session storage, where some sign-in libraries keep theirs | | `username`, `password` | HTTP authentication, for apps (or test environments) behind a login prompt | ```gherkin Scenario: A returning shop sees its parcels Given the shop web app with the following properties: | url | http://localhost:8400/portal | | local storage.token | ${env:SHOP_TOKEN} | When the "/parcels" page of the shop web app is opened Then the page shows "Signed in as Fjord Outdoor" ``` Where the token comes from is up to your test environment: a long-lived test token in the environment, one your identity provider issues to a test client, or a session your service creates for a seeded shop. Scenarios still share nothing: every scenario’s browser starts with its own copy. A step of your own can wrap either way in your own words, like `Given the shop is signed in as fjord-outdoor` ([Write custom steps](/next/guides/write-custom-steps/)). ## Keep secrets out of reports [Section titled “Keep secrets out of reports”](#keep-secrets-out-of-reports) Keep passwords and tokens in the environment, as `${env:…}`. Whatever a step takes from the environment is a secret: failure messages, the failure report and saved traces show it as `********`, down to the form data the browser sends and the headers of its requests. Screenshots and videos show the page as it is: a password field shows only dots, so type secrets into password fields. ## An identity provider next to your apps [Section titled “An identity provider next to your apps”](#an-identity-provider-next-to-your-apps) For your own tests, run [Dex](https://dexidp.io), a small OpenID Connect provider, in the Compose file with shops of your choosing. Your app signs in with it as it does with your real provider: compose.yaml ```yaml services: dex: image: ghcr.io/dexidp/dex:v2.45.1 command: ['dex', 'serve', '/etc/dex/config.yaml'] volumes: ['./dex.yaml:/etc/dex/config.yaml:ro'] ports: ['5556:5556'] parcels: build: ../app ports: ['8400:8400'] extra_hosts: ['dex.localhost:host-gateway'] # Dex, where the browsers find it ``` dex.yaml ```yaml issuer: http://dex.localhost:5556/dex storage: type: memory web: http: 0.0.0.0:5556 staticClients: - id: parcels-portal name: Parcels portal secret: portal-client-secret redirectURIs: ['http://localhost:8400/portal/callback'] enablePasswordDB: true staticPasswords: - email: orders@fjord-outdoor.example # htpasswd -nbBC 10 "" "$FJORD_OUTDOOR_PASSWORD" | tr -d ':\n' hash: '' username: Fjord Outdoor userID: fjord-outdoor connectors: - type: mockCallback id: mock name: Example ``` * The `issuer` is one address for both the browser and your app. The browsers, on your machine, take any `*.localhost` name for your machine, where Dex’s port is published; `extra_hosts` sends your app’s container there too. Give your app that issuer, the client’s `id` and `secret`, and its callback. An app that runs on your machine needs no `extra_hosts`. * Dex asks which way to sign in (`Log in with Email` or `Log in with Example`), then for the email address and password, then to grant access. `oauth2: {skipApprovalScreen: true}` leaves out the last page. * `Log in with Example` signs in a fixed test user, Kilgore Trout, without a password: enough for scenarios about what a signed-in shop sees rather than about signing in. * Dex keeps no session of its own: after signing out of your app, signing in asks for the password again. # Test webhooks and tokens > Sign the webhooks a scenario sends your service, check the signatures of the webhooks your service sends, and authorize requests with JSON Web Tokens and OAuth 2.0 client credentials tokens. Services call each other back: a courier calls your service as it delivers parcels, and your service calls each shop’s system in turn. Each side signs what it sends, so the other can trust it. And an API that shops’ systems call wants a token. The `rest` and `mock` packs cover both directions: * **Sign the requests a scenario sends,** as the other side would. * **Check the signatures of the webhooks your service sends,** on the WireMock that stands in for the receiver. * **Authorize requests with a token:** one axx signs, or one it gets from a token endpoint. ```gherkin Scenario: A shop hears of its parcel's delivery in a signed webhook Given a seeds/callback-delivered.yaml db seed And a POST request to /api/courier/callbacks And a request payload using an application/json content example named 'Delivered' And the request payload property reference is 'PX-WHK-9103' And the request is signed in the X-Courier-Signature header with the following properties: | key | ${env:COURIER_CALLBACK_KEY} | | value | sha256={signature} | When the request is executed Then the response status code is 204 And the mocked POST request to path /webhooks/linden-and-lace named delivered was received by shops And the mocked request named delivered on shops is signed as a standard webhook with the key '${env:SHOP_WEBHOOK_KEY}' ``` ## Sign what a scenario sends [Section titled “Sign what a scenario sends”](#sign-what-a-scenario-sends) A webhook is signed with an HMAC of what it sends, in a header. A table says how: ```gherkin Given the request is signed in the X-Courier-Signature header with the following properties: | key | ${env:COURIER_CALLBACK_KEY} | | value | sha256={signature} | ``` | Row | What it takes | Default | | ------------------ | -------------------------------------------------------------------------------------------------------------- | ------------- | | `key` | the secret the signature is made with (required) | | | `algorithm` | `hmac-sha256`, `hmac-sha1` or `hmac-sha512` | `hmac-sha256` | | `signs` | what is signed: a template of `{body}`, `{timestamp}` (Unix seconds), `{method}` and `{path}` (with the query) | `{body}` | | `encoding` | `hex` or `base64` | `hex` | | `value` | the header’s value: a template of `{signature}` and `{timestamp}` | `{signature}` | | `timestamp header` | a header that also carries `{timestamp}` | | The table covers the usual forms: | Service | `signs` | `value` | Other rows | | ------------------------------ | ----------------------- | ------------------------------ | ------------------------------------------------- | | GitHub (`X-Hub-Signature-256`) | `{body}` | `sha256={signature}` | | | Stripe (`Stripe-Signature`) | `{timestamp}.{body}` | `t={timestamp},v1={signature}` | | | Slack (`X-Slack-Signature`) | `v0:{timestamp}:{body}` | `v0={signature}` | `timestamp header` is `X-Slack-Request-Timestamp` | [Standard Webhooks](https://www.standardwebhooks.com) have a step of their own. It sets the `webhook-id`, `webhook-timestamp` and `webhook-signature` headers, with a key that is `whsec_` and the key in base64: ```gherkin Given the request is signed as a standard webhook with the key '${env:SHOP_WEBHOOK_KEY}' ``` The signature is made when the request is executed, over the body exactly as it is sent. Like the other request steps, both take `for 2nd ordered request` and `for request on `. ## Check what your service signed [Section titled “Check what your service signed”](#check-what-your-service-signed) The receiving end of your service’s webhooks is a WireMock ([Mock dependencies](/next/guides/mock-dependencies/)). Name the request first, then check its signature with the same table, or as a Standard Webhook: ```gherkin Then the mocked POST request to path /webhooks/linden-and-lace named delivered was received by shops And the payload properties for mocked request named delivered on shops are: | reference | PX-WHK-9103 | And the mocked request named delivered on shops is signed as a standard webhook with the key '${env:SHOP_WEBHOOK_KEY}' ``` Axx signs what WireMock received, byte for byte, with its method and its path, and compares the signatures. `{timestamp}` comes from the header’s value or from the timestamp header. The check fails unless the name matches a request, and every request it matches must be signed. So narrow the name to the scenario’s own data, as the payload properties above do, since other scenarios’ webhooks reach the same WireMock. ## Authorize requests with a token [Section titled “Authorize requests with a token”](#authorize-requests-with-a-token) Register a token under a name, then authorize requests with it: ```gherkin Given the shop token with the following properties: | key | ${env:SHOP_TOKEN_KEY} | | claim.shop | hawthorn-home | And a GET request to /api/shops/hawthorn-home/parcels And the request is authorized with the shop token ``` A token is one of two kinds: * **A JSON Web Token axx signs,** with rows: * `algorithm`: `HS256` (the default), `HS384`, `HS512`, `RS256` or `ES256`. * `key`: the shared secret, or for RS and ES, the PEM private key as a file of the project. * `key id`: the `kid` in the token’s header. * `claim.`: a claim, as a JSON value (`true`, `42`, `["parcels:read"]`) or text. * `expires in`: how long it lasts, `5m` unless it says otherwise. It is signed each time it is used, with `iat` and `exp` from then. * **An OAuth 2.0 client credentials token axx gets from a token endpoint,** with rows `token url`, `client id`, `client secret`, and optionally `scope` and `audience`. It is got once for the run, and again once it expires: ```gherkin Given the shop-system token with the following properties: | token url | http://localhost:8400/oauth/token | | client id | wisteria-way | | client secret | ${env:WISTERIA_CLIENT_SECRET} | ``` The request step sends `Authorization: Bearer `. In other packs’ property rows, `${token:}` stands for the token, as in a WebSocket’s header: ```gherkin Given the tracking websocket with the following properties: | url | ws://localhost:8400/portal/track/PX-LIV-5701/live | | header.Authorization | Bearer ${token:shop} | ``` It works in the rows of what a scenario registers for itself (the `websocket`, `sse`, `grpc`, `jsonrpc` and `graphql` packs), and in the `cli` pack’s command, arguments, input and environment. It doesn’t work in the rows of message brokers, whose connections are opened for the whole run before any scenario registers a token. ## Keep secrets secret [Section titled “Keep secrets secret”](#keep-secrets-secret) Signing keys, client secrets and tokens are masked in logs and failures, whether they come from `${env:..}` or are written in the table. Keep real keys out of feature files anyway: `${env:..}` reads them from the environment, which CI fills from its secrets. See the [rest](/next/references/packs/rest/) and [mock](/next/references/packs/mock/) packs’ references for every step. # Write custom steps > Add your own Gherkin steps as a Go pack that shares the context of Axx's packs. Custom steps sit next to the steps of Axx’s packs and are used the same way in feature files. They live in a **pack** of your own: Go code that builds on Axx’s core, like Axx’s packs, and works on the same context their steps use. ## Create a pack [Section titled “Create a pack”](#create-a-pack) Create a pack and add it to the project in one go: ```console $ axx pack new ./steps created the steps pack in ./steps and added it to axx-packs.yaml its example step: Then the steps pack is loaded ``` A pack is a Go package that exports `Pack()`. Its steps receive the scenario, and from it every pack’s context: the services registered in the scenario and their state. A custom step sees exactly what the steps of Axx’s packs set up, and their steps see what it adds. steps/pack.go ```go package steps import ( "github.com/nimbusxr/axx/core" "github.com/nimbusxr/axx/packs/sql" ) func Pack() core.Pack { return pack{} } type pack struct{} func (pack) Manifest() core.Manifest { return core.Manifest{ Name: "steps", Steps: []core.StepDef{{ ID: "steps.rows", Keyword: "Then", Expr: "the {word} table on {dbService} holds {int} row(s)", Doc: "Counts the rows of a table on a registered database.", Examples: []string{"Then the parcels.manifest_lines table on parcels-db holds 3 rows"}, Run: func(sc *core.Scenario, a core.Args) error { db := a.Value(1).(*sql.Service) // {dbService} resolves to the registered service sel, err := db.Query(sc.Context(), a.String(0), "SELECT * FROM "+a.String(0)) if err != nil { return err } if got := len(sel.Rows); got != a.Int(2) { return core.Fail("rows in "+a.String(0), a.Int(2), got) } return nil }, }}, } } ``` The contexts: | Pack | Context | Holds | | ------- | ------------------- | ------------------------------------------------------------------------------------------ | | `rest` | `rest.Context(sc)` | REST services, their requests in order, and each request’s response (`Exchange()`) | | `sql` | `sql.Context(sc)` | databases with their connection pool (`DB`), selections, triggers; `Query`, `AddSelection` | | `mongo` | `mongo.Context(sc)` | MongoDB databases (`DB()`) and their selections | | `mock` | `mock.Context(sc)` | mocked (WireMock) services | | `kafka` | `kafka.Context(sc)` | Kafka services, their topic clients (`Topic(name)`) and the events drafted for each topic | Every context has `Service(name...)` (no name: the default, the first registered), `Services()` and `AddService(...)`, and `sql.Connect`, `mongo.Connect` and `mock.NewService` create services the way the packs’ own steps do. Report failed expectations with `core.Fail(message, expected, actual)` so reports show both values, and keep state of your own in a `core.NewStateKey`: it is per scenario, like the packs’ contexts. A step parameter that names a file is a `{filepath}`, and a step whose `key | value` table has file properties names them in `TableTypes`: `map[string]string{"schema": "filepath"}` for a file the step reads, or `"url"` for a URL whose `file://` form names a file that may appear only during the run. Read the file with `sc.Suite().ResolvePath(a.String(0))`, which looks in the `resources` directories and next to `axx.yaml`: that is where editors look to link the value, complete its path and flag a missing file ([Set up your editor](/next/guides/set-up-your-editor/)). The first `axx` command that needs the project’s steps prepares Axx with the pack, like any other pack in `axx-packs.yaml` ([Choose packs](/next/guides/use-packs/)). ## Use the step [Section titled “Use the step”](#use-the-step) Once the pack is listed in `axx-packs.yaml`, its steps are used like any other step: `axx steps search` finds them, `axx validate` checks feature lines against them, and `axx skills install` adds them to the agent step index. `axx docs export --out ` writes the reference of the project’s packs, yours included: a page per pack, with its steps, their documentation and examples. ## Change a REST request [Section titled “Change a REST request”](#change-a-rest-request) A custom step can change a request the rest pack’s steps built, until it is executed: sign it, or give it a token the service expects. A request’s headers are the ones it is sent with: * `Header()` returns them, and `Set`, `Add` and `Del` change them, as with Go’s `http.Request.Header`. * `SetHeader(name, value)` does what the request header steps do: `Content-Type` and `Accept` replace an earlier value, and other headers get one more. * When the request is sent, it gets `Accept: */*`, its payload’s `Content-Type` and a `User-Agent` if it has none. `Exchange().RequestHeader` has the headers it was sent with, and `Exchange().Header` the response’s. This step signs a request’s payload for a service that checks an `X-Signature` header, like a carrier’s scan webhook: steps/pack.go ```go { ID: "steps.sign", Keyword: "Given", Expr: "the request is signed with the key {string}", Doc: "Signs the default REST service's request: an X-Signature header with the HMAC-SHA256 of its payload.", Examples: []string{"Given the request is signed with the key '${env:CARRIER_KEY}'"}, Run: func(sc *core.Scenario, a core.Args) error { svc, err := rest.Context(sc).Service() if err != nil { return err } req, err := svc.Request(0) if err != nil { return err } if req.Payload == nil { return errors.New("the request has no payload to sign") } mac := hmac.New(sha256.New, []byte(a.String(0))) mac.Write([]byte(*req.Payload)) req.Header().Set("X-Signature", "sha256="+hex.EncodeToString(mac.Sum(nil))) return nil }, }, ``` ```gherkin Scenario: A carrier's signed scan is accepted Given a POST request to /api/scans And a request payload using an application/json content example And the request is signed with the key '${env:CARRIER_KEY}' When the request is executed Then the response status code is 202 ``` The step signs the payload as the steps before it left it, so it comes after the payload steps. ## Give agents a tool [Section titled “Give agents a tool”](#give-agents-a-tool) A pack can also give coding agents tools of their own, in `axx mcp`: `Tools` in its manifest. A tool looks at the scenario an agent keeps open with `steps_try`, the one its steps ran in, and returns data (and images) for the agent. The web-core pack’s `web_page` shows the page the steps led to ([Set up agents](/next/guides/set-up-agents/#mcp-server)). ```go Tools: []core.Tool{{ Name: "steps_manifest_lines", Description: "The manifest lines the session's steps imported, as the importer stored them.", ReadOnly: true, Input: json.RawMessage(`{"type": "object", "properties": {}}`), Run: func(call *core.ToolCall) (*core.ToolResult, error) { db, err := sql.Context(call.Scenario).Service() if err != nil { return nil, err } sel, err := db.Query(call.Context, "manifest_lines", "SELECT * FROM parcels.manifest_lines") if err != nil { return nil, err } return &core.ToolResult{Data: map[string]any{"rows": sel.Rows}}, nil }, }}, ``` Name a tool after its pack, and describe what it gives and when to use it: agents choose tools by their descriptions. The input is checked against its JSON Schema before `Run`. ## Check what the pack needs [Section titled “Check what the pack needs”](#check-what-the-pack-needs) `axx doctor` runs a pack’s `Checks` in the projects that use it: that what its steps need is there, before a scenario finds out. The mobile packs check the Android SDK and Xcode this way. ```go Checks: []core.Check{{ Name: "carrier signing key", Run: func(ctx context.Context) core.CheckResult { if os.Getenv("CARRIER_KEY") == "" { return core.CheckResult{Status: core.CheckWarn, Detail: "CARRIER_KEY is not set", Hint: "set it to the key carriers sign their scans with"} } return core.CheckResult{Status: core.CheckOK, Detail: "CARRIER_KEY is set"} }, }}, ``` A check says what it found (`Detail`), and, when something is wrong, what puts it right (`Hint`). `CheckWarn` is for what only some scenarios need; `CheckFail` for what none can run without. # Configuration > The Axx configuration files - axx.yaml (every section, the services and lint keys, how files and profiles merge, interpolation) and axx-packs.yaml. An acceptance project has up to two configuration files, side by side: `axx.yaml` for the project and `axx-packs.yaml` for the packs it uses. Both are optional. ## axx.yaml [Section titled “axx.yaml”](#axxyaml) Without an `axx.yaml`, Axx runs `features/` with defaults. The complete, versioned definition is the JSON Schema: * online: [`https://axx.nimbusxr.us/schemas/v0/axx.schema.json`](/next/schemas/v0/axx.schema.json) * offline: `axx schema` (or `axx schema --out axx.schema.json`); `axx schema --outline` lists the keys, a line each Add this first line to get completion and validation in editors that use the YAML language server (VS Code, IntelliJ, Neovim): ```yaml # yaml-language-server: $schema=https://axx.nimbusxr.us/schemas/v0/axx.schema.json ``` ### A complete example [Section titled “A complete example”](#a-complete-example) axx.yaml ```yaml # yaml-language-server: $schema=https://axx.nimbusxr.us/schemas/v0/axx.schema.json version: 1 run: paths: [features] # feature files or directories tags: "not @wip" # default tag expression uses: [web-core] # only the scenarios that use one of these packs (a profile per platform) workers: auto # parallel scenarios: a number or auto (CPUs) exclusive: ["@isolated"] # these run alone, after the parallel phase order: defined # or random, random: timeouts: {step: 60s, scenario: 5m, hook: 2m} watch: false # show what scenarios do: browsers and devices on screen (axx run --watch) slowdown: 300ms # in a watched run, pause after each action (axx run --slowdown) reporters: [pretty, {junit: build/axx/junit.xml}] resources: ["."] # where seed, payload and schema paths in steps resolve, after the feature file's directory properties: # ${sys:name}; override with -D name=value local.host: localhost openapi: levels: # suite-wide OpenAPI validation levels validation.request.security.missing: IGNORE active: enabled: false # start only the services the selected scenarios' tags need onNoTags: fallback services: api: dir: . command: docker compose up --build ready: http: url: http://${sys:local.host}:8080/health timeout: 120s cleanup: docker compose down -v --remove-orphans lint: {} # axx lint rules fixtures: {} # axx fixtures settings profiles: # overlays: --profile ci, several in order (--profile ios,watch), or AXX_PROFILE ci: properties: {local.host: docker} ``` ### Sections [Section titled “Sections”](#sections) | Section | What it configures | Guide | | ---------------- | --------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | | `version` | the schema version of this file; currently `1` | | | `run` | paths, default tags, workers, exclusive tags, order, timeouts, default reporters | [Run in parallel](/next/guides/parallel-runs/), [Reports](/next/guides/reports/) | | `resources` | directories that relative file paths in steps resolve against, in order; the directory of `axx.yaml` is searched last | [Configure services](/next/guides/configure-services/#resource-paths) | | `properties` | values for `${sys:name}` | [Configure services](/next/guides/configure-services/#keep-values-out-of-feature-files) | | `openapi.levels` | default OpenAPI validation levels, by rule key | [Validate against OpenAPI](/next/guides/validate-openapi/) | | `active` | service startup by the selected scenarios’ tags and packs | [Manage the services under test](/next/guides/manage-services/#start-only-what-a-run-needs) | | `services` | the system under test ([keys](#services)) | [Manage the services under test](/next/guides/manage-services/) | | `packs` | settings for packs, keyed by pack name | each pack’s page under [Packs](/next/references/packs/) | | `lint` | test-data isolation rules ([keys](#lint)) | [Isolate test data](/next/guides/isolate-test-data/) | | `fixtures` | fixture factory settings | [Fixture factories](/next/guides/fixture-factories/) | | `profiles` | named overlays | [below](#finding-and-merging-files) | ### services [Section titled “services”](#services) Each key under `services:` names one service: something that runs on its own, which scenarios connect to. A program a scenario runs as its user would, like a command-line tool or an MCP server over stdio, belongs in its step (`command`), not here. | Key | Meaning | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `command` | How to start the service: a string (split into words, no shell) or an argv list. A command that exits `0` before the service is ready is fine: Axx keeps polling `ready`. | | `shell` | `true` runs `command` and `cleanup` through the shell, for pipes and `&&`. | | `dir`, `env` | Working directory (relative to `axx.yaml`) and extra environment. | | `dependsOn` | Services that must be ready first. Independent services start in parallel. | | `enabled` | `false` skips the service. Defaults to `true`. | | `ready` | Checks that must all pass: `http.url` (every URL returns 2xx), `tcp` (`host:port` accepts connections), `exec` (a command exits 0), `log` (a regular expression matches the service’s output). `timeout` defaults to 60s, `interval` to 1s. | | `stop` | `signal` (`SIGTERM` by default, or `SIGINT`) is sent to the service’s process group; after `grace` (default 10s) it is killed. | | `cleanup` | Runs after the service stops, even if it crashed or never became ready. | | `active.tags` | With `active.enabled`, the service starts only when a selected scenario has one of these tags. | | `active.uses` | With `active.enabled`, the service starts only when a selected scenario uses a step of one of these packs, like `[mobile-android]`. | | `debug` | `command`, `debugger` (`type`, `port`, `host`, `module`, `mode`), `onUnavailable` and `retry` for `axx run --debug` and `axx up --debug`. | ### lint [Section titled “lint”](#lint) Rules live under `lint.rules`; `lint.config` sets `baseDir` (patterns are relative to it) and the default `mode` (`error` fails, `warn` only reports); `lint.include` merges rule files such as `axx-lint.generated.yaml`. Each rule has: | Key | Meaning | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `name`, `description` | Shown in the report. | | `filePatterns` | Globs (`*`, `**`, `?`, `[abc]`, `{a,b}`), relative to `baseDir`. A leading `../` reaches outside it. The part before the first wildcard is a directory, and the rest matches the end of each path below it: `seeds/*.yaml` also finds YAML files in subdirectories of `seeds`. A pattern that ends in `/` matches folders, and all the files below each folder it matches count as one file: the folder of a scenario whose data spans several files (`seeds/*/`). | | `excludePatterns` | Globs removed from the match, against the whole path relative to `baseDir`, for example `**.fixture.yaml`. | | `type` | `regex` (default) or `jsonpath`. | | `regex` | The first capture group that matched is the value. | | `jsonPath` | A path in each file’s tree: `order.id`, `payments[0].id`, `payments[*].id`, an optional `$.` prefix, or any Jayway JsonPath. Files are read by their extension: JSON, YAML (`.yaml`, `.yml`), XML and CSV. A seed’s rows read alike in YAML, flat XML and CSV (`$['public.closures'][*]['id']`); in other XML, attributes are `@name` fields and an element with only text is a value. | | `validation` | `global-unique` (default): every occurrence is unique. `file-unique`: unique within each file. `cross-file-unique`: may repeat inside a file, never across files. A folder a folder pattern matches counts as one file. | | `mode` | Overrides the default mode for this rule. | | `ignoreValues` | Values that are shared on purpose. | ### Finding and merging files [Section titled “Finding and merging files”](#finding-and-merging-files) 1. With `-c path/to/axx.yaml`, that file. Otherwise Axx looks for `axx.yaml` (or `axx.yml`) in the working directory, then in each parent directory up to the repository root. 2. With `--profile NAME` or `AXX_PROFILE=NAME`, `profiles.NAME` is merged over the file, and then `axx.NAME.yaml` next to it, if either exists. Several profiles, comma-separated (`--profile ios,watch`), are merged in that order, each over the ones before: where they set different keys you get both, and where they set the same key the later one wins. A profile that exists in neither place is an error ([`AXX-E0104`](/next/references/error-codes/#axx-e0104)). 3. `axx.local.yaml` next to `axx.yaml`, if present, is merged last. Keep it out of version control for personal settings. 4. `--set path.to.key=value` sets any key for one run, over all of the above: `--set run.workers=1 --set packs.web-core.watch=true`. The value is read as YAML. 5. `-D name=value` overrides `properties`. Paths inside the file (`services.*.dir`, `resources`) are relative to the directory of `axx.yaml`. ### Interpolation [Section titled “Interpolation”](#interpolation) String values can reference the environment and properties: | Syntax | Value | | ---------------------- | -------------------------------------------------------------------------- | | `${env:NAME}` | environment variable `NAME` | | `${sys:name}` | property `name` | | `${sys:name:-default}` | `default` when `name` is not set; defaults can nest (`${sys:a:-${env:B}}`) | | `$${...}` | a literal `${...}` | The same syntax works in feature files: in service tables and step arguments. ### Errors [Section titled “Errors”](#errors) A syntax error or an invalid value stops every command with exit code `2` and points at the line: [`AXX-E0101`](/next/references/error-codes/#axx-e0101) for YAML syntax and [`AXX-E0102`](/next/references/error-codes/#axx-e0102) for values the schema rejects. ## axx-packs.yaml [Section titled “axx-packs.yaml”](#axx-packsyaml) The packs a project uses: `core`, which is always there, and the listed packs. Without the file, a project has no steps. `axx pack add`, `remove` and `new` edit it, and `axx init` creates it. axx-packs.yaml ```yaml packs: - rest # one of Axx's packs, by name - sql - ./steps # a pack in this repository (a Go package) - github.com/team/axx-grpc@v1.2.0 # a pack from a Go module; the version is optional ``` `axx-packs.lock` pins the versions of module packs; commit both files. [Choose packs](/next/guides/use-packs/) shows the commands. # JSON output > The --json envelope every Axx command prints, the error objects inside it, and the run report that axx run and the agent reporter produce. Every command accepts `--json` and then prints exactly one JSON document on stdout: the **envelope**. It is a frozen contract ([ADR 0003](https://github.com/nimbusxr/axx/blob/main/docs/adr/0003-cli-contract.md)): fields may be added, but removing or retyping a field requires a new `schemaVersion`. ## The envelope [Section titled “The envelope”](#the-envelope) ```json { "schemaVersion": 1, "command": "axx steps search", "ok": true, "data": { } } ``` | Field | Type | Meaning | | --------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | | `schemaVersion` | number | Envelope version; currently `1`. | | `command` | string | The command that ran, such as `axx run` or `axx steps search`. | | `ok` | boolean | `true` when the command succeeded (exit code `0`). | | `data` | object | The command’s result. Absent when the command could not produce one. | | `errors` | array | Errors that stopped the command. Absent when there are none: a run whose scenarios fail has `ok: false` and its report in `data`, without `errors`. | The process [exit code](/next/references/error-codes/#exit-codes) is the same with or without `--json`. ## Errors [Section titled “Errors”](#errors) ```json { "schemaVersion": 1, "command": "axx run", "ok": false, "errors": [ { "code": "AXX-E0202", "message": "invalid tag expression \"@x and\": Tag expression \"@x and\" could not be parsed because of syntax error: Expected operand.", "hint": "use expressions like \"@smoke and not @wip\"", "docs": "https://axx.nimbusxr.us/references/error-codes/#axx-e0202" } ] } ``` | Field | Meaning | | ---------- | -------------------------------------------------------------------------------------------------------------- | | `code` | A stable `AXX-Exxxx` code; see [error codes](/next/references/error-codes/). Branch on this, not on `message`. | | `message` | What went wrong, for humans. | | `location` | Optional `{file, line, column}`, for example a line in `axx.yaml` or a feature file. | | `hint` | Optional: what to do about it. | | `docs` | A link to the code’s entry in the error code reference. | ## The run report [Section titled “The run report”](#the-run-report) `axx run --json` puts the run report in `data`. `--format agent:FILE` writes the same object to a file. Only failures are listed, so the size depends on what broke, not on the size of the suite. This example leaves out the failure’s `logs` and `context`: ```json { "schemaVersion": 1, "axx": "0.1.0", "result": "failed", "durationMs": 7, "counts": { "scenarios": { "failed": 1, "passed": 1 }, "steps": { "failed": 1, "passed": 5 } }, "notRun": 0, "interrupted": false, "failures": [ { "scenario": "fails", "location": "features/demo.feature:6", "status": "failed", "step": { "keyword": "Then", "text": "the response status code is 201", "location": "features/demo.feature:9", "definition": "rest.response.status" }, "error": { "kind": "assertion", "message": "Expected status code <201> but was <200>.", "expected": 201, "actual": 200 }, "rerun": "axx run features/demo.feature:6" } ], "undefined": [] } ``` | Field | Meaning | | ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `result` | the worst scenario status: `passed`, `skipped`, `pending`, `undefined`, `ambiguous` or `failed`; `interrupted` when the run was cancelled | | `durationMs` | wall-clock time of the run | | `counts` | scenarios and steps by status (`passed`, `failed`, `skipped`, `undefined`, `ambiguous`, `pending`); zero counts are omitted | | `notRun` | selected scenarios that did not run, for example after `--fail-fast` or an interrupt | | `interrupted` | `true` when the run was cancelled | | `dryRun` | `true` for `axx run --dry-run`; absent otherwise | | `failures[]` | one entry per failed, undefined, ambiguous or pending scenario, in feature file order | | `failures[].location`, `.tags`, `.rerun` | where the scenario is, its tags, and the command that reruns only it | | `failures[].step` | the step that failed: keyword, text, location and the id of the matched `definition`; for a hook, `hook` is `before` or `after` | | `failures[].error.kind` | `assertion`, `error`, `timeout`, `panic`, `undefined`, `ambiguous` or `pending` | | `failures[].error.expected`, `.actual`, `.diff` | for assertions: the two values, as JSON, and a line diff when they are JSON documents or text of several lines | | `failures[].error.stack`, `.candidates`, `.suggestions` | the stack of a panic, the matching definitions of an ambiguous step, close expressions for an undefined one | | `failures[].logs`, `.context` | the scenario’s logs (the last 50), and pack context such as the last HTTP request and response | | `undefined[]` | undefined steps with suggestions, for writing them correctly | | `runErrors[]` | problems found after the scenarios, outside any of them (`source`: the pack, `message`); present only when there are some, and they make `result` `failed` | ## Other commands [Section titled “Other commands”](#other-commands) Each command documents its own `data`. The most useful for scripts and agents: | Command | `data` | | ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `axx steps search ` | `{query, steps[]}`; each step has `id`, `pack`, `expr`, `variants`, `keyword`, `arg`, `doc`, `examples`, `params` | | `axx validate` | `{files, scenarios, steps, problems[], warnings[]}`; each problem has `kind` (`undefined`, `ambiguous`, `argument`), `location`, `text`, and a `message`, `suggestions` or `candidates` where they apply; `warnings[]`, present when there are some, are findings of `axx lint`’s feature checks (kind `lint`), and `hints[]` name scenarios whose checks prove little (only a success status, or only that something did not happen); neither changes the exit code | | `axx doctor` | `{version, config, checks[]}`; each check has `name`, `status` (`ok`, `warn`, `fail`), `detail`, `hint` | | `axx env` | `{services[], up[]}`: the services axx started, each with `name`, `state` (`running`, `left over`, `not cleaned up`), `pid` and its `cleanup`; `up` lists those `axx up` holds | | `axx config show` | `{file, config, packs[]}`: the effective `axx.yaml` (profiles and `-D` applied, values of secret-looking keys masked) and the project’s packs | | `axx down` | `{stopped, cleanedUp}`: the services `axx up` kept running that it stopped, and the services an earlier run left behind that it stopped or cleaned up | | `axx lint` | `{baseDir, rules[], summary, notes[], hints[]}`; `hints[]`, present when there are some, suggest fixture factories and name scenarios whose checks prove little, and never change the exit code | | `axx init` | `{dryRun, files[], detected, agents}`; each file has `path`, `action` (`create`, `update`, `skip`), `reason`; `agents`, absent with `--no-agents`, has `detected[]` (agent ids), `skills[]` (like `files`) and `mcp[]` | | `axx mcp install` | `{dryRun, change}`; `change` and each entry of `axx init`’s `agents.mcp[]` has `agent`, `scope` (`project`, `user`), `path`, `action` (`create`, `update`, `skip`, or `manual` when `command` adds the server), `reason`, `command` | | `axx docs export` | `{files[]}` | | `axx fixtures generate` | `{baseDir, dryRun, written[], unchanged[]}`: every file the factories generate, relative to `baseDir`; with `--dry-run`, `written[]` is what it would write | | `axx fixtures check` | `{baseDir, checks, failures[], unmaterialized[]}`; each failure has `check`, `code` and `message`; `unmaterialized[]`, present when there are some, are ignored outputs this checkout has not generated yet | | `axx fixtures explain ` | `{file, factory, fixture, path, value, origins[]}`; `origins[]` is the value’s way into the fixture, outermost first, each with `kind` (`identity`, `reference`, `expression`, `fixture`, `prototype-overlay`, `prototype`, `defaults`, `schema`), `file`, `at` (where in the file) and `detail` | For editors and generated clients, the `axx.yaml` schema is at [`/schemas/v0/axx.schema.json`](/next/schemas/v0/axx.schema.json). # Packs > The packs Axx publishes: their steps, settings and tools, a page each. A pack gives Axx what a project tests: its steps, its settings in `axx.yaml`, and tools for coding agents. A project lists the packs it uses in `axx-packs.yaml` (`axx pack add `); a pack that builds on another brings it along. | Pack | For | | ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ | | [`core`](/next/references/packs/core/) | Parameter types every pack can use | | [`rest`](/next/references/packs/rest/) | REST requests and responses, OpenAPI validation | | [`mock`](/next/references/packs/mock/) | WireMock services and their OpenAPI contracts | | [`grpc`](/next/references/packs/grpc/) | gRPC calls and server streams: status, answers, metadata, read with the services’ protos or reflection | | [`jsonrpc`](/next/references/packs/jsonrpc/) | JSON-RPC 2.0 calls over HTTP: results and errors, checked against OpenRPC documents | | [`graphql`](/next/references/packs/graphql/) | GraphQL queries, mutations and subscriptions, of a federated graph or a subgraph, checked against the schema | | [`mcp`](/next/references/packs/mcp/) | MCP servers, over stdio or HTTP: their tools, checked against the tools’ schemas, their resources and prompts | | [`a2a`](/next/references/packs/a2a/) | A2A agents: the messages sent and streamed to them, and the tasks they run, with their states and artifacts | | [`sql`](/next/references/packs/sql/) | SQL databases: seeds, selections, locks, triggers | | [`mongo`](/next/references/packs/mongo/) | MongoDB: seeds and selections | | [`redis`](/next/references/packs/redis/) | Redis and its forks (Valkey, Dragonfly, KeyDB, Garnet): seeds and keys | | [`kafka`](/next/references/packs/kafka/) | Kafka events, Avro and the Schema Registry | | [`amqp`](/next/references/packs/amqp/) | AMQP queues and exchanges: RabbitMQ, and AMQP 1.0 brokers such as Artemis | | [`mqtt`](/next/references/packs/mqtt/) | MQTT topics: the messages devices and services publish | | [`nats`](/next/references/packs/nats/) | NATS subjects and JetStream streams | | [`logs`](/next/references/packs/logs/) | the entries services log | | [`files`](/next/references/packs/files/) | the files services write to a folder | | [`mail`](/next/references/packs/mail/) | the emails services send, read from Mailpit, or over POP3 or IMAP | | [`cli`](/next/references/packs/cli/) | commands: their exit codes and what they print | | [`websocket`](/next/references/packs/websocket/) | WebSocket connections: the messages sent and received, and how they close | | [`sse`](/next/references/packs/sse/) | server-sent event streams and the events they send | | [`asyncapi`](/next/references/packs/asyncapi/) | AsyncAPI contracts: the messages scenarios send and checks find, checked against them | | [`app-core`](/next/references/packs/app-core/) | apps, used as people use them, whatever they run on: launched, their controls tapped or clicked and filled, what they show checked | | [`mobile-core`](/next/references/packs/mobile-core/) | native mobile apps, used as people use them: launched or opened with a deep link, their controls tapped and filled, what they show checked | | [`mobile-android`](/next/references/packs/mobile-android/) | Android apps on emulators and devices, through the UiAutomator2 server: a device of its own and a clean app for each scenario | | [`mobile-ios`](/next/references/packs/mobile-ios/) | iOS apps on simulators, through WebDriverAgent: a simulator of its own and a clean app for each scenario | | [`desktop-core`](/next/references/packs/desktop-core/) | desktop apps, used as people use them: launched, their controls clicked and filled, what they show checked | | [`desktop-macos`](/next/references/packs/desktop-macos/) | macOS apps, through AX: a clean app for each scenario | | [`desktop-windows`](/next/references/packs/desktop-windows/) | Windows apps, through UI Automation and the Java Access Bridge: a clean app for each scenario | | [`desktop-linux`](/next/references/packs/desktop-linux/) | Linux apps, through AT-SPI, in desktops of Axx’s own: a clean app for each scenario | ## Web [Section titled “Web”](#web) | Pack | For | | ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------ | | [`web-core`](/next/references/packs/web-core/) | web apps in real browsers, driven by Playwright | | [`web-screenshots`](/next/references/packs/web-screenshots/) | pages that look as designed, compared with screenshots per platform | | [`web-a11y`](/next/references/packs/web-a11y/) | pages people can use whatever their abilities: axe-core audits and accessible structure | | [`web-network`](/next/references/packs/web-network/) | what pages fetch: failing, answered or slow requests, recordings, websocket messages | | [`web-lighthouse`](/next/references/packs/web-lighthouse/) | Lighthouse audits of pages: performance, accessibility, best practices and SEO scores, Core Web Vitals | | [`web-coverage`](/next/references/packs/web-coverage/) | how much of the web apps’ JavaScript the scenarios run: lcov and Istanbul reports, from Chromium | ## Models [Section titled “Models”](#models) | Pack | For | | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | | [`models-core`](/next/references/packs/models-core/) | stubs of the models apps and services run themselves: models of the real ones’ contracts whose answers are their mappings | | [`models-ios`](/next/references/packs/models-ios/) | prebuilt stubs for platform model calls in debug iOS Simulator apps | | [`models-macos`](/next/references/packs/models-macos/) | prebuilt stubs for Apple’s model calls in Mac apps’ test builds | | [`models-linux`](/next/references/packs/models-linux/) | stubs of the models Linux apps take from the system: Tesseract’s language data and speech-dispatcher’s voice | | [`models-windows`](/next/references/packs/models-windows/) | stubs of the models Windows apps take from the system: SAPI’s voice and the Windows AI APIs | | [`models-android`](/next/references/packs/models-android/) | prebuilt stubs for platform model calls in debuggable Android emulator apps | | [`models-web`](/next/references/packs/models-web/) | prebuilt stubs for browser model APIs, installed before application scripts | ## AWS [Section titled “AWS”](#aws) | Pack | For | | ------------------------------------------------------------ | ------------------------------------ | | [`aws-core`](/next/references/packs/aws-core/) | the AWS account the aws-\* packs use | | [`aws-s3`](/next/references/packs/aws-s3/) | S3 buckets and objects | | [`aws-sqs`](/next/references/packs/aws-sqs/) | SQS queues | | [`aws-sns`](/next/references/packs/aws-sns/) | SNS topics | | [`aws-eventbridge`](/next/references/packs/aws-eventbridge/) | EventBridge buses | | [`aws-dynamodb`](/next/references/packs/aws-dynamodb/) | DynamoDB tables | ## Google Cloud [Section titled “Google Cloud”](#google-cloud) | Pack | For | | -------------------------------------------------------- | --------------------------------------------- | | [`gcp-core`](/next/references/packs/gcp-core/) | the Google Cloud project the gcp-\* packs use | | [`gcp-storage`](/next/references/packs/gcp-storage/) | Cloud Storage buckets and objects | | [`gcp-pubsub`](/next/references/packs/gcp-pubsub/) | Pub/Sub topics | | [`gcp-bigquery`](/next/references/packs/gcp-bigquery/) | BigQuery tables | | [`gcp-firestore`](/next/references/packs/gcp-firestore/) | Firestore documents | ## Azure [Section titled “Azure”](#azure) | Pack | For | | -------------------------------------------------------------- | --------------------------------- | | [`azure-blob`](/next/references/packs/azure-blob/) | Blob Storage containers and blobs | | [`azure-servicebus`](/next/references/packs/azure-servicebus/) | Service Bus queues and topics | # A2A agents > Send messages to A2A agents (Agent2Agent protocol) as another agent sends them, and check what they answer: the tasks they run, with their states and their artifacts, as they answer or as they stream. Send messages to A2A agents (Agent2Agent protocol) as another agent sends them, and check what they answer: the tasks they run, with their states and their artifacts, as they answer or as they stream. ```gherkin Given the parcels a2a agent with the following properties: | url | http://localhost:8400 | When a message is sent to the parcels a2a agent: """ Where is PX-A2A-9201? """ Then the parcels a2a agent's task is completed And the parcels a2a agent's answer contains 'out for delivery' ``` * **The card is the contract:** it is read at `/.well-known/agent-card.json` (or from a file), checked for what A2A 1.0 requires, and says how to reach the agent: JSON-RPC, HTTP+JSON or gRPC. * **A conversation:** a message starts one; a reply continues the agent’s last task, which is how a task that asks for input (`input-required`) gets it. * **Streams belong to the scenario,** which ends them when it ends. The checks wait (10 seconds, or `within {duration}`) for what the stream has not sent yet. * **States are written as people say them:** `submitted`, `working`, `completed`, `input-required`, `auth-required`, `failed`, `canceled`, `rejected`. * **Secrets stay secret:** `${env:..}` values in the properties are masked in logs and failures, and `${token:..}` names a token of the scenario. Add it to a project with `axx pack add a2a`. ## Steps [Section titled “Steps”](#steps) ### `a2a.agent` [Section titled “a2a.agent”](#a2aagent) ```gherkin Given the {word} a2a agent with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register an A2A agent under a name: its card is read and checked for what A2A 1.0 requires, and says how to reach it (JSON-RPC, HTTP+JSON or gRPC). | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Values | Default | | --------------- | --------------------------------------------------------------------------------------------- | ------------------------------ | ------------------ | | `url` | the agent’s URL; its card is read at `/.well-known/agent-card.json` on its origin | | | | `card` | the agent’s card, when it is not there: a file of the project, or a URL | | | | `transport` | how to reach the agent, among the interfaces its card lists | `JSONRPC`, `HTTP+JSON`, `GRPC` | `the card's first` | | `header.` | a header sent with every request, such as `header.Authorization` | | | | `timeout` | how long a call may take, like `10s` | | `30s` | | `push url` | where the agent sends its tasks’ updates (a webhook, such as a WireMock the mock pack checks) | | | | `push token` | the token the agent sends with them, for the webhook to check | | | A `url` or a `card` is required. Values are expanded (`${env:..}`, `${sys:..}`, `${token:..}`); what `${env:..}` references expand to is masked everywhere. **Example:** ```gherkin Given the parcels a2a agent with the following properties: | url | http://localhost:8400 | Given the carrier a2a agent with the following properties: | url | https://agents.carrier.example | | transport | HTTP+JSON | | header.Authorization | Bearer ${token:carrier} | ``` *Since 0.1.5.* ### `a2a.card` [Section titled “a2a.card”](#a2acard) ```gherkin Then the {word} a2a agent's card has the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check the agent’s card: what another agent reads to find out what it does and how to reach it. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | A row is a path into the card (`name`, `capabilities.streaming`, `skills[0].id`) and its value. **Example:** ```gherkin Then the parcels a2a agent's card has the following properties: | name | Parcels agent | | capabilities.streaming | true | ``` *Since 0.1.5.* ### `a2a.skill` [Section titled “a2a.skill”](#a2askill) ```gherkin Then the {word} a2a agent has the {word} skill ``` Check that the agent’s card lists a skill, by its id. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin Then the parcels a2a agent has the track-parcel skill ``` *Since 0.1.5.* ### `a2a.send` [Section titled “a2a.send”](#a2asend) ```gherkin When a message is sent to the {word} a2a agent: """ ... """ ``` Send the doc string as a text message, which starts a conversation, and wait for the agent’s answer: a reply, or a task that ended or waits for input. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When a message is sent to the parcels a2a agent: """ Where is PX-A2A-9201? """ ``` *Since 0.1.5.* ### `a2a.send.file` [Section titled “a2a.send.file”](#a2asendfile) ```gherkin When the {filepath} message is sent to the {word} a2a agent ``` Send the message of a file of the project, an A2A message in JSON (its `parts`: text, data, files), which starts a conversation, and wait for the agent’s answer. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When the a2a/hold-request.json message is sent to the parcels a2a agent ``` *Since 0.1.5.* ### `a2a.stream` [Section titled “a2a.stream”](#a2astream) ```gherkin When a message is streamed to the {word} a2a agent: """ ... """ ``` Send the doc string as a text message, which starts a conversation, and take the agent’s answer as it streams: the checks read its updates as they come. The stream belongs to the scenario, which ends it when it ends. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When a message is streamed to the parcels a2a agent: """ Where is PX-A2A-9203? """ ``` *Since 0.1.5.* ### `a2a.reply` [Section titled “a2a.reply”](#a2areply) ```gherkin When a reply is sent to the {word} a2a agent: """ ... """ ``` Send the doc string as a text message that continues the agent’s last task: how a task that asks for input (`input-required`) gets it. Wait for the agent’s answer. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When a reply is sent to the parcels a2a agent: """ 2026-10-05 """ ``` *Since 0.1.5.* ### `a2a.cancel` [Section titled “a2a.cancel”](#a2acancel) ```gherkin When the {word} a2a agent is asked to cancel its task ``` Ask the agent to cancel its last task; check that it did with `the {word} a2a agent's task is canceled`. An agent that will not (a task that ended) fails the step with its error. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When the parcels a2a agent is asked to cancel its task ``` *Since 0.1.5.* ### `a2a.state` [Section titled “a2a.state”](#a2astate) ```gherkin Then [[within {duration} ]]the {word} a2a agent's task is {word} ``` Check the state of the agent’s last task: `submitted`, `working`, `completed`, `input-required`, `auth-required`, `failed`, `canceled` or `rejected`. * The check waits for a task still at work (10 seconds, or `within {duration}`), and fails at once for a task that ended or waits for input in another state. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} a2a agent's task is {word}` * `within {duration} the {word} a2a agent's task is {word}` **Example:** ```gherkin Then the parcels a2a agent's task is input-required Then within 30s the parcels a2a agent's task is completed ``` *Since 0.1.5.* ### `a2a.answer` [Section titled “a2a.answer”](#a2aanswer) ```gherkin Then the {word} a2a agent's answer contains {string} ``` Check that the agent’s answer has a text: its reply, or its task’s status message and artifacts. It waits for a task still at work. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Example:** ```gherkin Then the parcels a2a agent's answer contains 'out for delivery' ``` *Since 0.1.5.* ### `a2a.artifact` [Section titled “a2a.artifact”](#a2aartifact) ```gherkin Then the {word} a2a agent's task has an artifact where: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that an artifact of the agent’s last task has the table’s properties: `name`, `description`, `text` (its text parts), `data` (its first data part: `data.status`) and `parts`. It waits for a task still at work. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | A row is a path into the artifact, and its value. **Example:** ```gherkin Then the parcels a2a agent's task has an artifact where: | name | parcel-status | | data.status | OUT_FOR_DELIVERY | ``` *Since 0.1.5.* ### `a2a.update` [Section titled “a2a.update”](#a2aupdate) ```gherkin Then [[within {duration} ]]the {word} a2a agent's stream received an update where: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the stream of the message streamed last sent an update with the table’s properties. * A status update has `kind` `status`, its `state` and its `message`; an artifact update has `kind` `artifact` and its `artifact` (`artifact.name`, `artifact.text`, `artifact.data.status`…). * The check waits (10 seconds, or `within {duration}`), and fails at once when the stream has ended. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | A row is a path into the update, and its value. **Variants**, the parts in `[[...]]` said or left out: * `the {word} a2a agent's stream received an update where:` * `within {duration} the {word} a2a agent's stream received an update where:` **Example:** ```gherkin Then the parcels a2a agent's stream received an update where: | state | working | | message | Looking up PX-A2A-9203 | ``` *Since 0.1.5.* # AMQP > Send messages to AMQP queues and publish them to AMQP exchanges, and check the messages your services send there: RabbitMQ over AMQP 0-9-1, and brokers that speak AMQP 1.0, such as ActiveMQ Artemis. Send messages to AMQP queues and publish them to AMQP exchanges, and check the messages your services send there: RabbitMQ over AMQP 0-9-1, and brokers that speak AMQP 1.0, such as ActiveMQ Artemis. Register the broker once, usually in the `Background`; every AMQP step of the scenario uses it: ```gherkin Given the depot amqp broker with the following properties: | url | amqp://parcels:${env:RABBITMQ_PASSWORD}@localhost:5672/ | ``` Steps name a **queue** or an **exchange**: `the parcels.label-printed amqp queue`, `the printers amqp exchange`. Messages carry a routing key (`with the routing key '...'`, or a `routing key` row), headers (`header ` rows) and properties such as `content type`. A message whose body is JSON is sent as `application/json` unless a `content type` row says otherwise, and every message is persistent. * **Checking a queue receives from it:** Axx acknowledges each message it receives, as any consumer would, so check the queues your services **write** to and nothing else reads. * **Checking an exchange takes nothing from anyone:** for the exchanges a run checks, Axx declares a queue of its own (`axx-.`, exclusive, deleted with the run) once the services are up, and binds it with `#` and with the routing keys the checks name, which covers topic, fanout, headers and direct exchanges. * **A check only looks at the messages received since its scenario started,** so scenarios running in parallel check their own messages, by data unique to them. * **Publishing a message nothing receives fails the step:** the broker returns a message no queue is bound for, and the step says so, instead of the message going nowhere unnoticed. * **AMQP 1.0:** with `protocol | 1.0`, a queue is an anycast address and an exchange a multicast one, and the routing key is the message’s subject. RabbitMQ routes the same messages over 0-9-1 whatever protocol your services use, so keep the default for it. * **Secrets stay secret:** `${env:..}` values in the URL are masked in logs and failures. Add it to a project with `axx pack add amqp`. ## Steps [Section titled “Steps”](#steps) ### `amqp.broker` [Section titled “amqp.broker”](#amqpbroker) ```gherkin Given the {word} amqp broker with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register the AMQP broker the steps talk to. * The first broker registered is the one the scenario’s AMQP steps use. * Values expand `${env:..}` and `${sys:..}`; `${env:..}` values are masked. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Values | Default | | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | ------- | | `url` | the broker’s URL, with its user, password and virtual host: `amqp://user:password@host:5672/vhost`, or `amqps://` for TLS | | | | `protocol` | the AMQP version the broker speaks | `0-9-1`, `1.0` | `0-9-1` | | `asyncapi` | an AsyncAPI document, a file of the project or a URL, that the queues and exchanges follow: every message a scenario sends there, and every message a check finds, is checked against it (with the asyncapi pack) | | | **Example:** ```gherkin Given the depot amqp broker with the following properties: | url | amqp://parcels:${env:RABBITMQ_PASSWORD}@localhost:5672/ | Given the depot amqp broker with the following properties: | url | amqp://artemis:${env:ARTEMIS_PASSWORD}@localhost:5672 | | protocol | 1.0 | ``` *Since 0.1.5.* ### `amqp.queue.send` [Section titled “amqp.queue.send”](#amqpqueuesend) ```gherkin When a message is sent to the {word} amqp queue: """ ... """ ``` Send a message to the amqp queue; the doc string is its body. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When a message is sent to the parcels.label-printed amqp queue: """ {"reference": "PX-8103", "printer": "LEJ-3", "printedAt": "2026-09-28T08:15:00Z"} """ ``` *Since 0.1.5.* ### `amqp.queue.send.file` [Section titled “amqp.queue.send.file”](#amqpqueuesendfile) ```gherkin When the {filepath} message is sent to the {word} amqp queue[[ with the following properties:]] ``` Send a message whose body is the file to the amqp queue, with the properties of the table when the step has one. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Name | Takes | Default | | ---------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------ | | `header ` | a header sent with the message: its name after `header `, and its value | | | `content type` | the body’s media type | \`\`application/json`for a JSON body,`text/plain` otherwise` | | `correlation id` | the message’s correlation ID | | | `message id` | the message’s ID | | | `reply to` | where replies go: a queue’s name | | **Variants**, the parts in `[[...]]` said or left out: * `the {filepath} message is sent to the {word} amqp queue` * `the {filepath} message is sent to the {word} amqp queue with the following properties:` **Example:** ```gherkin When the amqp/label-printed.json message is sent to the parcels.label-printed amqp queue When the amqp/label-printed.json message is sent to the parcels.label-printed amqp queue with the following properties: | header printer | LEJ-3 | | content type | application/json | ``` *Since 0.1.5.* ### `amqp.queue.received` [Section titled “amqp.queue.received”](#amqpqueuereceived) ```gherkin Then [[within {duration} ]]the {word} amqp queue has a message where: | path | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the amqp queue has a message with those values, received since the scenario started. * The check waits for it: 10 seconds, or `within {duration}`. * Axx acknowledges the messages of a queue it checks, so check the queues your services write to. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a path into the message’s JSON body (a field name, a dotted path or a JSONPath) and the value it has, compared as text: `null` for null and `undefined` for absent. A row `header ` is instead on the header of that name sent with the message: `undefined` when there is none. Rows `routing key`, `content type`, `correlation id`, `message id` and `reply to` are on the message itself, not its body. **Variants**, the parts in `[[...]]` said or left out: * `the {word} amqp queue has a message where:` * `within {duration} the {word} amqp queue has a message where:` **Example:** ```gherkin Then within 30s the parcels.label-printed.rejected amqp queue has a message where: | reference | PX-8199 | | header reason | unknown parcel | ``` *Since 0.1.5.* ### `amqp.exchange.send` [Section titled “amqp.exchange.send”](#amqpexchangesend) ```gherkin When a message is published to the {word} amqp exchange[[ with the routing key {string}]]: """ ... """ ``` Send a message to the amqp exchange, with the routing key the step gives; the doc string is its body. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `a message is published to the {word} amqp exchange:` * `a message is published to the {word} amqp exchange with the routing key {string}:` **Example:** ```gherkin When a message is published to the printers amqp exchange: """ {"reference": "PX-8102", "printer": "LEJ-3", "printedAt": "2026-09-28T08:15:00Z"} """ When a message is published to the printers amqp exchange with the routing key 'printed.LEJ': """ {"reference": "PX-8102", "printer": "LEJ-3", "printedAt": "2026-09-28T08:15:00Z"} """ ``` *Since 0.1.5.* ### `amqp.exchange.send.file` [Section titled “amqp.exchange.send.file”](#amqpexchangesendfile) ```gherkin When the {filepath} message is published to the {word} amqp exchange[[ with the following properties:]] ``` Send a message whose body is the file to the amqp exchange, with the properties of the table when the step has one. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Name | Takes | Default | | ---------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------ | | `routing key` | the routing key the exchange routes the message by | | | `header ` | a header sent with the message: its name after `header `, and its value | | | `content type` | the body’s media type | \`\`application/json`for a JSON body,`text/plain` otherwise` | | `correlation id` | the message’s correlation ID | | | `message id` | the message’s ID | | | `reply to` | where replies go: a queue’s name | | **Variants**, the parts in `[[...]]` said or left out: * `the {filepath} message is published to the {word} amqp exchange` * `the {filepath} message is published to the {word} amqp exchange with the following properties:` **Example:** ```gherkin When the amqp/label-printed.json message is published to the printers amqp exchange When the amqp/label-printed.json message is published to the printers amqp exchange with the following properties: | routing key | printed.LEJ | | header printer | LEJ-3 | ``` *Since 0.1.5.* ### `amqp.exchange.received` [Section titled “amqp.exchange.received”](#amqpexchangereceived) ```gherkin Then [[within {duration} ]]the {word} amqp exchange has a message where: | path | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the amqp exchange has a message with those values, received since the scenario started. * The check waits for it: 10 seconds, or `within {duration}`. * Axx binds a queue of its own to an exchange it checks, for the run. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a path into the message’s JSON body (a field name, a dotted path or a JSONPath) and the value it has, compared as text: `null` for null and `undefined` for absent. A row `header ` is instead on the header of that name sent with the message: `undefined` when there is none. Rows `routing key`, `content type`, `correlation id`, `message id` and `reply to` are on the message itself, not its body. **Variants**, the parts in `[[...]]` said or left out: * `the {word} amqp exchange has a message where:` * `within {duration} the {word} amqp exchange has a message where:` **Example:** ```gherkin Then within 30s the labels amqp exchange has a message where: | routing key | print.express | | reference | PX-8101 | | header sender | maple-crafts | ``` *Since 0.1.5.* # Core > Apps, used as people use them, whatever they run on: launched, their controls tapped or clicked and filled, found by the names people see; what they show checked, waiting as apps take their time. Apps, used as people use them, whatever they run on: launched, their controls tapped or clicked and filled, found by the names people see; what they show checked, waiting as apps take their time. An app is registered by the pack of the platform it runs on: `mobile-android` and `mobile-ios` for phones, `desktop-macos`, `desktop-windows` and `desktop-linux` for desktops (`the courier android app with the following properties:`, `the depot macos app with the following properties:`). Every other step names the app the same way on every platform (`the courier app`, `the depot app`), so a feature for a desktop app lists its registration for each OS and runs unchanged on each: a registration for a platform the machine cannot run does nothing there. An app runs on one platform in a scenario; two apps may run on two, and a scenario can use a web app too. **Each scenario has a clean app:** the platform pack resets the app, and what it keeps, before the scenario starts. Controls are found by the name people see: a button’s text, a field’s label, a list item’s or a row’s text. When no name tells a control apart, `id=` or `xpath=` finds it. An app’s files are a folder of the files pack whose `owner` is the app (`app:depot`). Add it to a project with `axx pack add app-core`. ## Steps [Section titled “Steps”](#steps) ### `app-core.launch` [Section titled “app-core.launch”](#app-corelaunch) ```gherkin When the {word} app is launched ``` Launch the app, reset for the scenario by its platform’s pack (see the pack), or bring it to the front if it is already running. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When the courier app is launched When the depot app is launched ``` *Since 0.1.5.* ### `app-core.restart` [Section titled “app-core.restart”](#app-corerestart) ```gherkin When the {word} app is restarted ``` Stop the app and start it again, keeping what it stored: what survives a restart, like a sign-in, is still there. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When the courier app is restarted ``` *Since 0.1.5.* ### `app-core.tapped` [Section titled “app-core.tapped”](#app-coretapped) ```gherkin When the {string} {control} is tapped in the {word} app ``` Press a control, found by the name people see, or by an `id=` or `xpath=` selector: a tap on a phone, a click on a desktop. “tapped” and “clicked” are the same step: either word works on every platform. | Parameter | Takes | Values | For example | | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | | `"Get a quote"`, `'Express'` | | `{control}` | a kind of control an app shows, found by the name people see: a `button` by its text, a `field` by its label, a `list item` or a `row` by one of its texts, a `menu item` in its open menu, an `element` is anything, by its text or accessibility label; a phone’s `notification` is tapped, by a text in its title or its text. A kind an app’s platform does not have (a phone has no menu bar) fails the step | `button`, `field`, `checkbox`, `radio button`, `switch`, `tab`, `menu`, `menu item`, `list item`, `row`, `link`, `image`, `text`, `element`, `notification` | `button`, `list item` | | `{word}` | one word, with no spaces | | `parcels`, `PX-4101` | **Example:** ```gherkin When the "Sign in" button is tapped in the courier app When the "PX-MOB-9401" list item is tapped in the courier app ``` *Since 0.1.5.* ### `app-core.clicked` [Section titled “app-core.clicked”](#app-coreclicked) ```gherkin When the {string} {control} is clicked in the {word} app ``` Press a control, found by the name people see, or by an `id=` or `xpath=` selector: a tap on a phone, a click on a desktop. “tapped” and “clicked” are the same step: either word works on every platform. | Parameter | Takes | Values | For example | | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | | `"Get a quote"`, `'Express'` | | `{control}` | a kind of control an app shows, found by the name people see: a `button` by its text, a `field` by its label, a `list item` or a `row` by one of its texts, a `menu item` in its open menu, an `element` is anything, by its text or accessibility label; a phone’s `notification` is tapped, by a text in its title or its text. A kind an app’s platform does not have (a phone has no menu bar) fails the step | `button`, `field`, `checkbox`, `radio button`, `switch`, `tab`, `menu`, `menu item`, `list item`, `row`, `link`, `image`, `text`, `element`, `notification` | `button`, `list item` | | `{word}` | one word, with no spaces | | `parcels`, `PX-4101` | **Example:** ```gherkin When the "Register" button is clicked in the depot app When the "PX-DSK-4138" row is clicked in the depot app ``` *Since 0.2.0.* ### `app-core.fill` [Section titled “app-core.fill”](#app-corefill) ```gherkin When the {string} field in the {word} app is filled with {string} ``` Replace what a field holds with a text, as typing it would. `${env:..}` values are secrets: masked in logs and failures. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When the "Courier ID" field in the courier app is filled with "CR-LEJ-12" ``` *Since 0.1.5.* ### `app-core.scroll` [Section titled “app-core.scroll”](#app-corescroll) ```gherkin When the {string} {control} is scrolled into view in the {word} app ``` Scroll until a control shows: on a desktop, every area it is in, the outermost first; on a phone, the screen, down first, then up. | Parameter | Takes | Values | For example | | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | | `"Get a quote"`, `'Express'` | | `{control}` | a kind of control an app shows, found by the name people see: a `button` by its text, a `field` by its label, a `list item` or a `row` by one of its texts, a `menu item` in its open menu, an `element` is anything, by its text or accessibility label; a phone’s `notification` is tapped, by a text in its title or its text. A kind an app’s platform does not have (a phone has no menu bar) fails the step | `button`, `field`, `checkbox`, `radio button`, `switch`, `tab`, `menu`, `menu item`, `list item`, `row`, `link`, `image`, `text`, `element`, `notification` | `button`, `list item` | | `{word}` | one word, with no spaces | | `parcels`, `PX-4101` | **Example:** ```gherkin When the "PX-MOB-9412" list item is scrolled into view in the courier app ``` *Since 0.1.5.* ### `app-core.shows` [Section titled “app-core.shows”](#app-coreshows) ```gherkin Then [[within {duration} ]]the {word} app shows {string} ``` Check that the app shows a text: in a control’s text or label, whitespace collapsed. It waits for the text (10 seconds, or `within`). | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} app shows {string}` * `within {duration} the {word} app shows {string}` **Example:** ```gherkin Then the courier app shows "Today's deliveries" Then within 20s the courier app shows "Delivered" ``` *Since 0.1.5.* ### `app-core.hides` [Section titled “app-core.hides”](#app-corehides) ```gherkin Then [[within {duration} ]]the {word} app does not show {string} ``` Check that the app does not show a text, or no longer does: it waits for the text to go (10 seconds, or `within`). | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} app does not show {string}` * `within {duration} the {word} app does not show {string}` **Example:** ```gherkin Then the courier app does not show "PX-MOB-9401" ``` *Since 0.1.5.* ### `app-core.control.shown` [Section titled “app-core.control.shown”](#app-corecontrolshown) ```gherkin Then [[within {duration} ]]the {string} {control} is shown in the {word} app ``` Check that the app shows a control: it waits for it (10 seconds, or `within`). | Parameter | Takes | Values | For example | | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | | `"Get a quote"`, `'Express'` | | `{control}` | a kind of control an app shows, found by the name people see: a `button` by its text, a `field` by its label, a `list item` or a `row` by one of its texts, a `menu item` in its open menu, an `element` is anything, by its text or accessibility label; a phone’s `notification` is tapped, by a text in its title or its text. A kind an app’s platform does not have (a phone has no menu bar) fails the step | `button`, `field`, `checkbox`, `radio button`, `switch`, `tab`, `menu`, `menu item`, `list item`, `row`, `link`, `image`, `text`, `element`, `notification` | `button`, `list item` | | `{word}` | one word, with no spaces | | `parcels`, `PX-4101` | **Variants**, the parts in `[[...]]` said or left out: * `the {string} {control} is shown in the {word} app` * `within {duration} the {string} {control} is shown in the {word} app` **Example:** ```gherkin Then the "Mark delivered" button is shown in the courier app ``` *Since 0.1.5.* ### `app-core.control.enabled` [Section titled “app-core.control.enabled”](#app-corecontrolenabled) ```gherkin Then [[within {duration} ]]the {string} {control} is enabled in the {word} app ``` Check that a control can be used: it waits for it (10 seconds, or `within`). | Parameter | Takes | Values | For example | | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | | `"Get a quote"`, `'Express'` | | `{control}` | a kind of control an app shows, found by the name people see: a `button` by its text, a `field` by its label, a `list item` or a `row` by one of its texts, a `menu item` in its open menu, an `element` is anything, by its text or accessibility label; a phone’s `notification` is tapped, by a text in its title or its text. A kind an app’s platform does not have (a phone has no menu bar) fails the step | `button`, `field`, `checkbox`, `radio button`, `switch`, `tab`, `menu`, `menu item`, `list item`, `row`, `link`, `image`, `text`, `element`, `notification` | `button`, `list item` | | `{word}` | one word, with no spaces | | `parcels`, `PX-4101` | **Variants**, the parts in `[[...]]` said or left out: * `the {string} {control} is enabled in the {word} app` * `within {duration} the {string} {control} is enabled in the {word} app` **Example:** ```gherkin Then the "Sign in" button is enabled in the courier app ``` *Since 0.1.5.* ### `app-core.control.disabled` [Section titled “app-core.control.disabled”](#app-corecontroldisabled) ```gherkin Then [[within {duration} ]]the {string} {control} is disabled in the {word} app ``` Check that a control is shown but cannot be used, like a button until a form is complete. | Parameter | Takes | Values | For example | | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | | `"Get a quote"`, `'Express'` | | `{control}` | a kind of control an app shows, found by the name people see: a `button` by its text, a `field` by its label, a `list item` or a `row` by one of its texts, a `menu item` in its open menu, an `element` is anything, by its text or accessibility label; a phone’s `notification` is tapped, by a text in its title or its text. A kind an app’s platform does not have (a phone has no menu bar) fails the step | `button`, `field`, `checkbox`, `radio button`, `switch`, `tab`, `menu`, `menu item`, `list item`, `row`, `link`, `image`, `text`, `element`, `notification` | `button`, `list item` | | `{word}` | one word, with no spaces | | `parcels`, `PX-4101` | **Variants**, the parts in `[[...]]` said or left out: * `the {string} {control} is disabled in the {word} app` * `within {duration} the {string} {control} is disabled in the {word} app` **Example:** ```gherkin Then the "Sign in" button is disabled in the courier app ``` *Since 0.1.5.* ### `app-core.field.value` [Section titled “app-core.field.value”](#app-corefieldvalue) ```gherkin Then [[within {duration} ]]the {string} field in the {word} app has the value {string} ``` Check what a field holds: it waits for the value (10 seconds, or `within`). A password field holds its dots. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Variants**, the parts in `[[...]]` said or left out: * `the {string} field in the {word} app has the value {string}` * `within {duration} the {string} field in the {word} app has the value {string}` **Example:** ```gherkin Then the "Courier ID" field in the courier app has the value "CR-LEJ-12" ``` *Since 0.1.5.* ### `app-core.screenshot` [Section titled “app-core.screenshot”](#app-corescreenshot) ```gherkin Then [[within {duration} ]]the {word} app looks like the {string} screenshot ``` Check that the app looks like its screenshot, pixel by pixel (anti-aliasing aside), once it has settled. * Each platform draws apps its own way, so each device or desktop has its own screenshot, named after it (`delivered.android-parcels-pixel.png`); see the platform’s family pack for where they are kept. * Without one, the step takes it and fails: look at it, and keep it. * When the app looks different, the step attaches the screenshot, the app and their difference. * The check waits for the app to settle and look right: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} app looks like the {string} screenshot` * `within {duration} the {word} app looks like the {string} screenshot` **Example:** ```gherkin Then the courier app looks like the "delivered" screenshot ``` *Since 0.1.5.* ## Parameter types [Section titled “Parameter types”](#parameter-types) | Parameter | Takes | Values | For example | | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- | | `{control}` | a kind of control an app shows, found by the name people see: a `button` by its text, a `field` by its label, a `list item` or a `row` by one of its texts, a `menu item` in its open menu, an `element` is anything, by its text or accessibility label; a phone’s `notification` is tapped, by a text in its title or its text. A kind an app’s platform does not have (a phone has no menu bar) fails the step | `button`, `field`, `checkbox`, `radio button`, `switch`, `tab`, `menu`, `menu item`, `list item`, `row`, `link`, `image`, `text`, `element`, `notification` | `button`, `list item` | # AsyncAPI > Check messages against an AsyncAPI document: every message a scenario sends, which catches test data that drifted from the contract, and every message a check finds, which catches a service that broke it. Check messages against an AsyncAPI document: every message a scenario sends, which catches test data that drifted from the contract, and every message a check finds, which catches a service that broke it. A broker’s registration names the document in an `asyncapi` row, a file of the project or a URL: ```gherkin Given the parcels mqtt broker with the following properties: | url | mqtt://localhost:1883 | | asyncapi | asyncapi.yaml | ``` The row works on every messaging registration: Kafka services, AMQP, MQTT and NATS brokers, AWS accounts (SQS and SNS), Google Cloud projects (Pub/Sub), Service Bus namespaces, WebSockets and event streams. Without this pack in `axx-packs.yaml`, the row fails the registration. * **Documents:** AsyncAPI 2.x (2.6 and earlier) and 3.x, in YAML or JSON, with `$ref`s to other files and URLs. * **Channels:** a message belongs to the channel whose address is the message’s topic, queue, subject, exchange or routing key, or the path of a WebSocket or an event stream. A `{parameter}` in an address stands for any text without a slash: `depots/{depot}/scans` is the channel of `depots/LEJ/scans`. When channels say which servers they are on, only the servers of the message’s protocol count. * **Messages:** a message must be one of its channel’s: its payload and headers follow that message’s schemas, and its content type, when it has one, is the message’s. When a channel has several messages, the failure names the one the message comes closest to. * **Schemas:** JSON Schema (draft 7 unless the schema says otherwise, and the AsyncAPI schema format), or Avro, whose payloads are in Avro’s JSON encoding. Payloads are JSON unless their content type says YAML or text. Findings have keys and levels, like the REST pack’s OpenAPI findings: | Key | Found when | | --------------------------------------------- | ------------------------------------------------------------------------------------------------- | | `validation.channel.unknown` | the document has no channel for the message | | `validation.message.contentType` | the message’s content type is not its message’s | | `validation.message.payload.format` | the payload is not JSON (or YAML) | | `validation.message.payload.schema.` | the payload breaks a JSON Schema keyword (`required`, `type`, `enum`…), or `avro` its Avro schema | | `validation.message.headers.schema.` | the headers break a keyword of their schema | * **Levels:** `ERROR` (or `FAIL`) fails the step, `WARN` and `INFO` log the finding, `IGNORE` drops it. Every finding is an `ERROR` unless `packs.asyncapi.levels` in axx.yaml, or the scenario’s `the AsyncAPI validation levels are:`, says otherwise. * **Keys cover the keys below them:** `validation.message.payload` sets every payload finding; the most specific key set wins. Add it to a project with `axx pack add asyncapi`. ## Settings [Section titled “Settings”](#settings) In `axx.yaml`, under `packs.asyncapi`: | Setting | Takes | Values | | -------- | ---------------------------------------------------------------------------- | ---------------- | | `levels` | Validation key -> ERROR (or FAIL), WARN, INFO or IGNORE, for every scenario. | names and values | ## Steps [Section titled “Steps”](#steps) ### `asyncapi.levels` [Section titled “asyncapi.levels”](#asyncapilevels) ```gherkin Given the AsyncAPI validation levels are: | validation key | level | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Set the level of AsyncAPI validation findings for this scenario, on every contract it checks messages against. * A key also sets the keys below it: `validation.message.payload` relaxes `validation.message.payload.schema.required` too. The most specific key set wins. * The rows are merged over `packs.asyncapi.levels` of axx.yaml. * The pack’s documentation lists the keys, and what each level does. A row is a validation key and its level: `ERROR` (or `FAIL`), `WARN`, `INFO` or `IGNORE`. The key must be one the pack reports, or a prefix of such keys, like `validation.message.payload`: any other fails the step, which names the closest keys. **Example:** ```gherkin Given the AsyncAPI validation levels are: | validation.message.payload.schema.required | WARN | ``` *Since 0.1.5.* # Core > The AWS account the aws-* packs talk to, set up the way the AWS SDK is set up for the real services. The AWS account the aws-\* packs talk to, set up the way the AWS SDK is set up for the real services. Register the account once; every aws-\* step of the scenario uses it: ```gherkin Given the parcels aws account with the following properties: | region | eu-west-1 | | endpoint | http://localhost:4566 | ``` Without credentials in the table, the SDK finds them as it always does: the `AWS_*` environment variables, the shared files, then the container or instance role. `AWS_ENDPOINT_URL` is honored too, so the same features run against AWS and against an emulator. Values expand `${env:..}` and `${sys:..}`. Add it to a project with `axx pack add aws-core`. ## Steps [Section titled “Steps”](#steps) ### `aws-core.account` [Section titled “aws-core.account”](#aws-coreaccount) ```gherkin Given the {word} aws account with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register the AWS account the aws-\* steps talk to. * The first account registered is the default. * Without `access key id` and `secret access key`, the SDK finds credentials as it always does: the `AWS_*` environment variables, the shared files, then the container or instance role. * Values expand `${env:..}` and `${sys:..}`, so the same features run against AWS and against an emulator. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Default | | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `region` | the AWS region, like `eu-west-1` | *required* | | `endpoint` | where every service of the account is, instead of AWS: a local emulator, like `http://localhost:4566`, where S3 is addressed by path | | | `profile` | a profile of the shared AWS config and credentials files | | | `access key id` | the access key ID of static credentials, with `secret access key` | | | `secret access key` | the secret access key of static credentials, with `access key id` | | | `session token` | the session token of temporary static credentials | | | `asyncapi` | an AsyncAPI document, a file of the project or a URL, that the SQS queues and SNS topics follow: every message a scenario sends there, and every message a check finds, is checked against it (with the asyncapi pack) | | **Example:** ```gherkin Given the parcels aws account with the following properties: | region | eu-west-1 | | endpoint | http://localhost:4566 | | access key id | ${env:AWS_ACCESS_KEY_ID:-local} | | secret access key | ${env:AWS_SECRET_ACCESS_KEY:-local} | Given the parcels aws account with the following properties: | region | eu-central-1 | | profile | parcels-tests | ``` *Since 0.1.0.* # DynamoDB > Seed DynamoDB tables and check the items your services write. Seed DynamoDB tables and check the items your services write. The steps use the scenario’s AWS account (`the {word} aws account with the following properties:`, from aws-core). A **seed** is a YAML or JSON file that maps table names to the items to put, written as plain JSON (numbers become `N`, objects `M`, arrays `L`): ```yaml insured-parcels: - reference: PX-CLM-1001 declaredValue: 120 carrier: KESTREL ``` **Checks** wait (10 seconds unless `within {duration}` says otherwise) until the table has an item, or a number of items, meeting every condition: `attribute | value` rows with a dotted path into maps (`address.city`), compared as text, `null` for null and `undefined` for absent. Items are read with a scan, which suits the small tables of a test environment. Add it to a project with `axx pack add aws-dynamodb`; it builds on [`aws-core`](/next/references/packs/aws-core/), which comes with it. ## Steps [Section titled “Steps”](#steps) ### `aws-dynamodb.seed` [Section titled “aws-dynamodb.seed”](#aws-dynamodbseed) ```gherkin Given a {filepath} dynamodb seed ``` Put the items of a seed file into their tables: YAML or JSON that maps table names to lists of items. Items are plain JSON: numbers become `N`, objects `M` and arrays `L`. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | **Example:** ```gherkin Given a seeds/insured-parcels.yaml dynamodb seed ``` *Since 0.1.0.* ### `aws-dynamodb.item` [Section titled “aws-dynamodb.item”](#aws-dynamodbitem) ```gherkin Then [[within {duration} ]]the {word} dynamodb table has an item where: | attribute | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the table has an item with those values in those attributes. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row names an attribute of the item, or a dotted path into a map (`address.city`), and the value it has, compared as text: `null` for null and `undefined` for absent. Sets read as lists, and binary as base64. **Variants**, the parts in `[[...]]` said or left out: * `the {word} dynamodb table has an item where:` * `within {duration} the {word} dynamodb table has an item where:` **Example:** ```gherkin Then within 30s the claims dynamodb table has an item where: | id | CLM-4101 | | status | APPROVED | | amount | 89.5 | ``` *Since 0.1.0.* ### `aws-dynamodb.items` [Section titled “aws-dynamodb.items”](#aws-dynamodbitems) ```gherkin Then [[within {duration} ]]the {word} dynamodb table has {int} item(s) where: | attribute | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that exactly that many items of the table have those values in those attributes. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{int}` | a whole number | `200`, `3` | Each row names an attribute of the item, or a dotted path into a map (`address.city`), and the value it has, compared as text: `null` for null and `undefined` for absent. Sets read as lists, and binary as base64. **Variants**, the parts in `[[...]]` said or left out: * `the {word} dynamodb table has {int} item(s) where:` * `within {duration} the {word} dynamodb table has {int} item(s) where:` **Example:** ```gherkin Then the claims dynamodb table has 1 item where: | parcel | PX-4107 | ``` *Since 0.1.0.* # EventBridge > Put events on EventBridge buses and check the events your services put there. Put events on EventBridge buses and check the events your services put there. The steps use the scenario’s AWS account (`the {word} aws account with the following properties:`, from aws-core). An event has a detail type, a source and a JSON detail, as in `When a "Damage Reported" event from carrier.kestrel is put on the carrier-events eventbridge bus:` with the detail as the doc string. **Checking a bus** does not take events from anyone: for the buses a run checks, Axx adds a rule of its own (`axx--`, matching every event of the account) with a queue of its own as its target once the services are up, and removes both when the run ends. The conditions are paths into the event as EventBridge delivers it: `detail-type`, `source`, and `detail.` for the detail. A check only looks at the events received since its scenario started. Add it to a project with `axx pack add aws-eventbridge`; it builds on [`aws-core`](/next/references/packs/aws-core/), which comes with it. ## Steps [Section titled “Steps”](#steps) ### `aws-eventbridge.put` [Section titled “aws-eventbridge.put”](#aws-eventbridgeput) ```gherkin When a(n) {string} event from {word} is put on the {word} eventbridge bus: """ ... """ ``` Put an event with that detail type and source on the bus; the doc string is its detail, in JSON. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When a "Damage Reported" event from carrier.kestrel is put on the carrier-events eventbridge bus: """ {"parcel": "PX-4105", "carrier": "KESTREL", "note": "Crushed at the depot"} """ ``` *Since 0.1.0.* ### `aws-eventbridge.received` [Section titled “aws-eventbridge.received”](#aws-eventbridgereceived) ```gherkin Then [[within {duration} ]]the {word} eventbridge bus has an event where: | path | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the bus has an event with those values, put since the scenario started. * The check waits for it: 10 seconds, or `within {duration}`. * Axx adds a rule of its own to the bus for the run. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Path | Takes | | ---------------- | ---------------------------------------------------------------------------- | | `detail-type` | the event’s detail type, like `Claim Decided` | | `source` | the event’s source, like `parcels.claims` | | `detail.` | a field of the event’s detail, or a dotted path into it, like `detail.claim` | Values compare as text: `null` for null and `undefined` for absent. Other paths into the event, as EventBridge delivers it, work too, like `account` or `region`. **Variants**, the parts in `[[...]]` said or left out: * `the {word} eventbridge bus has an event where:` * `within {duration} the {word} eventbridge bus has an event where:` **Example:** ```gherkin Then within 30s the parcels eventbridge bus has an event where: | detail-type | Claim Decided | | source | parcels.claims | | detail.claim | CLM-4101 | | detail.decision | APPROVED | ``` *Since 0.1.0.* # S3 > Upload files to S3 buckets and check the objects your services write there. Upload files to S3 buckets and check the objects your services write there. The steps use the scenario’s AWS account (`the {word} aws account with the following properties:`, from aws-core). Checks wait for the object (10 seconds unless `within {duration}` says otherwise), since services write asynchronously: an upload that triggers processing (an S3 event notification to a queue, say) and the object that processing writes. Add it to a project with `axx pack add aws-s3`; it builds on [`aws-core`](/next/references/packs/aws-core/), which comes with it. ## Steps [Section titled “Steps”](#steps) ### `aws-s3.upload` [Section titled “aws-s3.upload”](#aws-s3upload) ```gherkin When the {filepath} file is uploaded to the {word} s3 bucket[[ as {path}]] ``` Upload a file to the s3 bucket, under the file’s name or the name given. Its content type follows the file’s extension. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"` | **Variants**, the parts in `[[...]]` said or left out: * `the {filepath} file is uploaded to the {word} s3 bucket` * `the {filepath} file is uploaded to the {word} s3 bucket as {path}` **Example:** ```gherkin When the invoices/kestrel-2026-09.csv file is uploaded to the carrier-drops s3 bucket When the invoices/kestrel-2026-09.csv file is uploaded to the carrier-drops s3 bucket as incoming/kestrel-2026-09.csv ``` *Since 0.1.0.* ### `aws-s3.has` [Section titled “aws-s3.has”](#aws-s3has) ```gherkin Then [[within {duration} ]]the {word} s3 bucket has a(n) object named {path} ``` Check that the s3 bucket has an object with that name. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} s3 bucket has a(n) object named {path}` * `within {duration} the {word} s3 bucket has a(n) object named {path}` **Example:** ```gherkin Then within 30s the carrier-drops s3 bucket has an object named disputes/kestrel-2026-09.csv ``` *Since 0.1.0.* ### `aws-s3.identical` [Section titled “aws-s3.identical”](#aws-s3identical) ```gherkin Then [[within {duration} ]]the {path} object in the {word} s3 bucket is identical to the {filepath} file ``` Check that the object exists with exactly the content of the file. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | **Variants**, the parts in `[[...]]` said or left out: * `the {path} object in the {word} s3 bucket is identical to the {filepath} file` * `within {duration} the {path} object in the {word} s3 bucket is identical to the {filepath} file` **Example:** ```gherkin Then the disputes/kestrel-2026-09.csv object in the carrier-drops s3 bucket is identical to the expected/kestrel-disputes.csv file ``` *Since 0.1.0.* ### `aws-s3.properties` [Section titled “aws-s3.properties”](#aws-s3properties) ```gherkin Then [[within {duration} ]]the {path} object in the {word} s3 bucket has the following properties: | path | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the object holds JSON with those values at those paths. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a path into the JSON (a property name, a dotted path or a JSONPath) and the value it has, compared as text: `null` for null and `undefined` for absent, as in the other JSON property steps. **Variants**, the parts in `[[...]]` said or left out: * `the {path} object in the {word} s3 bucket has the following properties:` * `within {duration} the {path} object in the {word} s3 bucket has the following properties:` **Example:** ```gherkin Then the summaries/kestrel-2026-09.json object in the carrier-drops s3 bucket has the following properties: | carrier | KESTREL | | lines | 14 | | totals.disputed | 5.25 | ``` *Since 0.1.0.* ### `aws-s3.contains` [Section titled “aws-s3.contains”](#aws-s3contains) ```gherkin Then [[within {duration} ]]the {path} object in the {word} s3 bucket contains {string} ``` Check that the text of the object contains the text. * The check waits for it: 10 seconds, or `within {duration}`. * Case matters; runs of spaces and line breaks count as one space. * The text is read by the object’s type: the text of a PDF’s pages, the paragraphs and tables of a Word document (.docx), the cells of every sheet of an Excel workbook (.xlsx), the text content (or the markup) of XML and HTML, or the text itself. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {path} object in the {word} s3 bucket contains {string}` * `within {duration} the {path} object in the {word} s3 bucket contains {string}` **Example:** ```gherkin Then the invoices/kestrel-2026-09.pdf object in the carrier-drops s3 bucket contains "Total due: 1284.50 EUR" ``` *Since 0.1.1.* ### `aws-s3.row` [Section titled “aws-s3.row”](#aws-s3row) ```gherkin Then [[within {duration} ]]the {path} object in the {word} s3 bucket has a row where: | column | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the table of the object has a row with those values in those columns. * The check waits for it: 10 seconds, or `within {duration}`. * The object is a CSV or TSV file, or an Excel workbook (.xlsx; its first sheet), whose first row names the columns. * Cells compare as text, as the workbook shows them, with runs of spaces as one space; an empty value matches an empty cell. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row names a column, as the object’s first row names it, and the value in that column. **Variants**, the parts in `[[...]]` said or left out: * `the {path} object in the {word} s3 bucket has a row where:` * `within {duration} the {path} object in the {word} s3 bucket has a row where:` **Example:** ```gherkin Then within 30s the disputes/kestrel-2026-09.csv object in the carrier-drops s3 bucket has a row where: | parcel | PX-5199 | | status | UNKNOWN_SHIPMENT | | billed | 4.10 | ``` *Since 0.1.1.* # SNS > Publish messages to SNS topics and check the messages your services publish. Publish messages to SNS topics and check the messages your services publish. The steps use the scenario’s AWS account (`the {word} aws account with the following properties:`, from aws-core). **Checking a topic** does not take messages from anyone: for the topics a run checks, Axx subscribes a queue of its own (`axx--`, with raw message delivery) once the services are up, and removes the subscription and the queue when the run ends. A check only looks at the messages received since its scenario started; match on data unique to the scenario, since scenarios run in parallel. Add it to a project with `axx pack add aws-sns`; it builds on [`aws-core`](/next/references/packs/aws-core/), which comes with it. ## Steps [Section titled “Steps”](#steps) ### `aws-sns.send` [Section titled “aws-sns.send”](#aws-snssend) ```gherkin When a message is published to the {word} sns topic: """ ... """ ``` Send a message to the sns topic; the doc string is its body. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When a message is published to the parcel-events sns topic: """ {"parcel": "PX-4104", "status": "DELIVERED", "deliveredAt": "2026-09-20T14:05:00Z"} """ ``` *Since 0.1.0.* ### `aws-sns.send.file` [Section titled “aws-sns.send.file”](#aws-snssendfile) ```gherkin When the {filepath} message is published to the {word} sns topic[[ with the following attributes:]] ``` Send a message whose body is the file to the sns topic, with the attributes of the table when the step has one. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is an attribute sent with the message, as text: its name and its value. **Variants**, the parts in `[[...]]` said or left out: * `the {filepath} message is published to the {word} sns topic` * `the {filepath} message is published to the {word} sns topic with the following attributes:` **Example:** ```gherkin When the messages/px-4104-delivered.json message is published to the parcel-events sns topic When the messages/px-4104-delivered.json message is published to the parcel-events sns topic with the following attributes: | eventType | ParcelDelivered | ``` *Since 0.1.0.* ### `aws-sns.received` [Section titled “aws-sns.received”](#aws-snsreceived) ```gherkin Then [[within {duration} ]]the {word} sns topic has a message where: | path | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the sns topic has a message with those values, received since the scenario started. * The check waits for it: 10 seconds, or `within {duration}`. * Axx subscribes a queue of its own to the topic for the run. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a path into the message’s JSON body (a field name, a dotted path or a JSONPath) and the value it has, compared as text: `null` for null and `undefined` for absent. A row `attribute ` is instead on the attribute of that name sent with the message: `undefined` when there is none. **Variants**, the parts in `[[...]]` said or left out: * `the {word} sns topic has a message where:` * `within {duration} the {word} sns topic has a message where:` **Example:** ```gherkin Then within 30s the claim-decisions sns topic has a message where: | claim | CLM-4101 | | decision | APPROVED | | attribute eventType | ClaimDecided | ``` *Since 0.1.0.* # SQS > Send messages to SQS queues and check the messages your services send to them. Send messages to SQS queues and check the messages your services send to them. The steps use the scenario’s AWS account (`the {word} aws account with the following properties:`, from aws-core). **Checking a queue receives from it**: Axx takes each message off the queue, as any consumer would. Check the queues your services **write** to (an outbound queue another system reads); a queue your service consumes is checked by what the service does with the messages. Axx starts receiving from the queues a run checks once the services are up, and a check only looks at the messages received since its scenario started. Match on data unique to the scenario: scenarios run in parallel and share the queue. Add it to a project with `axx pack add aws-sqs`; it builds on [`aws-core`](/next/references/packs/aws-core/), which comes with it. ## Steps [Section titled “Steps”](#steps) ### `aws-sqs.send` [Section titled “aws-sqs.send”](#aws-sqssend) ```gherkin When a message is sent to the {word} sqs queue: """ ... """ ``` Send a message to the sqs queue; the doc string is its body. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When a message is sent to the refund-results sqs queue: """ {"claim": "CLM-4106", "status": "PAID", "paidAt": "2026-09-24T10:00:00Z"} """ ``` *Since 0.1.0.* ### `aws-sqs.send.file` [Section titled “aws-sqs.send.file”](#aws-sqssendfile) ```gherkin When the {filepath} message is sent to the {word} sqs queue[[ with the following attributes:]] ``` Send a message whose body is the file to the sqs queue, with the attributes of the table when the step has one. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is an attribute sent with the message, as text: its name and its value. **Variants**, the parts in `[[...]]` said or left out: * `the {filepath} message is sent to the {word} sqs queue` * `the {filepath} message is sent to the {word} sqs queue with the following attributes:` **Example:** ```gherkin When the messages/refund-failed-clm-4109.json message is sent to the refund-results sqs queue When the messages/refund-failed-clm-4109.json message is sent to the refund-results sqs queue with the following attributes: | failureReason | ACCOUNT_CLOSED | ``` *Since 0.1.0.* ### `aws-sqs.received` [Section titled “aws-sqs.received”](#aws-sqsreceived) ```gherkin Then [[within {duration} ]]the {word} sqs queue has a message where: | path | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the sqs queue has a message with those values, received since the scenario started. * The check waits for it: 10 seconds, or `within {duration}`. * Axx takes the queue’s messages off it: check queues your services write to. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a path into the message’s JSON body (a field name, a dotted path or a JSONPath) and the value it has, compared as text: `null` for null and `undefined` for absent. A row `attribute ` is instead on the attribute of that name sent with the message: `undefined` when there is none. **Variants**, the parts in `[[...]]` said or left out: * `the {word} sqs queue has a message where:` * `within {duration} the {word} sqs queue has a message where:` **Example:** ```gherkin Then within 30s the refund-requests sqs queue has a message where: | claim | CLM-4101 | | amount | 89.5 | | attribute reason | DAMAGED | ``` *Since 0.1.0.* # Blob Storage > Upload files to Blob Storage containers and check the blobs your services write there. Upload files to Blob Storage containers and check the blobs your services write there. Register the storage account once, set up as the Azure SDK is set up for the real service; every blob step of the scenario uses it: ```gherkin Given the customs azure storage account with the following properties: | connection string | ${env:AZURE_STORAGE_CONNECTION_STRING} | ``` Checks wait for the blob (10 seconds unless `within {duration}` says otherwise), since services write asynchronously. Values expand `${env:..}` and `${sys:..}`. Add it to a project with `axx pack add azure-blob`. ## Steps [Section titled “Steps”](#steps) ### `azure-blob.account` [Section titled “azure-blob.account”](#azure-blobaccount) ```gherkin Given the {word} azure storage account with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register the storage account the blob steps talk to. * The first account registered is the default. * Give it a `connection string` or a `url`, not both. * Values expand `${env:..}` and `${sys:..}`. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `connection string` | the account’s connection string, or a local emulator’s, like `${env:AZURE_STORAGE_CONNECTION_STRING}` | | `url` | the account’s blob endpoint, like `https://.blob.core.windows.net`, signed in with the Azure default credential chain: environment, workload identity, managed identity, Azure CLI | **Example:** ```gherkin Given the customs azure storage account with the following properties: | connection string | ${env:AZURE_STORAGE_CONNECTION_STRING} | Given the customs azure storage account with the following properties: | url | ${env:CUSTOMS_STORAGE_URL} | ``` *Since 0.1.0.* ### `azure-blob.upload` [Section titled “azure-blob.upload”](#azure-blobupload) ```gherkin When the {filepath} file is uploaded to the {word} blob container[[ as {path}]] ``` Upload a file to the blob container, under the file’s name or the name given. Its content type follows the file’s extension. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"` | **Variants**, the parts in `[[...]]` said or left out: * `the {filepath} file is uploaded to the {word} blob container` * `the {filepath} file is uploaded to the {word} blob container as {path}` **Example:** ```gherkin When the invoices/kestrel-2026-09.csv file is uploaded to the declarations blob container When the invoices/kestrel-2026-09.csv file is uploaded to the declarations blob container as incoming/kestrel-2026-09.csv ``` *Since 0.1.0.* ### `azure-blob.has` [Section titled “azure-blob.has”](#azure-blobhas) ```gherkin Then [[within {duration} ]]the {word} blob container has a(n) blob named {path} ``` Check that the blob container has a blob with that name. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} blob container has a(n) blob named {path}` * `within {duration} the {word} blob container has a(n) blob named {path}` **Example:** ```gherkin Then within 30s the declarations blob container has a blob named disputes/kestrel-2026-09.csv ``` *Since 0.1.0.* ### `azure-blob.identical` [Section titled “azure-blob.identical”](#azure-blobidentical) ```gherkin Then [[within {duration} ]]the {path} blob in the {word} blob container is identical to the {filepath} file ``` Check that the blob exists with exactly the content of the file. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | **Variants**, the parts in `[[...]]` said or left out: * `the {path} blob in the {word} blob container is identical to the {filepath} file` * `within {duration} the {path} blob in the {word} blob container is identical to the {filepath} file` **Example:** ```gherkin Then the disputes/kestrel-2026-09.csv blob in the declarations blob container is identical to the expected/kestrel-disputes.csv file ``` *Since 0.1.0.* ### `azure-blob.properties` [Section titled “azure-blob.properties”](#azure-blobproperties) ```gherkin Then [[within {duration} ]]the {path} blob in the {word} blob container has the following properties: | path | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the blob holds JSON with those values at those paths. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a path into the JSON (a property name, a dotted path or a JSONPath) and the value it has, compared as text: `null` for null and `undefined` for absent, as in the other JSON property steps. **Variants**, the parts in `[[...]]` said or left out: * `the {path} blob in the {word} blob container has the following properties:` * `within {duration} the {path} blob in the {word} blob container has the following properties:` **Example:** ```gherkin Then the summaries/kestrel-2026-09.json blob in the declarations blob container has the following properties: | carrier | KESTREL | | lines | 14 | | totals.disputed | 5.25 | ``` *Since 0.1.0.* ### `azure-blob.contains` [Section titled “azure-blob.contains”](#azure-blobcontains) ```gherkin Then [[within {duration} ]]the {path} blob in the {word} blob container contains {string} ``` Check that the text of the blob contains the text. * The check waits for it: 10 seconds, or `within {duration}`. * Case matters; runs of spaces and line breaks count as one space. * The text is read by the blob’s type: the text of a PDF’s pages, the paragraphs and tables of a Word document (.docx), the cells of every sheet of an Excel workbook (.xlsx), the text content (or the markup) of XML and HTML, or the text itself. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {path} blob in the {word} blob container contains {string}` * `within {duration} the {path} blob in the {word} blob container contains {string}` **Example:** ```gherkin Then the invoices/kestrel-2026-09.pdf blob in the declarations blob container contains "Total due: 1284.50 EUR" ``` *Since 0.1.1.* ### `azure-blob.row` [Section titled “azure-blob.row”](#azure-blobrow) ```gherkin Then [[within {duration} ]]the {path} blob in the {word} blob container has a row where: | column | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the table of the blob has a row with those values in those columns. * The check waits for it: 10 seconds, or `within {duration}`. * The blob is a CSV or TSV file, or an Excel workbook (.xlsx; its first sheet), whose first row names the columns. * Cells compare as text, as the workbook shows them, with runs of spaces as one space; an empty value matches an empty cell. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row names a column, as the blob’s first row names it, and the value in that column. **Variants**, the parts in `[[...]]` said or left out: * `the {path} blob in the {word} blob container has a row where:` * `within {duration} the {path} blob in the {word} blob container has a row where:` **Example:** ```gherkin Then within 30s the disputes/kestrel-2026-09.csv blob in the declarations blob container has a row where: | parcel | PX-5199 | | status | UNKNOWN_SHIPMENT | | billed | 4.10 | ``` *Since 0.1.1.* # Service Bus > Send messages to Service Bus queues and topics, and check the messages your services send there. Send messages to Service Bus queues and topics, and check the messages your services send there. Register the namespace once, set up as the Azure SDK is set up for the real service; every Service Bus step of the scenario uses it: ```gherkin Given the customs service bus namespace with the following properties: | connection string | ${env:SERVICEBUS_CONNECTION_STRING} | ``` Steps name a **queue** or a **topic**: `the customs-filings service bus queue`, `the customs-events service bus topic`. Messages carry application properties (`with the following properties:`, and `property ` rows in checks). **Checking a queue receives from it**: Axx completes each message it receives, as any consumer would, so check the queues your services **write** to. **Checking a topic** takes nothing from anyone: for the topics a run checks, Axx creates a subscription of its own (`axx-`, through the management API, so it needs the Manage right) once the services are up, and deletes it when the run ends. A check only looks at the messages received since its scenario started. Add it to a project with `axx pack add azure-servicebus`. ## Steps [Section titled “Steps”](#steps) ### `azure-servicebus.namespace` [Section titled “azure-servicebus.namespace”](#azure-servicebusnamespace) ```gherkin Given the {word} service bus namespace with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register the Service Bus namespace the steps talk to. * The first namespace registered is the default. * Give it a `connection string` or a `namespace`, not both. * Values expand `${env:..}` and `${sys:..}`. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `connection string` | the namespace’s connection string, or a local emulator’s (with `UseDevelopmentEmulator=true`), like `${env:SERVICEBUS_CONNECTION_STRING}` | | `namespace` | the fully qualified namespace, like `.servicebus.windows.net`, signed in with the Azure default credential chain | | `management endpoint` | where the namespace’s management API is when it is not the namespace’s own host, as with local emulators | | `asyncapi` | an AsyncAPI document, a file of the project or a URL, that the queues and topics follow: every message a scenario sends there, and every message a check finds, is checked against it (with the asyncapi pack) | **Example:** ```gherkin Given the customs service bus namespace with the following properties: | connection string | ${env:SERVICEBUS_CONNECTION_STRING} | Given the customs service bus namespace with the following properties: | namespace | ${env:SERVICEBUS_NAMESPACE} | ``` *Since 0.1.0.* ### `azure-servicebus.send` [Section titled “azure-servicebus.send”](#azure-servicebussend) ```gherkin When a message is sent to the {word} service bus queue:/topic: """ ... """ ``` Send a message to the service bus queue or topic; the doc string is its body. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When a message is sent to the customs-filings service bus queue: """ {"declaration": "DEC-7103", "parcel": "PX-7103", "invoice": "DEC-7103.json"} """ ``` *Since 0.1.0.* ### `azure-servicebus.send.file` [Section titled “azure-servicebus.send.file”](#azure-servicebussendfile) ```gherkin When the {filepath} message is sent to the {word} service bus queue/topic[[ with the following properties:]] ``` Send a message whose body is the file to the service bus queue or topic, with the properties of the table when the step has one. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a property sent with the message, as text: its name and its value. **Variants**, the parts in `[[...]]` said or left out: * `the {filepath} message is sent to the {word} service bus queue/topic` * `the {filepath} message is sent to the {word} service bus queue/topic with the following properties:` **Example:** ```gherkin When the messages/filing-DEC-7101.json message is sent to the customs-filings service bus queue When the messages/filing-DEC-7101.json message is sent to the customs-filings service bus queue with the following properties: | broker | ACME-CUSTOMS | ``` *Since 0.1.0.* ### `azure-servicebus.received` [Section titled “azure-servicebus.received”](#azure-servicebusreceived) ```gherkin Then [[within {duration} ]]the {word} service bus queue/topic has a message where: | path | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the service bus queue or topic has a message with those values, received since the scenario started. * The check waits for it: 10 seconds, or `within {duration}`. * Axx completes the messages of a queue it checks, and subscribes to a topic for the run. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a path into the message’s JSON body (a field name, a dotted path or a JSONPath) and the value it has, compared as text: `null` for null and `undefined` for absent. A row `property ` is instead on the property of that name sent with the message: `undefined` when there is none. **Variants**, the parts in `[[...]]` said or left out: * `the {word} service bus queue/topic has a message where:` * `within {duration} the {word} service bus queue/topic has a message where:` **Example:** ```gherkin Then within 30s the duty-payments service bus queue has a message where: | declaration | DEC-7102 | | amount | 96 | | currency | EUR | | property broker | ACME-CUSTOMS | ``` *Since 0.1.0.* # Commands > Run commands, such as a command-line tool you ship or an operator's admin command, and check how they end and what they print. Run commands, such as a command-line tool you ship or an operator’s admin command, and check how they end and what they print. Register a command with `the {word} command with the following properties:`: the program and its first arguments, split like an app’s `command` in axx.yaml (double quotes group words; there is no shell), the folder it runs in, its environment and its timeout. Then run it, with more arguments and, if it reads them, lines of input, and check its exit code and its output (what it prints) or its error output (what it prints to stderr). * **A command runs in a folder of its own for each scenario**, unless its `dir` says otherwise, so parallel scenarios never write over each other’s files. The folder is removed when the scenario passes and kept when it fails. * **A program named by a relative path** (`bin/parcels`) is found from the directory of axx.yaml; a bare name (`psql`) on the PATH. * **The checks look at the command’s last run** in the scenario. A command that exits with a failing code does not fail the step that runs it: check its exit code. Only a command that cannot start, or runs past its timeout, fails the run step, and it shows what the command printed. * **Secrets stay secret:** `${env:..}` values in the properties, the arguments and the input are masked in logs, attachments and failures. * **Stopping:** a command that runs past its timeout is stopped with everything it started. A command that runs a process elsewhere, such as `docker compose exec`, stops, but the process it started may run on. Add it to a project with `axx pack add cli`. ## Steps [Section titled “Steps”](#steps) ### `cli.command` [Section titled “cli.command”](#clicommand) ```gherkin Given the {word} command with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register a command under a name: the program and its first arguments, where it runs, its environment and its timeout. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Default | | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | | `command` | the program and its first arguments, split like a service’s `command` in axx.yaml: double quotes group words, and there is no shell. A relative path to the program is from the directory of axx.yaml | *required* | | `dir` | the folder it runs in, relative to the directory of axx.yaml | `a folder of the scenario's own` | | `env.` | an environment variable, set over Axx’s own environment | | | `timeout` | how long a run may take before the command is stopped, like `30s` | `1m` | Values are expanded (`${env:..}`, `${sys:..}`); what `${env:..}` references expand to is masked everywhere. **Example:** ```gherkin Given the admin command with the following properties: | command | docker compose exec -T app parcels admin | | dir | ../infra | ``` *Since 0.1.5.* ### `cli.run` [Section titled “cli.run”](#clirun) ```gherkin When the {word} command is run[[ with {string}]] ``` Run the command, with more arguments after its own, split the same way. The step waits for the command to end; it fails only when the command cannot start or runs past its timeout. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} command is run` * `the {word} command is run with {string}` **Example:** ```gherkin When the admin command is run with 'label PX-ADM-6101' ``` *Since 0.1.5.* ### `cli.run.input` [Section titled “cli.run.input”](#cliruninput) ```gherkin When the {word} command is run with[[ {string} and]] the input: """ ... """ ``` Run the command with the doc string as its input (stdin), and more arguments if the step names them. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} command is run with the input:` * `the {word} command is run with {string} and the input:` **Example:** ```gherkin When the admin command is run with 'cancel -' and the input: """ PX-ADM-6103 PX-ADM-6104 """ ``` *Since 0.1.5.* ### `cli.exit` [Section titled “cli.exit”](#cliexit) ```gherkin Then the {word} command's exit code is {int} ``` Check the exit code of the command’s last run. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{int}` | a whole number | `200`, `3` | **Example:** ```gherkin Then the admin command's exit code is 0 ``` *Since 0.1.5.* ### `cli.output.contains` [Section titled “cli.output.contains”](#clioutputcontains) ```gherkin Then the {word} command's[[ error]] output contains {string} ``` Check that the command’s output (or its error output) contains the text. Runs of spaces and line breaks count as one space, in both. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} command's output contains {string}` * `the {word} command's error output contains {string}` **Example:** ```gherkin Then the admin command's error output contains 'PX-ADM-6102 is DISPATCHED' ``` *Since 0.1.5.* ### `cli.output.line` [Section titled “cli.output.line”](#clioutputline) ```gherkin Then the {word} command's[[ error]] output has a line matching {string} ``` Check that a line of the command’s output (or its error output) matches the regular expression (Java syntax), anywhere in the line; `^` and `$` are the line’s start and end. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} command's output has a line matching {string}` * `the {word} command's error output has a line matching {string}` **Example:** ```gherkin Then the admin command's output has a line matching 'cancelled PX-ADM-610[34]' ``` *Since 0.1.5.* ### `cli.output.is` [Section titled “cli.output.is”](#clioutputis) ```gherkin Then the {word} command's[[ error]] output is: """ ... """ ``` Check the command’s output (or its error output) is exactly the doc string. Windows line breaks count as plain ones, and so does one line break at the end. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} command's output is:` * `the {word} command's error output is:` **Example:** ```gherkin Then the admin command's output is: """ cancelled PX-ADM-6103 cancelled PX-ADM-6104 """ ``` *Since 0.1.5.* ### `cli.output.empty` [Section titled “cli.output.empty”](#clioutputempty) ```gherkin Then the {word} command's[[ error]] output is empty ``` Check that the command printed nothing at all to its output (or its error output). | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} command's output is empty` * `the {word} command's error output is empty` **Example:** ```gherkin Then the admin command's error output is empty ``` *Since 0.1.5.* ### `cli.output.properties` [Section titled “cli.output.properties”](#clioutputproperties) ```gherkin Then the {word} command's output has the following properties: | path | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check the JSON the command printed: each row is a path into it (a field name, a dotted path or a JSONPath) and the value it has. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Values compare as text: `null` is null and `undefined` is absent. **Example:** ```gherkin Then the admin command's output has the following properties: | shop | kestrel-books | | parcels[0].reference | PX-ADM-6105 | | parcels[0].serviceLevel | EXPRESS | ``` *Since 0.1.5.* ### `cli.output.identical` [Section titled “cli.output.identical”](#clioutputidentical) ```gherkin Then the {word} command's output is identical to the {filepath} file ``` Check that the command’s output is, byte for byte, the file of the project. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | **Example:** ```gherkin Then the admin command's output is identical to the labels/PX-ADM-6101.zpl file ``` *Since 0.1.5.* # Core > Parameter types every pack can use. Parameter types every pack can use. Every project has it, and every pack builds on it. ## Parameter types [Section titled “Parameter types”](#parameter-types) | Parameter | Takes | Values | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------ | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | | `5s`, `2m` | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | | `seeds/parcels.yaml`, `kafka/scan-delivered.json`, `"seeds/day one.yaml"` | | `{mimeType}` | a content type | `application/json`, `text/json`, `application/problem+json`, `application/x-www-form-urlencoded` | `application/json` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | | `1st`, `2nd`, `3rd` | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"`, `snap-*.json` | | `{pattern}` | a regular expression (Java syntax) with no spaces, which matches the whole value | | `PX-\d{4}`, `[A-Z]{2}-\d+` | # Core > Desktop apps on macOS, Windows and Linux, used as people use them, through the operating system's accessibility tree: what screen readers read. Axx starts the app as it ships, with no build for testing, and clicks and types with the pointer and the keyboard, as a person does. Desktop apps on macOS, Windows and Linux, used as people use them, through the operating system’s accessibility tree: what screen readers read. Axx starts the app as it ships, with no build for testing, and clicks and types with the pointer and the keyboard, as a person does. An app is registered for each OS it runs on, by that OS’s pack (`desktop-macos`, `desktop-windows`, `desktop-linux`), and every other step is `app-core`’s: a feature lists the app once for each OS and runs unchanged on each. This pack adds what only desktops do: a key pressed, a click at a place on a control, a drag. **Each scenario has the desktop, and a clean app.** A desktop has one screen, one pointer and one keyboard focus, so on macOS and Windows desktop scenarios take the machine’s desktop in turns (other runs on the machine wait too), while other scenarios run alongside them; on Linux, Axx runs desktops of its own. Before an app starts in a scenario, its home (a folder of the project’s `.axx/desktop`) is emptied, and what its OS keeps of it elsewhere is reset; after the scenario, the app and every process it started are stopped. An app’s files are a folder of the files pack whose `owner` is the app: `./` is where its OS keeps an app’s data, `~/` its home. **Screenshots** are of the app’s front window, as it draws itself, one for each platform and display scale (`arrivals.darwin@2x.png`). A focused field’s text cursor is left out of them, as the web pack leaves it out of its screenshots: it blinks. Add it to a project with `axx pack add desktop-core`; it builds on [`app-core`](/next/references/packs/app-core/), which comes with it. ## Settings [Section titled “Settings”](#settings) In `axx.yaml`, under `packs.desktop-core`: | Setting | Takes | Values | Default | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------- | ------- | | `screenshots.folder` | The folder of the apps’ screenshots, in the project. Unset, each feature keeps its screenshots in a screenshots folder next to its file. | text | | | `screenshots.platforms` | The platforms whose screenshots the project keeps: each compares its own; elsewhere the steps pass without comparing (default: every platform). | a list of `linux`, `darwin`, `windows` | | | `screenshots.tolerance` | The share of a screenshot’s pixels that may differ. | a number, 0 to 1 | `0` | | `screenshots.update` | Take every screenshot again, as the app looks now, instead of comparing. | `true`, `false` | | | `traces` | Which scenarios keep a trace of each app in .axx/desktop/traces: a page with its window after each step, and its controls where the scenario failed. | `failed`, `always`, `never` | `never` | | `videos` | Which scenarios keep a video of their desktop in .axx/desktop/videos, an MP4 file (H.264), and the run one of them all, run.mp4. OpenH264 Video Codec provided by Cisco Systems, Inc.: Axx downloads it from Cisco when a video is first kept; never turns it off. | `failed`, `always`, `never` | `never` | ## Steps [Section titled “Steps”](#steps) ### `desktop-core.key` [Section titled “desktop-core.key”](#desktop-corekey) ```gherkin When the {word} key is pressed in the {word} app ``` Press a key, with the keys held with it, in the app’s focused window: `Enter`, `Escape`, `Tab`, `Control+Shift+S`, as the web pack names them. `ControlOrMeta+Z` is Command on macOS and Control elsewhere, so a feature runs unchanged on each OS. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When the Escape key is pressed in the depot app When the ControlOrMeta+S key is pressed in the depot app ``` *Since 0.2.0.* ### `desktop-core.click-at` [Section titled “desktop-core.click-at”](#desktop-coreclick-at) ```gherkin When the {string} {control} in the {word} app is clicked at {int}, {int} ``` Click at a place on a control, in points from its top left, for what an app draws rather than names: a signature pad, a map, a canvas. | Parameter | Takes | Values | For example | | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | | `"Get a quote"`, `'Express'` | | `{control}` | a kind of control an app shows, found by the name people see: a `button` by its text, a `field` by its label, a `list item` or a `row` by one of its texts, a `menu item` in its open menu, an `element` is anything, by its text or accessibility label; a phone’s `notification` is tapped, by a text in its title or its text. A kind an app’s platform does not have (a phone has no menu bar) fails the step | `button`, `field`, `checkbox`, `radio button`, `switch`, `tab`, `menu`, `menu item`, `list item`, `row`, `link`, `image`, `text`, `element`, `notification` | `button`, `list item` | | `{word}` | one word, with no spaces | | `parcels`, `PX-4101` | | `{int}` | a whole number | | `200`, `3` | **Example:** ```gherkin When the "Courier signature" element in the depot app is clicked at 40, 40 ``` *Since 0.2.0.* ### `desktop-core.click-at-anchor` [Section titled “desktop-core.click-at-anchor”](#desktop-coreclick-at-anchor) ```gherkin When the {string} {control} in the {word} app is clicked at {int}, {int} from its {anchor} ``` Click at a place on a control, in points from its middle or a corner, for what keeps its place there as the control grows with the window or the screen: a picture an app shows in its middle, a button in its bottom right corner. Places run right and down, so one left of or above the point is negative. | Parameter | Takes | Values | For example | | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | | `"Get a quote"`, `'Express'` | | `{control}` | a kind of control an app shows, found by the name people see: a `button` by its text, a `field` by its label, a `list item` or a `row` by one of its texts, a `menu item` in its open menu, an `element` is anything, by its text or accessibility label; a phone’s `notification` is tapped, by a text in its title or its text. A kind an app’s platform does not have (a phone has no menu bar) fails the step | `button`, `field`, `checkbox`, `radio button`, `switch`, `tab`, `menu`, `menu item`, `list item`, `row`, `link`, `image`, `text`, `element`, `notification` | `button`, `list item` | | `{word}` | one word, with no spaces | | `parcels`, `PX-4101` | | `{int}` | a whole number | | `200`, `3` | | `{anchor}` | the point of a control that places on it are measured from: its `middle`, or a corner. Places run right and down from it, so one left of or above it is negative | `middle`, `top left`, `top right`, `bottom left`, `bottom right` | `middle`, `bottom right` | **Example:** ```gherkin When the "Courier signature" element in the depot app is clicked at -40, -20 from its bottom right ``` *Since 0.2.2.* ### `desktop-core.drag` [Section titled “desktop-core.drag”](#desktop-coredrag) ```gherkin When the pointer is dragged from {int}, {int} to {int}, {int} on the {string} {control} in the {word} app ``` Press the pointer at a place on a control, move it to another in steps, as a hand does, and release it there; places are in points from the control’s top left. | Parameter | Takes | Values | For example | | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | | `{int}` | a whole number | | `200`, `3` | | `{string}` | text in double or single quotes, which the step leaves out | | `"Get a quote"`, `'Express'` | | `{control}` | a kind of control an app shows, found by the name people see: a `button` by its text, a `field` by its label, a `list item` or a `row` by one of its texts, a `menu item` in its open menu, an `element` is anything, by its text or accessibility label; a phone’s `notification` is tapped, by a text in its title or its text. A kind an app’s platform does not have (a phone has no menu bar) fails the step | `button`, `field`, `checkbox`, `radio button`, `switch`, `tab`, `menu`, `menu item`, `list item`, `row`, `link`, `image`, `text`, `element`, `notification` | `button`, `list item` | | `{word}` | one word, with no spaces | | `parcels`, `PX-4101` | **Example:** ```gherkin When the pointer is dragged from 40, 30 to 140, 30 on the "Courier signature" element in the depot app ``` *Since 0.2.0.* ### `desktop-core.drag-from-anchor` [Section titled “desktop-core.drag-from-anchor”](#desktop-coredrag-from-anchor) ```gherkin When the pointer is dragged from {int}, {int} to {int}, {int} from the {anchor} of the {string} {control} in the {word} app ``` Drag on a control as a hand does, with places in points from its middle or a corner, for what keeps its place there as the control grows with the window or the screen. Places run right and down, so one left of or above the point is negative. | Parameter | Takes | Values | For example | | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | | `{int}` | a whole number | | `200`, `3` | | `{anchor}` | the point of a control that places on it are measured from: its `middle`, or a corner. Places run right and down from it, so one left of or above it is negative | `middle`, `top left`, `top right`, `bottom left`, `bottom right` | `middle`, `bottom right` | | `{string}` | text in double or single quotes, which the step leaves out | | `"Get a quote"`, `'Express'` | | `{control}` | a kind of control an app shows, found by the name people see: a `button` by its text, a `field` by its label, a `list item` or a `row` by one of its texts, a `menu item` in its open menu, an `element` is anything, by its text or accessibility label; a phone’s `notification` is tapped, by a text in its title or its text. A kind an app’s platform does not have (a phone has no menu bar) fails the step | `button`, `field`, `checkbox`, `radio button`, `switch`, `tab`, `menu`, `menu item`, `list item`, `row`, `link`, `image`, `text`, `element`, `notification` | `button`, `list item` | | `{word}` | one word, with no spaces | | `parcels`, `PX-4101` | **Example:** ```gherkin When the pointer is dragged from -60, 0 to 60, 0 from the middle of the "Courier signature" element in the depot app ``` *Since 0.2.2.* ## Parameter types [Section titled “Parameter types”](#parameter-types) | Parameter | Takes | Values | For example | | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | ------------------------ | | `{anchor}` | the point of a control that places on it are measured from: its `middle`, or a corner. Places run right and down from it, so one left of or above it is negative | `middle`, `top left`, `top right`, `bottom left`, `bottom right` | `middle`, `bottom right` | # Linux > Linux apps, through AT-SPI, the accessibility tree Orca reads: native apps (GTK 3, GTK 4), web views (WebKitGTK: Tauri, Wails), Electron, Qt, Java and Flutter, as they ship, under X11 or Wayland. Linux apps, through AT-SPI, the accessibility tree Orca reads: native apps (GTK 3, GTK 4), web views (WebKitGTK: Tauri, Wails), Electron, Qt, Java and Flutter, as they ship, under X11 or Wayland. **Axx runs desktops of its own**, `desktops` at once (1 by default): each a virtual screen (Xvfb), with its own session bus and accessibility bus for each scenario, and its own view of the file system, in which an app’s home is at the same path in every desktop. Nothing shows on your screen, and a feature never names a desktop: a scenario takes one, as a mobile scenario takes a device. It needs Xvfb, dbus-daemon and at-spi2-core (`axx doctor` checks them); more than one desktop needs user and mount namespaces, which a host that forbids them does not give, and then desktop scenarios run one at a time. **On Wayland** (`display: wayland`), each scenario’s desktop is a headless GNOME Shell of its own, the desktop most Linux users run, on the scenario’s session bus. Apps start in its GNOME session and pick Wayland or X11 as they do on a GNOME desktop: GTK, Qt 6 and Electron Wayland, Java and Qt 5 its Xwayland (Debian’s and Ubuntu’s Qt 5 is read on X11 only: with qtwayland5 installed, its registration’s `env.QT_QPA_PLATFORM` is `xcb`). The desktop portals answer them as GNOME’s do. GNOME Shell’s remote desktop takes the pointer’s and the keyboard’s events, and an extension of Axx’s, run from the session’s folder (nothing is installed), puts each window’s frame at the screen’s corner, shows no top bar, overview or notification, and takes screenshots. It needs gnome-shell and pipewire (and xwayland, for X11 apps), and a system bus: on a machine with systemd, its logind; in a container without logind, no /run/systemd/seats. **Each scenario has a desktop, and a clean app.** Before the app starts, its home is emptied, and it starts with `HOME` and the XDG folders in it; the scenario’s session bus runs with them too, so the app’s settings (dconf), keyring and portals start empty. Axx switches on what toolkits need to be read: the assistive technology announcement, the accessibility bus’s address on the screen (Qt 5), and java-atk-wrapper for Java apps where it is installed. Chromium (Electron, CEF) builds its tree from the environment, but leaves out what shows: an Electron app’s registration starts it with `--force-renderer-accessibility` in its `args`. An app’s `./` files are its home’s `.local/share`. Add it to a project with `axx pack add desktop-linux`; it builds on [`desktop-core`](/next/references/packs/desktop-core/), which comes with it. ## Settings [Section titled “Settings”](#settings) In `axx.yaml`, under `packs.desktop-linux`: | Setting | Takes | Values | Default | | ---------- | ------------------------------------------------------------------------------------------------------------------------------- | ----------------- | ------- | | `desktops` | How many Linux desktops run at once: desktop scenarios beyond them wait for one. More than one needs user and mount namespaces. | a number, 1 to 32 | `1` | | `display` | What Axx’s desktops run: x11, a virtual screen (Xvfb); or wayland, a headless GNOME Shell for each scenario. | `x11`, `wayland` | `x11` | ## Steps [Section titled “Steps”](#steps) ### `desktop-linux.app` [Section titled “desktop-linux.app”](#desktop-linuxapp) ```gherkin Given the {word} linux app with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register a Linux app. The app starts when a step launches it. On another OS, the registration does nothing, so a feature registers the app once for each OS it runs on. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Default | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | | `app` | the app: an executable of the project (an AppImage too), or a command on PATH | *required* | | `args` | the arguments it starts with, a quoted one with spaces; a file of the project is relative to axx.yaml | | | `env.` | an environment variable it starts with, like `env.PARCELS_API` | | | `locale` | its language and region, like `de-DE` | `en-US` | | `timezone` | its time zone, like `Europe/Berlin` | `UTC` | | `owner` | `system` for an app the desktop runs, or starts when another app asks it to, like GNOME Files (`nautilus`): the scenario reads it as it runs, and never resets, launches or stops it | | **Example:** ```gherkin Given the depot linux app with the following properties: | app | ../depot-desk/tauri/src-tauri/target/release/depot-desk | ``` *Since 0.2.0.* # macOS > macOS apps, through AX, the accessibility tree VoiceOver reads: native apps (AppKit, SwiftUI), web views (Tauri, Wails), Electron, Qt, Java and Flutter, as they ship. macOS apps, through AX, the accessibility tree VoiceOver reads: native apps (AppKit, SwiftUI), web views (Tauri, Wails), Electron, Qt, Java and Flutter, as they ship. macOS asks that the app running Axx be allowed to control the computer: the terminal, the IDE or the CI agent at the top of Axx’s process tree, in System Settings > Privacy & Security > Accessibility (Device Control & Data Access, on macOS 27), and Screen Recording for screenshots. `axx doctor` checks both, and names the app. Hosted macOS runners allow both already. **Each scenario has the desktop, and a clean app.** Desktop scenarios take the Mac’s desktop in turns: while they run, the screen, the pointer and the keyboard are theirs, so leave the Mac be, or run them in CI. Before the app starts, its home (`HOME` and `CFFIXED_USER_HOME`, which Foundation follows; Java’s `user.home`) is emptied, and its preferences, which macOS keeps for the user whatever the home, are emptied too: the domain of the app’s bundle identifier, and the domains its registration names (`preferences`: Qt’s `QSettings` names its own, after the app’s organization). Your own are kept aside during the run and put back when it ends (or as the next run starts, after one that was stopped). Window restoration is off. The keychain is not reset: a sign-in an app keeps there outlives the scenario. An app’s `./` files are its home’s `Library/Application Support`. Add it to a project with `axx pack add desktop-macos`; it builds on [`desktop-core`](/next/references/packs/desktop-core/), which comes with it. ## Steps [Section titled “Steps”](#steps) ### `desktop-macos.app` [Section titled “desktop-macos.app”](#desktop-macosapp) ```gherkin Given the {word} macos app with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register a macOS app. The app starts when a step launches it. On another OS, the registration does nothing, so a feature registers the app once for each OS it runs on. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Default | | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `app` | the app: a `.app` of the project, an installed app’s bundle identifier (`com.apple.TextEdit`), or an executable (a file of the project, or a command) | *required* | | `args` | the arguments it starts with, a quoted one with spaces; a file of the project is relative to axx.yaml | | | `env.` | an environment variable it starts with, like `env.PARCELS_API` | | | `preferences` | the preferences domains the app keeps its settings in beyond its bundle identifier’s, like `com.parcels-example.Depot desk` (Qt’s `QSettings` names its own), separated by commas: emptied before each scenario | | | `locale` | its language and region, like `de-DE` | `en-US` | | `timezone` | its time zone, like `Europe/Berlin` | `UTC` | | `owner` | `system` for an app the system runs already, like Finder (`com.apple.finder`): the scenario reads it as it runs, and never resets, launches or stops it; the windows it shows during the scenario are closed as the scenario ends | | **Example:** ```gherkin Given the depot macos app with the following properties: | app | ../depot-desk/appkit/build/Depot desk.app | ``` *Since 0.2.0.* # Windows > Windows apps, through UI Automation, the accessibility tree Narrator reads, and the Java Access Bridge for Java apps: native apps (Windows Forms, WPF, WinUI), web views (WebView2: Tauri, Wails), Electron, Qt, Java and Flutter, as they ship. Windows apps, through UI Automation, the accessibility tree Narrator reads, and the Java Access Bridge for Java apps: native apps (Windows Forms, WPF, WinUI), web views (WebView2: Tauri, Wails), Electron, Qt, Java and Flutter, as they ship. Axx runs apps on the signed-in user’s desktop: run it in a terminal there, or as a scheduled task for that user (`schtasks /it`), not as a service or over SSH, which have no desktop. Hosted Windows runners have one. `axx doctor` checks it. **Each scenario has the desktop, and a clean app.** Desktop scenarios take the machine’s desktop in turns: while they run, the screen, the pointer and the keyboard are theirs. Before the app starts, its home is emptied: `USERPROFILE`, and `APPDATA`, `LOCALAPPDATA`, `TEMP` and `TMP` in it, which apps’ data folders follow (Java’s `user.home` too); the registry keys the registration names are emptied too, your own kept aside during the run and put back when it ends. The profile’s own folders, like Documents, are Windows’ to resolve, and stay yours. Java apps are read through the Java Access Bridge of the Java that runs them, which Axx switches on for the launch. An app’s `./` files are its home’s `AppData\Roaming`. Add it to a project with `axx pack add desktop-windows`; it builds on [`desktop-core`](/next/references/packs/desktop-core/), which comes with it. ## Steps [Section titled “Steps”](#steps) ### `desktop-windows.app` [Section titled “desktop-windows.app”](#desktop-windowsapp) ```gherkin Given the {word} windows app with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register a Windows app. The app starts when a step launches it. On another OS, the registration does nothing, so a feature registers the app once for each OS it runs on. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Default | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | | `app` | the app: an `.exe` of the project, or a command | *required* | | `args` | the arguments it starts with, a quoted one with spaces; a file of the project is relative to axx.yaml | | | `env.` | an environment variable it starts with, like `env.PARCELS_API` | | | `registry` | the keys under `HKEY_CURRENT_USER` the app keeps its settings in, like `Software\Parcels\Depot desk`, separated by commas: emptied before each scenario | | | `locale` | its language and region: Windows has none of an app’s own, so the machine’s apply, and the run says so | | | `timezone` | its time zone: Windows has none of an app’s own, so the machine’s applies, and the run says so | | | `owner` | `system` for an app the system runs already, like File Explorer (`explorer.exe`): the scenario reads it as it runs, and never resets, launches or stops it; the windows it shows during the scenario are closed as the scenario ends | | **Example:** ```gherkin Given the depot windows app with the following properties: | app | ..\depot-desk\winforms\bin\Release\net8.0-windows\DepotDesk.exe | | registry | Software\Parcels\Depot desk | ``` *Since 0.2.0.* # Files > Check the files your services and apps write to a folder: an export directory, a volume they share with Axx, an app's own data. Check the files your services and apps write to a folder: an export directory, a volume they share with Axx, an app’s own data. Register the folder with `the {word} folder with the following properties:`. Its `path` is on this machine, relative to the directory of axx.yaml or absolute, or in the file context of its `owner`: `service:parcels` for a service of axx.yaml (relative to the folder Axx runs it in), `app:depot` for an app a scenario registers (`./` is where the app keeps its data, `~/` its home, wherever it runs). `${env:..}` and `${sys:..}` are expanded. A check names a file by its path in the folder, such as `manifests/M-KESTREL-0412/report.csv`, quoted when it has a space, and waits for it (10 seconds unless `within {duration}` says otherwise), since services write asynchronously. The checks are those of the storage packs’ objects: a file’s exact content, its JSON properties, its text, read by its type (PDF, Word, Excel, CSV, JSON, XML, HTML or plain text), and a row of its table (CSV, TSV or Excel). What a service or an app takes away is checked too: a folder `has no file named` one, or `is empty`, which wait for the files to go. An image an app saves is compared with its screenshot, pixel by pixel, one for each platform: `packs.files.screenshots` sets their `folder`, `tolerance`, `update` and `platforms`, as the app screenshots’ settings do. A scenario puts files in a folder too, as a service or an app would find them: `is a copy of` a file of the project, or `has the content:` of a doc string. `the {word} folder is emptied` removes what is in it, hidden files and subfolders too: a folder of the project, a service’s or an app’s, never one elsewhere on the machine. A service’s folder keeps what earlier runs wrote there, and Axx empties no folder unless a scenario says so: whether a check needs a file of its scenario’s own (named after data unique to it), or a folder emptied first, depends on the test. An app’s files are its own in each scenario. Add it to a project with `axx pack add files`. ## Settings [Section titled “Settings”](#settings) In `axx.yaml`, under `packs.files`: | Setting | Takes | Values | Default | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------- | ------- | | `screenshots.folder` | The folder of the images’ screenshots, in the project. Unset, each feature keeps its screenshots in a screenshots folder next to its file. | text | | | `screenshots.platforms` | The platforms whose screenshots the project keeps: each compares its own; elsewhere the step passes without comparing (default: every platform). | a list of `linux`, `darwin`, `windows` | | | `screenshots.tolerance` | The share of an image’s pixels that may differ. | a number, 0 to 1 | `0` | | `screenshots.update` | Take every screenshot again, as the image is now, instead of comparing. | `true`, `false` | | ## Steps [Section titled “Steps”](#steps) ### `files.folder` [Section titled “files.folder”](#filesfolder) ```gherkin Given the {word} folder with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register a folder under a name. The folder may appear only when a service first writes to it. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Default | | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | | `path` | the folder: with no `owner`, a path relative to the directory of axx.yaml, or absolute; with one, a path in its file context (`./exports`); `${env:..}` and `${sys:..}` are expanded | *required* | | `owner` | whose folder it is: `service:`, a service of axx.yaml, whose paths are relative to the folder Axx runs it in; or `app:`, an app the scenario registers, where `./` is where the app keeps its data and `~/` its home | `none: a folder on this machine` | **Example:** ```gherkin Given the exports folder with the following properties: | owner | service:parcels | | path | ./exports | Given the desk folder with the following properties: | owner | app:depot | | path | "./Parcels/Depot desk" | ``` *Since 0.1.1.* ### `files.copy` [Section titled “files.copy”](#filescopy) ```gherkin Given the {path} file in the {word} folder is a copy of the {filepath} file ``` Put a copy of a file of the project in the folder, under the path given, as a service or an app would find it there; the folders it is in are made, and a file there is replaced. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | **Example:** ```gherkin Given the manifests/M-KESTREL-0505/report.csv file in the exports folder is a copy of the reports/M-KESTREL-0505-earlier.csv file ``` *Since 0.2.3.* ### `files.content` [Section titled “files.content”](#filescontent) ```gherkin Given the {path} file in the {word} folder has the content: """ ... """ ``` Put a file with the doc string’s content in the folder, as a service or an app would find it there; the folders it is in are made, and a file there is replaced. `${env:..}` and `${sys:..}` are expanded. | Parameter | Takes | For example | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin Given the manifests/M-KESTREL-0505/summary.json file in the exports folder has the content: """json {"manifest": "M-KESTREL-0505", "lines": 1} """ ``` *Since 0.2.3.* ### `files.emptied` [Section titled “files.emptied”](#filesemptied) ```gherkin Given the {word} folder is emptied ``` Remove everything in the folder, hidden files and subfolders too, and keep the folder: a folder of the project, a service’s or an app’s. A folder elsewhere on this machine is a person’s, and is not emptied. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin Given the kestrel folder is emptied ``` *Since 0.2.3.* ### `files.has` [Section titled “files.has”](#fileshas) ```gherkin Then [[within {duration} ]]the {word} folder has a(n) file named {path} ``` Check that the folder has a file with that name. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} folder has a(n) file named {path}` * `within {duration} the {word} folder has a(n) file named {path}` **Example:** ```gherkin Then within 30s the exports folder has a file named disputes/kestrel-2026-09.csv ``` *Since 0.1.1.* ### `files.identical` [Section titled “files.identical”](#filesidentical) ```gherkin Then [[within {duration} ]]the {path} file in the {word} folder is identical to the {filepath} file ``` Check that the file exists with exactly the content of the file. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | **Variants**, the parts in `[[...]]` said or left out: * `the {path} file in the {word} folder is identical to the {filepath} file` * `within {duration} the {path} file in the {word} folder is identical to the {filepath} file` **Example:** ```gherkin Then the disputes/kestrel-2026-09.csv file in the exports folder is identical to the expected/kestrel-disputes.csv file ``` *Since 0.1.1.* ### `files.properties` [Section titled “files.properties”](#filesproperties) ```gherkin Then [[within {duration} ]]the {path} file in the {word} folder has the following properties: | path | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the file holds JSON with those values at those paths. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a path into the JSON (a property name, a dotted path or a JSONPath) and the value it has, compared as text: `null` for null and `undefined` for absent, as in the other JSON property steps. **Variants**, the parts in `[[...]]` said or left out: * `the {path} file in the {word} folder has the following properties:` * `within {duration} the {path} file in the {word} folder has the following properties:` **Example:** ```gherkin Then the summaries/kestrel-2026-09.json file in the exports folder has the following properties: | carrier | KESTREL | | lines | 14 | | totals.disputed | 5.25 | ``` *Since 0.1.1.* ### `files.contains` [Section titled “files.contains”](#filescontains) ```gherkin Then [[within {duration} ]]the {path} file in the {word} folder contains {string} ``` Check that the text of the file contains the text. * The check waits for it: 10 seconds, or `within {duration}`. * Case matters; runs of spaces and line breaks count as one space. * The text is read by the file’s type: the text of a PDF’s pages, the paragraphs and tables of a Word document (.docx), the cells of every sheet of an Excel workbook (.xlsx), the text content (or the markup) of XML and HTML, or the text itself. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {path} file in the {word} folder contains {string}` * `within {duration} the {path} file in the {word} folder contains {string}` **Example:** ```gherkin Then the invoices/kestrel-2026-09.pdf file in the exports folder contains "Total due: 1284.50 EUR" ``` *Since 0.1.1.* ### `files.row` [Section titled “files.row”](#filesrow) ```gherkin Then [[within {duration} ]]the {path} file in the {word} folder has a row where: | column | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the table of the file has a row with those values in those columns. * The check waits for it: 10 seconds, or `within {duration}`. * The file is a CSV or TSV file, or an Excel workbook (.xlsx; its first sheet), whose first row names the columns. * Cells compare as text, as the workbook shows them, with runs of spaces as one space; an empty value matches an empty cell. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row names a column, as the file’s first row names it, and the value in that column. **Variants**, the parts in `[[...]]` said or left out: * `the {path} file in the {word} folder has a row where:` * `within {duration} the {path} file in the {word} folder has a row where:` **Example:** ```gherkin Then within 30s the disputes/kestrel-2026-09.csv file in the exports folder has a row where: | parcel | PX-5199 | | status | UNKNOWN_SHIPMENT | | billed | 4.10 | ``` *Since 0.1.1.* ### `files.absent` [Section titled “files.absent”](#filesabsent) ```gherkin Then [[within {duration} ]]the {word} folder has no file named {path} ``` Check that the folder has no file with that name, or wait for it to go: 10 seconds, or `within {duration}`, for a service or an app that deletes or moves it. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} folder has no file named {path}` * `within {duration} the {word} folder has no file named {path}` **Example:** ```gherkin Then the exports folder has no file named manifests/M-KESTREL-0505/report.csv ``` *Since 0.2.3.* ### `files.empty` [Section titled “files.empty”](#filesempty) ```gherkin Then [[within {duration} ]]the {word} folder is empty ``` Check that the folder has no files, in its subfolders too, or wait for them to go: 10 seconds, or `within {duration}`. Hidden files (.DS\_Store, .gitkeep) are not counted, and a folder that is not there is empty. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} folder is empty` * `within {duration} the {word} folder is empty` **Example:** ```gherkin Then the exports folder is empty ``` *Since 0.2.3.* ### `files.screenshot` [Section titled “files.screenshot”](#filesscreenshot) ```gherkin Then [[within {duration} ]]the {path} file in the {word} folder looks like the {string} screenshot ``` Check that an image file looks like its screenshot, pixel by pixel (anti-aliasing aside): an image an app saved, like an annotated capture (PNG or JPEG). The check waits for the file: 10 seconds, or `within {duration}`. * Each platform draws images its own way, so each has its own screenshot, named after it (`checkout-annotated.darwin.png`), in a `screenshots` folder next to the feature file (or the folder `packs.files.screenshots` sets). * Without one, the step takes it and fails: look at it, and keep it. * When the image is different, the step attaches the screenshot, the image and their difference. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {path} file in the {word} folder looks like the {string} screenshot` * `within {duration} the {path} file in the {word} folder looks like the {string} screenshot` **Example:** ```gherkin Then the "snap-*.png" file in the inbox folder looks like the "checkout-annotated" screenshot ``` *Since 0.2.1.* # BigQuery > Seed BigQuery tables and check the rows your services write. Seed BigQuery tables and check the rows your services write. The steps use the scenario’s project (`the {word} gcp project with the following properties:`, from gcp-core). Tables are named `dataset.table` (in the project) or `project.dataset.table`. A **seed** is a YAML or JSON file that maps tables to the rows to insert (a streaming insert, `tabledata.insertAll`): ```yaml billing.carrier_rates: - carrier: KESTREL service: express price_per_kg: 1.35 ``` **Checks** wait (10 seconds unless `within {duration}` says otherwise) until the table has a row, or a number of rows, meeting every condition: `column | value` rows, with a dotted path into `RECORD` columns (`address.city`), compared as text: numbers as written, `NUMERIC` as its decimal, `TIMESTAMP` in RFC 3339 (`2026-09-24T09:30:00Z`), `DATE` as `2026-09-24`; `null` for NULL. A check reads the columns its conditions name, of up to 5,000 rows of the table: check the tables your scenarios write, not warehouse-size ones. Add it to a project with `axx pack add gcp-bigquery`; it builds on [`gcp-core`](/next/references/packs/gcp-core/), which comes with it. ## Steps [Section titled “Steps”](#steps) ### `gcp-bigquery.seed` [Section titled “gcp-bigquery.seed”](#gcp-bigqueryseed) ```gherkin Given a {filepath} bigquery seed ``` Insert the rows of a seed file into their tables: YAML or JSON that maps tables (`dataset.table` or `project.dataset.table`) to lists of rows. The rows are streamed in (`tabledata.insertAll`). | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | **Example:** ```gherkin Given a seeds/carrier-rates.yaml bigquery seed ``` *Since 0.1.0.* ### `gcp-bigquery.row` [Section titled “gcp-bigquery.row”](#gcp-bigqueryrow) ```gherkin Then [[within {duration} ]]the {word} bigquery table has a row where: | column | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the table has a row with those values in those columns. * The check waits for it: 10 seconds, or `within {duration}`. * Name the table `dataset.table`, or `project.dataset.table` for another project’s. * Values compare as text: numbers as written, `NUMERIC` as its decimal, `TIMESTAMP` in RFC 3339 (`2026-09-24T09:30:00Z`), `DATE` as `2026-09-24`, `DATETIME` as `2026-09-24T09:30:00`, `BYTES` in base64; `null` for NULL. * The check reads only the columns the step names, of up to 5,000 rows. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row names a column, or a dotted path into a `RECORD` column (`address.city`), and the value in it. **Variants**, the parts in `[[...]]` said or left out: * `the {word} bigquery table has a row where:` * `within {duration} the {word} bigquery table has a row where:` **Example:** ```gherkin Then within 30s the billing.invoice_lines bigquery table has a row where: | invoice | INV-2026-09-KES2 | | parcel | PX-5103 | | status | OVERCHARGED | | billed | 2.5 | | expected | 1.35 | ``` *Since 0.1.0.* ### `gcp-bigquery.rows` [Section titled “gcp-bigquery.rows”](#gcp-bigqueryrows) ```gherkin Then [[within {duration} ]]the {word} bigquery table has {int} row(s) where: | column | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that exactly that many rows of the table have those values in those columns. * The check waits for it: 10 seconds, or `within {duration}`. * Name the table `dataset.table`, or `project.dataset.table` for another project’s. * Values compare as text: numbers as written, `NUMERIC` as its decimal, `TIMESTAMP` in RFC 3339 (`2026-09-24T09:30:00Z`), `DATE` as `2026-09-24`, `DATETIME` as `2026-09-24T09:30:00`, `BYTES` in base64; `null` for NULL. * The check reads only the columns the step names, of up to 5,000 rows. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{int}` | a whole number | `200`, `3` | Each row names a column, or a dotted path into a `RECORD` column (`address.city`), and the value in it. **Variants**, the parts in `[[...]]` said or left out: * `the {word} bigquery table has {int} row(s) where:` * `within {duration} the {word} bigquery table has {int} row(s) where:` **Example:** ```gherkin Then the billing.invoice_lines bigquery table has 2 rows where: | invoice | INV-2026-09-KES1 | | status | MATCHED | ``` *Since 0.1.0.* # Core > The Google Cloud project the gcp-* packs talk to, set up the way the Google Cloud client libraries are set up for the real services. The Google Cloud project the gcp-\* packs talk to, set up the way the Google Cloud client libraries are set up for the real services. Register the project once; every gcp-\* step of the scenario uses it: ```gherkin Given the billing gcp project with the following properties: | project | parcels-billing | | endpoint | http://localhost:4588 | ``` Without `credentials`, the clients use Application Default Credentials, as they always do: `GOOGLE_APPLICATION_CREDENTIALS`, the gcloud login, or the workload’s service account. Values expand `${env:..}` and `${sys:..}`, so the same features run against Google Cloud and against an emulator. Add it to a project with `axx pack add gcp-core`. ## Steps [Section titled “Steps”](#steps) ### `gcp-core.project` [Section titled “gcp-core.project”](#gcp-coreproject) ```gherkin Given the {word} gcp project with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register the Google Cloud project the gcp-\* steps talk to. * The first project registered is the default. * Without `credentials`, the clients use Application Default Credentials: `GOOGLE_APPLICATION_CREDENTIALS`, the gcloud login, or the workload’s service account. * Values expand `${env:..}` and `${sys:..}`, so the same features run against Google Cloud and against an emulator. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Default | | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `project` | the project ID, like `parcels-billing` | *required* | | `endpoint` | where every service of the project is, instead of Google Cloud: a local emulator, like `http://localhost:4588`; the clients then connect without credentials, and without TLS to an `http://` endpoint | | | `credentials` | a service account key file, resolved against `resources`, like `keys/billing-tests.json` | | | `asyncapi` | an AsyncAPI document, a file of the project or a URL, that the Pub/Sub topics follow: every message a scenario sends there, and every message a check finds, is checked against it (with the asyncapi pack) | | **Example:** ```gherkin Given the billing gcp project with the following properties: | project | parcels-billing | | endpoint | http://localhost:4588 | Given the billing gcp project with the following properties: | project | parcels-billing | | credentials | keys/billing-tests.json | ``` *Since 0.1.0.* # Firestore > Seed Firestore and check the documents your services write. Seed Firestore and check the documents your services write. The steps use the scenario’s project (`the {word} gcp project with the following properties:`, from gcp-core) and its default database. A **seed** is a YAML or JSON file that maps collections to their documents, by ID (a collection can be nested: `shipments/SHP-1/scans`): ```yaml shipments: SHP-1001: carrier: KESTREL weightKg: 2.5 agreedPrice: 3.38 weighedAt: 2026-09-24T07:40:00Z manifest: "2026-09-24" ``` **Dates:** an unquoted YAML date is a Firestore timestamp (`weighedAt` above). A date alone (`2026-09-24`) is midnight UTC. A date and time can have fractional seconds and an offset (`2026-09-24T09:40:00.25+02:00`), and without an offset it is in UTC. A value tagged `!!timestamp` is a timestamp too. Quote a date to keep it a string (`manifest`). JSON has no date type, so the dates of a JSON seed are strings. **Checks** wait (10 seconds unless `within {duration}` says otherwise): for a document at a path (`invoices/INV-2026-09-KESTREL`) to have properties, or for a collection to have a document meeting every condition. Conditions and properties are `field | value` rows, with a dotted path into maps (`totals.billed`), compared as text: timestamps in RFC 3339 and UTC (`2026-09-24T07:40:00Z`), `null` for null and `undefined` for absent. A collection check reads up to 5,000 of its documents. Add it to a project with `axx pack add gcp-firestore`; it builds on [`gcp-core`](/next/references/packs/gcp-core/), which comes with it. ## Steps [Section titled “Steps”](#steps) ### `gcp-firestore.seed` [Section titled “gcp-firestore.seed”](#gcp-firestoreseed) ```gherkin Given a {filepath} firestore seed ``` Write the documents of a seed file: YAML or JSON that maps collections to their documents, by ID. Whole numbers are stored as integers, other numbers as doubles. Unquoted YAML dates (`2026-09-24`, `2026-09-24T07:40:00Z`) and values tagged `!!timestamp` are stored as timestamps; quote a date to store it as a string. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | **Example:** ```gherkin Given a seeds/shipments.yaml firestore seed ``` *Since 0.1.0.* ### `gcp-firestore.document` [Section titled “gcp-firestore.document”](#gcp-firestoredocument) ```gherkin Then [[within {duration} ]]the {word} firestore document has the following properties: | field | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the document at that path, like `invoices/INV-2026-09-KES1`, has those values in those fields. * The check waits for it: 10 seconds, or `within {duration}`. * Values compare as text: timestamps in RFC 3339 and UTC, references as their path, bytes in base64, a geo point as its `lat` and `lng`; `null` for null and `undefined` for absent. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row names a field of the document, or a dotted path into a map (`totals.billed`), and the value it has. **Variants**, the parts in `[[...]]` said or left out: * `the {word} firestore document has the following properties:` * `within {duration} the {word} firestore document has the following properties:` **Example:** ```gherkin Then within 30s the invoices/INV-2026-09-KES1 firestore document has the following properties: | status | RECONCILED | | lines | 2 | | totals.billed | 6.58 | | totals.disputed | 0 | ``` *Since 0.1.0.* ### `gcp-firestore.collection` [Section titled “gcp-firestore.collection”](#gcp-firestorecollection) ```gherkin Then [[within {duration} ]]the {word} firestore collection has a document where: | field | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the collection has a document with those values in those fields. * The check waits for it: 10 seconds, or `within {duration}`. * Values compare as text: timestamps in RFC 3339 and UTC, references as their path, bytes in base64, a geo point as its `lat` and `lng`; `null` for null and `undefined` for absent. * The check reads up to 5,000 documents of the collection. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row names a field of the document, or a dotted path into a map (`totals.billed`), and the value it has. **Variants**, the parts in `[[...]]` said or left out: * `the {word} firestore collection has a document where:` * `within {duration} the {word} firestore collection has a document where:` **Example:** ```gherkin Then the shipments firestore collection has a document where: | parcel | PX-5105 | | carrier | HERON | | weightKg | 3.2 | ``` *Since 0.1.0.* # Pub/Sub > Publish messages to Pub/Sub topics and check the messages your services publish. Publish messages to Pub/Sub topics and check the messages your services publish. The steps use the scenario’s project (`the {word} gcp project with the following properties:`, from gcp-core). **Checking a topic** does not take messages from anyone: for the topics a run checks, Axx creates a subscription of its own (`axx--`) once the services are up, and deletes it when the run ends. A check only looks at the messages received since its scenario started; match on data unique to the scenario, since scenarios run in parallel. The conditions are paths into the message data (JSON) and `attribute ` rows for its attributes. Add it to a project with `axx pack add gcp-pubsub`; it builds on [`gcp-core`](/next/references/packs/gcp-core/), which comes with it. ## Steps [Section titled “Steps”](#steps) ### `gcp-pubsub.send` [Section titled “gcp-pubsub.send”](#gcp-pubsubsend) ```gherkin When a message is published to the {word} pubsub topic: """ ... """ ``` Send a message to the pubsub topic; the doc string is its body. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When a message is published to the shipment-events pubsub topic: """ {"parcel": "PX-5105", "carrier": "HERON", "service": "express", "weightKg": 3.2} """ ``` *Since 0.1.0.* ### `gcp-pubsub.send.file` [Section titled “gcp-pubsub.send.file”](#gcp-pubsubsendfile) ```gherkin When the {filepath} message is published to the {word} pubsub topic[[ with the following attributes:]] ``` Send a message whose body is the file to the pubsub topic, with the attributes of the table when the step has one. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is an attribute sent with the message, as text: its name and its value. **Variants**, the parts in `[[...]]` said or left out: * `the {filepath} message is published to the {word} pubsub topic` * `the {filepath} message is published to the {word} pubsub topic with the following attributes:` **Example:** ```gherkin When the messages/px-5105-weighed.json message is published to the shipment-events pubsub topic When the messages/px-5105-weighed.json message is published to the shipment-events pubsub topic with the following attributes: | eventType | ShipmentWeighed | ``` *Since 0.1.0.* ### `gcp-pubsub.received` [Section titled “gcp-pubsub.received”](#gcp-pubsubreceived) ```gherkin Then [[within {duration} ]]the {word} pubsub topic has a message where: | path | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the pubsub topic has a message with those values, received since the scenario started. * The check waits for it: 10 seconds, or `within {duration}`. * Axx creates a subscription of its own to the topic for the run. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a path into the message’s JSON body (a field name, a dotted path or a JSONPath) and the value it has, compared as text: `null` for null and `undefined` for absent. A row `attribute ` is instead on the attribute of that name sent with the message: `undefined` when there is none. **Variants**, the parts in `[[...]]` said or left out: * `the {word} pubsub topic has a message where:` * `within {duration} the {word} pubsub topic has a message where:` **Example:** ```gherkin Then within 30s the invoice-events pubsub topic has a message where: | attribute eventType | InvoiceReconciled | | invoice | INV-2026-09-KES1 | | status | RECONCILED | ``` *Since 0.1.0.* # Cloud Storage > Upload files to Cloud Storage buckets and check the objects your services write there. Upload files to Cloud Storage buckets and check the objects your services write there. The steps use the scenario’s project (`the {word} gcp project with the following properties:`, from gcp-core). Checks wait for the object (10 seconds unless `within {duration}` says otherwise), since services write asynchronously: an upload that triggers processing (a Pub/Sub notification or an Eventarc trigger, say) and the object that processing writes. Add it to a project with `axx pack add gcp-storage`; it builds on [`gcp-core`](/next/references/packs/gcp-core/), which comes with it. ## Steps [Section titled “Steps”](#steps) ### `gcp-storage.upload` [Section titled “gcp-storage.upload”](#gcp-storageupload) ```gherkin When the {filepath} file is uploaded to the {word} gcs bucket[[ as {path}]] ``` Upload a file to the gcs bucket, under the file’s name or the name given. Its content type follows the file’s extension. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"` | **Variants**, the parts in `[[...]]` said or left out: * `the {filepath} file is uploaded to the {word} gcs bucket` * `the {filepath} file is uploaded to the {word} gcs bucket as {path}` **Example:** ```gherkin When the invoices/kestrel-2026-09.csv file is uploaded to the carrier-invoices gcs bucket When the invoices/kestrel-2026-09.csv file is uploaded to the carrier-invoices gcs bucket as incoming/kestrel-2026-09.csv ``` *Since 0.1.0.* ### `gcp-storage.has` [Section titled “gcp-storage.has”](#gcp-storagehas) ```gherkin Then [[within {duration} ]]the {word} gcs bucket has a(n) object named {path} ``` Check that the gcs bucket has an object with that name. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} gcs bucket has a(n) object named {path}` * `within {duration} the {word} gcs bucket has a(n) object named {path}` **Example:** ```gherkin Then within 30s the carrier-invoices gcs bucket has an object named disputes/kestrel-2026-09.csv ``` *Since 0.1.0.* ### `gcp-storage.identical` [Section titled “gcp-storage.identical”](#gcp-storageidentical) ```gherkin Then [[within {duration} ]]the {path} object in the {word} gcs bucket is identical to the {filepath} file ``` Check that the object exists with exactly the content of the file. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | **Variants**, the parts in `[[...]]` said or left out: * `the {path} object in the {word} gcs bucket is identical to the {filepath} file` * `within {duration} the {path} object in the {word} gcs bucket is identical to the {filepath} file` **Example:** ```gherkin Then the disputes/kestrel-2026-09.csv object in the carrier-invoices gcs bucket is identical to the expected/kestrel-disputes.csv file ``` *Since 0.1.0.* ### `gcp-storage.properties` [Section titled “gcp-storage.properties”](#gcp-storageproperties) ```gherkin Then [[within {duration} ]]the {path} object in the {word} gcs bucket has the following properties: | path | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the object holds JSON with those values at those paths. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a path into the JSON (a property name, a dotted path or a JSONPath) and the value it has, compared as text: `null` for null and `undefined` for absent, as in the other JSON property steps. **Variants**, the parts in `[[...]]` said or left out: * `the {path} object in the {word} gcs bucket has the following properties:` * `within {duration} the {path} object in the {word} gcs bucket has the following properties:` **Example:** ```gherkin Then the summaries/kestrel-2026-09.json object in the carrier-invoices gcs bucket has the following properties: | carrier | KESTREL | | lines | 14 | | totals.disputed | 5.25 | ``` *Since 0.1.0.* ### `gcp-storage.contains` [Section titled “gcp-storage.contains”](#gcp-storagecontains) ```gherkin Then [[within {duration} ]]the {path} object in the {word} gcs bucket contains {string} ``` Check that the text of the object contains the text. * The check waits for it: 10 seconds, or `within {duration}`. * Case matters; runs of spaces and line breaks count as one space. * The text is read by the object’s type: the text of a PDF’s pages, the paragraphs and tables of a Word document (.docx), the cells of every sheet of an Excel workbook (.xlsx), the text content (or the markup) of XML and HTML, or the text itself. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {path} object in the {word} gcs bucket contains {string}` * `within {duration} the {path} object in the {word} gcs bucket contains {string}` **Example:** ```gherkin Then the invoices/kestrel-2026-09.pdf object in the carrier-invoices gcs bucket contains "Total due: 1284.50 EUR" ``` *Since 0.1.1.* ### `gcp-storage.row` [Section titled “gcp-storage.row”](#gcp-storagerow) ```gherkin Then [[within {duration} ]]the {path} object in the {word} gcs bucket has a row where: | column | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the table of the object has a row with those values in those columns. * The check waits for it: 10 seconds, or `within {duration}`. * The object is a CSV or TSV file, or an Excel workbook (.xlsx; its first sheet), whose first row names the columns. * Cells compare as text, as the workbook shows them, with runs of spaces as one space; an empty value matches an empty cell. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row names a column, as the object’s first row names it, and the value in that column. **Variants**, the parts in `[[...]]` said or left out: * `the {path} object in the {word} gcs bucket has a row where:` * `within {duration} the {path} object in the {word} gcs bucket has a row where:` **Example:** ```gherkin Then within 30s the disputes/kestrel-2026-09.csv object in the carrier-invoices gcs bucket has a row where: | parcel | PX-5199 | | status | UNKNOWN_SHIPMENT | | billed | 4.10 | ``` *Since 0.1.1.* # GraphQL > Send queries, mutations and subscriptions to GraphQL services, and check their data and their errors, against the services' schemas. A federated graph's gateway and a subgraph alone are GraphQL services alike: register each at its URL. Send queries, mutations and subscriptions to GraphQL services, and check their data and their errors, against the services’ schemas. A federated graph’s gateway and a subgraph alone are GraphQL services alike: register each at its URL. ```gherkin Given the parcels graphql service with the following properties: | url | http://localhost:8400/graphql | | schema | introspection | ``` * **Operations** come from a file of the project (`.graphql`) or a doc string, with variables from a table: a variable takes the type the operation declares for it, so `10115` is text for a `String!`. * **The checks read the service’s last answer** in the scenario. An answer with errors does not fail the step that sends the operation: check its errors. * **Subscriptions** go over graphql-ws (the `graphql-transport-ws` protocol, on the url with `ws://` or `wss://`) or over server-sent events, and belong to the scenario, which ends them when it ends. * **The schema is the contract:** with a `schema` row (an SDL file of the project, a URL, or `introspection`), every operation and its variables are checked against it before they are sent, and every answer’s data against the operation’s types. A subgraph’s SDL may use the federation directives (`@key`, `@link`…). Findings have keys and levels, like the REST pack’s OpenAPI findings: | Key | Found when | | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | `validation.operation.` | the operation breaks a validation rule of the GraphQL spec (`FieldsOnCorrectType`, `KnownArgumentNames`, `ProvidedRequiredArguments`…) | | `validation.variables.missing` | a required variable is missing | | `validation.variables.type` | a variable is not of its type | | `validation.response.nonNull` | a non-null field is null, without an error that explains it | | `validation.response.type` | a field’s value is not of its type | | `validation.response.enum` | a value is not one of its enum’s | | `validation.response.missingField` | a field the operation selects is missing | | `validation.response.unknownField` | the data has a field the operation does not select | * **Levels:** `ERROR` (or `FAIL`) fails the step, `WARN` and `INFO` log the finding, `IGNORE` drops it. Every finding is an `ERROR` unless `packs.graphql.levels` in axx.yaml, or the scenario’s `the GraphQL validation levels are:`, says otherwise. * **Keys cover the keys below them:** `validation.operation` sets every operation finding; the most specific key set wins. * **Secrets stay secret:** `${env:..}` values in the properties are masked in logs and failures, and `${token:..}` names a token of the scenario. Add it to a project with `axx pack add graphql`. ## Settings [Section titled “Settings”](#settings) In `axx.yaml`, under `packs.graphql`: | Setting | Takes | Values | | -------- | ---------------------------------------------------------------------------- | ---------------- | | `levels` | Validation key -> ERROR (or FAIL), WARN, INFO or IGNORE, for every scenario. | names and values | ## Steps [Section titled “Steps”](#steps) ### `graphql.service` [Section titled “graphql.service”](#graphqlservice) ```gherkin Given the {word} graphql service with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register a GraphQL service under a name: a federated graph’s gateway, a subgraph, or any GraphQL API. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Values | Default | | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | -------------------------------------------------- | | `url` | the URL operations are posted to | | *required* | | `schema` | its schema, which operations and answers are checked against: an SDL file of the project, a URL, or `introspection` to ask the service | | | | `header.` | a header sent with every operation, such as `header.Authorization` | | | | `timeout` | how long an operation may take, like `5s` | | `10s` | | `subscriptions` | how subscriptions go | `graphql-ws`, `sse` | `graphql-ws` | | `subscriptions url` | where subscriptions go | | `the url, with `ws\://`or`wss\://` for graphql-ws` | Values are expanded (`${env:..}`, `${sys:..}`, `${token:..}`); what `${env:..}` references expand to is masked everywhere. **Example:** ```gherkin Given the parcels graphql service with the following properties: | url | http://localhost:8400/graphql | | schema | introspection | Given the graph graphql service with the following properties: | url | https://graph.parcels.example/graphql | | schema | graphql/supergraph.graphql | | header.Authorization | Bearer ${token:shop} | ``` *Since 0.1.5.* ### `graphql.send.file` [Section titled “graphql.send.file”](#graphqlsendfile) ```gherkin When the {filepath} query/mutation is sent to the {word} graphql service[[ with the following variables:]] ``` Send the query or the mutation of a file of the project, with the table’s variables when the step has one. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a variable, or a path into one (`address.postcode`), and its value: a value takes the type the operation declares for its variable, double quotes make it text, `null` is null and `undefined` leaves it out. **Variants**, the parts in `[[...]]` said or left out: * `the {filepath} query/mutation is sent to the {word} graphql service` * `the {filepath} query/mutation is sent to the {word} graphql service with the following variables:` **Example:** ```gherkin When the graphql/parcel.graphql query is sent to the parcels graphql service with the following variables: | reference | PX-GQL-7201 | When the graphql/hold-parcel.graphql mutation is sent to the parcels graphql service with the following variables: | reference | PX-GQL-7202 | | until | 2026-10-05 | ``` *Since 0.1.5.* ### `graphql.send.doc` [Section titled “graphql.send.doc”](#graphqlsenddoc) ```gherkin When a query/mutation is sent to the {word} graphql service: """ ... """ ``` Send the query or the mutation of the doc string. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When a query is sent to the parcels graphql service: """ { parcel(reference: "PX-GQL-7201") { status } } """ ``` *Since 0.1.5.* ### `graphql.subscribe` [Section titled “graphql.subscribe”](#graphqlsubscribe) ```gherkin When the {filepath} subscription is started on the {word} graphql service[[ with the following variables:]] ``` Start the subscription of a file of the project, with the table’s variables when the step has one. It belongs to the scenario, which ends it when it ends. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a variable, or a path into one (`address.postcode`), and its value: a value takes the type the operation declares for its variable, double quotes make it text, `null` is null and `undefined` leaves it out. **Variants**, the parts in `[[...]]` said or left out: * `the {filepath} subscription is started on the {word} graphql service` * `the {filepath} subscription is started on the {word} graphql service with the following variables:` **Example:** ```gherkin When the graphql/parcel-scanned.graphql subscription is started on the parcels graphql service with the following variables: | reference | PX-GQL-7203 | ``` *Since 0.1.5.* ### `graphql.no.errors` [Section titled “graphql.no.errors”](#graphqlnoerrors) ```gherkin Then the {word} graphql service answered without errors ``` Check that the service’s last answer has no errors. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin Then the parcels graphql service answered without errors ``` *Since 0.1.5.* ### `graphql.data` [Section titled “graphql.data”](#graphqldata) ```gherkin Then the {word} graphql service's data has the following properties: | path | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check the data of the service’s last answer. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a path into the data (a field, a dotted path like `parcel.shop.name`, or a JSONPath) and its value, compared as text: `null` for null and `undefined` for absent. **Example:** ```gherkin Then the parcels graphql service's data has the following properties: | parcel.status | IN_TRANSIT | | parcel.shop.name | Maple Home | ``` *Since 0.1.5.* ### `graphql.error` [Section titled “graphql.error”](#graphqlerror) ```gherkin Then the {word} graphql service answered an error where: | path | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the service’s last answer has an error with those values. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a path into the error (`message`, `path`, `extensions.code`) and its value, compared as text; the error’s `path` is its fields and indexes joined with dots, like `parcel.shop` or `parcels.0.reference`. **Example:** ```gherkin Then the parcels graphql service answered an error where: | path | holdParcel | | extensions.code | NOT_HOLDABLE | ``` *Since 0.1.5.* ### `graphql.received` [Section titled “graphql.received”](#graphqlreceived) ```gherkin Then [[within {duration} ]]the {word} graphql service's subscription received a message where: | path | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the service’s subscription received a message whose data has those values. * The check waits for it: 10 seconds, or `within {duration}`. * When the subscription has ended, it fails at once, saying why. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a path into the message’s data and its value, compared as text: `null` for null and `undefined` for absent. **Variants**, the parts in `[[...]]` said or left out: * `the {word} graphql service's subscription received a message where:` * `within {duration} the {word} graphql service's subscription received a message where:` **Example:** ```gherkin Then within 20s the parcels graphql service's subscription received a message where: | parcelScanned.reference | PX-GQL-7203 | | parcelScanned.status | OUT_FOR_DELIVERY | ``` *Since 0.1.5.* ### `graphql.levels` [Section titled “graphql.levels”](#graphqllevels) ```gherkin Given the GraphQL validation levels are: | validation key | level | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Set the level of GraphQL validation findings for this scenario, on every GraphQL service it sends operations to. * A key also sets the keys below it: `validation.operation` relaxes `validation.operation.FieldsOnCorrectType` too. The most specific key set wins. * The rows are merged over `packs.graphql.levels` of axx.yaml. * The pack’s documentation lists the keys, and what each level does. A row is a validation key and its level: `ERROR` (or `FAIL`), `WARN`, `INFO` or `IGNORE`. The key must be one the pack reports, or a prefix of such keys, like `validation.operation`: any other fails the step, which names the closest keys. **Example:** ```gherkin Given the GraphQL validation levels are: | validation.operation.FieldsOnCorrectType | IGNORE | ``` *Since 0.1.5.* # gRPC > Call gRPC services: unary calls and server streams, with requests built from JSON files or tables, and check their status, their answers, their metadata and the messages they stream. Call gRPC services: unary calls and server streams, with requests built from JSON files or tables, and check their status, their answers, their metadata and the messages they stream. Register the service once, usually in the `Background`: ```gherkin Given the tracking grpc service with the following properties: | address | localhost:8410 | ``` * **The contract is the service’s proto.** A `proto` row names its `.proto` file (compiled by Axx, with its imports) or a descriptor set (`protoc --descriptor_set_out --include_imports`); without one, Axx reads the service’s descriptors through its server reflection. Requests are built with them, so a field the request message does not have fails the step, and answers are read with them. * **Methods** are named `package.Service/Method`, as grpcurl names them; `Service/Method` or `Method` do when only one of the service’s methods has that name. * **Requests and answers are proto JSON:** fields by their JSON names (`lastScan.location`) or their proto names, enums by name, 64-bit numbers and bytes as proto JSON has them. A request’s table sets paths into it, as the REST pack’s request properties do. * **The checks read the service’s last call** in the scenario. A call that answers another status than OK does not fail the step that makes it: check its status. * **Server streams** belong to the scenario, which closes them when it ends. Their messages are checked as they come; their status once they end. * **Secrets stay secret:** `${env:..}` values in the properties, such as a token in `header.authorization`, are masked in logs and failures, and `${token:..}` names a token of the scenario. Add it to a project with `axx pack add grpc`. ## Steps [Section titled “Steps”](#steps) ### `grpc.service` [Section titled “grpc.service”](#grpcservice) ```gherkin Given the {word} grpc service with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register a gRPC service under a name. Its calls go over one connection per address for the run. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Values | Default | | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- | ---------- | | `address` | the service’s `host:port` | | *required* | | `proto` | its `.proto` file, compiled with its imports, or a descriptor set (`protoc --descriptor_set_out --include_imports`); without it, Axx reads the service’s server reflection | | | | `tls` | whether the connection is TLS, checked against the system’s certificates | `true`, `false` | `false` | | `header.` | metadata sent with every call, such as `header.authorization` | | | | `timeout` | the deadline of each unary call, like `5s` | | `10s` | Values are expanded (`${env:..}`, `${sys:..}`, `${token:..}`); what `${env:..}` references expand to is masked everywhere. **Example:** ```gherkin Given the tracking grpc service with the following properties: | address | localhost:8410 | Given the rating grpc service with the following properties: | address | rating.parcels.example:443 | | tls | true | | proto | protos/rating.proto | | header.authorization | Bearer ${env:RATING_TOKEN} | ``` *Since 0.1.5.* ### `grpc.call` [Section titled “grpc.call”](#grpccall) ```gherkin When the {word} method is called on the {word} grpc service ``` Call a method with an empty request message. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When the parcels.tracking.v1.Tracking/ListDepots method is called on the tracking grpc service ``` *Since 0.1.5.* ### `grpc.call.fields` [Section titled “grpc.call.fields”](#grpccallfields) ```gherkin When the {word} method is called on the {word} grpc service with the following fields: | path | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Call a method with a request message built from the table. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a path into the request (a field name, a dotted path like `lastScan.location`, or a JSONPath) and its value: a value takes the type of its field, double quotes make it text, `null` leaves the field unset and `undefined` removes it. **Example:** ```gherkin When the parcels.tracking.v1.Tracking/GetParcel method is called on the tracking grpc service with the following fields: | reference | PX-GRP-6101 | ``` *Since 0.1.5.* ### `grpc.call.file` [Section titled “grpc.call.file”](#grpccallfile) ```gherkin When the {word} method is called on the {word} grpc service with the {filepath} message[[ and the following fields:]] ``` Call a method with the request message of a JSON file of the project (proto JSON), with the table’s fields set into it when the step has one. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | Each row is a path into the request (a field name, a dotted path like `lastScan.location`, or a JSONPath) and its value: a value takes the type of its field, double quotes make it text, `null` leaves the field unset and `undefined` removes it. **Variants**, the parts in `[[...]]` said or left out: * `the {word} method is called on the {word} grpc service with the {filepath} message` * `the {word} method is called on the {word} grpc service with the {filepath} message and the following fields:` **Example:** ```gherkin When the parcels.tracking.v1.Tracking/GetParcel method is called on the tracking grpc service with the grpc/get-parcel.json message When the parcels.tracking.v1.Tracking/GetParcel method is called on the tracking grpc service with the grpc/get-parcel.json message and the following fields: | reference | PX-GRP-6102 | ``` *Since 0.1.5.* ### `grpc.status` [Section titled “grpc.status”](#grpcstatus) ```gherkin Then [[within {duration} ]]the {word} grpc service answered {word}[[ with a message containing {string}]] ``` Check the status the service’s last call answered: `OK`, `NOT_FOUND`, `INVALID_ARGUMENT`…, and the text its message contains. * A server stream answers when it ends: the check waits for that, 10 seconds or `within {duration}`. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} grpc service answered {word}` * `within {duration} the {word} grpc service answered {word}` * `the {word} grpc service answered {word} with a message containing {string}` * `within {duration} the {word} grpc service answered {word} with a message containing {string}` **Example:** ```gherkin Then the tracking grpc service answered OK Then the tracking grpc service answered NOT_FOUND with a message containing 'PX-GRP-6199' Then within 30s the tracking grpc service answered OK ``` *Since 0.1.5.* ### `grpc.answer` [Section titled “grpc.answer”](#grpcanswer) ```gherkin Then the {word} grpc service's answer has the following fields: | path | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check the answer of the service’s last call, a unary call that answered OK. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a path into the answer as proto JSON (a field’s JSON name, a dotted path like `lastScan.location`, or a JSONPath) and its value, compared as text: `null` for null and `undefined` for absent. Fields with their default values (`0`, `""`, `false`) are in the answer. **Example:** ```gherkin Then the tracking grpc service's answer has the following fields: | status | OUT_FOR_DELIVERY | | lastScan.location | Leipzig | ``` *Since 0.1.5.* ### `grpc.metadata` [Section titled “grpc.metadata”](#grpcmetadata) ```gherkin Then the {word} grpc service's answer has the following metadata: | name | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check the metadata the service’s last call answered with: its headers and trailers. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a name, in any case, and its value, compared as text: several values are joined with `, `; `undefined` for absent. **Example:** ```gherkin Then the tracking grpc service's answer has the following metadata: | x-served-by | tracking-v1 | ``` *Since 0.1.5.* ### `grpc.streamed` [Section titled “grpc.streamed”](#grpcstreamed) ```gherkin Then [[within {duration} ]]the {word} grpc service streamed a message where: | path | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the service’s last call, a server stream, sent a message with those values. * The check waits for it: 10 seconds, or `within {duration}`. * When the stream has ended, it fails at once, with the status it ended with. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a path into the message as proto JSON and its value, compared as text: `null` for null and `undefined` for absent. **Variants**, the parts in `[[...]]` said or left out: * `the {word} grpc service streamed a message where:` * `within {duration} the {word} grpc service streamed a message where:` **Example:** ```gherkin Then within 20s the tracking grpc service streamed a message where: | reference | PX-GRP-6103 | | status | OUT_FOR_DELIVERY | ``` *Since 0.1.5.* # JSON-RPC > Call JSON-RPC 2.0 services over HTTP, and check their results and their errors, against their OpenRPC documents. Call JSON-RPC 2.0 services over HTTP, and check their results and their errors, against their OpenRPC documents. Register the service once, usually in the `Background`: ```gherkin Given the depots jsonrpc service with the following properties: | url | http://localhost:8400/rpc | | openrpc | rpc.discover | ``` * **Params** come from a table (by name, set as the REST pack’s request properties are) or a doc string (an object by name, or an array by position). * **The checks read the service’s last call** in the scenario. A call that answers an error does not fail the step that makes it: check its error. * **OpenRPC:** an `openrpc` row names the service’s OpenRPC document: a file of the project, a URL, or `rpc.discover` to ask the service for it. Every call’s params are then checked against its method before it is sent, and every result and error when it comes back. A call that breaks the document fails the step that makes it, and names the rule it broke. OpenRPC findings have keys and levels, like the REST pack’s OpenAPI findings: | Key | Found when | | ------------------------------------ | ---------------------------------------------------------------------------------------------------------------- | | `validation.method.unknown` | the document has no such method | | `validation.params.structure` | the params are by name for a method that takes them by position, or the other way round | | `validation.params.missing` | a required param is missing | | `validation.params.unknown` | a param the method does not have | | `validation.params.schema.` | a param breaks a keyword of its schema (`required`, `type`, `pattern`…) | | `validation.result.schema.` | the result breaks a keyword of its schema | | `validation.error.unknown` | the error is not one the method declares, nor one of JSON-RPC’s own (-32700, -32600 to -32603, -32000 to -32099) | * **Levels:** `ERROR` (or `FAIL`) fails the step, `WARN` and `INFO` log the finding, `IGNORE` drops it. Every finding is an `ERROR` unless `packs.jsonrpc.openrpc.levels` in axx.yaml, or the scenario’s `the OpenRPC validation levels are:`, says otherwise. * **Keys cover the keys below them:** `validation.params` sets every params finding; the most specific key set wins. * **Secrets stay secret:** `${env:..}` values in the properties are masked in logs and failures, and `${token:..}` names a token of the scenario. Add it to a project with `axx pack add jsonrpc`. ## Settings [Section titled “Settings”](#settings) In `axx.yaml`, under `packs.jsonrpc`: | Setting | Takes | Values | | ---------------- | ---------------------------------------------------------------------------- | ---------------- | | `openrpc.levels` | Validation key -> ERROR (or FAIL), WARN, INFO or IGNORE, for every scenario. | names and values | ## Steps [Section titled “Steps”](#steps) ### `jsonrpc.service` [Section titled “jsonrpc.service”](#jsonrpcservice) ```gherkin Given the {word} jsonrpc service with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register a JSON-RPC 2.0 service, which takes calls over HTTP, under a name. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Default | | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `url` | the URL calls are posted to | *required* | | `header.` | a header sent with every call, such as `header.Authorization` | | | `timeout` | how long a call may take, like `5s` | `10s` | | `openrpc` | the service’s OpenRPC document, which its calls are checked against: a file of the project, a URL, or `rpc.discover` to ask the service for it | | Values are expanded (`${env:..}`, `${sys:..}`, `${token:..}`); what `${env:..}` references expand to is masked everywhere. **Example:** ```gherkin Given the depots jsonrpc service with the following properties: | url | http://localhost:8400/rpc | | openrpc | rpc.discover | Given the depots jsonrpc service with the following properties: | url | https://depots.parcels.example/rpc | | header.Authorization | Bearer ${env:DEPOTS_TOKEN} | | openrpc | openrpc/depots.json | ``` *Since 0.1.5.* ### `jsonrpc.call` [Section titled “jsonrpc.call”](#jsonrpccall) ```gherkin When the {word} method is called on the {word} jsonrpc service ``` Call a method without params. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When the depot.list method is called on the depots jsonrpc service ``` *Since 0.1.5.* ### `jsonrpc.call.params` [Section titled “jsonrpc.call.params”](#jsonrpccallparams) ```gherkin When the {word} method is called on the {word} jsonrpc service with the following params: | path | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Call a method with params by name, from the table. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a param, or a path into one (`address.postcode`), and its value: numbers and booleans are JSON’s, double quotes make a value text, `null` is null and `undefined` leaves it out. **Example:** ```gherkin When the parcel.hold method is called on the depots jsonrpc service with the following params: | reference | PX-RPC-7101 | | until | 2026-10-02 | ``` *Since 0.1.5.* ### `jsonrpc.call.doc` [Section titled “jsonrpc.call.doc”](#jsonrpccalldoc) ```gherkin When the {word} method is called on the {word} jsonrpc service with the params: """ ... """ ``` Call a method with the params of the doc string: an object, by name, or an array, by position. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When the parcel.get method is called on the depots jsonrpc service with the params: """ ["PX-RPC-7102"] """ ``` *Since 0.1.5.* ### `jsonrpc.result` [Section titled “jsonrpc.result”](#jsonrpcresult) ```gherkin Then the {word} jsonrpc service's result has the following properties: | path | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check the result of the service’s last call. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a path into the result (a field name, a dotted path or a JSONPath) and its value, compared as text: `null` for null and `undefined` for absent. **Example:** ```gherkin Then the depots jsonrpc service's result has the following properties: | reference | PX-RPC-7101 | | status | ON_HOLD | ``` *Since 0.1.5.* ### `jsonrpc.result.is` [Section titled “jsonrpc.result.is”](#jsonrpcresultis) ```gherkin Then the {word} jsonrpc service's result is {string} ``` Check the whole result of the service’s last call: a text’s text, or the JSON of any other result. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Example:** ```gherkin Then the depots jsonrpc service's result is 'true' ``` *Since 0.1.5.* ### `jsonrpc.error` [Section titled “jsonrpc.error”](#jsonrpcerror) ```gherkin Then the {word} jsonrpc service answered the error {int}[[ with a message containing {string}]] ``` Check the error code the service’s last call answered, and the text its message contains. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{int}` | a whole number | `200`, `3` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} jsonrpc service answered the error {int}` * `the {word} jsonrpc service answered the error {int} with a message containing {string}` **Example:** ```gherkin Then the depots jsonrpc service answered the error -32602 Then the depots jsonrpc service answered the error -32010 with a message containing 'out for delivery' ``` *Since 0.1.5.* ### `jsonrpc.error.properties` [Section titled “jsonrpc.error.properties”](#jsonrpcerrorproperties) ```gherkin Then the {word} jsonrpc service's error has the following properties: | path | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check the error the service’s last call answered: its `code`, its `message` and its `data`. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a path into the error object (`code`, `message`, `data.reason`) and its value, compared as text: `null` for null and `undefined` for absent. **Example:** ```gherkin Then the depots jsonrpc service's error has the following properties: | code | -32010 | | data.status | OUT_FOR_DELIVERY | ``` *Since 0.1.5.* ### `jsonrpc.openrpc.levels` [Section titled “jsonrpc.openrpc.levels”](#jsonrpcopenrpclevels) ```gherkin Given the OpenRPC validation levels are: | validation key | level | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Set the level of OpenRPC validation findings for this scenario, on every JSON-RPC service it calls. * A key also sets the keys below it: `validation.params` relaxes `validation.params.schema.required` too. The most specific key set wins. * The rows are merged over `packs.jsonrpc.openrpc.levels` of axx.yaml. * The pack’s documentation lists the keys, and what each level does. A row is a validation key and its level: `ERROR` (or `FAIL`), `WARN`, `INFO` or `IGNORE`. The key must be one the pack reports, or a prefix of such keys, like `validation.params`: any other fails the step, which names the closest keys. **Example:** ```gherkin Given the OpenRPC validation levels are: | validation.params | IGNORE | ``` *Since 0.1.5.* # Kafka > Publish events to Kafka topics (plain text or Confluent Avro through a Schema Registry) and assert on the records of a topic. Publish events to Kafka topics (plain text or Confluent Avro through a Schema Registry) and assert on the records of a topic. A **kafka service** names a cluster; a **topic client** gives one topic its producer and consumer configuration, written as Java client properties (`producer.*`, `consumer.*`); **kafka events** are drafts (key, headers, payload) you build and then publish; a **named kafka event** (`the parcel-events kafka event named registered ...`) is an expectation that one record of the topic must meet. Consumer assertions scan every record of the topic from its first offset, re-checking for up to 30 seconds as records arrive. Avro values are compared through Apache Avro’s text form of them (GenericData.toString), so JSONPath expectations such as `$.reference` work on Avro records as on JSON; union values appear without their wrapper. **Configuration** (`packs.kafka` in axx.yaml): `timeout` (default `30s`), `maxRecords` kept per topic (default 100000), `lenientUnions` (accept Avro union values without their `{"": value}` wrapper when exactly one branch fits). **Client properties.** A topic client’s table takes Java Kafka client properties, prefixed `producer.` or `consumer.`: `a(n) {word} kafka topic client with the following properties:` lists those Axx applies. Without them, a client writes and reads keys and payloads as text, reads every record of the topic, and uses no consumer group. Java client properties Axx accepts without effect: * `group.id`, `group.instance.id`, `group.protocol`, `group.remote.assignor`, `enable.auto.commit`, `auto.commit.interval.ms`, `session.timeout.ms`, `heartbeat.interval.ms`, `max.poll.interval.ms`, `max.poll.records`, `partition.assignment.strategy`, `internal.leave.group.on.close`, `exclude.internal.topics`, `default.api.timeout.ms`, `client.rack`, `check.crcs`, `internal.throw.on.fetch.stable.offset.unsupported`: Axx reads each topic from the start without a consumer group, and never commits offsets. * `batch.size`, `max.block.ms`, `metadata.max.idle.ms`, `partitioner.ignore.keys`, `partitioner.adaptive.partitioning.enable`, `partitioner.availability.timeout.ms`, `transaction.timeout.ms`, `compression.gzip.level`, `compression.lz4.level`, `compression.zstd.level`: Axx sizes batches by `max.request.size`, and publishes one event at a time. * `client.dns.lookup`, `receive.buffer.bytes`, `send.buffer.bytes`, `reconnect.backoff.ms`, `reconnect.backoff.max.ms`, `retry.backoff.max.ms`, `socket.connection.setup.timeout.max.ms`, `metadata.recovery.strategy`, `metrics.num.samples`, `metrics.recording.level`, `metrics.sample.window.ms`, `auto.include.jmx.reporter`, `enable.metrics.push`, `ssl.provider`, `ssl.cipher.suites`, `ssl.keymanager.algorithm`, `ssl.trustmanager.algorithm`, `ssl.secure.random.implementation`, `sasl.kerberos.service.name`, `sasl.login.connect.timeout.ms`, `sasl.login.read.timeout.ms`, `sasl.login.retry.backoff.ms`, `sasl.login.retry.backoff.max.ms`, `sasl.login.refresh.window.factor`, `sasl.login.refresh.window.jitter`, `sasl.login.refresh.min.period.seconds`, `sasl.login.refresh.buffer.seconds`, `key.serializer.encoding`, `value.serializer.encoding`, `serializer.encoding`, `key.deserializer.encoding`, `value.deserializer.encoding`, `deserializer.encoding`: they tune Java, and Axx has no counterpart for them; its text is always UTF-8. * `latest.compatibility.strict`, `id.compatibility.strict`, `avro.remove.java.properties`, `avro.use.logical.type.converters`, `avro.reflection.allow.null`, `max.schemas.per.subject`, `use.latest.with.metadata`, `auto.register.schemas.retry`: Axx writes and reads generic Avro records. Those that fail the step, since they name Java classes: `context.name.strategy`, `interceptor.classes`, `sasl.client.callback.handler.class`, `sasl.login.callback.handler.class`, `sasl.login.class`, `security.providers`, `specific.avro.key.type`, `specific.avro.value.type`, `ssl.engine.factory.class`, `metric.reporters` other than JmxReporter, and any other `*.class` or `*.classes` property. Add it to a project with `axx pack add kafka`. ## Settings [Section titled “Settings”](#settings) In `axx.yaml`, under `packs.kafka`: | Setting | Takes | Values | Default | | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- | -------- | | `lenientUnions` | Accept Avro union values written without their {“branch type”: value} wrapper when exactly one branch fits; Avro’s JSON encoding needs the wrapper. | `true`, `false` | `false` | | `maxRecords` | Records kept in memory per topic; older records are dropped and can no longer match. | a number | `100000` | | `timeout` | How long a consumer assertion waits for a matching record, like 45s. | text | `30s` | ## Steps [Section titled “Steps”](#steps) ### `kafka.service` [Section titled “kafka.service”](#kafkaservice) ```gherkin Given the {word} kafka service with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register a Kafka cluster. The first one registered in a scenario is the default for steps without `on the {word} kafka service`. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Default | | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `brokers` | the cluster’s brokers, a comma-separated list of `host:port`; `${env:..}` and `${sys:..}` are expanded | *required* | | `asyncapi` | an AsyncAPI document, a file of the project or a URL, that the topics follow: every message a scenario sends there, and every message a check finds, is checked against it (with the asyncapi pack) | | Any other property is ignored, with a warning. **Example:** ```gherkin Given the events kafka service with the following properties: | brokers | localhost:9092 | ``` ### `kafka.client` [Section titled “kafka.client”](#kafkaclient) ```gherkin Given the {word} kafka topic client ``` Create a topic client on the default Kafka service with the default configuration: string keys and values, records read from the start of the topic. A topic can have one client per service in a scenario. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin Given the parcel-events kafka topic client ``` ### `kafka.client.props` [Section titled “kafka.client.props”](#kafkaclientprops) ```gherkin Given a(n) {word} kafka topic client[[ on the {word} kafka service]] with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Create a topic client configured with Java Kafka client properties: `producer.` rows for publishing, `consumer.` rows for assertions. * For Avro, set the Confluent Avro serializer or deserializer and the `schema.registry.url`, as in the examples. * A topic can have one client per service in a scenario. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Values | Default | | ------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | | `.bootstrap.servers` | the brokers the client connects to, a comma-separated list; the service’s `brokers` by default | | | | `.client.id` | the name the client gives the brokers | | `axx` | | `producer.key.serializer` | how the producer writes keys: `StringSerializer` as text, `ByteArraySerializer` as the text’s bytes, `KafkaAvroSerializer` as an Avro string | `org.apache.kafka.common.serialization.StringSerializer`, `org.apache.kafka.common.serialization.ByteArraySerializer`, `io.confluent.kafka.serializers.KafkaAvroSerializer` | `org.apache.kafka.common.serialization.StringSerializer` | | `producer.value.serializer` | how the producer writes payloads: `StringSerializer` and `ByteArraySerializer` as text, `KafkaAvroSerializer` as Avro, which publishing with a schema needs | `org.apache.kafka.common.serialization.StringSerializer`, `org.apache.kafka.common.serialization.ByteArraySerializer`, `io.confluent.kafka.serializers.KafkaAvroSerializer` | `org.apache.kafka.common.serialization.StringSerializer` | | `consumer.key.deserializer` | how assertions read keys: `StringDeserializer` and `ByteArrayDeserializer` as text, `KafkaAvroDeserializer` as Avro | `org.apache.kafka.common.serialization.StringDeserializer`, `org.apache.kafka.common.serialization.ByteArrayDeserializer`, `io.confluent.kafka.serializers.KafkaAvroDeserializer` | `org.apache.kafka.common.serialization.StringDeserializer` | | `consumer.value.deserializer` | how assertions read payloads: `StringDeserializer` and `ByteArrayDeserializer` as text, `KafkaAvroDeserializer` as Avro, whose properties are checked on Avro’s text form of the record | `org.apache.kafka.common.serialization.StringDeserializer`, `org.apache.kafka.common.serialization.ByteArrayDeserializer`, `io.confluent.kafka.serializers.KafkaAvroDeserializer` | `org.apache.kafka.common.serialization.StringDeserializer` | | `.schema.registry.url` | the Schema Registry’s URLs, a comma-separated list, which the Avro serializers need; `user:password@` in a URL signs in with basic auth | | | | `.basic.auth.credentials.source` | where the registry’s basic auth credentials come from: the `URL`, `basic.auth.user.info` (`USER_INFO`), or the SASL username and password (`SASL_INHERIT`) | `URL`, `USER_INFO`, `SASL_INHERIT` | `URL` | | `.basic.auth.user.info` | `user:password` for the registry, with `basic.auth.credentials.source=USER_INFO` | | | | `.schema.registry.basic.auth.user.info` | the older name of `basic.auth.user.info` | | | | `.bearer.auth.credentials.source` | where the registry’s bearer token comes from: only `STATIC_TOKEN`, the `bearer.auth.token` | `STATIC_TOKEN` | | | `.bearer.auth.token` | a bearer token for the registry | | | | `producer.auto.register.schemas` | `true` registers the schema under its subject; `false` looks its ID up, and fails if it is not registered | | `true` | | `producer.use.latest.version` | `true` writes with the subject’s latest schema and its ID, with `auto.register.schemas=false` | | `false` | | `producer.normalize.schemas` | `true` has the registry normalize the schema when registering it or looking it up | | `false` | | `producer.use.schema.id` | the ID of the schema to write with, with `auto.register.schemas=false` | | | | `producer.key.subject.name.strategy` | the subject of a key’s schema: `-key` (`TopicNameStrategy`); the record strategies fail, since a key is an Avro string | `io.confluent.kafka.serializers.subject.TopicNameStrategy`, `io.confluent.kafka.serializers.subject.RecordNameStrategy`, `io.confluent.kafka.serializers.subject.TopicRecordNameStrategy` | `io.confluent.kafka.serializers.subject.TopicNameStrategy` | | `producer.value.subject.name.strategy` | the subject of a payload’s schema: `-value` (`TopicNameStrategy`), the record’s full name (`RecordNameStrategy`), or `-` (`TopicRecordNameStrategy`) | `io.confluent.kafka.serializers.subject.TopicNameStrategy`, `io.confluent.kafka.serializers.subject.RecordNameStrategy`, `io.confluent.kafka.serializers.subject.TopicRecordNameStrategy` | `io.confluent.kafka.serializers.subject.TopicNameStrategy` | | `.schema.reflection` | only `false`: reflection needs Java classes | `false` | | | `consumer.specific.avro.reader` | only `false`: Axx has no generated classes and reads generic records | `false` | | | `.schema.registry.ssl.` | an `ssl.` property of this table, for HTTPS to the Schema Registry | | | | `producer.acks` | the acknowledgements a publish waits for: from every in-sync replica (`all`, `-1`), from none (`0`) or from the leader (`1`); `0` and `1` turn idempotence off unless `enable.idempotence` is set, as in Java | `all`, `-1`, `0`, `1` | `all` | | `producer.enable.idempotence` | `false` turns idempotent publishing off; it is on by default with `acks=all`, and `true` needs `acks=all` | | | | `producer.compression.type` | how the producer compresses what it sends | `none`, `gzip`, `snappy`, `lz4`, `zstd` | `none` | | `producer.linger.ms` | how long the producer waits to fill a batch, in milliseconds, as in Java | | `5` | | `producer.max.request.size` | the largest batch the producer sends, in bytes | | | | `producer.buffer.memory` | the most bytes of records the producer holds before they are sent | | | | `producer.delivery.timeout.ms` | how long a publish may take, retries included, in milliseconds | | | | `producer.retries` | how many times a failed publish is retried | | | | `producer.max.in.flight.requests.per.connection` | how many publish requests may wait for an answer from a broker at once, with idempotence off | | | | `producer.partitioner.class` | how records are spread over the partitions: by the key’s murmur2 hash (`DefaultPartitioner`), in turn (`RoundRobinPartitioner`) or a batch at a time (`UniformStickyPartitioner`); other classes fail the step | `org.apache.kafka.clients.producer.internals.DefaultPartitioner`, `org.apache.kafka.clients.producer.RoundRobinPartitioner`, `org.apache.kafka.clients.producer.UniformStickyPartitioner` | `org.apache.kafka.clients.producer.internals.DefaultPartitioner` | | `producer.transactional.id` | nothing: it fails the step, since the publish steps run no transactions and a transactional producer cannot send | | | | `consumer.auto.offset.reset` | the records assertions consider: every record of the topic (`earliest`), or those produced after the assertion starts (`latest`); `none` fails the step, since Axx reads without a consumer group | `earliest`, `latest` | `earliest` | | `consumer.isolation.level` | `read_committed` leaves out the records of transactions that are open or were aborted | `read_uncommitted`, `read_committed` | `read_uncommitted` | | `consumer.allow.auto.create.topics` | `true` lets reading a topic that does not exist create it; producers always may, as in Java | | `true` | | `consumer.fetch.max.wait.ms` | how long a broker may wait to fill a fetch, in milliseconds | | | | `consumer.fetch.min.bytes` | the fewest bytes a broker answers a fetch with | | | | `consumer.fetch.max.bytes` | the most bytes a broker answers a fetch with | | | | `consumer.max.partition.fetch.bytes` | the most bytes of one partition a broker answers a fetch with | | | | `.security.protocol` | how the client connects: in plain text or over TLS (`SSL`), and signed in with SASL (`SASL_`) or not | `PLAINTEXT`, `SSL`, `SASL_PLAINTEXT`, `SASL_SSL` | `PLAINTEXT` | | `.sasl.mechanism` | how the client signs in with SASL; GSSAPI and OAUTHBEARER are not supported | `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512` | `PLAIN` | | `.sasl.jaas.config` | the username and password the client signs in with, in a `PlainLoginModule` or `ScramLoginModule` entry; a `SASL_` protocol needs it | | | | `.ssl.truststore.location` | the CA certificates the client trusts: a JKS, PKCS12 or PEM file, resolved against `resources` | | | | `.ssl.truststore.password` | the truststore’s password, which a JKS file may do without, as in Java | | | | `.ssl.truststore.type` | the truststore’s format, for a file Axx does not recognize by its content as JKS, PKCS12 or PEM | `JKS`, `PKCS12`, `PEM` | `JKS` | | `.ssl.truststore.certificates` | CA certificates the client trusts, in PEM, written in the cell | | | | `.ssl.keystore.location` | the client’s certificate and key, for mutual TLS: a JKS, PKCS12 or PEM file, resolved against `resources` | | | | `.ssl.keystore.password` | the keystore’s password | | | | `.ssl.key.password` | the private key’s password, for JKS key entries and encrypted PEM keys; the keystore’s password by default | | | | `.ssl.keystore.type` | the keystore’s format, for a file Axx does not recognize by its content as JKS, PKCS12 or PEM | `JKS`, `PKCS12`, `PEM` | `JKS` | | `.ssl.keystore.key` | the client’s private key, in PEM, written in the cell; with `ssl.keystore.certificate.chain` | | | | `.ssl.keystore.certificate.chain` | the client’s certificate chain, in PEM, written in the cell | | | | `.ssl.endpoint.identification.algorithm` | `https` checks the broker’s host name against its certificate; an empty value skips that check, and the certificate is still verified | | `https` | | `.ssl.protocol` | the oldest TLS version the client accepts; `TLS` is `TLSv1.2` | `TLS`, `TLSv1.2`, `TLSv1.3` | `TLSv1.2` | | `.ssl.enabled.protocols` | the TLS versions the client may use, a comma-separated list of `TLSv1.2` and `TLSv1.3`; older versions are left out | | | | `.metadata.max.age.ms` | how often the client refreshes what it knows of the cluster, in milliseconds; assertions refresh every 5 seconds without it | | | | `.connections.max.idle.ms` | how long a connection may stay idle before the client closes it, in milliseconds | | | | `.socket.connection.setup.timeout.ms` | how long connecting to a broker may take, in milliseconds | | | | `.request.timeout.ms` | how long a broker has to answer a request, in milliseconds | | | | `.retry.backoff.ms` | how long the client waits before it retries a request, in milliseconds | | | | `producer.` | any other Java producer property: those the pack’s description lists have no effect or fail the step, and the others are ignored, with a warning | | | | `consumer.` | any other Java consumer property: those the pack’s description lists have no effect or fail the step, and the others are ignored, with a warning | | | `` is `producer`, for publishing, or `consumer`, for assertions: Axx applies the Java Kafka client property after the prefix to that client. Values are expanded (`${env:..}`, `${sys:..}`); an empty value fails the step, and a row with neither prefix is ignored, with a warning. **Variants**, the parts in `[[...]]` said or left out: * `a(n) {word} kafka topic client with the following properties:` * `a(n) {word} kafka topic client on the {word} kafka service with the following properties:` **Example:** ```gherkin Given a depot-scans kafka topic client with the following properties: | producer.value.serializer | io.confluent.kafka.serializers.KafkaAvroSerializer | | producer.schema.registry.url | http://localhost:9081 | Given a parcel-events kafka topic client on the events kafka service with the following properties: | consumer.value.deserializer | io.confluent.kafka.serializers.KafkaAvroDeserializer | | consumer.schema.registry.url | http://localhost:9081 | ``` ### `kafka.event` [Section titled “kafka.event”](#kafkaevent) ```gherkin Given a(n)[[ {ordinal} ordered]] {word} kafka event[[ on {word} kafka service]] ``` Draft a new event of the topic: the steps that follow set its key, headers and payload, which starts as `{}`. * Without an ordinal, the event is appended. * With one, it must be the next position: `a 2nd ordered` after one event. * An ordinal equal to the number of events the topic has still appends, with a warning. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | -------------------- | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Variants**, the parts in `[[...]]` said or left out: * `a(n) {word} kafka event` * `a(n) {ordinal} ordered {word} kafka event` * `a(n) {word} kafka event on {word} kafka service` * `a(n) {ordinal} ordered {word} kafka event on {word} kafka service` **Example:** ```gherkin Given a depot-scans kafka event Given a 2nd ordered depot-scans kafka event Given a depot-scans kafka event on events kafka service ``` ### `kafka.event.key` [Section titled “kafka.event.key”](#kafkaeventkey) ```gherkin Given the[[ {ordinal} ordered]] {word} kafka event key is {word}[[ on the {word} kafka service]] ``` Set the key of a drafted event. With `the {ordinal} ordered` the step works on that event of the topic (`1st` is the first event created); without it, on the first event. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | -------------------- | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} kafka event key is {word}` * `the {ordinal} ordered {word} kafka event key is {word}` * `the {word} kafka event key is {word} on the {word} kafka service` * `the {ordinal} ordered {word} kafka event key is {word} on the {word} kafka service` **Example:** ```gherkin Given the depot-scans kafka event key is PX-4101 Given the 2nd ordered depot-scans kafka event key is PX-4102 on the events kafka service ``` ### `kafka.event.headers` [Section titled “kafka.event.headers”](#kafkaeventheaders) ```gherkin Given the[[ {ordinal} ordered]] {word} kafka event headers[[ on the {word} kafka service]] are: | header | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Set headers of a drafted event. With `the {ordinal} ordered` the step works on that event of the topic (`1st` is the first event created); without it, on the first event. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | -------------------- | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | A row sets a header of the event: setting a header again replaces its value, and an empty cell sends the text `null`. **Variants**, the parts in `[[...]]` said or left out: * `the {word} kafka event headers are:` * `the {ordinal} ordered {word} kafka event headers are:` * `the {word} kafka event headers on the {word} kafka service are:` * `the {ordinal} ordered {word} kafka event headers on the {word} kafka service are:` **Example:** ```gherkin Given the depot-scans kafka event headers are: | X-Event-Type | ParcelScanned | | X-Depot | Leipzig | Given the 1st ordered depot-scans kafka event headers on the events kafka service are: | X-Event-Type | ParcelScanned | ``` ### `kafka.event.payload.resource` [Section titled “kafka.event.payload.resource”](#kafkaeventpayloadresource) ```gherkin Given the[[ {ordinal} ordered]] {word} kafka event payload is a(n) {filepath} resource[[ on the {word} kafka service]] ``` Set the payload of a drafted event to the contents of a file. * For Avro events, the file is Avro’s JSON encoding of the record, with unions as `{"": value}`. * With `the {ordinal} ordered` the step works on that event of the topic (`1st` is the first event created); without it, on the first event. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} kafka event payload is a(n) {filepath} resource` * `the {ordinal} ordered {word} kafka event payload is a(n) {filepath} resource` * `the {word} kafka event payload is a(n) {filepath} resource on the {word} kafka service` * `the {ordinal} ordered {word} kafka event payload is a(n) {filepath} resource on the {word} kafka service` **Example:** ```gherkin Given the depot-scans kafka event payload is a kafka/scan-delivered.json resource Given the 3rd ordered depot-scans kafka event payload is a kafka/scan-out-for-delivery.json resource on the events kafka service ``` ### `kafka.event.properties.first` [Section titled “kafka.event.properties.first”](#kafkaeventpropertiesfirst) ```gherkin Given the[[ {ordinal} ordered]] kafka event payload properties[[ on the {word} kafka service]] are: | JSONPath | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Set JSONPath properties of an event of the service’s **first** topic client, the first one created in the scenario. * Every property must already exist in the payload. * With `the {ordinal} ordered` the step works on that event of the topic (`1st` is the first event created); without it, on the first event. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | -------------------- | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | A row sets the property at the JSONPath to the value, always as a string; an empty cell sets JSON null. **Variants**, the parts in `[[...]]` said or left out: * `the kafka event payload properties are:` * `the {ordinal} ordered kafka event payload properties are:` * `the kafka event payload properties on the {word} kafka service are:` * `the {ordinal} ordered kafka event payload properties on the {word} kafka service are:` **Example:** ```gherkin Given the kafka event payload properties are: | $.scanId | SC-4101-1 | | $.parcelRef | PX-4101 | Given the 2nd ordered kafka event payload properties on the events kafka service are: | $.scanId | SC-4101-2 | | $.parcelRef | PX-4101 | ``` ### `kafka.event.properties` [Section titled “kafka.event.properties”](#kafkaeventproperties) ```gherkin Given the {word} kafka event payload properties[[ on the {word} kafka service]] are: | JSONPath | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Set JSONPath properties of the topic’s first event. Every property must already exist in the payload: set it in the payload file first. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | A row sets the property at the JSONPath to the value, always as a string; an empty cell sets JSON null. **Variants**, the parts in `[[...]]` said or left out: * `the {word} kafka event payload properties are:` * `the {word} kafka event payload properties on the {word} kafka service are:` **Example:** ```gherkin Given the depot-scans kafka event payload properties are: | $.scanId | SC-4101-1 | | $.parcelRef | PX-4101 | Given the depot-scans kafka event payload properties on the events kafka service are: | $.location | Leipzig | ``` ### `kafka.event.properties.ordinal` [Section titled “kafka.event.properties.ordinal”](#kafkaeventpropertiesordinal) ```gherkin Given the {ordinal} ordered {word} kafka event payload properties[[ on the {word} kafka service]] are: | JSONPath | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Set JSONPath properties of that event of the topic (`1st` is the first event created). Every property must already exist in the payload: set it in the payload file first. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | -------------------- | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | A row sets the property at the JSONPath to the value, always as a string; an empty cell sets JSON null. **Variants**, the parts in `[[...]]` said or left out: * `the {ordinal} ordered {word} kafka event payload properties are:` * `the {ordinal} ordered {word} kafka event payload properties on the {word} kafka service are:` **Example:** ```gherkin Given the 2nd ordered depot-scans kafka event payload properties are: | $.scanId | SC-4101-2 | | $.parcelRef | PX-4101 | ``` *Since 0.1.0.* ### `kafka.event.property.null` [Section titled “kafka.event.property.null”](#kafkaeventpropertynull) ```gherkin Given the[[ {ordinal} ordered]] {word} kafka event payload property {word} is null[[ on the {word} kafka service]] ``` Set a JSONPath property of a drafted event’s payload to JSON null; the property must exist. With `the {ordinal} ordered` the step works on that event of the topic (`1st` is the first event created); without it, on the first event. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | -------------------- | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} kafka event payload property {word} is null` * `the {ordinal} ordered {word} kafka event payload property {word} is null` * `the {word} kafka event payload property {word} is null on the {word} kafka service` * `the {ordinal} ordered {word} kafka event payload property {word} is null on the {word} kafka service` **Example:** ```gherkin Given the depot-scans kafka event payload property $.location is null Given the 2nd ordered depot-scans kafka event payload property $.location is null on the events kafka service ``` ### `kafka.event.publish.schema` [Section titled “kafka.event.publish.schema”](#kafkaeventpublishschema) ```gherkin When the[[ {ordinal} ordered]] {word} kafka event is published using schema {filepath}[[ on the {word} kafka service]] ``` Publish a drafted event as Confluent Avro, with an `.avsc` schema file. * The payload, Avro’s JSON encoding of the record, is read with the schema; a payload that does not fit it fails the step with the JSONPath of the mismatch. * The schema is registered, or looked up, in the Schema Registry under the subject of `value.subject.name.strategy`, `-value` by default. * The record is written as magic byte 0, the schema ID and the Avro binary. * The topic client needs `producer.value.serializer=io.confluent.kafka.serializers.KafkaAvroSerializer` and `producer.schema.registry.url`. * With `the {ordinal} ordered` the step works on that event of the topic (`1st` is the first event created); without it, on the first event. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} kafka event is published using schema {filepath}` * `the {ordinal} ordered {word} kafka event is published using schema {filepath}` * `the {word} kafka event is published using schema {filepath} on the {word} kafka service` * `the {ordinal} ordered {word} kafka event is published using schema {filepath} on the {word} kafka service` **Example:** ```gherkin When the depot-scans kafka event is published using schema schemas/depot-scan.avsc When the 2nd ordered depot-scans kafka event is published using schema schemas/depot-scan.avsc on the events kafka service ``` ### `kafka.event.publish` [Section titled “kafka.event.publish”](#kafkaeventpublish) ```gherkin When the[[ {ordinal} ordered]] {word} kafka event is published[[ on the {word} kafka service]] ``` Publish a drafted event as it is: its key, its headers and its payload’s text. * The producer’s value serializer writes the payload: `StringSerializer` by default, or `ByteArraySerializer`. * For Avro, use `is published using schema`: with `KafkaAvroSerializer`, this step fails. * With `the {ordinal} ordered` the step works on that event of the topic (`1st` is the first event created); without it, on the first event. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | -------------------- | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} kafka event is published` * `the {ordinal} ordered {word} kafka event is published` * `the {word} kafka event is published on the {word} kafka service` * `the {ordinal} ordered {word} kafka event is published on the {word} kafka service` **Example:** ```gherkin When the depot-scans kafka event is published When the 2nd ordered depot-scans kafka event is published on the events kafka service ``` *Since 0.1.0.* ### `kafka.consumed.key` [Section titled “kafka.consumed.key”](#kafkaconsumedkey) ```gherkin Then the {word} kafka event named {word} key is {word}[[ on the {word} kafka service]] ``` Expect the label’s record to have this key. * The consumer’s key deserializer decides how keys read. * The expectation joins those of the label (`named {word}`) in the scenario: **one record** of the topic must meet **every** one of them, key, payload properties and headers together. * Records are read from the start of the topic, or with `consumer.auto.offset.reset=latest`, only those produced after the step starts. * The step fails after 30 seconds (`packs.kafka.timeout`) without a match; the failure lists the label’s expectations and the latest records, each with the reason it did not match. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} kafka event named {word} key is {word}` * `the {word} kafka event named {word} key is {word} on the {word} kafka service` **Example:** ```gherkin Then the parcel-events kafka event named registered key is PX-4101 Then the parcel-events kafka event named registered key is PX-4101 on the events kafka service ``` ### `kafka.consumed.properties` [Section titled “kafka.consumed.properties”](#kafkaconsumedproperties) ```gherkin Then the {word} kafka event named {word} payload properties[[ on the {word} kafka service]] are: | JSONPath | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Expect JSONPath properties of the label’s record payload. * Values are typed: `"text"` is a string, `null` JSON null, `12` an integer, `1.5` a decimal, `true` and `false` booleans, `{...}` and `[...]` JSON; anything else is a string. * Numbers must match in type: `2` does not equal `2.0`. * An empty cell fails the step: write `null` for JSON null, `""` for an empty string. * The expectation joins those of the label (`named {word}`) in the scenario: **one record** of the topic must meet **every** one of them, key, payload properties and headers together. * Records are read from the start of the topic, or with `consumer.auto.offset.reset=latest`, only those produced after the step starts. * The step fails after 30 seconds (`packs.kafka.timeout`) without a match; the failure lists the label’s expectations and the latest records, each with the reason it did not match. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | A row expects the property at the JSONPath to have the value. **Variants**, the parts in `[[...]]` said or left out: * `the {word} kafka event named {word} payload properties are:` * `the {word} kafka event named {word} payload properties on the {word} kafka service are:` **Example:** ```gherkin Then the parcel-events kafka event named registered payload properties are: | $.reference | PX-4101 | | $.weightGrams | 1200 | | $.serviceLevel | STANDARD | Then the parcel-events kafka event named registered payload properties on the events kafka service are: | $.reference | PX-4101 | | $.source | api | ``` ### `kafka.consumed.headers` [Section titled “kafka.consumed.headers”](#kafkaconsumedheaders) ```gherkin Then the {word} kafka event named {word} headers[[ on the {word} kafka service]] are: | header | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Expect headers of the label’s record. * Each header must occur exactly once, with exactly this value. * The expectation joins those of the label (`named {word}`) in the scenario: **one record** of the topic must meet **every** one of them, key, payload properties and headers together. * Records are read from the start of the topic, or with `consumer.auto.offset.reset=latest`, only those produced after the step starts. * The step fails after 30 seconds (`packs.kafka.timeout`) without a match; the failure lists the label’s expectations and the latest records, each with the reason it did not match. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | A row expects a header of the record to have the value; an empty cell fails the step. **Variants**, the parts in `[[...]]` said or left out: * `the {word} kafka event named {word} headers are:` * `the {word} kafka event named {word} headers on the {word} kafka service are:` **Example:** ```gherkin Then the parcel-events kafka event named registered headers are: | X-Event-Type | ParcelRegistered | Then the parcel-events kafka event named registered headers on the events kafka service are: | X-Event-Type | ParcelRegistered | ``` ### `kafka.consumed.headers.match` [Section titled “kafka.consumed.headers.match”](#kafkaconsumedheadersmatch) ```gherkin Then the {word} kafka event named {word} headers match: | header | pattern | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Expect headers of the label’s record to match regular expressions. * A header must have one distinct value: unlike the other steps, this one ignores repeated identical values of a header. * The expectation joins those of the label (`named {word}`) in the scenario: **one record** of the topic must meet **every** one of them, key, payload properties and headers together. * Records are read from the start of the topic, or with `consumer.auto.offset.reset=latest`, only those produced after the step starts. * The step fails after 30 seconds (`packs.kafka.timeout`) without a match; the failure lists the label’s expectations and the latest records, each with the reason it did not match. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | A row expects a header of the record to match a regular expression (Java syntax), over its whole value; an empty cell fails the step. **Example:** ```gherkin Then the parcel-events kafka event named registered headers match: | X-Event-Type | Parcel[A-Za-z]+ | ``` ### `kafka.consumed.headers.match.service` [Section titled “kafka.consumed.headers.match.service”](#kafkaconsumedheadersmatchservice) ```gherkin Then the {word} kafka event named {word} headers on the {word} kafka service match: | header | pattern | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Expect headers of the label’s record, on a topic client of that Kafka service, to match regular expressions. * A header must have one distinct value: unlike the other steps, this one ignores repeated identical values of a header. * The expectation joins those of the label (`named {word}`) in the scenario: **one record** of the topic must meet **every** one of them, key, payload properties and headers together. * Records are read from the start of the topic, or with `consumer.auto.offset.reset=latest`, only those produced after the step starts. * The step fails after 30 seconds (`packs.kafka.timeout`) without a match; the failure lists the label’s expectations and the latest records, each with the reason it did not match. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | A row expects a header of the record to match a regular expression (Java syntax), over its whole value; an empty cell fails the step. **Example:** ```gherkin Then the parcel-events kafka event named registered headers on the events kafka service match: | X-Event-Type | Parcel[A-Za-z]+ | ``` *Since 0.1.0.* # Logs > Assert the entries your services log, read from files or sent to Axx over UDP, TCP or HTTP. Assert the entries your services log, read from files or sent to Axx over UDP, TCP or HTTP. A **log** is where a service’s log lines are: a file Axx reads, or an address Axx listens on while services send their lines to it. Register it with `the {word} log with the following properties:`, then assert the entries a scenario must produce. Only what the log received after the scenario registered it counts, and every step waits for its entries (10 seconds unless `within {duration}` says otherwise). Use logs to prove that something did **not** happen: have the service log its decision (“line ML-KES-0413-1 rejected: duplicate reference”) and assert that entry, rather than waiting and hoping nothing arrives. Match on data unique to the scenario (a reference, an id): scenarios run in parallel and share the log. **The url** says where the lines are: | url | Axx | | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------ | | `file:///var/log/app.log`, `file://logs/app.log` | reads what is appended to the file (relative to axx.yaml); `file://.axx/logs/services.log` is the console of the services Axx starts | | `udp://0.0.0.0:5140` | listens; each datagram is one or more lines (e.g. Docker’s syslog log driver, a service’s syslog handler) | | `tcp://0.0.0.0:5150` | listens; newline-delimited or octet-counted (RFC 6587) messages | | `http://0.0.0.0:5160/logs`, `https://...` | listens; the body of each POST or PUT to that path (e.g. a Fluent Bit or Vector http output). https uses a self-signed certificate | Axx opens listeners before it starts the services, for the log steps of the scenarios in the run, so services can send from the start; `axx up` keeps them open between runs. Services in containers reach them at `host.docker.internal`. **Patterns** are regular expressions (Java syntax), searched in the log’s text, not matched against whole lines: `^` and `$` match at line boundaries, every match counts, and a pattern can span lines (`\n`, or `(?s)` to let `.` match newlines), which covers multi-line entries such as stack traces. Add it to a project with `axx pack add logs`. ## Steps [Section titled “Steps”](#steps) ### `logs.log` [Section titled “logs.log”](#logslog) ```gherkin Given the {word} log with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register a log under a name. Assertions only look at what the log receives from then on. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Default | | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | | `url` | where the log’s lines are: a file Axx reads (`file://`, relative to the directory of axx.yaml or absolute), or an address Axx listens on (`udp://`, `tcp://`, `http://`, `https://`); `${env:..}` and `${sys:..}` are expanded | *required* | Any other property fails the step. **Example:** ```gherkin Given the parcels log with the following properties: | url | udp://0.0.0.0:5140 | Given the console log with the following properties: | url | file://.axx/logs/services.log | ``` ### `logs.entry` [Section titled “logs.entry”](#logsentry) ```gherkin Then [[within {duration} ]]the {word} log has an entry matching {string} ``` Wait (10s, or the given time) until the log has a match for the regular expression. The pattern is searched in the log’s text: `^` and `$` match at line boundaries and a pattern can span lines. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} log has an entry matching {string}` * `within {duration} the {word} log has an entry matching {string}` **Example:** ```gherkin Then the parcels log has an entry matching 'registration refused reference=PX-EVT-4003' Then within 30s the parcels log has an entry matching 'manifest line processed line=ML-MAP-0019-3 status=IMPORTED' ``` ### `logs.entries` [Section titled “logs.entries”](#logsentries) ```gherkin Then [[within {duration} ]]the {word} log has entries matching: | pattern | ``` Wait until the log has a match for every regular expression in the table. Each row needs a match of its own: the same pattern in two rows needs two matches. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | A row is a regular expression the log must have a match for. **Variants**, the parts in `[[...]]` said or left out: * `the {word} log has entries matching:` * `within {duration} the {word} log has entries matching:` **Example:** ```gherkin Then the console log has entries matching: | msg="manifest line processed" line=ML-FJORD-0101-1 status=REJECTED | | msg="manifest line processed" line=ML-FJORD-0101-2 status=REJECTED | ``` ### `logs.count` [Section titled “logs.count”](#logscount) ```gherkin Then [[within {duration} ]]the {word} log has {int} entry/entries matching {string} ``` Wait until the log has the given number of matches for the regular expression, for example one per retry. More matches than that fail the step. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{int}` | a whole number | `200`, `3` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} log has {int} entry/entries matching {string}` * `within {duration} the {word} log has {int} entry/entries matching {string}` **Example:** ```gherkin Then the parcels log has 2 entries matching 'storing parcel PX-DBF-3002 failed, retrying' ``` ### `logs.across` [Section titled “logs.across”](#logsacross) ```gherkin Then [[within {duration} ]]the logs have entries matching: | log | pattern | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Wait until every log in the table has a match for its regular expression, for example the service’s own entry and its dependency’s. Each row needs a match of its own. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | ----------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | A row names a log registered in the scenario, and a regular expression it must have a match for. **Variants**, the parts in `[[...]]` said or left out: * `the logs have entries matching:` * `within {duration} the logs have entries matching:` **Example:** ```gherkin Then the logs have entries matching: | parcels | msg="registration refused; not announced" reference=PX-ADR-1104 | | console | GET /v1/postcodes/DE/00012 | ``` # Email > Check the emails your services send: who they went to and from, their subject, their text, their HTML, their attachments and their headers. Check the emails your services send: who they went to and from, their subject, their text, their HTML, their attachments and their headers. Your service sends its mail as it always does, by SMTP, to the mail server of your test environment: smtp4dev, Mailpit or any other. Axx reads the mailbox that catches it. Register it once, usually in the `Background`; every mail step of the scenario uses it: ```gherkin Given the shops mailbox with the following properties: | url | http://localhost:8025 | ``` * **Any mailbox, a real one included,** is read over POP3 (`pop3://`, `pop3s://`) or IMAP (`imap://`, `imaps://`): smtp4dev, GreenMail, Inbucket, Dovecot, a staging inbox. * **Mailpit** is read through its API (`http://` or `https://`). * **A check only looks at the emails that arrived since its scenario started,** and waits for one that meets it: 10 seconds, or `within {duration}`. For a run, Axx reads each mailbox the checks name from when the services are up, and never deletes or marks what it reads. * **Scenarios share the mailbox,** so each checks its own mail, by a recipient or a subject unique to it. * **Refused mail** is a stub of the mail server’s, as WireMock’s mappings are for HTTP: smtp4dev refuses the recipients its `RecipientValidationExpression` says to, with the code it gives, and a scenario that uses such a recipient sees how your service copes. * **Mail sent through a provider’s HTTP API** (SendGrid, SES, Mailgun, Postmark) is not in a mailbox: mock the provider with WireMock and check the request with the mock pack. * **Secrets stay secret:** the password, and `${env:..}` values, are masked in logs and failures. Add it to a project with `axx pack add mail`. ## Steps [Section titled “Steps”](#steps) ### `mail.mailbox` [Section titled “mail.mailbox”](#mailmailbox) ```gherkin Given the {word} mailbox with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register the mailbox the steps read. * The first mailbox registered is the one the scenario’s mail steps use. * Values expand `${env:..}` and `${sys:..}`; the password is masked. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Default | | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `url` | where the mailbox is: Mailpit’s address, `http://localhost:8025`, or `pop3://`, `pop3s://`, `imap://` or `imaps://` and the mail server’s host | *required* | | `username` | the user Axx signs in as: for POP3 and IMAP, and for a Mailpit behind a password | | | `password` | its password, like `${env:MAILBOX_PASSWORD}` | | | `folder` | the IMAP folder | `INBOX` | **Example:** ```gherkin Given the shops mailbox with the following properties: | url | http://localhost:8025 | Given the shops mailbox with the following properties: | url | imaps://imap.parcels.example | | username | shop-notifications@parcels.example | | password | ${env:MAILBOX_PASSWORD} | ``` *Since 0.1.5.* ### `mail.received` [Section titled “mail.received”](#mailreceived) ```gherkin Then [[within {duration} ]]the {word} mailbox has an email where: | field | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the mailbox has an email with those values, arrived since the scenario started. * The check waits for it: 10 seconds, or `within {duration}`. * Addresses compare without regard to case; `text` and `html` are text the body contains, with runs of spaces and line breaks as one space. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Field | Takes | | --------------- | ----------------------------------------- | | `to` | an address the email went to | | `cc` | an address it was copied to | | `from` | the address it came from | | `subject` | its subject, all of it | | `text` | text its plain-text body contains | | `html` | text its HTML body contains, tags and all | | `attachment` | the file name of an attachment | | `header ` | a header’s value: its name after `header` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} mailbox has an email where:` * `within {duration} the {word} mailbox has an email where:` **Example:** ```gherkin Then within 10s the shops mailbox has an email where: | to | orders@hawthorn-home.example | | subject | Parcel PX-MAIL-9701 registered | | attachment | PX-MAIL-9701-label.zpl | ``` *Since 0.1.5.* # MCP servers > Call the tools of MCP servers (Model Context Protocol) as an AI assistant calls them, and check their results, with the tools' own schemas as the contract. Read their resources and request their prompts. Call the tools of MCP servers (Model Context Protocol) as an AI assistant calls them, and check their results, with the tools’ own schemas as the contract. Read their resources and request their prompts. ```gherkin Given the parcels mcp server with the following properties: | url | http://localhost:8400/mcp | When the track_parcel tool is called on the parcels mcp server with the following arguments: | reference | PX-MCP-9101 | Then the track_parcel tool's result is not an error And the track_parcel tool's result has the following properties: | status | OUT_FOR_DELIVERY | ``` * **Over stdio or streamable HTTP:** a `command` runs the server for the scenario, which ends it when it ends; a `url` reaches it. The session agrees on the protocol version with the server: 2026-07-28, or an earlier one the server speaks. * **Run it as its clients do:** a server assistants start over stdio is the scenario’s `command`, like a command-line tool, and not a service in axx.yaml; a server that runs on its own over HTTP is a service, and the `url` reaches it. * **A tool’s schemas are the contract:** its arguments take the types its input schema gives them (`10115` is text for a string), and are checked against it before the call; its structured result is checked against its output schema. * **Two kinds of failure:** a tool’s result that is an error (the tool ran, and says what went wrong), and a call the server refused (an unknown tool, invalid parameters), with its JSON-RPC error code. Findings have keys and levels, like the REST pack’s OpenAPI findings: | Key | Found when | | --------------------------------------- | ------------------------------------------------------------------------- | | `validation.arguments.schema.` | the arguments break the tool’s input schema (`required`, `type`, `enum`…) | | `validation.result.schema.` | the structured result breaks the tool’s output schema | | `validation.result.missing` | a tool with an output schema answered no structured result | * **Levels:** `ERROR` (or `FAIL`) fails the step, `WARN` and `INFO` log the finding, `IGNORE` drops it. Every finding is an `ERROR` unless `packs.mcp.levels` in axx.yaml, or the scenario’s `the MCP validation levels are:`, says otherwise. * **Secrets stay secret:** `${env:..}` values in the properties are masked in logs and failures, and `${token:..}` names a token of the scenario. Add it to a project with `axx pack add mcp`. ## Settings [Section titled “Settings”](#settings) In `axx.yaml`, under `packs.mcp`: | Setting | Takes | Values | | -------- | ---------------------------------------------------------------------------- | ---------------- | | `levels` | Validation key -> ERROR (or FAIL), WARN, INFO or IGNORE, for every scenario. | names and values | ## Steps [Section titled “Steps”](#steps) ### `mcp.server` [Section titled “mcp.server”](#mcpserver) ```gherkin Given the {word} mcp server with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register an MCP server under a name, and connect to it: over stdio (a `command` the scenario runs, and ends when it ends) or over streamable HTTP (a `url`). The session agrees on the protocol version with the server. * A server its clients start over stdio is the scenario’s `command`, as assistants run it, not a service in axx.yaml; a server that runs on its own over HTTP is a service, which the `url` reaches. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Default | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | | `command` | the command that runs the server over stdio, split into words without a shell, like `docker compose exec -T app parcels mcp` | | | `url` | the server’s streamable HTTP endpoint | | | `dir` | the folder the command runs in, relative to the project | `a folder of the scenario's own` | | `env.` | an environment variable of the command, such as `env.PARCELS_DB_URL` | | | `header.` | a header sent with every request to the url, such as `header.Authorization` | | | `timeout` | how long connecting, or a call, may take, like `10s` | `30s` | | `protocol version` | the MCP version to speak, to check that the server still answers an earlier one, like `2025-06-18` | `the latest the server speaks` | A `command` or a `url` is required. Values are expanded (`${env:..}`, `${sys:..}`, `${token:..}`); what `${env:..}` references expand to is masked everywhere. **Example:** ```gherkin Given the parcels mcp server with the following properties: | url | http://localhost:8400/mcp | Given the parcels mcp server with the following properties: | command | ${sys:parcels.mcp} | | dir | ../infra | ``` *Since 0.1.5.* ### `mcp.tool` [Section titled “mcp.tool”](#mcptool) ```gherkin Then the {word} mcp server has the {word} tool[[ with the following properties:]] ``` Check that the server lists the tool, and that it has the table’s properties: what an AI assistant reads before it calls the tool, like its description and its hints. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | A row is a path into the tool as the server lists it (`description`, `annotations.readOnlyHint`, `inputSchema.required[0]`) and its value. **Variants**, the parts in `[[...]]` said or left out: * `the {word} mcp server has the {word} tool` * `the {word} mcp server has the {word} tool with the following properties:` **Example:** ```gherkin Then the parcels mcp server has the track_parcel tool Then the parcels mcp server has the track_parcel tool with the following properties: | title | Track a parcel | | annotations.readOnlyHint | true | ``` *Since 0.1.5.* ### `mcp.noTool` [Section titled “mcp.noTool”](#mcpnotool) ```gherkin Then the {word} mcp server does not have the {word} tool ``` Check that the server does not list the tool: one it must not offer an AI assistant. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin Then the parcels mcp server does not have the cancel_parcel tool ``` *Since 0.1.5.* ### `mcp.call` [Section titled “mcp.call”](#mcpcall) ```gherkin When the {word} tool is called on the {word} mcp server[[ with the following arguments:]] ``` Call the tool, with the table’s arguments when the step has one. The arguments are checked against the tool’s input schema before they are sent, and its structured result against its output schema. * A result that is an error does not fail the step: check it with `the {word} tool's result is an error`. * Nor does a call the server refuses: check it with `the {word} tool's call failed with the error code {int}`. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is an argument, or a path into one (`address.postcode`), and its value: a value takes the type the tool’s input schema gives its argument, double quotes make it text, `null` is null and `undefined` leaves it out. **Variants**, the parts in `[[...]]` said or left out: * `the {word} tool is called on the {word} mcp server` * `the {word} tool is called on the {word} mcp server with the following arguments:` **Example:** ```gherkin When the track_parcel tool is called on the parcels mcp server with the following arguments: | reference | PX-MCP-9101 | ``` *Since 0.1.5.* ### `mcp.call.file` [Section titled “mcp.call.file”](#mcpcallfile) ```gherkin When the {word} tool is called on the {word} mcp server with the {filepath} arguments ``` Call the tool with the arguments of a file of the project: a JSON or YAML object of the arguments’ names and values. They are checked against the tool’s input schema before they are sent. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | **Example:** ```gherkin When the hold_parcel tool is called on the parcels mcp server with the mcp/hold-parcel.json arguments ``` *Since 0.1.5.* ### `mcp.notError` [Section titled “mcp.notError”](#mcpnoterror) ```gherkin Then the {word} tool's result[[ on the {word} mcp server]] is not an error ``` Check that the tool answered, with a result that is not an error. The result is the tool’s latest in the scenario, on the server the step names, or on the only server the scenario called the tool on. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} tool's result is not an error` * `the {word} tool's result on the {word} mcp server is not an error` **Example:** ```gherkin Then the track_parcel tool's result is not an error ``` *Since 0.1.5.* ### `mcp.error` [Section titled “mcp.error”](#mcperror) ```gherkin Then the {word} tool's result[[ on the {word} mcp server]] is an error ``` Check that the tool answered with a result that is an error: the tool ran, and says what went wrong, for the assistant to tell the user or try again. Check what it says with `contains`. The result is the tool’s latest in the scenario, on the server the step names, or on the only server the scenario called the tool on. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} tool's result is an error` * `the {word} tool's result on the {word} mcp server is an error` **Example:** ```gherkin Then the hold_parcel tool's result is an error ``` *Since 0.1.5.* ### `mcp.contains` [Section titled “mcp.contains”](#mcpcontains) ```gherkin Then the {word} tool's result[[ on the {word} mcp server]] contains {string} ``` Check that the tool’s result has a text: in its content (text, and embedded resources’ text), or in its structured content’s JSON. The result is the tool’s latest in the scenario, on the server the step names, or on the only server the scenario called the tool on. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} tool's result contains {string}` * `the {word} tool's result on the {word} mcp server contains {string}` **Example:** ```gherkin Then the hold_parcel tool's result contains 'already out for delivery' ``` *Since 0.1.5.* ### `mcp.properties` [Section titled “mcp.properties”](#mcpproperties) ```gherkin Then the {word} tool's result[[ on the {word} mcp server]] has the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check the tool’s structured result: its structured content, or the JSON of its text for a tool that answers JSON as text. * Paths are dotted (`lastScan.location`) or JSONPath; `null` and `undefined` work as in the other property steps. * The result is the tool’s latest in the scenario, on the server the step names, or on the only server the scenario called the tool on. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | A row is a path into the structured result, and its value. **Variants**, the parts in `[[...]]` said or left out: * `the {word} tool's result has the following properties:` * `the {word} tool's result on the {word} mcp server has the following properties:` **Example:** ```gherkin Then the track_parcel tool's result has the following properties: | status | OUT_FOR_DELIVERY | | lastLocation | Leipzig | ``` *Since 0.1.5.* ### `mcp.refused` [Section titled “mcp.refused”](#mcprefused) ```gherkin Then the {word} tool's call[[ on the {word} mcp server]] failed with the error code {int} ``` Check that the server refused the call, with a JSON-RPC error of that code: `-32602` for a tool it does not have or invalid parameters, `-32603` for an internal error. The result is the tool’s latest in the scenario, on the server the step names, or on the only server the scenario called the tool on. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{int}` | a whole number | `200`, `3` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} tool's call failed with the error code {int}` * `the {word} tool's call on the {word} mcp server failed with the error code {int}` **Example:** ```gherkin Then the cancel_parcel tool's call failed with the error code -32602 ``` *Since 0.1.5.* ### `mcp.resource.read` [Section titled “mcp.resource.read”](#mcpresourceread) ```gherkin When the {word} resource is read from the {word} mcp server ``` Read a resource of the server, by its URI. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When the parcels://PX-MCP-9103/label resource is read from the parcels mcp server ``` *Since 0.1.5.* ### `mcp.resource.contains` [Section titled “mcp.resource.contains”](#mcpresourcecontains) ```gherkin Then the {word} resource contains {string} ``` Check that the text of the resource, as the scenario last read it, has a text. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Example:** ```gherkin Then the parcels://PX-MCP-9103/label resource contains 'PX-MCP-9103' ``` *Since 0.1.5.* ### `mcp.resource.properties` [Section titled “mcp.resource.properties”](#mcpresourceproperties) ```gherkin Then the {word} resource has the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check the resource, as the scenario last read it, when its text is JSON. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | A row is a path into the resource’s JSON, and its value. **Example:** ```gherkin Then the parcels://PX-MCP-9103/label resource has the following properties: | reference | PX-MCP-9103 | | service | EXPRESS | ``` *Since 0.1.5.* ### `mcp.prompt` [Section titled “mcp.prompt”](#mcpprompt) ```gherkin When the {word} prompt is requested from the {word} mcp server[[ with the following arguments:]] ``` Request a prompt of the server, with the table’s arguments when the step has one: the messages it offers an AI assistant. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | A row is an argument of the prompt, and its value, as text. **Variants**, the parts in `[[...]]` said or left out: * `the {word} prompt is requested from the {word} mcp server` * `the {word} prompt is requested from the {word} mcp server with the following arguments:` **Example:** ```gherkin When the delivery_update prompt is requested from the parcels mcp server with the following arguments: | reference | PX-MCP-9104 | ``` *Since 0.1.5.* ### `mcp.prompt.contains` [Section titled “mcp.prompt.contains”](#mcppromptcontains) ```gherkin Then the {word} prompt contains {string} ``` Check that the text of the prompt’s messages, as the scenario last requested it, has a text. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Example:** ```gherkin Then the delivery_update prompt contains 'PX-MCP-9104' ``` *Since 0.1.5.* ### `mcp.levels` [Section titled “mcp.levels”](#mcplevels) ```gherkin Given the MCP validation levels are: | validation key | level | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Relax, for this scenario, the checks of tools’ arguments and results against their schemas. A row is a validation key and its level: `ERROR` (or `FAIL`), `WARN`, `INFO` or `IGNORE`. The key must be one the pack reports, or a prefix of such keys, like `validation.arguments`: any other fails the step, which names the closest keys. **Example:** ```gherkin Given the MCP validation levels are: | validation.arguments.schema.required | IGNORE | ``` *Since 0.1.5.* # Android > Android apps, on emulators Axx starts or on devices adb lists, driven by the UiAutomator2 server ([Appium](https://appium.io)'s), which Axx downloads on first use, installs on each device and keeps running while the device is in use. It needs the Android SDK (ANDROID_HOME): its emulator and its devices (AVDs), and adb. Android apps, on emulators Axx starts or on devices adb lists, driven by the UiAutomator2 server ([Appium](https://appium.io)’s), which Axx downloads on first use, installs on each device and keeps running while the device is in use. It needs the Android SDK (ANDROID\_HOME): its emulator and its devices (AVDs), and adb. **Each scenario has a device to itself.** Axx starts emulators of a device (AVD) as scenarios need them, read-only so that several run at once and nothing a scenario changes outlives it, up to `devices` at once (1 by default: Android scenarios then take turns). A scenario leases a device for its whole run. **Each scenario starts from a clean app.** Before the app starts, its data is cleared (and with it what it stored, its permissions and its notifications), the permissions the registration names are granted, the photos Axx put in the library before are taken out (and the registration’s put in), so are the documents in Downloads, the system’s file picker starts afresh (on an emulator Axx started), and the device’s language, time zone and location are set. There is no switch that skips it: a scenario is one journey, and that journey is its own. **What the camera sees.** On an emulator Axx starts, the back camera renders the AVD’s environment, which Axx sets for each scenario: what `the app's camera sees ` gives it (a picture, a video or a glTF scene, seen from where the phone is in it), or else the emulator’s own room. In a debuggable app that has ARCore, an agent Axx attaches as the app starts answers ARCore from the scene. An emulator started from a quick-boot snapshot saved with another camera starts afresh the first time a scenario’s camera sees something. Add it to a project with `axx pack add mobile-android`; it builds on [`mobile-core`](/next/references/packs/mobile-core/), which comes with it. ## Settings [Section titled “Settings”](#settings) In `axx.yaml`, under `packs.mobile-android`: | Setting | Takes | Values | | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | | `bootTimeout` | How long an emulator may take to boot, like 3m. Default 3m. | text | | `devices` | How many emulators of a device (AVD) run at once: each scenario has one to itself, so it is how many Android scenarios run at once. Default 1. | a number | | `emulatorArgs` | More arguments for the emulators Axx starts, like \[-gpu, host]. On Linux they draw without a GPU (-gpu swiftshader\_indirect) unless these name one. | a list of text | ## Steps [Section titled “Steps”](#steps) ### `mobile-android.app` [Section titled “mobile-android.app”](#mobile-androidapp) ```gherkin Given the {word} android app with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register an Android app, and the device it runs on. The app starts when a step launches it or opens a link in it. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Default | | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | | `apk` | the app: an APK, a file of the project | | | `package` | the app’s package; with no `apk`, an app the device has | | | `activity` | the activity the app starts with | `the APK's launcher activity` | | `device` | an emulator’s device (AVD) Axx starts, like `Pixel_9`, or a device adb lists, by its serial | *required* | | `permissions` | the permissions the app has from the start, like `POST_NOTIFICATIONS, ACCESS_FINE_LOCATION` | | | `locale` | the device’s language and region, like `de-DE` | `en-US` | | `timezone` | the device’s time zone, like `Europe/Berlin` | `UTC` | | `host ports` | ports of the machine Axx runs on that the app reaches as `localhost` on the device, like `5500, 8090`: the app calls `http://localhost:5500` as it does on a developer’s device, and gets the service there | | | `location` | where an emulator says it is, every second as a phone’s GPS does: a latitude and a longitude, like `51.3397, 12.3731` | `where an emulator starts (37.4219983, -122.084), whatever the scenario before set` | | `photos` | image or video files put in the device’s photo library before the app starts, for it to pick, like `photos/begin.jpg, photos/end.jpg` (found as a step finds a file: next to the feature file first; put in DCIM/axx): the library shows each by the time it was taken, from the file’s own metadata. The next scenario’s reset takes them out again | `none` | | `documents` | files put in the device’s Downloads before the app starts, where the system’s file picker shows them, for the app to pick, like `documents/safe-place-authorization.pdf` (found as a step finds a file: next to the feature file first; the phone has them by their names). The picker opens at Recent: tap Show roots, then Downloads, then the file. The next scenario’s reset takes them out again: on an emulator Axx started, Downloads is emptied, whatever an app saved there too | `none` | | `appium` | an Appium server of the project’s own or a device farm’s, which then runs the device, in place of `device` | | | `capability.` | an Appium capability for that server, like a device farm’s options, as text or JSON | | **Example:** ```gherkin Given the courier android app with the following properties: | apk | ../courier/android/app/build/outputs/apk/debug/app-debug.apk | | device | Pixel_9 | | permissions | POST_NOTIFICATIONS | ``` *Since 0.1.5.* ### `mobile-android.back` [Section titled “mobile-android.back”](#mobile-androidback) ```gherkin When the {word} app's back button is pressed ``` Press Android’s back button: back to the previous screen, or out of the app from its first. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When the courier app's back button is pressed ``` *Since 0.1.5.* # Core > Native mobile apps, used as people use them: launched, or opened with a deep link straight at the screen a scenario tests; their controls tapped and filled, found by the names people see; what they show checked, waiting as apps take their time. Native mobile apps, used as people use them: launched, or opened with a deep link straight at the screen a scenario tests; their controls tapped and filled, found by the names people see; what they show checked, waiting as apps take their time. The steps every app takes, on a phone or a desktop, are `app-core`‘s, which this pack carries out for phones; its own are the steps of phones alone: deep links, the background, swipes, the system’s dialogs and notifications. The apps are registered, and run, by the platform packs: `mobile-android` (Android emulators and devices) and, later, `mobile-ios`. They drive the apps through the platforms’ own drivers: the UiAutomator2 server on Android, which Axx runs itself, and WebDriverAgent on iOS, through [Appium](https://appium.io); Axx downloads them on first use. **Each scenario has a device to itself, and a clean app:** the platform pack leases a device for the scenario and resets the app, and what it can see, before the scenario starts. Every step names its app, so a scenario can drive two, and use a web app too. Controls are found by the name people see: a button’s text, a field’s label, a list item’s text. When no name tells a control apart, `id=` (the Android resource ID or the iOS accessibility identifier) or `xpath=` finds it. Add it to a project with `axx pack add mobile-core`; it builds on [`app-core`](/next/references/packs/app-core/), which comes with it. ## Settings [Section titled “Settings”](#settings) In `axx.yaml`, under `packs.mobile-core`: | Setting | Takes | Values | Default | | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | ------- | | `screenshots.folder` | The folder of the apps’ screenshots, in the project. Unset, each feature keeps its screenshots in a screenshots folder next to its file. | text | | | `screenshots.tolerance` | The share of a screenshot’s pixels that may differ. | a number, 0 to 1 | `0` | | `screenshots.update` | Take every screenshot again, as the app looks now, instead of comparing. | `true`, `false` | | | `traces` | Which scenarios keep a trace of each app in .axx/mobile/traces: its screen after each step, and its controls where it failed. | `failed`, `always`, `never` | `never` | | `videos` | Which scenarios keep a video in .axx/mobile/videos: their phones as they ran, each touch drawn, with their steps beside them. | `failed`, `always`, `never` | `never` | ## Steps [Section titled “Steps”](#steps) ### `mobile-core.link` [Section titled “mobile-core.link”](#mobile-corelink) ```gherkin When the {word} app is opened with the {string} link ``` Open a link in the app, as tapping it in another app would: a deep link jumps straight to the screen a scenario tests. The app is launched first when it is not running. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Example:** ```gherkin When the courier app is opened with the "parcels-courier://deliveries/PX-MOB-9401" link ``` *Since 0.1.5.* ### `mobile-core.background` [Section titled “mobile-core.background”](#mobile-corebackground) ```gherkin When the {word} app is sent to the background ``` Send the app to the background, as switching to another app does. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When the courier app is sent to the background ``` *Since 0.1.5.* ### `mobile-core.foreground` [Section titled “mobile-core.foreground”](#mobile-coreforeground) ```gherkin When the {word} app is brought back ``` Bring the app back to the front from the background, where it left off. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When the courier app is brought back ``` *Since 0.1.5.* ### `mobile-core.swipe` [Section titled “mobile-core.swipe”](#mobile-coreswipe) ```gherkin When the {word} app is swiped {direction} ``` Swipe across the screen: down, from the top, refreshes a list that pulls to refresh; left and right page through what pages. | Parameter | Takes | Values | For example | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | -------------------- | | `{word}` | one word, with no spaces | | `parcels`, `PX-4101` | | `{direction}` | where a swipe goes, as the finger moves (`down` from the top pulls a list to refresh); or which way a phone moves or turns in the scene its camera sees, where it also moves `forward` and `back` | `up`, `down`, `left`, `right`, `forward`, `back` | `down`, `left` | **Example:** ```gherkin When the courier app is swiped down ``` *Since 0.1.5.* ### `mobile-core.dialog.accepted` [Section titled “mobile-core.dialog.accepted”](#mobile-coredialogaccepted) ```gherkin When the {word} app's dialog is accepted ``` Answer the dialog the system shows over the app, like a request for a permission: accepted allows, dismissed does not. It waits for the dialog. A dialog of the app’s own is part of its screen: tap its buttons. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When the courier app's dialog is accepted ``` *Since 0.1.5.* ### `mobile-core.dialog.dismissed` [Section titled “mobile-core.dialog.dismissed”](#mobile-coredialogdismissed) ```gherkin When the {word} app's dialog is dismissed ``` Answer the dialog the system shows over the app, like a request for a permission: accepted allows, dismissed does not. It waits for the dialog. A dialog of the app’s own is part of its screen: tap its buttons. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When the courier app's dialog is dismissed ``` *Since 0.1.5.* ### `mobile-core.dialog.shows` [Section titled “mobile-core.dialog.shows”](#mobile-coredialogshows) ```gherkin Then [[within {duration} ]]the {word} app's dialog shows {string} ``` Check the text of the dialog the system shows over the app: it waits for the dialog (10 seconds, or `within`). | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} app's dialog shows {string}` * `within {duration} the {word} app's dialog shows {string}` **Example:** ```gherkin Then the courier app's dialog shows "send you notifications" ``` *Since 0.1.5.* ### `mobile-core.camera-sees` [Section titled “mobile-core.camera-sees”](#mobile-corecamera-sees) ```gherkin Given the {word} app's camera sees {filepath} ``` The app’s camera sees a 3D scene, a glTF file (.glb or .gltf), or an image (PNG or JPEG, and HEIC on iOS) or a video (MP4 or MOV; H.264 on Android), as the phone held upright sees it: Axx plays the camera of an iOS simulator, which has none, and of an Android emulator it starts, and, from a scene, the app’s AR. A scene’s camera is where the phone is, which the phone’s steps move; the app’s AR finds the scene’s surfaces and objects where they are. A video plays in a loop. A running app starts again, to see it. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | **Example:** ```gherkin Given the courier app's camera sees scenes/parcel-on-porch.glb Given the courier app's camera sees labels/label-PX-IOS-9621.png ``` *Since 0.1.5.* ### `mobile-core.phone-to-camera` [Section titled “mobile-core.phone-to-camera”](#mobile-corephone-to-camera) ```gherkin When the {word} app's phone moves to the scene's {string} camera ``` Put the phone where one of the scene’s cameras is, looking the way it looks: a place the scene names, with a camera there. The app’s camera sees the scene from there. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Example:** ```gherkin When the courier app's phone moves to the scene's "crouched beside the parcel" camera ``` *Since 0.1.5.* ### `mobile-core.phone-path` [Section titled “mobile-core.phone-path”](#mobile-corephone-path) ```gherkin When the {word} app's phone {travel} the scene's {string} path[[ at {speed}]] ``` Carry the phone along one of the scene’s paths: an animation that moves a camera (the phone’s, or another the phone follows), as it was recorded. With no speed, the phone keeps to the animation’s own times; `at 2x` goes twice as fast; `at 1.4 m/s` keeps one speed along the way. `travels` waits for the phone to arrive; `starts traveling` lets the scenario go on while it moves, and a later step waits for it to reach the end. The phone stays where the path ends. | Parameter | Takes | Values | For example | | ---------- | --------------------------------------------------------------------------------------------------------------------------- | ----------------------------- | ----------------------------- | | `{word}` | one word, with no spaces | | `parcels`, `PX-4101` | | `{travel}` | whether the step waits for the phone to arrive (`travels`), or lets the scenario go on while it moves (`starts traveling`) | `travels`, `starts traveling` | `travels`, `starts traveling` | | `{string}` | text in double or single quotes, which the step leaves out | | `"Get a quote"`, `'Express'` | | `{speed}` | how fast a phone travels: `25 mph`, `40 km/h` or `11 m/s`; or, for a GPX file with times, how many times its own pace: `4x` | | `25 mph`, `40 km/h` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} app's phone {travel} the scene's {string} path` * `the {word} app's phone {travel} the scene's {string} path at {speed}` **Example:** ```gherkin When the courier app's phone travels the scene's "up the garden path" path When the courier app's phone starts traveling the scene's "up the garden path" path at 2x ``` *Since 0.1.5.* ### `mobile-core.phone-move` [Section titled “mobile-core.phone-move”](#mobile-corephone-move) ```gherkin When the {word} app's phone moves {float} m {direction} ``` Step with the phone, as the person holding it steps, still facing the same way, at a walking pace: `forward` and `back` along the ground the way the phone looks, `left` and `right` across it, `up` and `down`. | Parameter | Takes | Values | For example | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | -------------------- | | `{word}` | one word, with no spaces | | `parcels`, `PX-4101` | | `{float}` | a decimal number | | `2.5`, `0.1` | | `{direction}` | where a swipe goes, as the finger moves (`down` from the top pulls a list to refresh); or which way a phone moves or turns in the scene its camera sees, where it also moves `forward` and `back` | `up`, `down`, `left`, `right`, `forward`, `back` | `down`, `left` | **Example:** ```gherkin When the courier app's phone moves 1.5 m right ``` *Since 0.1.5.* ### `mobile-core.phone-turn` [Section titled “mobile-core.phone-turn”](#mobile-corephone-turn) ```gherkin When the {word} app's phone turns {int} degrees {direction} ``` Turn the phone where it stands, as the person holding it turns: `left` and `right` about the upright, keeping its tilt; `up` and `down`, tilting it. | Parameter | Takes | Values | For example | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | -------------------- | | `{word}` | one word, with no spaces | | `parcels`, `PX-4101` | | `{int}` | a whole number | | `200`, `3` | | `{direction}` | where a swipe goes, as the finger moves (`down` from the top pulls a list to refresh); or which way a phone moves or turns in the scene its camera sees, where it also moves `forward` and `back` | `up`, `down`, `left`, `right`, `forward`, `back` | `down`, `left` | **Example:** ```gherkin When the courier app's phone turns 90 degrees left ``` *Since 0.1.5.* ### `mobile-core.camera-tap` [Section titled “mobile-core.camera-tap”](#mobile-corecamera-tap) ```gherkin When the {string} in the {word} app's camera view is tapped ``` Tap what the app’s camera shows, by its name in the scene the camera sees: the tap lands where the object shows on the screen (its middle, or where the scene’s \_tap point for it says), as a finger would. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When the "left end of the parcel" in the courier app's camera view is tapped ``` *Since 0.1.5.* ### `mobile-core.notification` [Section titled “mobile-core.notification”](#mobile-corenotification) ```gherkin Then [[within {duration} ]]the {word} app shows a notification {string} ``` Check that the device shows a notification with a text, in its title or its text: it opens the notifications (Android’s notification shade, iOS’s Notification Center), looks, and closes them. It waits for the notification (10 seconds, or `within`). | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} app shows a notification {string}` * `within {duration} the {word} app shows a notification {string}` **Example:** ```gherkin Then the courier app shows a notification "PX-MOB-9401 delivered" ``` *Since 0.1.5.* ### `mobile-core.notification-action` [Section titled “mobile-core.notification-action”](#mobile-corenotification-action) ```gherkin When the {string} action of the {string} notification is tapped in the {word} app ``` Tap one of a notification’s actions, the notification found by a text in its title or its text: on Android a button under it (opened out first, if it is not), on iOS in the menu a long press shows. It waits for the notification (10 seconds). The notification itself, as a person opens it, is tapped with `the "PX-MOB-9401 delivered" notification is tapped in the courier app`. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When the "Next delivery" action of the "PX-MOB-9401 delivered" notification is tapped in the courier app ``` *Since 0.1.5.* ### `mobile-core.background-task` [Section titled “mobile-core.background-task”](#mobile-corebackground-task) ```gherkin When the {string} background task is run in the {word} app ``` Have the phone run one of the app’s background tasks now, as it does when the task’s time comes: the app’s own code runs, in the app, and does what it does then (a sync, a reminder). Axx only moves the moment: the app must have scheduled the task (Axx waits up to 15 seconds for it to, as an app does a moment after it launches), and on Android the conditions it set (a network, say) still apply. The task is named as the app names it: on Android, its WorkManager work’s unique name or one of its tags (each is tagged with its worker’s class); on iOS, its identifier (BGTaskSchedulerPermittedIdentifiers). The step waits for the task to be done: on iOS, until the app says so, in the 30 seconds iOS gives a task; on Android, until the work ends. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When the "round-reminder" background task is run in the courier app When the "example.parcels.courier.round-reminder" background task is run in the courier app ``` *Since 0.1.5.* ### `mobile-core.travel-through` [Section titled “mobile-core.travel-through”](#mobile-coretravel-through) ```gherkin When the {word} app's phone {travel} at {speed} through: | latitude | longitude | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Move the phone along a route, as its GPS reports a person walking or driving it: from the first point of the table to the last, each a latitude and a longitude, at a steady speed. The app sees where the phone is, how fast it moves and which way. `travels` waits for the phone to arrive; `starts traveling` lets the scenario go on while it moves, and a later step waits for it to reach the end. The phone stays at the last point. | Parameter | Takes | Values | For example | | ---------- | --------------------------------------------------------------------------------------------------------------------------- | ----------------------------- | ----------------------------- | | `{word}` | one word, with no spaces | | `parcels`, `PX-4101` | | `{travel}` | whether the step waits for the phone to arrive (`travels`), or lets the scenario go on while it moves (`starts traveling`) | `travels`, `starts traveling` | `travels`, `starts traveling` | | `{speed}` | how fast a phone travels: `25 mph`, `40 km/h` or `11 m/s`; or, for a GPX file with times, how many times its own pace: `4x` | | `25 mph`, `40 km/h` | A row is a point the route goes through, in order. **Example:** ```gherkin When the courier app's phone travels at 30 km/h through: | 51.3397 | 12.3731 | | 51.3397 | 12.3760 | ``` *Since 0.1.5.* ### `mobile-core.travel-route` [Section titled “mobile-core.travel-route”](#mobile-coretravel-route) ```gherkin When the {word} app's phone {travel} the {filepath} route[[ at {speed}]] ``` Move the phone along a GPX file’s route, as its GPS reports it: its track, else its route, else its waypoints. With no speed, the phone keeps to the file’s own times; `at 4x` goes four times as fast; `at 25 mph` keeps one speed throughout. `travels` waits for the phone to arrive; `starts traveling` lets the scenario go on while it moves. | Parameter | Takes | Values | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------- | ------------------------------------------------- | | `{word}` | one word, with no spaces | | `parcels`, `PX-4101` | | `{travel}` | whether the step waits for the phone to arrive (`travels`), or lets the scenario go on while it moves (`starts traveling`) | `travels`, `starts traveling` | `travels`, `starts traveling` | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | | `{speed}` | how fast a phone travels: `25 mph`, `40 km/h` or `11 m/s`; or, for a GPX file with times, how many times its own pace: `4x` | | `25 mph`, `40 km/h` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} app's phone {travel} the {filepath} route` * `the {word} app's phone {travel} the {filepath} route at {speed}` **Example:** ```gherkin When the courier app's phone travels the routes/grimmaische-strasse.gpx route When the courier app's phone starts traveling the routes/grimmaische-strasse.gpx route at 4x ``` *Since 0.1.5.* ### `mobile-core.route-end` [Section titled “mobile-core.route-end”](#mobile-coreroute-end) ```gherkin Then [[within {duration} ]]the {word} app's phone reaches the end of its route ``` Wait for the phone a `starts traveling` step set off to arrive at the last point of its route, or the end of the scene’s path it travels (the route’s own time and 10 seconds, or `within`). | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} app's phone reaches the end of its route` * `within {duration} the {word} app's phone reaches the end of its route` **Example:** ```gherkin Then within 2m the courier app's phone reaches the end of its route ``` *Since 0.1.5.* ## Parameter types [Section titled “Parameter types”](#parameter-types) | Parameter | Takes | Values | For example | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | ----------------------------- | | `{direction}` | where a swipe goes, as the finger moves (`down` from the top pulls a list to refresh); or which way a phone moves or turns in the scene its camera sees, where it also moves `forward` and `back` | `up`, `down`, `left`, `right`, `forward`, `back` | `down`, `left`, `forward` | | `{speed}` | how fast a phone travels: `25 mph`, `40 km/h` or `11 m/s`; or, for a GPX file with times, how many times its own pace: `4x` | | `25 mph`, `40 km/h`, `4x` | | `{travel}` | whether the step waits for the phone to arrive (`travels`), or lets the scenario go on while it moves (`starts traveling`) | `travels`, `starts traveling` | `travels`, `starts traveling` | # iOS > iOS apps, on simulators Axx runs, driven through WebDriverAgent, the app on the simulator that drives yours: Axx downloads the build of it that the [Appium](https://appium.io) project publishes on first use, and starts it on each simulator itself. It needs a Mac with Xcode and an iOS simulator runtime. iOS apps, on simulators Axx runs, driven through WebDriverAgent, the app on the simulator that drives yours: Axx downloads the build of it that the [Appium](https://appium.io) project publishes on first use, and starts it on each simulator itself. It needs a Mac with Xcode and an iOS simulator runtime. **Each scenario has a simulator to itself.** Axx makes simulators as scenarios need them, up to `devices` at once (1 by default: iOS scenarios then take turns), each a clone, deleted when the run ends: for a device type, like iPhone 16, of Axx’s own simulator of it, which it sets up once and keeps in its cache (no scenario ever runs on it); for a simulator the project set up, of that one. A scenario leases a simulator for its whole run. **Each scenario starts from a clean app.** Before the app starts, it is installed afresh (its data and its notifications go with the old one), the simulator’s keychain and the app’s permissions are reset, the permissions the registration names are granted, the documents a scenario before put On My iPhone are taken out (and the registration’s put in), and the location is set. A dialog a scenario leaves unanswered, like a request for notifications, is dismissed as it ends: it would outlive the app and take the next scenario’s taps. The app starts in the registration’s language, region and time zone. There is no switch that skips it: a scenario is one journey, and that journey is its own. Add it to a project with `axx pack add mobile-ios`; it builds on [`mobile-core`](/next/references/packs/mobile-core/), which comes with it. ## Settings [Section titled “Settings”](#settings) In `axx.yaml`, under `packs.mobile-ios`: | Setting | Takes | Values | | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- | | `bootTimeout` | How long a simulator may take to boot, like 30m. The simulator Axx sets up once, to clone, takes over ten minutes on GitHub’s macOS runners; its clones boot in seconds, and are ready once iOS serves apps’ requests for notifications (seconds more, or a minute on a busy Mac). Default 20m. | text | | `devices` | How many simulators of a device run at once: each scenario has one to itself, so it is how many iOS scenarios run at once. Default 1. | a number | | `keep` | Keep the simulators booted after the run, for the next run to start at once, instead of making them for each run: in Xcode’s set, named Axx (n), where Device Hub lists them. Each scenario still has its simulator to itself, and its app reset. A watched run (axx run –watch) keeps them. Default false. | `true`, `false` | ## Steps [Section titled “Steps”](#steps) ### `mobile-ios.app` [Section titled “mobile-ios.app”](#mobile-iosapp) ```gherkin Given the {word} ios app with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register an iOS app, and the simulator it runs on. The app starts when a step launches it or opens a link in it. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Default | | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- | | `app` | the app: a simulator build (a `.app` folder, or it zipped), a file of the project | | | `bundle id` | the app’s bundle identifier; with no `app`, an app the simulator has | `the app's own` | | `device` | a device type Axx makes simulators of, like `iPhone 16` or `iPhone 16, iOS 18.1` (the newest iOS Xcode has, by default); a simulator the project set up, by its name, which Axx clones; or a simulator’s UDID | *required* | | `permissions` | what the app may use from the start, like `location, photos` (`xcrun simctl help privacy` lists them); notifications cannot be granted: the app asks, and the scenario answers the dialog | | | `locale` | the app’s language and region, like `de-DE` | `en-US` | | `timezone` | the app’s time zone, like `Europe/Berlin` | `UTC` | | `location` | where the simulator says it is: a latitude and a longitude, like `51.3397, 12.3731` | `none` | | `photos` | image or video files put in the simulator’s photo library before the app starts, for it to pick, like `photos/begin.jpg, photos/end.jpg` (found as a step finds a file: next to the feature file first): the library shows each by the time it was taken, from the file’s own metadata. A simulator Axx makes, or keeps for runs, is not used again after a scenario that put photos in it (its library cannot be emptied); one of yours keeps them | `none` | | `documents` | files put On My iPhone, in the simulator’s Files, before the app starts, where the document picker shows them, for the app to pick, like `documents/safe-place-authorization.pdf` (found as a step finds a file: next to the feature file first; the phone has them by their names). In the picker, tap Browse, then On My iPhone, then the file as a list item, by its name without its extension. The next scenario’s reset takes them out again: on a simulator Axx makes, On My iPhone is emptied, whatever an app saved there too | `none` | | `appium` | an Appium server of the project’s own or a device farm’s, which then runs the device, in place of `device` | | | `capability.` | an Appium capability for that server, like a device farm’s options, as text or JSON | | **Example:** ```gherkin Given the courier ios app with the following properties: | app | ../courier/ios/build/Build/Products/Debug-iphonesimulator/Courier.app | | device | iPhone 16 | ``` *Since 0.1.5.* # Mocks > Verify the requests WireMock mocks received. Stubs are defined in WireMock mapping files; Axx only verifies. Verify the requests WireMock mocks received. Stubs are defined in WireMock mapping files; Axx only verifies. With the Axx WireMock image (`ghcr.io/nimbusxr/axx-wiremock`), every call to a mock is checked against the mocked service’s OpenAPI contract. A step that checks a call that broke the contract fails; a call that broke it, and that no step checks, fails the run after the scenarios. The image also mocks AI models, in the format of each request (OpenAI’s API and the servers that speak it, Anthropic, Gemini, Bedrock, Ollama): its stubs answer with the `model-request` matcher and the `model-answer` transformer, and the `the mocked ... model ...` steps check what the service asked the model: the texts, the tools and the schema. Add it to a project with `axx pack add mock`. ## Steps [Section titled “Steps”](#steps) ### `mock.service` [Section titled “mock.service”](#mockservice) ```gherkin Given the mocked {word} service with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register a WireMock server, to check the requests it received. The first mocked service registered in a scenario is the default one. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Default | | -------- | ---------------------------------------------------------------------------------------------------------- | ---------- | | `url` | the WireMock server’s URL, with its admin API at `/__admin` below it; it can use `${env:…}` and `${sys:…}` | *required* | Any other property fails the step. **Example:** ```gherkin Given the mocked addresses service with the following properties: | url | http://localhost:8081 | ``` ### `mock.received` [Section titled “mock.received”](#mockreceived) ```gherkin Then the mocked {word} request to {word} named {word} was received by {mockedService} ``` Check that the mocked service received a request at least once, and name it for the steps that follow. * The request is its method and its exact URL, query string included. * The method is `GET`, `POST`, `PUT`, `DELETE`, `PATCH`, `OPTIONS` or `HEAD`, in capitals. * With the Axx WireMock image, a call that broke the service’s OpenAPI contract fails the step. | Parameter | Takes | For example | | ----------------- | ---------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | **Example:** ```gherkin Then the mocked GET request to /v1/postcodes/DE/10115 named postcode-check was received by addresses ``` ### `mock.received.path` [Section titled “mock.received.path”](#mockreceivedpath) ```gherkin Then the mocked {word} request to path {word} named {word} was received by {mockedService} ``` Check that the mocked service received a request to a path at least once, whatever its query string, and name it for the steps that follow. * For requests whose query changes from call to call, such as a request ID: check the query parameters that matter with `the query parameters for mocked request named ... are:`. * The method is `GET`, `POST`, `PUT`, `DELETE`, `PATCH`, `OPTIONS` or `HEAD`, in capitals. * With the Axx WireMock image, a call that broke the service’s OpenAPI contract fails the step. | Parameter | Takes | For example | | ----------------- | ---------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | **Example:** ```gherkin Then the mocked POST request to path /v1/collections named collection was received by courier ``` ### `mock.openapi.levels` [Section titled “mock.openapi.levels”](#mockopenapilevels) ```gherkin Given the OpenAPI validation levels for the mocked {mockedService} service are: | validation key | level | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Relax the mocked service’s OpenAPI contract for this scenario: a finding that would fail the mock step that checks the call is reported at the level its row sets instead. * `WARN` logs the finding; `INFO` and `IGNORE` drop it. * A key also covers the keys below it: `validation.response.body` covers `validation.response.body.schema.required`. * It applies to the calls this scenario’s mock steps check. A scenario whose mock steps check no call to the service logs a warning. * A stub that is off-contract on purpose is better relaxed in its own metadata (`openApiValidationLevels`), which applies wherever it answers. * This is the dependency’s contract: your own service’s is relaxed with `the OpenAPI validation levels are:`. | Parameter | Takes | For example | | ----------------- | ---------------------------------------------------- | ----------- | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | A row is a validation key and its level: `ERROR` (or `FAIL`), `WARN`, `INFO` or `IGNORE`. The key must be one the Axx WireMock extension reports, or a prefix of such keys, like `validation.response.body`: any other fails the step, which names the closest keys. **Example:** ```gherkin Given the OpenAPI validation levels for the mocked addresses service are: | validation.response.body.schema.additionalProperties | WARN | ``` ### `mock.count.exactly` [Section titled “mock.count.exactly”](#mockcountexactly) ```gherkin Then the mocked request named {word}[[ on {mockedService}]] was received exactly {int} time(s) ``` Check that the mocked service received the named request exactly that many times. | Parameter | Takes | For example | | ----------------- | ---------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | | `{int}` | a whole number | `200`, `3` | **Variants**, the parts in `[[...]]` said or left out: * `the mocked request named {word} was received exactly {int} time(s)` * `the mocked request named {word} on {mockedService} was received exactly {int} time(s)` **Example:** ```gherkin Then the mocked request named postcode-check was received exactly 1 time ``` ### `mock.count.atLeast` [Section titled “mock.count.atLeast”](#mockcountatleast) ```gherkin Then the mocked request named {word}[[ on {mockedService}]] was received at least {int} time(s) ``` Check that the mocked service received the named request at least that many times. | Parameter | Takes | For example | | ----------------- | ---------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | | `{int}` | a whole number | `200`, `3` | **Variants**, the parts in `[[...]]` said or left out: * `the mocked request named {word} was received at least {int} time(s)` * `the mocked request named {word} on {mockedService} was received at least {int} time(s)` **Example:** ```gherkin Then the mocked request named postcode-check was received at least 1 time ``` ### `mock.count.atMost` [Section titled “mock.count.atMost”](#mockcountatmost) ```gherkin Then the mocked request named {word}[[ on {mockedService}]] was received at most {int} time(s) ``` Check that the mocked service received the named request at most that many times. | Parameter | Takes | For example | | ----------------- | ---------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | | `{int}` | a whole number | `200`, `3` | **Variants**, the parts in `[[...]]` said or left out: * `the mocked request named {word} was received at most {int} time(s)` * `the mocked request named {word} on {mockedService} was received at most {int} time(s)` **Example:** ```gherkin Then the mocked request named postcode-check was received at most 1 time ``` ### `mock.notReceived` [Section titled “mock.notReceived”](#mocknotreceived) ```gherkin Then the mocked {word} request to {word} named {word}[[ on {mockedService}]] was not received ``` Check that the mocked service received no request with that method and exact URL, and name it for the steps that follow. | Parameter | Takes | For example | | ----------------- | ---------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | **Variants**, the parts in `[[...]]` said or left out: * `the mocked {word} request to {word} named {word} was not received` * `the mocked {word} request to {word} named {word} on {mockedService} was not received` **Example:** ```gherkin Then the mocked GET request to /v1/postcodes/DE/12489 named skipped-check was not received ``` ### `mock.notReceived.path` [Section titled “mock.notReceived.path”](#mocknotreceivedpath) ```gherkin Then the mocked {word} request to path {word} named {word}[[ on {mockedService}]] was not received ``` Check that the mocked service received no request with that method to a path, whatever its query string, and name it for the steps that follow. * Every request the mocked service received counts, other scenarios’ too: the path must be the scenario’s own, like one with its parcel’s reference. For a path every scenario calls, check that none of its requests has the scenario’s data with `none of the mocked ... requests to path ... have ...`. * The method is `GET`, `POST`, `PUT`, `DELETE`, `PATCH`, `OPTIONS` or `HEAD`, in capitals. | Parameter | Takes | For example | | ----------------- | ---------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | **Variants**, the parts in `[[...]]` said or left out: * `the mocked {word} request to path {word} named {word} was not received` * `the mocked {word} request to path {word} named {word} on {mockedService} was not received` **Example:** ```gherkin Then the mocked POST request to path /v1/parcels/PX-WEB-5302/returns named no-return on courier was not received ``` ### `mock.header.is` [Section titled “mock.header.is”](#mockheaderis) ```gherkin Then the header {word} for mocked request named {word}[[ on {mockedService}]] is {string} ``` Check that the named request was received with a header of that value. Later steps on the named request check this header too. | Parameter | Takes | For example | | ----------------- | ---------------------------------------------------------- | ---------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the header {word} for mocked request named {word} is {string}` * `the header {word} for mocked request named {word} on {mockedService} is {string}` **Example:** ```gherkin Then the header X-Api-Key for mocked request named postcode-check is 'example-address-key' ``` ### `mock.headers.are` [Section titled “mock.headers.are”](#mockheadersare) ```gherkin Then the headers for mocked request named {word} on {mockedService} are: | header | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the named request was received with every header of the table, each with its value. Later steps on the named request check these headers too. | Parameter | Takes | For example | | ----------------- | ---------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | A row is a header’s name and the value it must have. **Example:** ```gherkin Then the headers for mocked request named postcode-check on addresses are: | X-Api-Key | example-address-key | | Accept | application/json | ``` ### `mock.header.matches` [Section titled “mock.header.matches”](#mockheadermatches) ```gherkin Then the header {word} for mocked request named {word} on {mockedService} matches {pattern} ``` Check that the named request was received with a header that matches a regular expression. * WireMock evaluates the expression, in Java syntax; it must match the whole value. * Later steps on the named request check this header too. | Parameter | Takes | For example | | ----------------- | -------------------------------------------------------------------------------- | -------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | | `{pattern}` | a regular expression (Java syntax) with no spaces, which matches the whole value | `PX-\d{4}`, `[A-Z]{2}-\d+` | **Example:** ```gherkin Then the header Accept for mocked request named postcode-check on addresses matches ^application/json$ ``` ### `mock.headers.match` [Section titled “mock.headers.match”](#mockheadersmatch) ```gherkin Then the headers for mocked request named {word} on {mockedService} match: | header | regular expression | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the named request was received with headers that match regular expressions, a row each. * WireMock evaluates the expressions, in Java syntax; each must match the whole value. * Later steps on the named request check these headers too. | Parameter | Takes | For example | | ----------------- | ---------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | A row is a header’s name and a regular expression its value must match. **Example:** ```gherkin Then the headers for mocked request named postcode-check on addresses match: | X-Api-Key | example-.+ | | Accept | application/json.* | ``` ### `mock.header.missing` [Section titled “mock.header.missing”](#mockheadermissing) ```gherkin Then the header {word} for mocked request named {word} on {mockedService} is missing ``` Check that the named request was received without that header. Later steps on the named request check this header too. | Parameter | Takes | For example | | ----------------- | ---------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | **Example:** ```gherkin Then the header Authorization for mocked request named postcode-check on addresses is missing ``` ### `mock.headers.missing` [Section titled “mock.headers.missing”](#mockheadersmissing) ```gherkin Then the headers for mocked request named {word} on {mockedService} are missing: | header | ``` Check that the named request was received without any of the headers the table names. Later steps on the named request check these headers too. | Parameter | Takes | For example | | ----------------- | ---------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | A row names a header, in the table’s only column. **Example:** ```gherkin Then the headers for mocked request named postcode-check on addresses are missing: | Authorization | | Cookie | ``` ### `mock.properties.are` [Section titled “mock.properties.are”](#mockpropertiesare) ```gherkin Then the payload properties for mocked request named {word}[[ on {mockedService}]] are: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the named request was received with a JSON body that has every property of the table, each with its value. * A property is a JSONPath, like `deliverTo.postcode` or `$.lines[0].reference`; WireMock reads it. * Values compare as text: `800` matches the number 800, and `"10115"` the string 10115. `undefined` means the body has no such property, to check that a request leaves something out. * Later steps on the named request check these too, so its counts count only the requests that have them. | Parameter | Takes | For example | | ----------------- | ---------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | A row is a property’s JSONPath and the value it must have, or `undefined`. **Variants**, the parts in `[[...]]` said or left out: * `the payload properties for mocked request named {word} are:` * `the payload properties for mocked request named {word} on {mockedService} are:` **Example:** ```gherkin Then the payload properties for mocked request named collection on courier are: | reference | PX-REG-1401 | | deliverTo.postcode | "10115" | | recipient | undefined | ``` ### `mock.query.are` [Section titled “mock.query.are”](#mockqueryare) ```gherkin Then the query parameters for mocked request named {word}[[ on {mockedService}]] are: | parameter | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the named request was received with every query parameter of the table, each with its value. * Name the request by its path (`the mocked ... request to path ...`): a request named by its whole URL matches its query already. * `undefined` means the query has no such parameter. * Later steps on the named request check these too, so its counts count only the requests that have them. | Parameter | Takes | For example | | ----------------- | ---------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | A row is a query parameter’s name and the value it must have, or `undefined`. **Variants**, the parts in `[[...]]` said or left out: * `the query parameters for mocked request named {word} are:` * `the query parameters for mocked request named {word} on {mockedService} are:` **Example:** ```gherkin Then the query parameters for mocked request named collection on courier are: | slot | same-day | | reference | undefined | ``` ### `mock.form.are` [Section titled “mock.form.are”](#mockformare) ```gherkin Then the form fields for mocked request named {word}[[ on {mockedService}]] are: | field | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the named request was received with a form-encoded body that has every field of the table, each with its value. * `undefined` means the form has no such field. * Later steps on the named request check these too, so its counts count only the requests that have them. | Parameter | Takes | For example | | ----------------- | ---------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | A row is a field’s name and the value it must have, or `undefined`. **Variants**, the parts in `[[...]]` said or left out: * `the form fields for mocked request named {word} are:` * `the form fields for mocked request named {word} on {mockedService} are:` **Example:** ```gherkin Then the form fields for mocked request named pickup-notice on courier are: | reference | PX-WEB-5401 | | day | Friday | ``` ### `mock.query.none` [Section titled “mock.query.none”](#mockquerynone) ```gherkin Then none of the mocked {word} requests to path {word}[[ on {mockedService}]] have the query parameters: | parameter | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the mocked service received no request with that method to a path, whatever its query string, that has every query parameter of the table, each with its value. * Every request the mocked service received counts, other scenarios’ too: put the scenario’s own data in the table, like its parcel’s reference, next to what must not be there. * A check that something did not happen proves little on its own: check what the scenario did send, too. * `undefined` means the request has no such parameter. | Parameter | Takes | For example | | ----------------- | ---------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | A row is a query parameter’s name and the value it has, or `undefined`. **Variants**, the parts in `[[...]]` said or left out: * `none of the mocked {word} requests to path {word} have the query parameters:` * `none of the mocked {word} requests to path {word} on {mockedService} have the query parameters:` **Example:** ```gherkin Then none of the mocked POST requests to path /v1/collections on courier have the query parameters: | slot | same-day | | reference | PX-REG-1402 | ``` ### `mock.properties.none` [Section titled “mock.properties.none”](#mockpropertiesnone) ```gherkin Then none of the mocked {word} requests to path {word}[[ on {mockedService}]] have the payload properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the mocked service received no request with that method to a path, whatever its query string, that has every payload property of the table, each with its value. * Every request the mocked service received counts, other scenarios’ too: put the scenario’s own data in the table, like its parcel’s reference, next to what must not be there. * A check that something did not happen proves little on its own: check what the scenario did send, too. * `undefined` means the request has no such property. | Parameter | Takes | For example | | ----------------- | ---------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | A row is a property’s JSONPath, like `deliverTo.postcode`, and the value it has, or `undefined`. **Variants**, the parts in `[[...]]` said or left out: * `none of the mocked {word} requests to path {word} have the payload properties:` * `none of the mocked {word} requests to path {word} on {mockedService} have the payload properties:` **Example:** ```gherkin Then none of the mocked POST requests to path /v1/collections on courier have the payload properties: | reference | PX-REG-1301 | ``` ### `mock.form.none` [Section titled “mock.form.none”](#mockformnone) ```gherkin Then none of the mocked {word} requests to path {word}[[ on {mockedService}]] have the form fields: | field | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the mocked service received no request with that method to a path, whatever its query string, that has every form field of the table, each with its value. * Every request the mocked service received counts, other scenarios’ too: put the scenario’s own data in the table, like its parcel’s reference, next to what must not be there. * A check that something did not happen proves little on its own: check what the scenario did send, too. * `undefined` means the request has no such field. | Parameter | Takes | For example | | ----------------- | ---------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | A row is a form field’s name and the value it has, or `undefined`. **Variants**, the parts in `[[...]]` said or left out: * `none of the mocked {word} requests to path {word} have the form fields:` * `none of the mocked {word} requests to path {word} on {mockedService} have the form fields:` **Example:** ```gherkin Then none of the mocked POST requests to path /v1/pickups on courier have the form fields: | reference | PX-WEB-5302 | ``` ### `mock.signed` [Section titled “mock.signed”](#mocksigned) ```gherkin Then the mocked request named {word}[[ on {mockedService}]] is signed in the {word} header with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the named request was received, and that every request it names is signed in the header: an HMAC of its body, or of what the `signs` template gives, made with the key. * Axx signs what the mock received, byte for byte, its method and its path with the query, and compares the signatures. * `{timestamp}` is read from the header’s value, or from the timestamp header. * The key expands `${env:..}`, and is masked. | Parameter | Takes | For example | | ----------------- | ---------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | | Property | Takes | Values | Default | | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------- | ------------- | | `key` | the secret the signature is made with, like `${env:COURIER_WEBHOOK_KEY}` | | *required* | | `algorithm` | the HMAC | `hmac-sha256`, `hmac-sha1`, `hmac-sha512` | `hmac-sha256` | | `signs` | what is signed: a template of `{body}`, `{timestamp}` (Unix seconds), `{method}` and `{path}` (with the query), like `{timestamp}.{body}` | | `{body}` | | `encoding` | how the signature is written | `hex`, `base64` | `hex` | | `value` | the header’s value: a template of `{signature}` and `{timestamp}`, like `sha256={signature}` or `t={timestamp},v1={signature}` | | `{signature}` | | `timestamp header` | a header that also carries `{timestamp}`, as Slack’s `X-Slack-Request-Timestamp` does | | | **Variants**, the parts in `[[...]]` said or left out: * `the mocked request named {word} is signed in the {word} header with the following properties:` * `the mocked request named {word} on {mockedService} is signed in the {word} header with the following properties:` **Example:** ```gherkin Then the mocked request named status-callback on shops is signed in the X-Parcels-Signature header with the following properties: | key | ${env:SHOP_WEBHOOK_KEY} | | value | sha256={signature} | ``` *Since 0.1.5.* ### `mock.webhook` [Section titled “mock.webhook”](#mockwebhook) ```gherkin Then the mocked request named {word}[[ on {mockedService}]] is signed as a standard webhook with the key {string} ``` Check that the named request was received, and that every request it names is a Standard Webhook (standardwebhooks.com) signed with the key: its `webhook-signature` has the signature of its `webhook-id`, `webhook-timestamp` and body. * The key is `whsec_` and the key in base64, as Standard Webhooks give it; it expands `${env:..}`, and is masked. | Parameter | Takes | For example | | ----------------- | ---------------------------------------------------------- | ---------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the mocked request named {word} is signed as a standard webhook with the key {string}` * `the mocked request named {word} on {mockedService} is signed as a standard webhook with the key {string}` **Example:** ```gherkin Then the mocked request named delivered on shops is signed as a standard webhook with the key '${env:SHOP_WEBHOOK_KEY}' ``` *Since 0.1.5.* ### `mock.model.asked` [Section titled “mock.model.asked”](#mockmodelasked) ```gherkin Then the mocked {mockedService} model was asked about {string}[[ {int} time(s)]] ``` Check that the mocked model was asked about a text: that a request to it had the text in its conversation, at least once, or as many times as the step says. * The model is a WireMock with the Axx image’s model mock: its stubs are mapping files, and it answers in the format of the request, whichever provider’s API the service speaks (OpenAI’s and the servers that speak it, Anthropic, Gemini, Bedrock, Ollama). Register it as any mocked service. * A request is about a text when its conversation has it: its system prompt, its messages, and the tools’ calls and results, JSON-escaped text included. Every request since the run started counts, other scenarios’ too: name the scenario’s own data, like its parcel’s reference. * A request that continues an earlier response of OpenAI’s Responses API (`previous_response_id`) is about what that response’s requests were about too. | Parameter | Takes | For example | | ----------------- | ---------------------------------------------------------- | ---------------------------- | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | | `{int}` | a whole number | `200`, `3` | **Variants**, the parts in `[[...]]` said or left out: * `the mocked {mockedService} model was asked about {string}` * `the mocked {mockedService} model was asked about {string} {int} time(s)` **Example:** ```gherkin Then the mocked models model was asked about 'PX-AI-8102' Then the mocked models model was asked about 'PX-AI-8104' 2 times ``` *Since 0.1.5.* ### `mock.model.contains` [Section titled “mock.model.contains”](#mockmodelcontains) ```gherkin Then the mocked {mockedService} model's request about {string} contains {string} ``` Check that a request the mocked model was asked about a text sent another: in its messages, its tools’ results, its tools or its settings. * The model is a WireMock with the Axx image’s model mock: its stubs are mapping files, and it answers in the format of the request, whichever provider’s API the service speaks (OpenAI’s and the servers that speak it, Anthropic, Gemini, Bedrock, Ollama). Register it as any mocked service. * A request is about a text when its conversation has it: its system prompt, its messages, and the tools’ calls and results, JSON-escaped text included. Every request since the run started counts, other scenarios’ too: name the scenario’s own data, like its parcel’s reference. * A request that continues an earlier response of OpenAI’s Responses API (`previous_response_id`) is about what that response’s requests were about too. | Parameter | Takes | For example | | ----------------- | ---------------------------------------------------------- | ---------------------------- | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Example:** ```gherkin Then the mocked models model's request about 'PX-AI-8102' contains 'OUT_FOR_DELIVERY' ``` *Since 0.1.5.* ### `mock.model.notContains` [Section titled “mock.model.notContains”](#mockmodelnotcontains) ```gherkin Then the mocked {mockedService} model's request about {string} does not contain {string} ``` Check that no request the mocked model was asked about a text sent another, anywhere: in its messages, its tools’ results, its tools or its settings. The model must have been asked about the text, or the check proves nothing and fails. * The model is a WireMock with the Axx image’s model mock: its stubs are mapping files, and it answers in the format of the request, whichever provider’s API the service speaks (OpenAI’s and the servers that speak it, Anthropic, Gemini, Bedrock, Ollama). Register it as any mocked service. * A request is about a text when its conversation has it: its system prompt, its messages, and the tools’ calls and results, JSON-escaped text included. Every request since the run started counts, other scenarios’ too: name the scenario’s own data, like its parcel’s reference. * A request that continues an earlier response of OpenAI’s Responses API (`previous_response_id`) is about what that response’s requests were about too. | Parameter | Takes | For example | | ----------------- | ---------------------------------------------------------- | ---------------------------- | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Example:** ```gherkin Then the mocked models model's request about 'PX-AI-8102' does not contain 'Lindenweg 14' ``` *Since 0.1.5.* ### `mock.model.tool` [Section titled “mock.model.tool”](#mockmodeltool) ```gherkin Then the mocked {mockedService} model was offered the {word} tool in the request about {string} ``` Check that a request the mocked model was asked about a text offered it the tool, by its name. * The model is a WireMock with the Axx image’s model mock: its stubs are mapping files, and it answers in the format of the request, whichever provider’s API the service speaks (OpenAI’s and the servers that speak it, Anthropic, Gemini, Bedrock, Ollama). Register it as any mocked service. * A request is about a text when its conversation has it: its system prompt, its messages, and the tools’ calls and results, JSON-escaped text included. Every request since the run started counts, other scenarios’ too: name the scenario’s own data, like its parcel’s reference. * A request that continues an earlier response of OpenAI’s Responses API (`previous_response_id`) is about what that response’s requests were about too. | Parameter | Takes | For example | | ----------------- | ---------------------------------------------------------- | ---------------------------- | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Example:** ```gherkin Then the mocked models model was offered the track_parcel tool in the request about 'PX-AI-8102' ``` *Since 0.1.5.* ### `mock.model.schema` [Section titled “mock.model.schema”](#mockmodelschema) ```gherkin Then the mocked {mockedService} model was asked for the {filepath} schema in the request about {string} ``` Check that a request the mocked model was asked about a text asked for its answer in the JSON Schema of a file of the project (JSON or YAML): the structured output the request asks for is that schema, whatever the order of its keys. * The model is a WireMock with the Axx image’s model mock: its stubs are mapping files, and it answers in the format of the request, whichever provider’s API the service speaks (OpenAI’s and the servers that speak it, Anthropic, Gemini, Bedrock, Ollama). Register it as any mocked service. * A request is about a text when its conversation has it: its system prompt, its messages, and the tools’ calls and results, JSON-escaped text included. Every request since the run started counts, other scenarios’ too: name the scenario’s own data, like its parcel’s reference. * A request that continues an earlier response of OpenAI’s Responses API (`previous_response_id`) is about what that response’s requests were about too. | Parameter | Takes | For example | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Example:** ```gherkin Then the mocked models model was asked for the schemas/address.json schema in the request about 'Lindenweg 14' ``` *Since 0.1.5.* ### `mock.mcp.called` [Section titled “mock.mcp.called”](#mockmcpcalled) ```gherkin Then the mocked {mockedService} mcp server's {word} tool was called[[ {int} time(s)]] with the following arguments: | argument | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the service called the tool of a mocked MCP server with the table’s arguments: at least once, or as many times as the step says. * Every call since the run started counts, other scenarios’ too: put the scenario’s own data in the table, like its parcel’s reference. * Paths are dotted (`address.postcode`) or JSONPath, and values compare as text; `null` and `undefined` work as in the other property steps. | Parameter | Takes | For example | | ----------------- | ---------------------------------------------------- | -------------------- | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{int}` | a whole number | `200`, `3` | A row is a path into the call’s arguments, and its value. **Variants**, the parts in `[[...]]` said or left out: * `the mocked {mockedService} mcp server's {word} tool was called with the following arguments:` * `the mocked {mockedService} mcp server's {word} tool was called {int} time(s) with the following arguments:` **Example:** ```gherkin Then the mocked partner-carrier mcp server's shipment_status tool was called with the following arguments: | reference | PX-MCP-9301 | Then the mocked partner-carrier mcp server's shipment_status tool was called 1 time with the following arguments: | reference | PX-MCP-9301 | ``` *Since 0.1.5.* ### `mock.a2a.sent` [Section titled “mock.a2a.sent”](#mocka2asent) ```gherkin Then the mocked {mockedService} a2a agent was sent a message containing {string}[[ {int} time(s)]] ``` Check that the service sent the mocked A2A agent a message whose text has the text (its text parts, and its data parts as JSON): at least once, or as many times as the step says. * Every message since the run started counts, other scenarios’ too: look for the scenario’s own data, like its parcel’s reference. * Messages sent and streamed count alike, over JSON-RPC or HTTP+JSON. | Parameter | Takes | For example | | ----------------- | ---------------------------------------------------------- | ---------------------------- | | `{mockedService}` | the name of a mocked service the scenario registered | `addresses` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | | `{int}` | a whole number | `200`, `3` | **Variants**, the parts in `[[...]]` said or left out: * `the mocked {mockedService} a2a agent was sent a message containing {string}` * `the mocked {mockedService} a2a agent was sent a message containing {string} {int} time(s)` **Example:** ```gherkin Then the mocked partner-carrier a2a agent was sent a message containing 'PX-A2A-9301' Then the mocked partner-carrier a2a agent was sent a message containing 'PX-A2A-9301' 1 time ``` *Since 0.1.5.* ## Parameter types [Section titled “Parameter types”](#parameter-types) | Parameter | Takes | Values | For example | | ----------------- | ---------------------------------------------------- | ------ | ----------- | | `{mockedService}` | the name of a mocked service the scenario registered | | `addresses` | # models-android > Answer SpeechRecognizer, ML Kit built-in image labeling, text recognition, object detection and selfie segmentation, and ML Kit Gemini Nano text generation in debuggable emulator apps from prebuilt .axxmodel files in axx-models. Speech uses log-mel matching for app-provided audio and the microphone step's precomputed matches for queued recordings. Microphone scenarios play recordings into AudioRecord, converted to its rate, channels and PCM format, then silence. MediaRecorder audio capture uses documented JNI registration for its public native controls and public codecs to write the requested file on stop. It supports pause/resume, amplitude, reset/release, paths, File and FileDescriptor outputs, preserving the app's descriptor ownership and microphone permission. It requires the public MediaRecorder(Context) constructor and Java reset/release methods; video, automatic duration/file-size limits and output rotation fail explicitly. The ART agent attaches before application binding. Vision inputs use InputImage.fromBitmap; custom ML Kit model clients retain their native loader. Text generation includes single-chunk streaming through the Prompt API. Answer SpeechRecognizer, ML Kit built-in image labeling, text recognition, object detection and selfie segmentation, and ML Kit Gemini Nano text generation in debuggable emulator apps from prebuilt .axxmodel files in axx-models. Speech uses log-mel matching for app-provided audio and the microphone step’s precomputed matches for queued recordings. Microphone scenarios play recordings into AudioRecord, converted to its rate, channels and PCM format, then silence. MediaRecorder audio capture uses documented JNI registration for its public native controls and public codecs to write the requested file on stop. It supports pause/resume, amplitude, reset/release, paths, File and FileDescriptor outputs, preserving the app’s descriptor ownership and microphone permission. It requires the public MediaRecorder(Context) constructor and Java reset/release methods; video, automatic duration/file-size limits and output rotation fail explicitly. The ART agent attaches before application binding. Vision inputs use InputImage.fromBitmap; custom ML Kit model clients retain their native loader. Text generation includes single-chunk streaming through the Prompt API. Add it to a project with `axx pack add models-android`; it builds on [`models-core`](/next/references/packs/models-core/) and [`mobile-android`](/next/references/packs/mobile-android/) and [`app-core`](/next/references/packs/app-core/), which comes with it. # Core > Stub the models your apps and services run themselves (embedded models): an image classifier on a phone, a model a service loads with ONNX Runtime. A stub is a model in the real one's format, with its inputs and outputs, whose answers are its mappings: the app's runtime loads it as it loads the real model, and an input like one of the mappings' gets that mapping's answer. A stub is a few kilobytes and answers at once, the same every time, where a real model fills a simulator's or an emulator's memory and answers as it likes. Models a service reaches over HTTP are WireMock's: see the model mocks of the WireMock image. Stub the models your apps and services run themselves (embedded models): an image classifier on a phone, a model a service loads with ONNX Runtime. A stub is a model in the real one’s format, with its inputs and outputs, whose answers are its mappings: the app’s runtime loads it as it loads the real model, and an input like one of the mappings’ gets that mapping’s answer. A stub is a few kilobytes and answers at once, the same every time, where a real model fills a simulator’s or an emulator’s memory and answers as it likes. Models a service reaches over HTTP are WireMock’s: see the model mocks of the WireMock image. A stub’s directory has its model’s contract, `contract.json`, what the app relies on from the model: `axx models contract ` writes it from the real model, once, and the project keeps it, so no test needs the real model. Its `mappings` folder has the mappings, as WireMock’s has its stubs: each JSON file a mapping, or several under `mappings`. A mapping’s `input` is a picture (`image`) or a tensor’s values (`tensor`), and its `output` the answer, in the terms of the model’s task (a classifier’s `label` and `score`) or as the outputs’ values (`tensors`). An input matches the mapping whose input it is most like, when it is alike enough (the contract’s `match.similarity`), and the default mapping (`"default": true`) answers the others. `axx models build -o ` writes the stub, which the app’s test build ships in place of the real model: an Android app’s test assets, an iOS app’s test target, a service’s test image. Stubs are written in the real model’s format: ONNX, TFLite (LiteRT), Core ML, ExecuTorch, TensorFlow (SavedModel and TensorFlow\.js), Keras, TorchScript, MNN, ncnn, OpenVINO, GGUF (llama.cpp), safetensors (transformers), LiteRT-LM, CTranslate2 (faster-whisper), whisper.cpp and OpenAI Whisper; Tesseract; Piper; and Apple’s Foundation Models, whose stub is a Swift type an app’s test build gives its sessions. A pack adds formats, tasks and kinds of input of its own: see the stubs guide. Add it to a project with `axx pack add models-core`. ## Steps [Section titled “Steps”](#steps) ### `models-core.microphone` [Section titled “models-core.microphone”](#models-coremicrophone) ```gherkin When the {word} app's microphone hears the {string} file ``` The app’s microphone hears a recording (a WAV file). Recordings wait, in order, until the app listens: the app’s platform speech stub answers the mapping the recording matches (Axx matches it as the stub’s keys were made), and a web page’s own microphone (getUserMedia) records it. The file’s name selects no answer. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Example:** ```gherkin When the courier app's microphone hears the "recordings/door.wav" file ``` *Since 0.2.0.* ### `models-core.speaker` [Section titled “models-core.speaker”](#models-corespeaker) ```gherkin Then [[within {duration} ]]the {word} app's speaker plays the {string} file ``` Check that the app has spoken a recording (a WAV file): an utterance of its speech has the recording’s sound. The app speaks with a voice stub, which speaks a mapping’s recording for a text that has the mapping’s key: the file is the recording the app’s words are spoken as. The app’s platform pack hears its speech (models-linux: speech-dispatcher’s, in the desktops Axx runs; models-windows: SAPI’s). The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} app's speaker plays the {string} file` * `within {duration} the {word} app's speaker plays the {string} file` **Example:** ```gherkin Then within 10s the depot app's speaker plays the "recordings/handover-rules.wav" file ``` *Since 0.2.0.* ## Commands [Section titled “Commands”](#commands) What the pack adds to Axx’s command line, in a project that uses it. | Command | | | --------------------------------------------------------- | --------------------------------------------------------------- | | [`axx models contract`](/next/references/cli/axx-models/) | Write a model’s contract, what a stub of it keeps to | | [`axx models build`](/next/references/cli/axx-models/) | Build a stub, a model of its contract that answers its mappings | # models-ios > Answer supported Speech and Vision calls in debug iOS Simulator apps from prebuilt .axxmodel files in the app's axx-models folder. Speech uses log-mel matching for app-provided audio and the microphone step's precomputed matches for queued recordings. Microphone scenarios feed AVAudioEngine input taps and connections, AVAudioRecorder files (including duration, pause and metering), and AVCaptureSession audio data outputs, then silence. A tap-fed recognition shares its capture's recording and finishes when that recording ends or the app ends audio. Input nodes use 48 kHz mono float PCM independently of the host microphone; taps receive their requested PCM format. Audio data output requires a simulator audio capture device and microphone permission. Foundation Models text sessions, of the system model and of Apple's server model (PrivateCloudComputeLanguageModel; a session of a model of the app's own is left as it is), use the public provider API on iOS 27 or later; a stub whose contract gives format.details.availability (deviceNotEligible, appleIntelligenceNotEnabled or modelNotReady) is an iPhone without the system model, which reports it unavailable. Copy stubs before launching the app; restarting reloads them. Answer supported Speech and Vision calls in debug iOS Simulator apps from prebuilt .axxmodel files in the app’s axx-models folder. Speech uses log-mel matching for app-provided audio and the microphone step’s precomputed matches for queued recordings. Microphone scenarios feed AVAudioEngine input taps and connections, AVAudioRecorder files (including duration, pause and metering), and AVCaptureSession audio data outputs, then silence. A tap-fed recognition shares its capture’s recording and finishes when that recording ends or the app ends audio. Input nodes use 48 kHz mono float PCM independently of the host microphone; taps receive their requested PCM format. Audio data output requires a simulator audio capture device and microphone permission. Foundation Models text sessions, of the system model and of Apple’s server model (PrivateCloudComputeLanguageModel; a session of a model of the app’s own is left as it is), use the public provider API on iOS 27 or later; a stub whose contract gives format.details.availability (deviceNotEligible, appleIntelligenceNotEnabled or modelNotReady) is an iPhone without the system model, which reports it unavailable. Copy stubs before launching the app; restarting reloads them. Add it to a project with `axx pack add models-ios`; it builds on [`models-core`](/next/references/packs/models-core/) and [`mobile-ios`](/next/references/packs/mobile-ios/) and [`app-core`](/next/references/packs/app-core/), which comes with it. # models-linux > Answer the models Linux apps take from the system, in the desktops desktop-linux runs, from stubs in the app's axx-models folder (in its `./` files, its home's .local/share): Tesseract's language data (a stub of the tesseract format, like `eng.traineddata`), which Tesseract reads from the folder its documented TESSDATA_PREFIX names, and the voice of the desktop's speech service, speech-dispatcher (a Piper voice stub: its `.onnx` and `.onnx.json`), which speaks the app's text with Piper (its `piper` program, on PATH) into the folder the speaker step hears. Axx gives the app's speech-dispatcher its user configuration, in the app's home (`.config/speech-dispatcher`): the voice is its only one, and no sound plays (its audio is ALSA's null device, which needs speech-dispatcher's audio plugins). Copy stubs before launching the app; restarting reloads them. Answer the models Linux apps take from the system, in the desktops desktop-linux runs, from stubs in the app’s axx-models folder (in its `./` files, its home’s .local/share): Tesseract’s language data (a stub of the tesseract format, like `eng.traineddata`), which Tesseract reads from the folder its documented TESSDATA\_PREFIX names, and the voice of the desktop’s speech service, speech-dispatcher (a Piper voice stub: its `.onnx` and `.onnx.json`), which speaks the app’s text with Piper (its `piper` program, on PATH) into the folder the speaker step hears. Axx gives the app’s speech-dispatcher its user configuration, in the app’s home (`.config/speech-dispatcher`): the voice is its only one, and no sound plays (its audio is ALSA’s null device, which needs speech-dispatcher’s audio plugins). Copy stubs before launching the app; restarting reloads them. Add it to a project with `axx pack add models-linux`; it builds on [`models-core`](/next/references/packs/models-core/) and [`desktop-linux`](/next/references/packs/desktop-linux/) and [`app-core`](/next/references/packs/app-core/), which comes with it. # models-macos > Answer Mac apps' calls to Apple's models from prebuilt .axxmodel files in the app's axx-models folder (in its `./` files, its home's Library/Application Support): Vision (image classification, text recognition, human and animal rectangles, person segmentation), Speech (SFSpeechRecognizer) and Foundation Models text sessions, of the system model and of Apple's server model (PrivateCloudComputeLanguageModel; a session of a model of the app's own is left as it is), as models-ios does in the iOS Simulator, with the same contracts (`macos.*` APIs). desktop-macos loads Axx's library into the app as it starts, from its environment (DYLD_INSERT_LIBRARIES): macOS lets it into an app signed without the hardened runtime, or with the `com.apple.security.cs.allow-dyld-environment-variables` and `com.apple.security.cs.disable-library-validation` entitlements (the app's test build's), which Axx checks before it starts the app. Speech uses log-mel matching for app-provided audio and the microphone step's precomputed matches for queued recordings; microphone scenarios feed AVAudioEngine input taps and connections, AVAudioRecorder files and AVCaptureSession audio data outputs, then silence (a capture session still opens the Mac's audio capture device, which needs one, and the user's permission to use it). A Mac has no test's grant of speech recognition or the microphone: a stubbed recognizer, and the microphone Axx feeds, are authorized. A stub whose contract gives format.details.availability (deviceNotEligible, appleIntelligenceNotEnabled or modelNotReady) is a Mac without the system model, which reports it unavailable. Foundation Models stubs need macOS 27 or later; the rest, macOS 15. Copy stubs before launching the app; restarting reloads them. Answer Mac apps’ calls to Apple’s models from prebuilt .axxmodel files in the app’s axx-models folder (in its `./` files, its home’s Library/Application Support): Vision (image classification, text recognition, human and animal rectangles, person segmentation), Speech (SFSpeechRecognizer) and Foundation Models text sessions, of the system model and of Apple’s server model (PrivateCloudComputeLanguageModel; a session of a model of the app’s own is left as it is), as models-ios does in the iOS Simulator, with the same contracts (`macos.*` APIs). desktop-macos loads Axx’s library into the app as it starts, from its environment (DYLD\_INSERT\_LIBRARIES): macOS lets it into an app signed without the hardened runtime, or with the `com.apple.security.cs.allow-dyld-environment-variables` and `com.apple.security.cs.disable-library-validation` entitlements (the app’s test build’s), which Axx checks before it starts the app. Speech uses log-mel matching for app-provided audio and the microphone step’s precomputed matches for queued recordings; microphone scenarios feed AVAudioEngine input taps and connections, AVAudioRecorder files and AVCaptureSession audio data outputs, then silence (a capture session still opens the Mac’s audio capture device, which needs one, and the user’s permission to use it). A Mac has no test’s grant of speech recognition or the microphone: a stubbed recognizer, and the microphone Axx feeds, are authorized. A stub whose contract gives format.details.availability (deviceNotEligible, appleIntelligenceNotEnabled or modelNotReady) is a Mac without the system model, which reports it unavailable. Foundation Models stubs need macOS 27 or later; the rest, macOS 15. Copy stubs before launching the app; restarting reloads them. Add it to a project with `axx pack add models-macos`; it builds on [`models-core`](/next/references/packs/models-core/) and [`desktop-macos`](/next/references/packs/desktop-macos/) and [`app-core`](/next/references/packs/app-core/), which comes with it. # models-web > Answer a browser's built-in AI (the Prompt API's LanguageModel, Summarizer, Writer, Rewriter, Translator, LanguageDetector, Proofreader) and SpeechRecognition from prebuilt .axxmodel files in the web app's axx-models folder, and play the microphone step's recordings into the page's own microphone (getUserMedia). Copy the files into the app's folder before opening its first page. The adapter runs before the page's scripts, in its frames of its origin and its popups. Answer a browser’s built-in AI (the Prompt API’s LanguageModel, Summarizer, Writer, Rewriter, Translator, LanguageDetector, Proofreader) and SpeechRecognition from prebuilt .axxmodel files in the web app’s axx-models folder, and play the microphone step’s recordings into the page’s own microphone (getUserMedia). Copy the files into the app’s folder before opening its first page. The adapter runs before the page’s scripts, in its frames of its origin and its popups. Add it to a project with `axx pack add models-web`; it builds on [`models-core`](/next/references/packs/models-core/) and [`web-core`](/next/references/packs/web-core/) and [`app-core`](/next/references/packs/app-core/), which comes with it. # models-windows > Answer the models Windows apps take from Windows, from the stubs in the app's axx-models folder (in its `./` files, its home's AppData\Roaming), in the desktops desktop-windows runs: the voice they speak with, SAPI's, from a Piper voice stub, and the Windows AI APIs, from stubs of them. **The voice.** SAPI's (System.Speech's, and native SAPI's), from a Piper voice stub (its `.onnx` and `.onnx.json`). As the app starts, Axx adds its voice engine (built with the .NET SDK, 8 or later, into Axx's cache) to SAPI's voices, as `axx voice`, makes it the user's default voice, and gives the app the stub: the voice speaks each text with Piper (its Windows release, which Axx downloads, pinned) into SAPI's output and into the folder the speaker step hears. SAPI lists the voices in HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Speech\Voices\Tokens only, so Axx runs as an administrator, as GitHub's Windows runners do, or as a user who may create keys there; the scenario's end takes the voice out again and puts the user's default voice back. The engine is 64-bit .NET: it speaks in 64-bit apps. **The Windows AI APIs.** The Windows App SDK's: TextRecognizer, ImageDescriptionGenerator, ImageObjectExtractor, and the language model, Phi Silica, with its skills, TextSummarizer, TextRewriter and TextToTableConverter (`windows.ai.*`: `axx models contract` writes their contracts), in a self-contained build (WindowsAppSDKSelfContained), whose own folder has their files, which its manifest maps their classes to. As the app starts, Axx puts its library in place of the app's file of each API the folder has a stub of (Microsoft.Windows.AI.Imaging.dll, Microsoft.Windows.AI.Text.dll), and the app's own back as the scenario ends: a class the folder has a stub of is the library's, any other the app's file's; the language model's are the library's together, with a stub of the model or a skill. Axx builds the library once for each app, against the app's .winmd files, with C++/WinRT (its release, which Axx downloads, pinned) and MSVC (Visual Studio's, or its Build Tools', C++ tools and a Windows 11 SDK). The library answers as Windows does: only an app with package identity whose package declares the systemAIModels capability. A stub of a PC without the model gives the reason (format.details.availability) in Windows' words: NotSupportedOnCurrentSystem, or NotReady, which EnsureReadyAsync makes ready. Answer the models Windows apps take from Windows, from the stubs in the app’s axx-models folder (in its `./` files, its home’s AppData\Roaming), in the desktops desktop-windows runs: the voice they speak with, SAPI’s, from a Piper voice stub, and the Windows AI APIs, from stubs of them. **The voice.** SAPI’s (System.Speech’s, and native SAPI’s), from a Piper voice stub (its `.onnx` and `.onnx.json`). As the app starts, Axx adds its voice engine (built with the .NET SDK, 8 or later, into Axx’s cache) to SAPI’s voices, as `axx voice`, makes it the user’s default voice, and gives the app the stub: the voice speaks each text with Piper (its Windows release, which Axx downloads, pinned) into SAPI’s output and into the folder the speaker step hears. SAPI lists the voices in HKEY\_LOCAL\_MACHINE\SOFTWARE\Microsoft\Speech\Voices\Tokens only, so Axx runs as an administrator, as GitHub’s Windows runners do, or as a user who may create keys there; the scenario’s end takes the voice out again and puts the user’s default voice back. The engine is 64-bit .NET: it speaks in 64-bit apps. **The Windows AI APIs.** The Windows App SDK’s: TextRecognizer, ImageDescriptionGenerator, ImageObjectExtractor, and the language model, Phi Silica, with its skills, TextSummarizer, TextRewriter and TextToTableConverter (`windows.ai.*`: `axx models contract` writes their contracts), in a self-contained build (WindowsAppSDKSelfContained), whose own folder has their files, which its manifest maps their classes to. As the app starts, Axx puts its library in place of the app’s file of each API the folder has a stub of (Microsoft.Windows.AI.Imaging.dll, Microsoft.Windows.AI.Text.dll), and the app’s own back as the scenario ends: a class the folder has a stub of is the library’s, any other the app’s file’s; the language model’s are the library’s together, with a stub of the model or a skill. Axx builds the library once for each app, against the app’s .winmd files, with C++/WinRT (its release, which Axx downloads, pinned) and MSVC (Visual Studio’s, or its Build Tools’, C++ tools and a Windows 11 SDK). The library answers as Windows does: only an app with package identity whose package declares the systemAIModels capability. A stub of a PC without the model gives the reason (format.details.availability) in Windows’ words: NotSupportedOnCurrentSystem, or NotReady, which EnsureReadyAsync makes ready. Add it to a project with `axx pack add models-windows`; it builds on [`models-core`](/next/references/packs/models-core/) and [`desktop-windows`](/next/references/packs/desktop-windows/) and [`app-core`](/next/references/packs/app-core/), which comes with it. # MongoDB > Register MongoDB databases, seed collections from JSON files, and query and assert on documents. Register MongoDB databases, seed collections from JSON files, and query and assert on documents. Add it to a project with `axx pack add mongo`. ## Steps [Section titled “Steps”](#steps) ### `mongo.service` [Section titled “mongo.service”](#mongoservice) ```gherkin Given a(n) {word} mongo database with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register a MongoDB database. The first one registered in a scenario is the default. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Default | | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `url` | the connection string, which must name the database, like `mongodb://localhost:27017/parcels`; its `authSource` defaults to that database | *required* | | `user` | the user Axx connects as | *required* | | `password` | the user’s password | *required* | Every value can take `${env:…}` and `${sys:…}` references. **Example:** ```gherkin Given a tracking-db mongo database with the following properties: | url | mongodb://localhost:27017/parcels?authSource=admin | | user | parcels | | password | parcels | ``` ### `mongo.seed` [Section titled “mongo.seed”](#mongoseed) ```gherkin Given a {filepath} mongo db seed ``` Insert the documents of a JSON file into the default MongoDB database. The file maps each collection’s name to an array of its documents, in Extended JSON: `{"$date": "2026-05-05T06:40:00Z"}` is a date and `{"$oid": ...}` an ObjectId. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | **Example:** ```gherkin Given a seeds/scans-in-transit.json mongo db seed ``` ### `mongo.seed.named` [Section titled “mongo.seed.named”](#mongoseednamed) ```gherkin Given a {filepath} MongoDB seed for {word} ``` Insert the documents of a JSON file into the named MongoDB database; the file is as for the default database’s seed. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin Given a seeds/scans-in-transit.json MongoDB seed for tracking-db ``` ### `mongo.seed.named.alt` [Section titled “mongo.seed.named.alt”](#mongoseednamedalt) ```gherkin Given a {filepath} mongo db seed for {word} ``` Insert the documents of a JSON file into the named MongoDB database; the file is as for the default database’s seed. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin Given a seeds/scans-in-transit.json mongo db seed for tracking-db ``` *Since 0.1.0.* ### `mongo.find` [Section titled “mongo.find”](#mongofind) ```gherkin Then a[[ {ordinal}]] selection of documents is retrieved from the {word} collection[[ on {mongoService}]] where: | field | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Find the documents that match and keep them as the next selection, for the steps that check it. Selections are numbered in the order they are retrieved, whatever ordinal the step says; `the selection` means the first. | Parameter | Takes | For example | | ---------------- | ---------------------------------------------------------------------- | -------------------- | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{mongoService}` | the name of a MongoDB database registered in the scenario | `tracking-db` | Each row names a field, dotted (`recipient.city`) for one inside a document, and the value it holds: JSON when it reads as JSON (`3`, `true`, `null`, `"3"`, `{"$oid": "..."}`), text otherwise. **Variants**, the parts in `[[...]]` said or left out: * `a selection of documents is retrieved from the {word} collection where:` * `a {ordinal} selection of documents is retrieved from the {word} collection where:` * `a selection of documents is retrieved from the {word} collection on {mongoService} where:` * `a {ordinal} selection of documents is retrieved from the {word} collection on {mongoService} where:` **Example:** ```gherkin Then a selection of documents is retrieved from the scans collection where: | parcelRef | PX-TRK-3001 | | status | IN_TRANSIT | ``` *Since 0.1.0.* ### `mongo.find.poll` [Section titled “mongo.find.poll”](#mongofindpoll) ```gherkin Then within {duration} a[[ {ordinal}]] selection of at least {int} document(s) is retrieved from the {word} collection[[ on {mongoService}]] where: | field | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Find the documents that match, again every 500ms, until at least that many come back, and keep them as the next selection. * The step fails when the time is up with fewer documents. * An error from MongoDB fails the step only when the time is up. * Selections are numbered in the order they are retrieved, whatever ordinal the step says; `the selection` means the first. | Parameter | Takes | For example | | ---------------- | ---------------------------------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{int}` | a whole number | `200`, `3` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{mongoService}` | the name of a MongoDB database registered in the scenario | `tracking-db` | Each row names a field, dotted (`recipient.city`) for one inside a document, and the value it holds: JSON when it reads as JSON (`3`, `true`, `null`, `"3"`, `{"$oid": "..."}`), text otherwise. **Variants**, the parts in `[[...]]` said or left out: * `within {duration} a selection of at least {int} document(s) is retrieved from the {word} collection where:` * `within {duration} a {ordinal} selection of at least {int} document(s) is retrieved from the {word} collection where:` * `within {duration} a selection of at least {int} document(s) is retrieved from the {word} collection on {mongoService} where:` * `within {duration} a {ordinal} selection of at least {int} document(s) is retrieved from the {word} collection on {mongoService} where:` **Example:** ```gherkin Then within 10s a selection of at least 1 document is retrieved from the tracking collection where: | _id | PX-TRK-3001 | | scanCount | 2 | ``` *Since 0.1.0.* ### `mongo.docs.eq` [Section titled “mongo.docs.eq”](#mongodocseq) ```gherkin Then the[[ {ordinal}]] selection[[ on {mongoService}]] has {int} document(s) ``` Check that a selection of documents has exactly the given number of documents. `the selection` means the first selection of the scenario. | Parameter | Takes | For example | | ---------------- | ---------------------------------------------------------------------- | ------------- | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{mongoService}` | the name of a MongoDB database registered in the scenario | `tracking-db` | | `{int}` | a whole number | `200`, `3` | **Variants**, the parts in `[[...]]` said or left out: * `the selection has {int} document(s)` * `the {ordinal} selection has {int} document(s)` * `the selection on {mongoService} has {int} document(s)` * `the {ordinal} selection on {mongoService} has {int} document(s)` **Example:** ```gherkin Then the selection has 2 documents Then the 2nd selection on tracking-db has 1 document ``` *Since 0.1.0.* ### `mongo.docs.gt` [Section titled “mongo.docs.gt”](#mongodocsgt) ```gherkin Then the[[ {ordinal}]] selection[[ on {mongoService}]] has more than {int} document(s) ``` Check that a selection of documents has more than the given number of documents. `the selection` means the first selection of the scenario. | Parameter | Takes | For example | | ---------------- | ---------------------------------------------------------------------- | ------------- | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{mongoService}` | the name of a MongoDB database registered in the scenario | `tracking-db` | | `{int}` | a whole number | `200`, `3` | **Variants**, the parts in `[[...]]` said or left out: * `the selection has more than {int} document(s)` * `the {ordinal} selection has more than {int} document(s)` * `the selection on {mongoService} has more than {int} document(s)` * `the {ordinal} selection on {mongoService} has more than {int} document(s)` **Example:** ```gherkin Then the selection has more than 2 documents Then the 2nd selection on tracking-db has more than 1 document ``` *Since 0.1.0.* ### `mongo.docs.lt` [Section titled “mongo.docs.lt”](#mongodocslt) ```gherkin Then the[[ {ordinal}]] selection[[ on {mongoService}]] has fewer than {int} document(s) ``` Check that a selection of documents has fewer than the given number of documents. `the selection` means the first selection of the scenario. | Parameter | Takes | For example | | ---------------- | ---------------------------------------------------------------------- | ------------- | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{mongoService}` | the name of a MongoDB database registered in the scenario | `tracking-db` | | `{int}` | a whole number | `200`, `3` | **Variants**, the parts in `[[...]]` said or left out: * `the selection has fewer than {int} document(s)` * `the {ordinal} selection has fewer than {int} document(s)` * `the selection on {mongoService} has fewer than {int} document(s)` * `the {ordinal} selection on {mongoService} has fewer than {int} document(s)` **Example:** ```gherkin Then the selection has fewer than 2 documents Then the 2nd selection on tracking-db has fewer than 1 document ``` *Since 0.1.0.* ### `mongo.doc.are` [Section titled “mongo.doc.are”](#mongodocare) ```gherkin Then the {ordinal} document for the[[ {ordinal}]] selection[[ on {mongoService}]] properties are: | JSONPath | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check the fields of one document of a selection, read as JSON: an ObjectId is its hex string, and a date is in ISO-8601 UTC, like `2026-05-05T06:40:00.000Z`. | Parameter | Takes | For example | | ---------------- | ---------------------------------------------------------------------- | ------------- | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{mongoService}` | the name of a MongoDB database registered in the scenario | `tracking-db` | Each row names a JSONPath, like `lastLocation` or `scans[0].status`, and the value there as text, like `true` or `2`: `null` for null, `undefined` for a field the document lacks. **Variants**, the parts in `[[...]]` said or left out: * `the {ordinal} document for the selection properties are:` * `the {ordinal} document for the {ordinal} selection properties are:` * `the {ordinal} document for the selection on {mongoService} properties are:` * `the {ordinal} document for the {ordinal} selection on {mongoService} properties are:` **Example:** ```gherkin Then the 1st document for the selection properties are: | status | DELIVERED | | lastLocation | Leipzig | | delivered | true | ``` *Since 0.1.0.* ### `mongo.doc.match` [Section titled “mongo.doc.match”](#mongodocmatch) ```gherkin Then the {ordinal} document for the[[ {ordinal}]] selection[[ on {mongoService}]] properties match: | JSONPath | pattern | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check the fields of one document of a selection, like the `properties are` step, with regular expressions (Java syntax) that must match the whole value, as text. | Parameter | Takes | For example | | ---------------- | ---------------------------------------------------------------------- | ------------- | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{mongoService}` | the name of a MongoDB database registered in the scenario | `tracking-db` | Each row names a JSONPath, like `_id` or `scans[0].status`, and a regular expression its value must match. **Variants**, the parts in `[[...]]` said or left out: * `the {ordinal} document for the selection properties match:` * `the {ordinal} document for the {ordinal} selection properties match:` * `the {ordinal} document for the selection on {mongoService} properties match:` * `the {ordinal} document for the {ordinal} selection on {mongoService} properties match:` **Example:** ```gherkin Then the 1st document for the 2nd selection on tracking-db properties match: | _id | PX-TRK-\d{4} | | lastLocation | Hamburg.* | ``` *Since 0.1.0.* ## Parameter types [Section titled “Parameter types”](#parameter-types) | Parameter | Takes | Values | For example | | ---------------- | --------------------------------------------------------- | ------ | ------------- | | `{mongoService}` | the name of a MongoDB database registered in the scenario | | `tracking-db` | # MQTT > Publish messages to MQTT topics, and check the messages your services publish there, over MQTT 5 (Mosquitto, EMQX, HiveMQ, RabbitMQ's MQTT plugin). Publish messages to MQTT topics, and check the messages your services publish there, over MQTT 5 (Mosquitto, EMQX, HiveMQ, RabbitMQ’s MQTT plugin). Register the broker once, usually in the `Background`; every MQTT step of the scenario uses it: ```gherkin Given the depots mqtt broker with the following properties: | url | mqtt://localhost:1883 | | username | depot-scanners | | password | ${env:MQTT_PASSWORD} | ``` Steps name a **topic**: `the depots/LEJ/scans mqtt topic`. Messages carry user properties (`property ` rows), and are published at QoS 1, not retained, unless the `qos` and `retain` rows say otherwise. * **A check subscribes for the run:** for the topics a run checks, Axx subscribes once the services are up, with a connection of its own. A topic in a check can be a filter with `+` and `#`: `the depots/+/alerts mqtt topic`, and a `topic` row checks the topic a message came on. * **A check only looks at the messages received since its scenario started,** so scenarios running in parallel check their own messages, by data unique to them. * **Retained messages from before never count:** Axx subscribes without the messages the broker keeps for new subscribers, which earlier runs may have left. * **Secrets stay secret:** `${env:..}` values in the properties are masked in logs and failures. Add it to a project with `axx pack add mqtt`. ## Steps [Section titled “Steps”](#steps) ### `mqtt.broker` [Section titled “mqtt.broker”](#mqttbroker) ```gherkin Given the {word} mqtt broker with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register the MQTT broker the steps talk to. * The first broker registered is the one the scenario’s MQTT steps use. * Values expand `${env:..}` and `${sys:..}`; `${env:..}` values are masked. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Default | | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | | `url` | the broker’s URL: `mqtt://host:1883`, `mqtts://` for TLS, or `ws://` and `wss://` for MQTT over WebSockets | | | `username` | the user Axx connects as | | | `password` | its password, like `${env:MQTT_PASSWORD}` | | | `client id` | the start of the client IDs Axx connects with, one per connection | `axx-` | | `asyncapi` | an AsyncAPI document, a file of the project or a URL, that the topics follow: every message a scenario sends there, and every message a check finds, is checked against it (with the asyncapi pack) | | **Example:** ```gherkin Given the depots mqtt broker with the following properties: | url | mqtt://localhost:1883 | Given the depots mqtt broker with the following properties: | url | mqtts://mqtt.parcels.example:8883 | | username | depot-scanners | | password | ${env:MQTT_PASSWORD} | ``` *Since 0.1.5.* ### `mqtt.send` [Section titled “mqtt.send”](#mqttsend) ```gherkin When a message is published to the {word} mqtt topic: """ ... """ ``` Send a message to the mqtt topic; the doc string is its body. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When a message is published to the depots/LEJ/scans mqtt topic: """ {"scanId": "SC-8201-1", "parcelRef": "PX-8201", "status": "OUT_FOR_DELIVERY", "location": "Leipzig"} """ ``` *Since 0.1.5.* ### `mqtt.send.file` [Section titled “mqtt.send.file”](#mqttsendfile) ```gherkin When the {filepath} message is published to the {word} mqtt topic[[ with the following properties:]] ``` Send a message whose body is the file to the mqtt topic, with the properties of the table when the step has one. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Name | Takes | Values | Default | | ------------------ | -------------------------------------------------------------------------------- | --------------- | ------- | | `qos` | the quality of service it is published at | `0`, `1`, `2` | `1` | | `retain` | whether the broker keeps it for new subscribers | `true`, `false` | `false` | | `property ` | a user property sent with the message: its name after `property `, and its value | | | | `content type` | the body’s media type | | | | `response topic` | the topic a reply goes to | | | | `correlation data` | what ties a reply to its request, as text | | | **Variants**, the parts in `[[...]]` said or left out: * `the {filepath} message is published to the {word} mqtt topic` * `the {filepath} message is published to the {word} mqtt topic with the following properties:` **Example:** ```gherkin When the mqtt/scan-unknown.json message is published to the depots/LEJ/scans mqtt topic When the mqtt/scan-unknown.json message is published to the depots/LEJ/scans mqtt topic with the following properties: | property scanner | LEJ-HANDHELD-7 | | qos | 1 | ``` *Since 0.1.5.* ### `mqtt.received` [Section titled “mqtt.received”](#mqttreceived) ```gherkin Then [[within {duration} ]]the {word} mqtt topic has a message where: | path | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the mqtt topic has a message with those values, received since the scenario started. * The check waits for it: 10 seconds, or `within {duration}`. * Axx subscribes to a topic it checks for the run, without the retained messages earlier runs left. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a path into the message’s JSON body (a field name, a dotted path or a JSONPath) and the value it has, compared as text: `null` for null and `undefined` for absent. A row `property ` is instead on the property of that name sent with the message: `undefined` when there is none. Rows `topic`, `content type`, `response topic` and `correlation data` are on the message itself, not its body. **Variants**, the parts in `[[...]]` said or left out: * `the {word} mqtt topic has a message where:` * `within {duration} the {word} mqtt topic has a message where:` **Example:** ```gherkin Then within 30s the depots/+/alerts mqtt topic has a message where: | topic | depots/LEJ/alerts | | parcelRef | PX-8299 | | property scanner | LEJ-HANDHELD-7 | ``` *Since 0.1.5.* # NATS > Publish messages to NATS subjects, and check the messages your services publish on subjects and into JetStream streams. Publish messages to NATS subjects, and check the messages your services publish on subjects and into JetStream streams. Register the server once, usually in the `Background`; every NATS step of the scenario uses it: ```gherkin Given the tracking nats server with the following properties: | url | nats://localhost:4222 | | token | ${env:NATS_TOKEN} | ``` Steps name a **subject** or a **stream**: `the deliveries.confirmed nats subject`, `the TRACKING nats stream`. Messages carry headers (`with the following headers:`, and `header ` rows in checks). A message published to a subject a stream captures is published through JetStream, and the step waits for the stream to store it. * **A subject check subscribes for the run:** for the subjects a run checks, Axx subscribes once the services are up. A subject in a check can have the wildcards `*` and `>`: `the tracking.> nats subject`, and a `subject` row checks the subject a message came on. * **A stream check reads only what is new:** for the streams a run checks, Axx reads each with an ordered consumer of its own that starts with the messages stored after the services are up, so what earlier runs stored never counts. It takes nothing from the stream’s other consumers; a work-queue stream allows no other consumer, so check its subject instead. * **A check only looks at the messages received since its scenario started,** so scenarios running in parallel check their own messages, by data unique to them. * **Secrets stay secret:** `${env:..}` values in the properties are masked in logs and failures. Add it to a project with `axx pack add nats`. ## Steps [Section titled “Steps”](#steps) ### `nats.server` [Section titled “nats.server”](#natsserver) ```gherkin Given the {word} nats server with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register the NATS server the steps talk to. * The first server registered is the one the scenario’s NATS steps use. * Give it at most one of a `token`, a `username` and `password`, and `creds`. * Values expand `${env:..}` and `${sys:..}`; `${env:..}` values are masked. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `url` | the server’s URL, `nats://host:4222` or `tls://` for TLS; several, separated by commas, for a cluster | | `token` | the token Axx connects with, like `${env:NATS_TOKEN}` | | `username` | the user Axx connects as | | `password` | its password | | `creds` | a credentials file (JWT and seed), relative to the directory of axx.yaml | | `asyncapi` | an AsyncAPI document, a file of the project or a URL, that the subjects follow: every message a scenario sends there, and every message a check finds, is checked against it (with the asyncapi pack) | **Example:** ```gherkin Given the tracking nats server with the following properties: | url | nats://localhost:4222 | Given the tracking nats server with the following properties: | url | tls://nats.parcels.example:4222 | | creds | nats/axx.creds | ``` *Since 0.1.5.* ### `nats.subject.send` [Section titled “nats.subject.send”](#natssubjectsend) ```gherkin When a message is published to the {word} nats subject: """ ... """ ``` Send a message to the nats subject; the doc string is its body. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When a message is published to the deliveries.confirmed nats subject: """ {"reference": "PX-8301", "signedBy": "H. Wolf", "deliveredAt": "2026-09-28T10:12:00Z"} """ ``` *Since 0.1.5.* ### `nats.subject.send.file` [Section titled “nats.subject.send.file”](#natssubjectsendfile) ```gherkin When the {filepath} message is published to the {word} nats subject[[ with the following headers:]] ``` Send a message whose body is the file to the nats subject, with the headers of the table when the step has one. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a header sent with the message, as text: its name and its value. **Variants**, the parts in `[[...]]` said or left out: * `the {filepath} message is published to the {word} nats subject` * `the {filepath} message is published to the {word} nats subject with the following headers:` **Example:** ```gherkin When the nats/delivery-confirmed.json message is published to the deliveries.confirmed nats subject When the nats/delivery-confirmed.json message is published to the deliveries.confirmed nats subject with the following headers: | courier | CR-LEJ-12 | ``` *Since 0.1.5.* ### `nats.subject.received` [Section titled “nats.subject.received”](#natssubjectreceived) ```gherkin Then [[within {duration} ]]the {word} nats subject has a message where: | path | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the nats subject has a message with those values, received since the scenario started. * The check waits for it: 10 seconds, or `within {duration}`. * Axx subscribes to a subject it checks for the run. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a path into the message’s JSON body (a field name, a dotted path or a JSONPath) and the value it has, compared as text: `null` for null and `undefined` for absent. A row `header ` is instead on the header of that name sent with the message: `undefined` when there is none. A row `subject` is on the message itself, not its body. **Variants**, the parts in `[[...]]` said or left out: * `the {word} nats subject has a message where:` * `within {duration} the {word} nats subject has a message where:` **Example:** ```gherkin Then within 30s the tracking.> nats subject has a message where: | subject | tracking.PX-8301 | | status | DELIVERED | | header source | courier | ``` *Since 0.1.5.* ### `nats.stream.received` [Section titled “nats.stream.received”](#natsstreamreceived) ```gherkin Then [[within {duration} ]]the {word} nats stream has a message where: | path | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the nats stream has a message with those values, received since the scenario started. * The check waits for it: 10 seconds, or `within {duration}`. * Axx reads a stream it checks with an ordered consumer of its own, from the messages stored after the services are up. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a path into the message’s JSON body (a field name, a dotted path or a JSONPath) and the value it has, compared as text: `null` for null and `undefined` for absent. A row `header ` is instead on the header of that name sent with the message: `undefined` when there is none. A row `subject` is on the message itself, not its body. **Variants**, the parts in `[[...]]` said or left out: * `the {word} nats stream has a message where:` * `within {duration} the {word} nats stream has a message where:` **Example:** ```gherkin Then within 30s the TRACKING nats stream has a message where: | subject | tracking.PX-8302 | | status | OUT_FOR_DELIVERY | ``` *Since 0.1.5.* # Redis > Seed keys into Redis, and check the keys your services write there: their values, their JSON properties or hash fields, and that they are gone. Every server that speaks Redis's protocol works: Redis, Valkey, Dragonfly, KeyDB, Garnet. Seed keys into Redis, and check the keys your services write there: their values, their JSON properties or hash fields, and that they are gone. Every server that speaks Redis’s protocol works: Redis, Valkey, Dragonfly, KeyDB, Garnet. Register the server once, usually in the `Background`; every Redis step of the scenario uses it: ```gherkin Given the cache redis server with the following properties: | url | redis://localhost:6379/0 | ``` * **A key is shown as JSON,** whatever its type: a string holding JSON as that JSON (other text as a string), a hash as an object of its fields, a list as an array, a set as a sorted array, and a sorted set as an object of its members and their scores. `has the following properties:` reads paths into it. * **Checks wait** for the key to be as they say, 10 seconds unless `within {duration}` says otherwise. * **Scenarios share the server,** so each seeds and checks keys of its own, named after data unique to it, such as a parcel reference. * **Secrets stay secret:** a password in the URL, or from `${env:..}`, is masked in logs and failures. Add it to a project with `axx pack add redis`. ## Steps [Section titled “Steps”](#steps) ### `redis.server` [Section titled “redis.server”](#redisserver) ```gherkin Given the {word} redis server with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register the Redis server the steps talk to. * The first server registered is the one the scenario’s Redis steps use. * Values expand `${env:..}` and `${sys:..}`; `${env:..}` values are masked. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Default | | ---------- | ----------------------------------------------------------------------------- | ---------- | | `url` | the server’s URL: `redis://user:password@host:6379/0`, or `rediss://` for TLS | *required* | | `database` | the database number, instead of the URL’s | | **Example:** ```gherkin Given the cache redis server with the following properties: | url | redis://localhost:6379/0 | ``` *Since 0.1.5.* ### `redis.seed` [Section titled “redis.seed”](#redisseed) ```gherkin Given a {filepath} redis seed ``` Write the keys of a YAML or JSON file to the scenario’s Redis server, replacing any keys of those names. The file maps each key to one of `value` (text, or JSON for an object or an array), `hash` (its fields), `list`, `set` or `sorted set` (members and their scores), and optionally a `ttl` (like `10m`) after which it expires. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | **Example:** ```gherkin Given a seeds/quote-cache.yaml redis seed ``` *Since 0.1.5.* ### `redis.value` [Section titled “redis.value”](#redisvalue) ```gherkin Then [[within {duration} ]]the {word} redis key has the value {string} ``` Check that the key holds that text. The check waits for it: 10 seconds, or `within {duration}`. * The key must be a string key; for the others, check their properties. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} redis key has the value {string}` * `within {duration} the {word} redis key has the value {string}` **Example:** ```gherkin Then within 10s the quote:DE-1:STANDARD:1200 redis key has the value '690' ``` *Since 0.1.5.* ### `redis.properties` [Section titled “redis.properties”](#redisproperties) ```gherkin Then [[within {duration} ]]the {word} redis key has the following properties: | path | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the key, shown as JSON, has the properties of the table. The check waits for it: 10 seconds, or `within {duration}`. * A string key holding JSON is that JSON; a hash is an object of its fields; a list an array; a set a sorted array; a sorted set an object of its members and their scores. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a path into the key’s JSON (a field name, a dotted path or a JSONPath) and the value it has, compared as text: `null` for null and `undefined` for absent. **Variants**, the parts in `[[...]]` said or left out: * `the {word} redis key has the following properties:` * `within {duration} the {word} redis key has the following properties:` **Example:** ```gherkin Then within 10s the quote:DE-1:STANDARD:1200 redis key has the following properties: | priceCents | 690 | | currency | EUR | Then the shop:hawthorn-home redis key has the following properties: | tier | gold | ``` *Since 0.1.5.* ### `redis.absent` [Section titled “redis.absent”](#redisabsent) ```gherkin Then [[within {duration} ]]the {word} redis key does not exist ``` Check that the key does not exist, or waits for it to go: 10 seconds, or `within {duration}`, for a service that deletes it or lets it expire. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} redis key does not exist` * `within {duration} the {word} redis key does not exist` **Example:** ```gherkin Then within 10s the tracking:PX-RDS-9603 redis key does not exist ``` *Since 0.1.5.* # REST > Send HTTP requests to REST services, validate them against the services' OpenAPI specifications, and assert on the responses. Send HTTP requests to REST services, validate them against the services’ OpenAPI specifications, and assert on the responses. **Services.** Register services with `the service with the following properties:` (`url`, optional `openapi`). The first service registered in a scenario is the default one; the other steps name a service with `on `. **Requests.** Each service keeps its requests in the order they are added: `a GET request to /path` adds the first (default) request and `a 2nd ordered POST request to /path` the second. Header, payload, execution and response steps address a request with `for 2nd ordered request` / `for 2nd ordered response` (default: the first). A request is executed once; its response stays available to every later step. **Payloads.** A payload starts from an OpenAPI content example (the named one, or the first in document order; `externalValue` examples are read relative to the specification) or from an empty template `{}`, and is edited with the payload property steps (JSONPath, typed values, `null`/`undefined`). `application/x-www-form-urlencoded` payloads are sent form-encoded. **Execution.** Requests go through one HTTP client shared by the run (connections are reused and closed at the end), honor the step timeout, and do not verify TLS certificates unless `packs.rest.tls.verify: true` is set in axx.yaml. Without a Content-Type header a payload is sent with its payload step’s media type. **Redirects.** A GET or HEAD request follows redirects (301, 302, 303, 307 and 308); a request with another method follows only `303 See Other`, with a GET, as a browser does after a form. A request that follows a redirect has the response it leads to: the response steps check that one, and the redirect responses, with their `Location` and `Set-Cookie` headers, are not kept. The other redirects of a POST, PUT, PATCH or DELETE (301, 302, 307 and 308) are its response: check the status and the `Location` header, and send the request it leads to as the next ordered request. **Cookies.** The pack keeps no cookies: a `Set-Cookie` response header is never sent back, not even to where a redirect leads, so no request carries a cookie from another scenario. Send a cookie with the `Cookie` request header: it goes along when the request follows a redirect on the same host or to a subdomain of it. **OpenAPI validation.** When a service has an `openapi` specification (OpenAPI 3.0 or 3.1, a URL or a file; parsed once per run), each executed request and its response are validated after sending. Every finding has a key in the style of the swagger request validator (the one the WireMock extension uses), and a level: * `ERROR` (alias `FAIL`, the default for every key) fails the execute step, listing every error with its key; * `WARN` and `INFO` are logged on the step; * `IGNORE` drops the finding. Levels come from `openapi.levels` in axx.yaml and are overridden per scenario with `the OpenAPI validation levels are:`. A key also sets every more specific key (`validation.request.body` covers `validation.request.body.schema.required`); the most specific configured key wins. A key must be one of the keys below, or a prefix of them: any other, in the step or in axx.yaml, is an error that names the closest keys. | Key | Reported when | | -------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | `validation.request.path.missing` | no path of the specification matches the request path | | `validation.request.operation.notAllowed` | the path exists but not for the request method | | `validation.request.body.missing` | the operation requires a body and the request has none | | `validation.request.body.unexpected` | the request has a body the operation does not declare | | `validation.request.contentType.notAllowed` | the request Content-Type is not declared for the operation | | `validation.request.body.schema.{keyword}` | the request body violates a schema keyword (type, required, enum, format, pattern, minimum, maxLength, additionalProperties, oneOf, …) | | `validation.request.body.schema.invalidJson` | the request body cannot be parsed | | `validation.request.body.schema.processingError` | the request body schema cannot be compiled or the body cannot be read | | `validation.request.parameter.missing` | a required path parameter is missing | | `validation.request.parameter.query.missing` | a required query parameter is missing | | `validation.request.parameter.header.missing` | a required header parameter is missing | | `validation.request.parameter.cookie.missing` | a required cookie parameter is missing | | `validation.request.parameter.schema.{keyword}` | a parameter value violates its schema (type, enum, format, pattern, minimum, …) | | `validation.request.parameter.schema.invalidJson` | a JSON (content) parameter cannot be parsed | | `validation.request.parameter.schema.processingError` | a parameter’s schema cannot be compiled | | `validation.request.parameter.collection.invalidFormat` | an array or object parameter is serialized in the wrong style | | `validation.request.parameter.collection.tooManyItems` | an array parameter has more than maxItems items | | `validation.request.parameter.collection.tooFewItems` | an array parameter has fewer than minItems items | | `validation.request.parameter.collection.duplicateItems` | an array parameter with uniqueItems repeats an item | | `validation.request.parameter.{in}.invalid` | any other parameter problem; {in} is `path`, `query`, `header` or `cookie` | | `validation.request.security.missing` | the credentials a security requirement needs are absent | | `validation.request.security.invalid` | credentials are present but do not match the security scheme | | `validation.response.status.unknown` | the response status is not documented for the operation (and there is no default) | | `validation.response.contentType.notAllowed` | the response Content-Type is not declared for the status | | `validation.response.body.missing` | the response declares a body schema but has no body | | `validation.response.body.unexpected` | a response to HEAD has a body | | `validation.response.body.schema.{keyword}` | the response body violates a schema keyword | | `validation.response.body.schema.invalidJson` | the response body cannot be parsed | | `validation.response.body.schema.processingError` | the response body schema cannot be compiled or the body cannot be read | | `validation.response.header.missing` | a required response header is missing | | `validation.response.header.schema.{keyword}` | a response header value violates its schema | | `validation.request.unknownError` | anything else the validator reports about the request | | `validation.response.unknownError` | anything else the validator reports about the response | Schema keywords use the draft-4 names: `const` is reported as `enum`, `exclusiveMinimum`/`exclusiveMaximum` as `minimum`/`maximum`, `unevaluatedProperties` as `additionalProperties`. A schema failure that names no keyword is keyed `unknownError` (`validation.request.body.schema.unknownError`). When a scenario fails, its failure context (`rest`) shows the last request and response (headers, bodies truncated to 2 KB) and the OpenAPI findings. Add it to a project with `axx pack add rest`. ## Settings [Section titled “Settings”](#settings) In `axx.yaml`, under `packs.rest`: | Setting | Takes | Values | Default | | ---------------- | -------------------------------------------------------------------------------------------------- | ---------------- | ------- | | `openapi.levels` | Validation key -> ERROR (or FAIL), WARN, INFO or IGNORE. Merged over the top-level openapi.levels. | names and values | | | `tls.verify` | Verify TLS certificates. | `true`, `false` | `false` | ## Steps [Section titled “Steps”](#steps) ### `rest.service` [Section titled “rest.service”](#restservice) ```gherkin Given the {word} service with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register a REST service: where its requests go, and the OpenAPI specification they are checked against. * The first service registered in a scenario is the default one. * With an `openapi` specification, every request executed and its response are validated against it, and request payloads can start from its examples. * Values can use `${env:…}` and `${sys:…}`, which are expanded. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Default | | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `url` | the base URL requests go to, like `http://localhost:8400`; a request’s path is added to it | *required* | | `openapi` | the service’s OpenAPI 3.0 or 3.1 specification: a URL, or a file of the project (relative to the `resources` directories or to axx.yaml’s directory) | | Any other property fails the step. **Example:** ```gherkin Given the parcels service with the following properties: | url | http://localhost:8400 | | openapi | http://localhost:8400/openapi.json | ``` ### `rest.openapi.levels` [Section titled “rest.openapi.levels”](#restopenapilevels) ```gherkin Given the OpenAPI validation levels[[ on {service}]] are: | validation key | level | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Set the level of OpenAPI validation findings for this scenario, on the default or the named service. * A key also sets the keys below it: `validation.request.body` relaxes `validation.request.body.schema.required` too. The most specific key set wins. * The rows are merged over `openapi.levels` of axx.yaml. * The pack’s documentation lists the keys, and what each level does. | Parameter | Takes | For example | | ----------- | -------------------------------------------------- | ----------- | | `{service}` | the name of a REST service the scenario registered | `parcels` | A row is a validation key and its level: `ERROR` (or `FAIL`), `WARN`, `INFO` or `IGNORE`. The key must be one the pack reports, or a prefix of such keys, like `validation.request.body`: any other fails the step, which names the closest keys. **Variants**, the parts in `[[...]]` said or left out: * `the OpenAPI validation levels are:` * `the OpenAPI validation levels on {service} are:` **Example:** ```gherkin Given the OpenAPI validation levels are: | validation.request.body.schema.maximum | IGNORE | | validation.response.header.missing | WARN | Given the OpenAPI validation levels on parcels are: | validation.request.body.schema.required | WARN | ``` ### `rest.request` [Section titled “rest.request”](#restrequest) ```gherkin Given a(n) {word} request to {word}[[ on {service}]] ``` Add a request to the default or the named service: its method, and its path below the service’s `url`. * The path can have a query string, like `/api/parcels?sender=kestrel-books`. A whole URL replaces the service’s `url`. * This is the service’s first (default) request; the ordered form adds more. | Parameter | Takes | For example | | ----------- | -------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{service}` | the name of a REST service the scenario registered | `parcels` | **Variants**, the parts in `[[...]]` said or left out: * `a(n) {word} request to {word}` * `a(n) {word} request to {word} on {service}` **Example:** ```gherkin Given a GET request to /api/parcels/PX-1001 Given a DELETE request to /api/parcels/PX-1001 on parcels ``` ### `rest.request.ordered` [Section titled “rest.request.ordered”](#restrequestordered) ```gherkin Given a {ordinal} ordered {word} request to {word}[[ on {service}]] ``` Add the Nth request of a service. Requests are numbered in the order they are added: the 1st ordered request is the default request, and the Nth can only be added once N-1 exist. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | -------------------- | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{service}` | the name of a REST service the scenario registered | `parcels` | **Variants**, the parts in `[[...]]` said or left out: * `a {ordinal} ordered {word} request to {word}` * `a {ordinal} ordered {word} request to {word} on {service}` **Example:** ```gherkin Given a 2nd ordered GET request to /api/parcels/PX-1001 Given a 1st ordered POST request to /api/parcels on parcels ``` ### `rest.request.header` [Section titled “rest.request.header”](#restrequestheader) ```gherkin Given the request header {word} is {string}[[ for {ordinal} ordered request]] ``` Set a request header. * `Content-Type` and `Accept` replace an earlier value; other headers may be added more than once, and are all sent. * Without an ordinal it applies to the service’s first (default) request; `for 2nd ordered request`, to its second. * It uses the default service, the first one registered; `rest.request.header.on` names a service. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | ---------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | **Variants**, the parts in `[[...]]` said or left out: * `the request header {word} is {string}` * `the request header {word} is {string} for {ordinal} ordered request` **Example:** ```gherkin Given the request header Content-Type is 'application/json' ``` ### `rest.request.header.on` [Section titled “rest.request.header.on”](#restrequestheaderon) ```gherkin Given the request header {word} is {string} for[[ {ordinal} ordered]] request on {service} ``` `rest.request.header` on a named service: `for request on ` addresses the service’s first (default) request, `for 2nd ordered request on ` its second one. Everything else works like `rest.request.header`. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | ---------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{service}` | the name of a REST service the scenario registered | `parcels` | **Variants**, the parts in `[[...]]` said or left out: * `the request header {word} is {string} for request on {service}` * `the request header {word} is {string} for {ordinal} ordered request on {service}` **Example:** ```gherkin Given the request header Accept is 'application/json' for 1st ordered request on parcels ``` ### `rest.request.headers` [Section titled “rest.request.headers”](#restrequestheaders) ```gherkin Given the request headers[[ for {ordinal} ordered request]] are: | header | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Set request headers, a row each. * A name may repeat. Like the single-header step, `Content-Type` and `Accept` replace an earlier value, and other headers are all sent. * Without an ordinal it applies to the service’s first (default) request; `for 2nd ordered request`, to its second. * It uses the default service, the first one registered; `rest.request.headers.on` names a service. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | ------------ | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | A row is a header’s name and its value. **Variants**, the parts in `[[...]]` said or left out: * `the request headers are:` * `the request headers for {ordinal} ordered request are:` **Example:** ```gherkin Given the request headers are: | Accept | application/json | | Accept-Language | de-DE | ``` ### `rest.request.headers.on` [Section titled “rest.request.headers.on”](#restrequestheaderson) ```gherkin Given the request headers for[[ {ordinal} ordered]] request on {service} are: | header | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. `rest.request.headers` on a named service: `for request on ` addresses the service’s first (default) request, `for 2nd ordered request on ` its second one. Everything else works like `rest.request.headers`. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | ------------ | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{service}` | the name of a REST service the scenario registered | `parcels` | A row is a header’s name and its value. **Variants**, the parts in `[[...]]` said or left out: * `the request headers for request on {service} are:` * `the request headers for {ordinal} ordered request on {service} are:` **Example:** ```gherkin Given the request headers for request on parcels are: | Content-Type | application/json | | Accept | application/json | ``` ### `rest.request.payload.empty` [Section titled “rest.request.payload.empty”](#restrequestpayloadempty) ```gherkin Given a request payload using a(n) {mimeType} empty content template[[ for {ordinal} ordered request]] ``` Start the request payload from an empty JSON object, `{}`, for the payload property steps to fill. * It needs no OpenAPI specification. * With `application/x-www-form-urlencoded`, the properties are sent form-encoded, and nested objects and arrays as JSON text. * A request takes one payload step. * Without an ordinal it applies to the service’s first (default) request; `for 2nd ordered request`, to its second. * It uses the default service, the first one registered; `rest.request.payload.empty.on` names a service. | Parameter | Takes | Values | For example | | ------------ | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | ------------------ | | `{mimeType}` | a content type | `application/json`, `text/json`, `application/problem+json`, `application/x-www-form-urlencoded` | `application/json` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | | `1st`, `2nd` | **Variants**, the parts in `[[...]]` said or left out: * `a request payload using a(n) {mimeType} empty content template` * `a request payload using a(n) {mimeType} empty content template for {ordinal} ordered request` **Example:** ```gherkin Given a request payload using an application/json empty content template ``` ### `rest.request.payload.empty.on` [Section titled “rest.request.payload.empty.on”](#restrequestpayloademptyon) ```gherkin Given a request payload using a(n) {mimeType} empty content template for[[ {ordinal} ordered]] request on {service} ``` `rest.request.payload.empty` on a named service: `for request on ` addresses the service’s first (default) request, `for 2nd ordered request on ` its second one. Everything else works like `rest.request.payload.empty`. | Parameter | Takes | Values | For example | | ------------ | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | ------------------ | | `{mimeType}` | a content type | `application/json`, `text/json`, `application/problem+json`, `application/x-www-form-urlencoded` | `application/json` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | | `1st`, `2nd` | | `{service}` | the name of a REST service the scenario registered | | `parcels` | **Variants**, the parts in `[[...]]` said or left out: * `a request payload using a(n) {mimeType} empty content template for request on {service}` * `a request payload using a(n) {mimeType} empty content template for {ordinal} ordered request on {service}` **Example:** ```gherkin Given a request payload using an application/json empty content template for request on parcels ``` ### `rest.request.payload.example` [Section titled “rest.request.payload.example”](#restrequestpayloadexample) ```gherkin Given a request payload using a(n) {mimeType} content example[[ named {string}]][[ for {ordinal} ordered request]] ``` Start the request payload from a request body example of the service’s OpenAPI specification. * With `named ''` it is the example of that name; without, the first example in document order, or the media type’s single `example`. * The example is looked up under the operation that matches the request’s method and path, for the media type. * An example with an `externalValue` is read relative to the specification. * The service needs its `openapi` property, and the request must be added first. * A request takes one payload step. * Without an ordinal it applies to the service’s first (default) request; `for 2nd ordered request`, to its second. * It uses the default service, the first one registered; `rest.request.payload.example.on` names a service. | Parameter | Takes | Values | For example | | ------------ | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | ---------------------------- | | `{mimeType}` | a content type | `application/json`, `text/json`, `application/problem+json`, `application/x-www-form-urlencoded` | `application/json` | | `{string}` | text in double or single quotes, which the step leaves out | | `"Get a quote"`, `'Express'` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | | `1st`, `2nd` | **Variants**, the parts in `[[...]]` said or left out: * `a request payload using a(n) {mimeType} content example` * `a request payload using a(n) {mimeType} content example named {string}` * `a request payload using a(n) {mimeType} content example for {ordinal} ordered request` * `a request payload using a(n) {mimeType} content example named {string} for {ordinal} ordered request` **Example:** ```gherkin Given a request payload using an application/json content example named 'Standard parcel' ``` ### `rest.request.payload.example.on` [Section titled “rest.request.payload.example.on”](#restrequestpayloadexampleon) ```gherkin Given a request payload using a(n) {mimeType} content example[[ named {string}]] for[[ {ordinal} ordered]] request on {service} ``` `rest.request.payload.example` on a named service: `for request on ` addresses the service’s first (default) request, `for 2nd ordered request on ` its second one. Everything else works like `rest.request.payload.example`. | Parameter | Takes | Values | For example | | ------------ | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | ---------------------------- | | `{mimeType}` | a content type | `application/json`, `text/json`, `application/problem+json`, `application/x-www-form-urlencoded` | `application/json` | | `{string}` | text in double or single quotes, which the step leaves out | | `"Get a quote"`, `'Express'` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | | `1st`, `2nd` | | `{service}` | the name of a REST service the scenario registered | | `parcels` | **Variants**, the parts in `[[...]]` said or left out: * `a request payload using a(n) {mimeType} content example for request on {service}` * `a request payload using a(n) {mimeType} content example named {string} for request on {service}` * `a request payload using a(n) {mimeType} content example for {ordinal} ordered request on {service}` * `a request payload using a(n) {mimeType} content example named {string} for {ordinal} ordered request on {service}` **Example:** ```gherkin Given a request payload using an application/json content example for 1st ordered request on parcels ``` ### `rest.request.payload.resource` [Section titled “rest.request.payload.resource”](#restrequestpayloadresource) ```gherkin Given a request payload using a(n) {mimeType} {filepath} resource[[ for {ordinal} ordered request]] ``` Start the request payload from a file of the project, such as a fixture factory’s output: it is sent with the media type. * The file is found like other files steps name: relative to the `resources` directories, or to axx.yaml’s directory. * It is sent as it is, and a JSON payload can then be changed with the payload property steps. With `application/x-www-form-urlencoded`, the file is a JSON object whose properties are sent form-encoded. * It needs no OpenAPI specification. * A request takes one payload step. * Without an ordinal it applies to the service’s first (default) request; `for 2nd ordered request`, to its second. * It uses the default service, the first one registered; `rest.request.payload.resource.on` names a service. | Parameter | Takes | Values | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------- | | `{mimeType}` | a content type | `application/json`, `text/json`, `application/problem+json`, `application/x-www-form-urlencoded` | `application/json` | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | | `1st`, `2nd` | **Variants**, the parts in `[[...]]` said or left out: * `a request payload using a(n) {mimeType} {filepath} resource` * `a request payload using a(n) {mimeType} {filepath} resource for {ordinal} ordered request` **Example:** ```gherkin Given a request payload using an application/json requests/order-7731.json resource ``` ### `rest.request.payload.resource.on` [Section titled “rest.request.payload.resource.on”](#restrequestpayloadresourceon) ```gherkin Given a request payload using a(n) {mimeType} {filepath} resource for[[ {ordinal} ordered]] request on {service} ``` `rest.request.payload.resource` on a named service: `for request on ` addresses the service’s first (default) request, `for 2nd ordered request on ` its second one. Everything else works like `rest.request.payload.resource`. | Parameter | Takes | Values | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------- | | `{mimeType}` | a content type | `application/json`, `text/json`, `application/problem+json`, `application/x-www-form-urlencoded` | `application/json` | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | | `1st`, `2nd` | | `{service}` | the name of a REST service the scenario registered | | `parcels` | **Variants**, the parts in `[[...]]` said or left out: * `a request payload using a(n) {mimeType} {filepath} resource for request on {service}` * `a request payload using a(n) {mimeType} {filepath} resource for {ordinal} ordered request on {service}` **Example:** ```gherkin Given a request payload using an application/json requests/order-7732.json resource for 2nd ordered request on parcels ``` ### `rest.request.property` [Section titled “rest.request.property”](#restrequestproperty) ```gherkin Given the request payload property {word} is {string}[[ for {ordinal} ordered request]] ``` Set a property of the request payload, by its JSONPath, like `weightGrams`, `recipient.postcode` or `$.recipient.name`. * A value in double quotes inside the quotes, like `'"42"'`, is always a string. * Any other value takes the type of the property’s current value: a string, a boolean, an integer, a number, or an object or array parsed from JSON text. * A property that does not exist yet, or is null, takes the type its value reads as: `true`, `42`, `1.5`, `{...}`, `[...]`, or else a string. * A property inside an object the payload lacks creates the object: `recipient.name` in a payload without `recipient`. A path through an array (`items[2].sku`) needs the array. * A payload step must come first. * Without an ordinal it applies to the service’s first (default) request; `for 2nd ordered request`, to its second. * It uses the default service, the first one registered; `rest.request.property.on` names a service. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | ---------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | **Variants**, the parts in `[[...]]` said or left out: * `the request payload property {word} is {string}` * `the request payload property {word} is {string} for {ordinal} ordered request` **Example:** ```gherkin Given the request payload property sender is 'kestrel-books' ``` ### `rest.request.property.on` [Section titled “rest.request.property.on”](#restrequestpropertyon) ```gherkin Given the request payload property {word} is {string} for[[ {ordinal} ordered]] request on {service} ``` `rest.request.property` on a named service: `for request on ` addresses the service’s first (default) request, `for 2nd ordered request on ` its second one. Everything else works like `rest.request.property`. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | ---------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{service}` | the name of a REST service the scenario registered | `parcels` | **Variants**, the parts in `[[...]]` said or left out: * `the request payload property {word} is {string} for request on {service}` * `the request payload property {word} is {string} for {ordinal} ordered request on {service}` **Example:** ```gherkin Given the request payload property serviceLevel is 'EXPRESS' for request on parcels ``` ### `rest.request.properties` [Section titled “rest.request.properties”](#restrequestproperties) ```gherkin Given the request payload properties[[ for {ordinal} ordered request]] are: | JSONPath | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Set properties of the request payload, a row each, in order, like the single-property step. * `null` sets JSON null, and `undefined` removes the property, in any case. * `"null"` and `"undefined"`, in double quotes, are the strings. * A property inside an object the payload lacks creates the object, so a table can list `recipient.name` and `recipient.postcode` on an empty payload. * Without an ordinal it applies to the service’s first (default) request; `for 2nd ordered request`, to its second. * It uses the default service, the first one registered; `rest.request.properties.on` names a service. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | ------------ | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | A row is a property’s JSONPath and its value. **Variants**, the parts in `[[...]]` said or left out: * `the request payload properties are:` * `the request payload properties for {ordinal} ordered request are:` **Example:** ```gherkin Given the request payload properties are: | reference | PX-4101 | | weightGrams | 1200 | | recipient.postcode | "53111" | ``` ### `rest.request.properties.on` [Section titled “rest.request.properties.on”](#restrequestpropertieson) ```gherkin Given the request payload properties for[[ {ordinal} ordered]] request on {service} are: | JSONPath | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. `rest.request.properties` on a named service: `for request on ` addresses the service’s first (default) request, `for 2nd ordered request on ` its second one. Everything else works like `rest.request.properties`. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | ------------ | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{service}` | the name of a REST service the scenario registered | `parcels` | A row is a property’s JSONPath and its value. **Variants**, the parts in `[[...]]` said or left out: * `the request payload properties for request on {service} are:` * `the request payload properties for {ordinal} ordered request on {service} are:` **Example:** ```gherkin Given the request payload properties for 1st ordered request on parcels are: | sender | lark-ceramics | | recipient.street | undefined | ``` ### `rest.request.property.null` [Section titled “rest.request.property.null”](#restrequestpropertynull) ```gherkin Given the request payload property {word} is null[[ for {ordinal} ordered request]] ``` Set an existing property of the request payload to JSON null. * Without an ordinal it applies to the service’s first (default) request; `for 2nd ordered request`, to its second. * It uses the default service, the first one registered; `rest.request.property.null.on` names a service. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | **Variants**, the parts in `[[...]]` said or left out: * `the request payload property {word} is null` * `the request payload property {word} is null for {ordinal} ordered request` **Example:** ```gherkin Given the request payload property recipient.street is null ``` ### `rest.request.property.null.on` [Section titled “rest.request.property.null.on”](#restrequestpropertynullon) ```gherkin Given the request payload property {word} is null for[[ {ordinal} ordered]] request on {service} ``` `rest.request.property.null` on a named service: `for request on ` addresses the service’s first (default) request, `for 2nd ordered request on ` its second one. Everything else works like `rest.request.property.null`. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{service}` | the name of a REST service the scenario registered | `parcels` | **Variants**, the parts in `[[...]]` said or left out: * `the request payload property {word} is null for request on {service}` * `the request payload property {word} is null for {ordinal} ordered request on {service}` **Example:** ```gherkin Given the request payload property recipient.street is null for request on parcels ``` ### `rest.execute` [Section titled “rest.execute”](#restexecute) ```gherkin When the[[ {ordinal} ordered]] request is executed[[ on {service}]] ``` Send a request, and keep its response for the response steps. * With an OpenAPI specification, the request and its response are validated after sending. Findings at level `ERROR` fail the step, which lists them all with their keys; `WARN` and `INFO` findings are logged. * The payload is sent as it is, or form-encoded for `application/x-www-form-urlencoded`. * Without a `Content-Type` header, the request has the payload’s media type; without an `Accept` header, `*/*`. * A GET or HEAD request follows redirects, and a request with another method a `303 See Other` (with a GET): the response steps then check the response it leads to. Other redirects are the response, with their `Location` header. * No cookies are kept: a `Set-Cookie` response header is not sent back. Send one with the `Cookie` request header. * The request honors the step timeout, and is executed once. * Without an ordinal it sends the service’s first (default) request; `the 2nd ordered request`, its second. * Without `on {service}` it uses the default service, the first one registered. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | ------------ | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{service}` | the name of a REST service the scenario registered | `parcels` | **Variants**, the parts in `[[...]]` said or left out: * `the request is executed` * `the request is executed on {service}` * `the {ordinal} ordered request is executed` * `the {ordinal} ordered request is executed on {service}` **Example:** ```gherkin When the request is executed When the 2nd ordered request is executed on parcels ``` ### `rest.response.status` [Section titled “rest.response.status”](#restresponsestatus) ```gherkin Then the[[ {ordinal} ordered]] response status code is {int}[[ on {service}]] ``` Check the HTTP status code of a response. * Without an ordinal it checks the response to the first (default) request; `the 2nd ordered response`, the response to the second. * Without `on {service}` it uses the default service, the first one registered. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | ------------ | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{int}` | a whole number | `200`, `3` | | `{service}` | the name of a REST service the scenario registered | `parcels` | **Variants**, the parts in `[[...]]` said or left out: * `the response status code is {int}` * `the response status code is {int} on {service}` * `the {ordinal} ordered response status code is {int}` * `the {ordinal} ordered response status code is {int} on {service}` **Example:** ```gherkin Then the response status code is 200 Then the 2nd ordered response status code is 201 on parcels ``` ### `rest.response.body.contains` [Section titled “rest.response.body.contains”](#restresponsebodycontains) ```gherkin Then the response body contains {string}[[ for {ordinal} ordered response]] ``` Check that the response body contains the text. * Without an ordinal it checks the response to the first (default) request; `for 2nd ordered response`, the response to the second. * It uses the default service, the first one registered; `rest.response.body.contains.on` names a service. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | **Variants**, the parts in `[[...]]` said or left out: * `the response body contains {string}` * `the response body contains {string} for {ordinal} ordered response` **Example:** ```gherkin Then the response body contains 'already registered' ``` ### `rest.response.body.contains.on` [Section titled “rest.response.body.contains.on”](#restresponsebodycontainson) ```gherkin Then the response body contains {string} for[[ {ordinal} ordered]] response on {service} ``` `rest.response.body.contains` on a named service: `for response on ` addresses the service’s first (default) response, `for 2nd ordered response on ` its second one. Everything else works like `rest.response.body.contains`. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{service}` | the name of a REST service the scenario registered | `parcels` | **Variants**, the parts in `[[...]]` said or left out: * `the response body contains {string} for response on {service}` * `the response body contains {string} for {ordinal} ordered response on {service}` **Example:** ```gherkin Then the response body contains 'already registered' for 2nd ordered response on parcels ``` ### `rest.response.header.is` [Section titled “rest.response.header.is”](#restresponseheaderis) ```gherkin Then the response header {word} is {string}[[ for {ordinal} ordered response]] ``` Check that a response header has the value. * The header’s name is matched in any case. * With the header repeated, one of its values must be the value. * Without an ordinal it checks the response to the first (default) request; `for 2nd ordered response`, the response to the second. * It uses the default service, the first one registered; `rest.response.header.is.on` names a service. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | ---------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | **Variants**, the parts in `[[...]]` said or left out: * `the response header {word} is {string}` * `the response header {word} is {string} for {ordinal} ordered response` **Example:** ```gherkin Then the response header Content-Type is 'application/json' ``` ### `rest.response.header.is.on` [Section titled “rest.response.header.is.on”](#restresponseheaderison) ```gherkin Then the response header {word} is {string} for[[ {ordinal} ordered]] response on {service} ``` `rest.response.header.is` on a named service: `for response on ` addresses the service’s first (default) response, `for 2nd ordered response on ` its second one. Everything else works like `rest.response.header.is`. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | ---------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{service}` | the name of a REST service the scenario registered | `parcels` | **Variants**, the parts in `[[...]]` said or left out: * `the response header {word} is {string} for response on {service}` * `the response header {word} is {string} for {ordinal} ordered response on {service}` **Example:** ```gherkin Then the response header Content-Type is 'application/json' for response on parcels ``` ### `rest.response.header.matches` [Section titled “rest.response.header.matches”](#restresponseheadermatches) ```gherkin Then the response header {word} matches {pattern}[[ for {ordinal} ordered response]] ``` Check that a response header matches a regular expression. * The regular expression is in Java syntax, and must match the whole value. * With the header repeated, one of its values must match. * Without an ordinal it checks the response to the first (default) request; `for 2nd ordered response`, the response to the second. * It uses the default service, the first one registered; `rest.response.header.matches.on` names a service. | Parameter | Takes | For example | | ----------- | -------------------------------------------------------------------------------- | -------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{pattern}` | a regular expression (Java syntax) with no spaces, which matches the whole value | `PX-\d{4}`, `[A-Z]{2}-\d+` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | **Variants**, the parts in `[[...]]` said or left out: * `the response header {word} matches {pattern}` * `the response header {word} matches {pattern} for {ordinal} ordered response` **Example:** ```gherkin Then the response header Content-Type matches ^application/json.*$ ``` ### `rest.response.header.matches.on` [Section titled “rest.response.header.matches.on”](#restresponseheadermatcheson) ```gherkin Then the response header {word} matches {pattern} for[[ {ordinal} ordered]] response on {service} ``` `rest.response.header.matches` on a named service: `for response on ` addresses the service’s first (default) response, `for 2nd ordered response on ` its second one. Everything else works like `rest.response.header.matches`. | Parameter | Takes | For example | | ----------- | -------------------------------------------------------------------------------- | -------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{pattern}` | a regular expression (Java syntax) with no spaces, which matches the whole value | `PX-\d{4}`, `[A-Z]{2}-\d+` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{service}` | the name of a REST service the scenario registered | `parcels` | **Variants**, the parts in `[[...]]` said or left out: * `the response header {word} matches {pattern} for response on {service}` * `the response header {word} matches {pattern} for {ordinal} ordered response on {service}` **Example:** ```gherkin Then the response header Content-Type matches ^application/json$ for 1st ordered response on parcels ``` ### `rest.response.header.missing` [Section titled “rest.response.header.missing”](#restresponseheadermissing) ```gherkin Then the response header {word} is missing[[ for {ordinal} ordered response]] ``` Check that the response has no header of that name. * Without an ordinal it checks the response to the first (default) request; `for 2nd ordered response`, the response to the second. * It uses the default service, the first one registered; `rest.response.header.missing.on` names a service. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | **Variants**, the parts in `[[...]]` said or left out: * `the response header {word} is missing` * `the response header {word} is missing for {ordinal} ordered response` **Example:** ```gherkin Then the response header Content-Length is missing ``` ### `rest.response.header.missing.on` [Section titled “rest.response.header.missing.on”](#restresponseheadermissingon) ```gherkin Then the response header {word} is missing for[[ {ordinal} ordered]] response on {service} ``` `rest.response.header.missing` on a named service: `for response on ` addresses the service’s first (default) response, `for 2nd ordered response on ` its second one. Everything else works like `rest.response.header.missing`. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{service}` | the name of a REST service the scenario registered | `parcels` | **Variants**, the parts in `[[...]]` said or left out: * `the response header {word} is missing for response on {service}` * `the response header {word} is missing for {ordinal} ordered response on {service}` **Example:** ```gherkin Then the response header Retry-After is missing for response on parcels ``` ### `rest.response.headers.are` [Section titled “rest.response.headers.are”](#restresponseheadersare) ```gherkin Then the response headers[[ for {ordinal} ordered response]] are: | header | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check response headers, a row each, like the single-header step. * A name may repeat. * Every row is checked, and every mismatch reported. * Without an ordinal it checks the response to the first (default) request; `for 2nd ordered response`, the response to the second. * It uses the default service, the first one registered; `rest.response.headers.are.on` names a service. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | ------------ | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | A row is a header’s name and the value it must have. **Variants**, the parts in `[[...]]` said or left out: * `the response headers are:` * `the response headers for {ordinal} ordered response are:` **Example:** ```gherkin Then the response headers are: | Content-Type | application/json | | Location | /api/parcels/PX-4101 | ``` ### `rest.response.headers.are.on` [Section titled “rest.response.headers.are.on”](#restresponseheadersareon) ```gherkin Then the response headers for[[ {ordinal} ordered]] response on {service} are: | header | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. `rest.response.headers.are` on a named service: `for response on ` addresses the service’s first (default) response, `for 2nd ordered response on ` its second one. Everything else works like `rest.response.headers.are`. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | ------------ | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{service}` | the name of a REST service the scenario registered | `parcels` | A row is a header’s name and the value it must have. **Variants**, the parts in `[[...]]` said or left out: * `the response headers for response on {service} are:` * `the response headers for {ordinal} ordered response on {service} are:` **Example:** ```gherkin Then the response headers for 1st ordered response on parcels are: | Content-Type | application/problem+json | | Retry-After | 5 | ``` ### `rest.response.headers.match` [Section titled “rest.response.headers.match”](#restresponseheadersmatch) ```gherkin Then the response headers[[ for {ordinal} ordered response]] match: | header | regular expression | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that response headers match regular expressions, a row each. * Each regular expression is in Java syntax, and must match the whole value. * Every row is checked, and every mismatch reported. * Without an ordinal it checks the response to the first (default) request; `for 2nd ordered response`, the response to the second. * It uses the default service, the first one registered; `rest.response.headers.match.on` names a service. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | ------------ | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | A row is a header’s name and a regular expression its value must match. **Variants**, the parts in `[[...]]` said or left out: * `the response headers match:` * `the response headers for {ordinal} ordered response match:` **Example:** ```gherkin Then the response headers match: | Content-Type | application/json.* | | Location | /api/parcels/PX-[0-9]{4} | ``` ### `rest.response.headers.match.on` [Section titled “rest.response.headers.match.on”](#restresponseheadersmatchon) ```gherkin Then the response headers for[[ {ordinal} ordered]] response on {service} match: | header | regular expression | ``` The table may start with the row naming its columns, as above, or go straight to its rows. `rest.response.headers.match` on a named service: `for response on ` addresses the service’s first (default) response, `for 2nd ordered response on ` its second one. Everything else works like `rest.response.headers.match`. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | ------------ | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{service}` | the name of a REST service the scenario registered | `parcels` | A row is a header’s name and a regular expression its value must match. **Variants**, the parts in `[[...]]` said or left out: * `the response headers for response on {service} match:` * `the response headers for {ordinal} ordered response on {service} match:` **Example:** ```gherkin Then the response headers for response on parcels match: | Retry-After | [0-9]+ | ``` ### `rest.response.headers.missing` [Section titled “rest.response.headers.missing”](#restresponseheadersmissing) ```gherkin Then the response headers[[ for {ordinal} ordered response]] are missing: | header | ``` Check that the response has none of the headers the table names. * Without an ordinal it checks the response to the first (default) request; `for 2nd ordered response`, the response to the second. * It uses the default service, the first one registered; `rest.response.headers.missing.on` names a service. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | ------------ | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | A row names a header, in its first cell. **Variants**, the parts in `[[...]]` said or left out: * `the response headers are missing:` * `the response headers for {ordinal} ordered response are missing:` **Example:** ```gherkin Then the response headers are missing: | Retry-After | | Set-Cookie | ``` ### `rest.response.headers.missing.on` [Section titled “rest.response.headers.missing.on”](#restresponseheadersmissingon) ```gherkin Then the response headers for[[ {ordinal} ordered]] response on {service} are missing: | header | ``` `rest.response.headers.missing` on a named service: `for response on ` addresses the service’s first (default) response, `for 2nd ordered response on ` its second one. Everything else works like `rest.response.headers.missing`. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | ------------ | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{service}` | the name of a REST service the scenario registered | `parcels` | A row names a header, in its first cell. **Variants**, the parts in `[[...]]` said or left out: * `the response headers for response on {service} are missing:` * `the response headers for {ordinal} ordered response on {service} are missing:` **Example:** ```gherkin Then the response headers for response on parcels are missing: | Retry-After | ``` ### `rest.response.property.is` [Section titled “rest.response.property.is”](#restresponsepropertyis) ```gherkin Then the response payload property {word} is {string}[[ for {ordinal} ordered response]] ``` Check a property of a JSON response, by its JSONPath, like `status`, `recipient.postcode` or `[?(@.sender=='kestrel-books')].reference`. * An indefinite path, like a filter, reads as a list. * The response must be JSON: `application/json`, `text/json` or any `+json` type, whatever its charset. * A value is compared with its JSON type. Text, like `REGISTERED`, is a string, and so is a value in double quotes, like `"42"`. * `42` is an integer, `42L` a long and `1.5` a number. An integer never equals a decimal: `5` is not `5.0`. * `true` and `false` are booleans. `{...}` and `[...]` are a JSON object and a JSON array; an object’s members may come in any order. * Without an ordinal it checks the response to the first (default) request; `for 2nd ordered response`, the response to the second. * It uses the default service, the first one registered; `rest.response.property.is.on` names a service. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | ---------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | **Variants**, the parts in `[[...]]` said or left out: * `the response payload property {word} is {string}` * `the response payload property {word} is {string} for {ordinal} ordered response` **Example:** ```gherkin Then the response payload property status is 'REGISTERED' ``` ### `rest.response.property.is.on` [Section titled “rest.response.property.is.on”](#restresponsepropertyison) ```gherkin Then the response payload property {word} is {string} for[[ {ordinal} ordered]] response on {service} ``` `rest.response.property.is` on a named service: `for response on ` addresses the service’s first (default) response, `for 2nd ordered response on ` its second one. Everything else works like `rest.response.property.is`. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | ---------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{service}` | the name of a REST service the scenario registered | `parcels` | **Variants**, the parts in `[[...]]` said or left out: * `the response payload property {word} is {string} for response on {service}` * `the response payload property {word} is {string} for {ordinal} ordered response on {service}` **Example:** ```gherkin Then the response payload property status is 'REGISTERED' for 1st ordered response on parcels ``` ### `rest.response.property.null` [Section titled “rest.response.property.null”](#restresponsepropertynull) ```gherkin Then the response payload property {word} is null[[ for {ordinal} ordered response]] ``` Check that a property of the response payload exists, and is JSON null. * Without an ordinal it checks the response to the first (default) request; `for 2nd ordered response`, the response to the second. * It uses the default service, the first one registered; `rest.response.property.null.on` names a service. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | **Variants**, the parts in `[[...]]` said or left out: * `the response payload property {word} is null` * `the response payload property {word} is null for {ordinal} ordered response` **Example:** ```gherkin Then the response payload property lastLocation is null ``` ### `rest.response.property.null.on` [Section titled “rest.response.property.null.on”](#restresponsepropertynullon) ```gherkin Then the response payload property {word} is null for[[ {ordinal} ordered]] response on {service} ``` `rest.response.property.null` on a named service: `for response on ` addresses the service’s first (default) response, `for 2nd ordered response on ` its second one. Everything else works like `rest.response.property.null`. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{service}` | the name of a REST service the scenario registered | `parcels` | **Variants**, the parts in `[[...]]` said or left out: * `the response payload property {word} is null for response on {service}` * `the response payload property {word} is null for {ordinal} ordered response on {service}` **Example:** ```gherkin Then the response payload property lastLocation is null for response on parcels ``` ### `rest.response.property.undefined` [Section titled “rest.response.property.undefined”](#restresponsepropertyundefined) ```gherkin Then the response payload property {word} is undefined[[ for {ordinal} ordered response]] ``` Check that a property of the response payload does not exist. * An indefinite path always exists: it reads as a list, which may be empty. * Without an ordinal it checks the response to the first (default) request; `for 2nd ordered response`, the response to the second. * It uses the default service, the first one registered; `rest.response.property.undefined.on` names a service. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | **Variants**, the parts in `[[...]]` said or left out: * `the response payload property {word} is undefined` * `the response payload property {word} is undefined for {ordinal} ordered response` **Example:** ```gherkin Then the response payload property recipient.street is undefined ``` ### `rest.response.property.undefined.on` [Section titled “rest.response.property.undefined.on”](#restresponsepropertyundefinedon) ```gherkin Then the response payload property {word} is undefined for[[ {ordinal} ordered]] response on {service} ``` `rest.response.property.undefined` on a named service: `for response on ` addresses the service’s first (default) response, `for 2nd ordered response on ` its second one. Everything else works like `rest.response.property.undefined`. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{service}` | the name of a REST service the scenario registered | `parcels` | **Variants**, the parts in `[[...]]` said or left out: * `the response payload property {word} is undefined for response on {service}` * `the response payload property {word} is undefined for {ordinal} ordered response on {service}` **Example:** ```gherkin Then the response payload property recipient.street is undefined for 1st ordered response on parcels ``` ### `rest.response.property.matches` [Section titled “rest.response.property.matches”](#restresponsepropertymatches) ```gherkin Then the response payload property {word} matches {pattern}[[ for {ordinal} ordered response]] ``` Check that a property of the response payload is a string that matches a regular expression. * The regular expression is in Java syntax, and must match the whole value. * Without an ordinal it checks the response to the first (default) request; `for 2nd ordered response`, the response to the second. * It uses the default service, the first one registered; `rest.response.property.matches.on` names a service. | Parameter | Takes | For example | | ----------- | -------------------------------------------------------------------------------- | -------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{pattern}` | a regular expression (Java syntax) with no spaces, which matches the whole value | `PX-\d{4}`, `[A-Z]{2}-\d+` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | **Variants**, the parts in `[[...]]` said or left out: * `the response payload property {word} matches {pattern}` * `the response payload property {word} matches {pattern} for {ordinal} ordered response` **Example:** ```gherkin Then the response payload property barcode matches ^PX[0-9]{11}$ ``` ### `rest.response.property.matches.on` [Section titled “rest.response.property.matches.on”](#restresponsepropertymatcheson) ```gherkin Then the response payload property {word} matches {pattern} for[[ {ordinal} ordered]] response on {service} ``` `rest.response.property.matches` on a named service: `for response on ` addresses the service’s first (default) response, `for 2nd ordered response on ` its second one. Everything else works like `rest.response.property.matches`. | Parameter | Takes | For example | | ----------- | -------------------------------------------------------------------------------- | -------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{pattern}` | a regular expression (Java syntax) with no spaces, which matches the whole value | `PX-\d{4}`, `[A-Z]{2}-\d+` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{service}` | the name of a REST service the scenario registered | `parcels` | **Variants**, the parts in `[[...]]` said or left out: * `the response payload property {word} matches {pattern} for response on {service}` * `the response payload property {word} matches {pattern} for {ordinal} ordered response on {service}` **Example:** ```gherkin Then the response payload property barcode matches ^PX[0-9]{11}$ for response on parcels ``` ### `rest.response.properties.are` [Section titled “rest.response.properties.are”](#restresponsepropertiesare) ```gherkin Then the response payload properties[[ for {ordinal} ordered response]] are: | JSONPath | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check properties of a JSON response, a row each, like the single-property step. * `null` and `undefined`, in any case, check for JSON null and for no such property; `"null"`, in double quotes, is the string. * Every row is checked, and every mismatch reported. * A value is compared with its JSON type. Text, like `REGISTERED`, is a string, and so is a value in double quotes, like `"42"`. * `42` is an integer, `42L` a long and `1.5` a number. An integer never equals a decimal: `5` is not `5.0`. * `true` and `false` are booleans. `{...}` and `[...]` are a JSON object and a JSON array; an object’s members may come in any order. * Without an ordinal it checks the response to the first (default) request; `for 2nd ordered response`, the response to the second. * It uses the default service, the first one registered; `rest.response.properties.are.on` names a service. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | ------------ | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | A row is a property’s JSONPath and the value it must have. **Variants**, the parts in `[[...]]` said or left out: * `the response payload properties are:` * `the response payload properties for {ordinal} ordered response are:` **Example:** ```gherkin Then the response payload properties are: | reference | PX-4101 | | status | REGISTERED | | zone | DE-1 | ``` ### `rest.response.properties.are.on` [Section titled “rest.response.properties.are.on”](#restresponsepropertiesareon) ```gherkin Then the response payload properties for[[ {ordinal} ordered]] response on {service} are: | JSONPath | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. `rest.response.properties.are` on a named service: `for response on ` addresses the service’s first (default) response, `for 2nd ordered response on ` its second one. Everything else works like `rest.response.properties.are`. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | ------------ | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{service}` | the name of a REST service the scenario registered | `parcels` | A row is a property’s JSONPath and the value it must have. **Variants**, the parts in `[[...]]` said or left out: * `the response payload properties for response on {service} are:` * `the response payload properties for {ordinal} ordered response on {service} are:` **Example:** ```gherkin Then the response payload properties for 1st ordered response on parcels are: | serviceLevel | STANDARD | | recipient.name | Ada Lovelace | | recipient.street | undefined | ``` ### `rest.response.properties.match` [Section titled “rest.response.properties.match”](#restresponsepropertiesmatch) ```gherkin Then the response payload properties[[ for {ordinal} ordered response]] match: | JSONPath | regular expression | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that properties of the response payload are strings that match regular expressions, a row each. * Each regular expression is in Java syntax, and must match the whole value. * Every row is checked, and every mismatch reported. * Without an ordinal it checks the response to the first (default) request; `for 2nd ordered response`, the response to the second. * It uses the default service, the first one registered; `rest.response.properties.match.on` names a service. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | ------------ | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | A row is a property’s JSONPath and a regular expression its value must match. **Variants**, the parts in `[[...]]` said or left out: * `the response payload properties match:` * `the response payload properties for {ordinal} ordered response match:` **Example:** ```gherkin Then the response payload properties match: | barcode | ^PX[0-9]{11}$ | | signature | ^[0-9a-f]{64}$ | ``` ### `rest.response.properties.match.on` [Section titled “rest.response.properties.match.on”](#restresponsepropertiesmatchon) ```gherkin Then the response payload properties for[[ {ordinal} ordered]] response on {service} match: | JSONPath | regular expression | ``` The table may start with the row naming its columns, as above, or go straight to its rows. `rest.response.properties.match` on a named service: `for response on ` addresses the service’s first (default) response, `for 2nd ordered response on ` its second one. Everything else works like `rest.response.properties.match`. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | ------------ | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{service}` | the name of a REST service the scenario registered | `parcels` | A row is a property’s JSONPath and a regular expression its value must match. **Variants**, the parts in `[[...]]` said or left out: * `the response payload properties for response on {service} match:` * `the response payload properties for {ordinal} ordered response on {service} match:` **Example:** ```gherkin Then the response payload properties for 2nd ordered response on parcels match: | [0].reference | PX-[0-9]{4} | ``` ### `rest.token` [Section titled “rest.token”](#resttoken) ```gherkin Given the {word} token with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register a bearer token under a name: a JSON Web Token Axx signs, or an OAuth 2.0 client credentials token Axx gets from a token endpoint. * A request uses it with `the request is authorized with the {word} token`; the steps of other packs that take header rows, like the websocket and sse packs, with `${token:}`. * A JSON Web Token is signed each time it is used, with `iat` and `exp` from then; the claims of the table add to them or replace them. * A client credentials token is got once for the run, and again once it expires. * Values expand `${env:..}` and `${sys:..}`; the token and its secrets are masked in logs and failures. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Values | Default | | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | ------- | | `algorithm` | how Axx signs a JSON Web Token | `HS256`, `HS384`, `HS512`, `RS256`, `ES256` | `HS256` | | `key` | the secret an HS token is signed with, like `${env:SHOP_TOKEN_KEY}`, or the PEM private key (a file of the project) of an RS or ES token | | | | `key id` | the `kid` in the token’s header | | | | `claim.` | a claim of the token: a JSON value (`true`, `42`, `["parcels:read"]`) or text | | | | `expires in` | how long the token lasts from when it is used, like `5m` | | `5m` | | `token url` | an OAuth 2.0 token endpoint Axx gets a client credentials token from, instead of signing one | | | | `client id` | the client’s ID, for the token endpoint | | | | `client secret` | the client’s secret, like `${env:SHOP_CLIENT_SECRET}` | | | | `scope` | the scopes asked for, separated by spaces | | | | `audience` | the API the token is for, for token endpoints that ask for one | | | Give a `key` for a token Axx signs, or a `token url`, `client id` and `client secret` for one it gets, not both. **Example:** ```gherkin Given the shop token with the following properties: | key | ${env:SHOP_TOKEN_KEY} | | claim.shop | maple-crafts | Given the shop-system token with the following properties: | token url | http://localhost:8400/oauth/token | | client id | maple-crafts | | client secret | ${env:SHOP_CLIENT_SECRET} | | scope | parcels:read | ``` *Since 0.1.5.* ### `rest.request.token` [Section titled “rest.request.token”](#restrequesttoken) ```gherkin Given the request is authorized with the {word} token[[ for {ordinal} ordered request]] ``` Send the request with the token of that name: `Authorization: Bearer `. * The token is had when the request is executed: a JSON Web Token is signed then. * Without an ordinal it applies to the service’s first (default) request; `for 2nd ordered request`, to its second. * It uses the default service, the first one registered; `rest.request.token.on` names a service. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | **Variants**, the parts in `[[...]]` said or left out: * `the request is authorized with the {word} token` * `the request is authorized with the {word} token for {ordinal} ordered request` **Example:** ```gherkin Given the request is authorized with the shop token ``` *Since 0.1.5.* ### `rest.request.token.on` [Section titled “rest.request.token.on”](#restrequesttokenon) ```gherkin Given the request is authorized with the {word} token for[[ {ordinal} ordered]] request on {service} ``` `rest.request.token` on a named service: `for request on ` addresses the service’s first (default) request, `for 2nd ordered request on ` its second one. Everything else works like `rest.request.token`. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{service}` | the name of a REST service the scenario registered | `parcels` | **Variants**, the parts in `[[...]]` said or left out: * `the request is authorized with the {word} token for request on {service}` * `the request is authorized with the {word} token for {ordinal} ordered request on {service}` **Example:** ```gherkin Given the request is authorized with the shop token for 2nd ordered request on parcels ``` *Since 0.1.5.* ### `rest.request.signed` [Section titled “rest.request.signed”](#restrequestsigned) ```gherkin Given the request is signed in the {word} header[[ for {ordinal} ordered request]] with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Sign the request as webhooks are signed: an HMAC of its body, or of what the `signs` template gives, in the header. * The signature is made when the request is executed, over the body as it is sent. * `sha256={signature}` is GitHub’s form; `t={timestamp},v1={signature}` with `{timestamp}.{body}` signed, Stripe’s. * The key expands `${env:..}`, and is masked. * Without an ordinal it applies to the service’s first (default) request; `for 2nd ordered request`, to its second. * It uses the default service, the first one registered; `rest.request.signed.on` names a service. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | Property | Takes | Values | Default | | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------- | ------------- | | `key` | the secret the signature is made with, like `${env:COURIER_WEBHOOK_KEY}` | | *required* | | `algorithm` | the HMAC | `hmac-sha256`, `hmac-sha1`, `hmac-sha512` | `hmac-sha256` | | `signs` | what is signed: a template of `{body}`, `{timestamp}` (Unix seconds), `{method}` and `{path}` (with the query), like `{timestamp}.{body}` | | `{body}` | | `encoding` | how the signature is written | `hex`, `base64` | `hex` | | `value` | the header’s value: a template of `{signature}` and `{timestamp}`, like `sha256={signature}` or `t={timestamp},v1={signature}` | | `{signature}` | | `timestamp header` | a header that also carries `{timestamp}`, as Slack’s `X-Slack-Request-Timestamp` does | | | **Variants**, the parts in `[[...]]` said or left out: * `the request is signed in the {word} header with the following properties:` * `the request is signed in the {word} header for {ordinal} ordered request with the following properties:` **Example:** ```gherkin Given the request is signed in the X-Courier-Signature header with the following properties: | key | ${env:COURIER_WEBHOOK_KEY} | | value | sha256={signature} | ``` *Since 0.1.5.* ### `rest.request.signed.on` [Section titled “rest.request.signed.on”](#restrequestsignedon) ```gherkin Given the request is signed in the {word} header for[[ {ordinal} ordered]] request on {service} with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. `rest.request.signed` on a named service: `for request on ` addresses the service’s first (default) request, `for 2nd ordered request on ` its second one. Everything else works like `rest.request.signed`. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{service}` | the name of a REST service the scenario registered | `parcels` | | Property | Takes | Values | Default | | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------- | ------------- | | `key` | the secret the signature is made with, like `${env:COURIER_WEBHOOK_KEY}` | | *required* | | `algorithm` | the HMAC | `hmac-sha256`, `hmac-sha1`, `hmac-sha512` | `hmac-sha256` | | `signs` | what is signed: a template of `{body}`, `{timestamp}` (Unix seconds), `{method}` and `{path}` (with the query), like `{timestamp}.{body}` | | `{body}` | | `encoding` | how the signature is written | `hex`, `base64` | `hex` | | `value` | the header’s value: a template of `{signature}` and `{timestamp}`, like `sha256={signature}` or `t={timestamp},v1={signature}` | | `{signature}` | | `timestamp header` | a header that also carries `{timestamp}`, as Slack’s `X-Slack-Request-Timestamp` does | | | **Variants**, the parts in `[[...]]` said or left out: * `the request is signed in the {word} header for request on {service} with the following properties:` * `the request is signed in the {word} header for {ordinal} ordered request on {service} with the following properties:` **Example:** ```gherkin Given the request is signed in the Stripe-Signature header for request on parcels with the following properties: | key | ${env:PAYMENTS_WEBHOOK_KEY} | | signs | {timestamp}.{body} | | value | t={timestamp},v1={signature} | ``` *Since 0.1.5.* ### `rest.request.webhook` [Section titled “rest.request.webhook”](#restrequestwebhook) ```gherkin Given the request is signed as a standard webhook with the key {string}[[ for {ordinal} ordered request]] ``` Sign the request as a Standard Webhook (standardwebhooks.com): its `webhook-id`, `webhook-timestamp` and `webhook-signature` headers. * The key is `whsec_` and the key in base64, as Standard Webhooks give it; it expands `${env:..}`, and is masked. * The signature is made when the request is executed, over the body as it is sent. * Without an ordinal it applies to the service’s first (default) request; `for 2nd ordered request`, to its second. * It uses the default service, the first one registered; `rest.request.webhook.on` names a service. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | **Variants**, the parts in `[[...]]` said or left out: * `the request is signed as a standard webhook with the key {string}` * `the request is signed as a standard webhook with the key {string} for {ordinal} ordered request` **Example:** ```gherkin Given the request is signed as a standard webhook with the key '${env:SHOP_WEBHOOK_KEY}' ``` *Since 0.1.5.* ### `rest.request.webhook.on` [Section titled “rest.request.webhook.on”](#restrequestwebhookon) ```gherkin Given the request is signed as a standard webhook with the key {string} for[[ {ordinal} ordered]] request on {service} ``` `rest.request.webhook` on a named service: `for request on ` addresses the service’s first (default) request, `for 2nd ordered request on ` its second one. Everything else works like `rest.request.webhook`. | Parameter | Takes | For example | | ----------- | ---------------------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{service}` | the name of a REST service the scenario registered | `parcels` | **Variants**, the parts in `[[...]]` said or left out: * `the request is signed as a standard webhook with the key {string} for request on {service}` * `the request is signed as a standard webhook with the key {string} for {ordinal} ordered request on {service}` **Example:** ```gherkin Given the request is signed as a standard webhook with the key '${env:SHOP_WEBHOOK_KEY}' for request on shops ``` *Since 0.1.5.* ## Parameter types [Section titled “Parameter types”](#parameter-types) | Parameter | Takes | Values | For example | | ----------- | -------------------------------------------------- | ------ | ----------- | | `{service}` | the name of a REST service the scenario registered | | `parcels` | # SQL > Seed, query and assert on relational databases. PostgreSQL is fully supported (including JSONB and trigger fault injection); MySQL/MariaDB and SQL Server support seeds, selections, row counts and locks, and SQLite all of them but locks. JDBC URLs (jdbc:postgresql://...) are accepted as-is. Seed, query and assert on relational databases. PostgreSQL is fully supported (including JSONB and trigger fault injection); MySQL/MariaDB and SQL Server support seeds, selections, row counts and locks, and SQLite all of them but locks. JDBC URLs (jdbc:postgresql://…) are accepted as-is. Add it to a project with `axx pack add sql`. ## Steps [Section titled “Steps”](#steps) ### `sql.service` [Section titled “sql.service”](#sqlservice) ```gherkin Given a(n) {word} database with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register a database. * The first database registered in a scenario is the default. * The scheme of the `url` says which database it is: PostgreSQL (`postgres://`, `postgresql://`, `jdbc:postgresql:`), MySQL or MariaDB (`mysql://`, `jdbc:mysql:`, `jdbc:mariadb:`), SQL Server (`sqlserver://`, `jdbc:sqlserver:`) or SQLite (`sqlite:`, `file:`, `jdbc:sqlite:`). * A PostgreSQL URL connects without TLS unless it sets `sslmode`; its `currentSchema` sets the default schema. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Default | | ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `url` | where the database is: a native or JDBC URL, like `postgres://localhost:5432/parcels` | *required* | | `user` | the user Axx connects as; SQLite ignores it, but the table needs it all the same | *required* | | `password` | the user’s password; SQLite ignores it, but the table needs it all the same | *required* | | `schema` | a schema, kept for code that builds on the pack; the steps don’t use it, so name tables `schema.table` (or set a PostgreSQL URL’s `currentSchema`) | | `url`, `user` and `password` can take `${env:…}` and `${sys:…}` references. **Example:** ```gherkin Given a parcels-db database with the following properties: | url | postgres://localhost:5432/parcels | | user | parcels | | password | parcels | ``` ### `sql.seed` [Section titled “sql.seed”](#sqlseed) ```gherkin Given a {filepath} db seed[[ on {dbService}]] ``` Insert the rows of a dataset file, in the file’s order and in one transaction. * A YAML file (`.yaml`, `.yml`) maps each `schema.table` to a list of rows, each a mapping of column to value. * A flat XML file (`.xml`) has a `` element per row inside ``. * A JSON file (`.json`) maps each `schema.table` to an array of rows. * A CSV file (`.csv`) stands for its directory, which has a `.csv` file per table, in the order its `table-ordering.txt` lists or else by name; a first line names the columns, and `null` is NULL. The step takes the directory itself too. * An Excel file (`.xlsx`) has a sheet per table, named after it, with a first row that names the columns; an empty cell is NULL. * A value can be `[null]` for NULL, `[UNIX_TIMESTAMP]` for the seconds since 1970, or a time counted from now, like `[DAY,NOW]`, `[DAY,PLUS,1]` or `[HOUR,MINUS,2]` (the unit is `DAY`, `HOUR`, `MIN` or `SEC`). It is the machine’s local time; PostgreSQL also gets its offset, so a `timestamptz` column holds that very moment. * In YAML and JSON, a value that is a list or an object is stored as JSON text. * A row the database refuses, like one whose key exists already, fails the step, and nothing of the file is written. * Seeded rows are never deleted. | Parameter | Takes | For example | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | | `{dbService}` | the name of a database registered in the scenario | `parcels-db` | **Variants**, the parts in `[[...]]` said or left out: * `a {filepath} db seed` * `a {filepath} db seed on {dbService}` **Example:** ```gherkin Given a seeds/manifest-kestrel.yaml db seed Given a seeds/dispatching.yaml db seed on parcels-db Given a seeds/manifests/csv/bulk-manifest/parcels.manifest_lines.csv db seed ``` ### `sql.lock` [Section titled “sql.lock”](#sqllock) ```gherkin Given the rows in the {word} table[[ on {dbService}]] are locked where: | column | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Lock the rows that match, with `SELECT ... FOR UPDATE` on a separate connection. * The locks hold until the row locks are released or the scenario ends. * Rows locked elsewhere, like by another scenario, are waited for 10 seconds at most; then the step fails. * SQLite has no row locks: the step fails on it. | Parameter | Takes | For example | | ------------- | ------------------------------------------------- | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{dbService}` | the name of a database registered in the scenario | `parcels-db` | Each row names a column and the value the locked rows hold in it, or `null` for none (`IS NULL`); values are escaped. **Variants**, the parts in `[[...]]` said or left out: * `the rows in the {word} table are locked where:` * `the rows in the {word} table on {dbService} are locked where:` **Example:** ```gherkin Given the rows in the parcels.parcels table are locked where: | reference | PX-DSP-2001 | ``` ### `sql.unlock` [Section titled “sql.unlock”](#sqlunlock) ```gherkin Then the row locks[[ on {dbService}]] are released ``` Release all the row locks the lock step took on the database. | Parameter | Takes | For example | | ------------- | ------------------------------------------------- | ------------ | | `{dbService}` | the name of a database registered in the scenario | `parcels-db` | **Variants**, the parts in `[[...]]` said or left out: * `the row locks are released` * `the row locks on {dbService} are released` **Example:** ```gherkin Then the row locks are released Then the row locks on parcels-db are released ``` ### `sql.select` [Section titled “sql.select”](#sqlselect) ```gherkin Then a[[ {ordinal}]] selection of rows is retrieved from the {word} table[[ on {dbService}]] where: | column | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Select the rows that match (`SELECT * ... WHERE`) and keep them as the next selection, for the steps that check it. Selections are numbered in the order they are retrieved, whatever ordinal the step says; `the selection` means the first. | Parameter | Takes | For example | | ------------- | ---------------------------------------------------------------------- | -------------------- | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{dbService}` | the name of a database registered in the scenario | `parcels-db` | Each row names a column and the value the selected rows hold in it, or `null` for none (`IS NULL`); values are escaped. **Variants**, the parts in `[[...]]` said or left out: * `a selection of rows is retrieved from the {word} table where:` * `a {ordinal} selection of rows is retrieved from the {word} table where:` * `a selection of rows is retrieved from the {word} table on {dbService} where:` * `a {ordinal} selection of rows is retrieved from the {word} table on {dbService} where:` **Example:** ```gherkin Then a selection of rows is retrieved from the parcels.pickups table where: | reference | PX-WEB-5302 | | day | Friday | Then a 2nd selection of rows is retrieved from the parcels.manifest_lines table on parcels-db where: | manifest_id | M-KESTREL-0412 | | error | null | ``` ### `sql.select.poll` [Section titled “sql.select.poll”](#sqlselectpoll) ```gherkin Then within {duration} a[[ {ordinal}]] selection of at least {int} row(s) is retrieved from the {word} table[[ on {dbService}]] where: | column | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Select the rows that match, again every 500ms, until at least that many come back, and keep them as the next selection. * The step fails when the time is up with fewer rows. * Selections are numbered in the order they are retrieved, whatever ordinal the step says; `the selection` means the first. | Parameter | Takes | For example | | ------------- | ---------------------------------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{int}` | a whole number | `200`, `3` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{dbService}` | the name of a database registered in the scenario | `parcels-db` | Each row names a column and the value the selected rows hold in it, or `null` for none (`IS NULL`); values are escaped. **Variants**, the parts in `[[...]]` said or left out: * `within {duration} a selection of at least {int} row(s) is retrieved from the {word} table where:` * `within {duration} a {ordinal} selection of at least {int} row(s) is retrieved from the {word} table where:` * `within {duration} a selection of at least {int} row(s) is retrieved from the {word} table on {dbService} where:` * `within {duration} a {ordinal} selection of at least {int} row(s) is retrieved from the {word} table on {dbService} where:` **Example:** ```gherkin Then within 10s a selection of at least 2 rows is retrieved from the parcels.manifest_lines table where: | manifest_id | M-KESTREL-0412 | | status | IMPORTED | ``` ### `sql.select.jsonb` [Section titled “sql.select.jsonb”](#sqlselectjsonb) ```gherkin Then a[[ {ordinal}]] selection of rows is retrieved from the {word} table[[ on {dbService}]] where the {word} jsonb column contains: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Select the rows whose JSONB column contains the properties (`@>`), and keep them as the next selection. * The database must be PostgreSQL. * Selections are numbered in the order they are retrieved, whatever ordinal the step says; `the selection` means the first. | Parameter | Takes | For example | | ------------- | ---------------------------------------------------------------------- | -------------------- | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{dbService}` | the name of a database registered in the scenario | `parcels-db` | Each row names a property, dotted (`recipient.city`) for one inside an object, and the value it holds, compared as a JSON string (`null` is JSON null), so it never matches a number or a boolean. **Variants**, the parts in `[[...]]` said or left out: * `a selection of rows is retrieved from the {word} table where the {word} jsonb column contains:` * `a {ordinal} selection of rows is retrieved from the {word} table where the {word} jsonb column contains:` * `a selection of rows is retrieved from the {word} table on {dbService} where the {word} jsonb column contains:` * `a {ordinal} selection of rows is retrieved from the {word} table on {dbService} where the {word} jsonb column contains:` **Example:** ```gherkin Then a selection of rows is retrieved from the parcels.parcels table where the details jsonb column contains: | manifestId | M-KESTREL-0412 | | source | manifest | ``` ### `sql.json.are` [Section titled “sql.json.are”](#sqljsonare) ```gherkin Then the {ordinal} row {word} property for the[[ {ordinal}]] selection[[ on {dbService}]] json properties are: | JSONPath | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check the JSON a column holds in one row of a selection. `the 1st row details property` is the `details` column of the selection’s first row. | Parameter | Takes | For example | | ------------- | ---------------------------------------------------------------------- | -------------------- | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{dbService}` | the name of a database registered in the scenario | `parcels-db` | Each row names a JSONPath, like `source` or `$.recipient.city`, and the value there as text, like `true` or `850`: `null` for JSON null, `undefined` for a property the JSON lacks. **Variants**, the parts in `[[...]]` said or left out: * `the {ordinal} row {word} property for the selection json properties are:` * `the {ordinal} row {word} property for the {ordinal} selection json properties are:` * `the {ordinal} row {word} property for the selection on {dbService} json properties are:` * `the {ordinal} row {word} property for the {ordinal} selection on {dbService} json properties are:` **Example:** ```gherkin Then the 1st row details property for the selection json properties are: | source | portal | | signature | true | | customsInvoice | undefined | ``` ### `sql.json.match` [Section titled “sql.json.match”](#sqljsonmatch) ```gherkin Then the {ordinal} row {word} property for the[[ {ordinal}]] selection[[ on {dbService}]] json properties match: | JSONPath | pattern | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check the JSON a column holds in one row of a selection, like the `json properties are` step, with regular expressions (Java syntax) that must match the whole value, as text. | Parameter | Takes | For example | | ------------- | ---------------------------------------------------------------------- | -------------------- | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{dbService}` | the name of a database registered in the scenario | `parcels-db` | Each row names a JSONPath, like `zone` or `$.recipient.postcode`, and a regular expression its value must match. **Variants**, the parts in `[[...]]` said or left out: * `the {ordinal} row {word} property for the selection json properties match:` * `the {ordinal} row {word} property for the {ordinal} selection json properties match:` * `the {ordinal} row {word} property for the selection on {dbService} json properties match:` * `the {ordinal} row {word} property for the {ordinal} selection on {dbService} json properties match:` **Example:** ```gherkin Then the 1st row recipient property for the 2nd selection json properties match: | postcode | \d{5} | | country | [A-Z]{2} | ``` ### `sql.rows.eq` [Section titled “sql.rows.eq”](#sqlrowseq) ```gherkin Then the[[ {ordinal}]] selection[[ on {dbService}]] has {int} row(s) ``` Check that a selection has exactly the given number of rows. `the selection` means the first selection of the scenario. | Parameter | Takes | For example | | ------------- | ---------------------------------------------------------------------- | ------------ | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{dbService}` | the name of a database registered in the scenario | `parcels-db` | | `{int}` | a whole number | `200`, `3` | **Variants**, the parts in `[[...]]` said or left out: * `the selection has {int} row(s)` * `the {ordinal} selection has {int} row(s)` * `the selection on {dbService} has {int} row(s)` * `the {ordinal} selection on {dbService} has {int} row(s)` **Example:** ```gherkin Then the selection has 2 rows Then the 2nd selection on parcels-db has 1 row ``` ### `sql.rows.gt` [Section titled “sql.rows.gt”](#sqlrowsgt) ```gherkin Then the[[ {ordinal}]] selection[[ on {dbService}]] has more than {int} row(s) ``` Check that a selection has more than the given number of rows. `the selection` means the first selection of the scenario. | Parameter | Takes | For example | | ------------- | ---------------------------------------------------------------------- | ------------ | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{dbService}` | the name of a database registered in the scenario | `parcels-db` | | `{int}` | a whole number | `200`, `3` | **Variants**, the parts in `[[...]]` said or left out: * `the selection has more than {int} row(s)` * `the {ordinal} selection has more than {int} row(s)` * `the selection on {dbService} has more than {int} row(s)` * `the {ordinal} selection on {dbService} has more than {int} row(s)` **Example:** ```gherkin Then the selection has more than 2 rows Then the 2nd selection on parcels-db has more than 1 row ``` ### `sql.rows.lt` [Section titled “sql.rows.lt”](#sqlrowslt) ```gherkin Then the[[ {ordinal}]] selection[[ on {dbService}]] has fewer than {int} row(s) ``` Check that a selection has fewer than the given number of rows. `the selection` means the first selection of the scenario. | Parameter | Takes | For example | | ------------- | ---------------------------------------------------------------------- | ------------ | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{dbService}` | the name of a database registered in the scenario | `parcels-db` | | `{int}` | a whole number | `200`, `3` | **Variants**, the parts in `[[...]]` said or left out: * `the selection has fewer than {int} row(s)` * `the {ordinal} selection has fewer than {int} row(s)` * `the selection on {dbService} has fewer than {int} row(s)` * `the {ordinal} selection on {dbService} has fewer than {int} row(s)` **Example:** ```gherkin Then the selection has fewer than 2 rows Then the 2nd selection on parcels-db has fewer than 1 row ``` ### `sql.trigger.raise` [Section titled “sql.trigger.raise”](#sqltriggerraise) ```gherkin Given a(n)[[ {ordinal} ordered]] before insert trigger on the {word} table[[ on {dbService}]] will raise a(n) {sqlState} exception where: | column | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Make inserts of matching rows into the table fail with that SQLSTATE, to test how the service handles database errors. * A BEFORE INSERT trigger raises the error, so the database must be PostgreSQL. * Triggers are numbered in the order they are created, whatever ordinal the step says. * The trigger is dropped when the scenario ends. | Parameter | Takes | For example | | ------------- | -------------------------------------------------------------------------------------------------------- | -------------------- | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{dbService}` | the name of a database registered in the scenario | `parcels-db` | | `{sqlState}` | a five-character SQLSTATE error code, like `40001` (serialization failure) or `23505` (unique violation) | `40001`, `23505` | Each row names a column and the value the inserted row holds in it, or `null` for none (`IS NULL`); values are escaped. **Variants**, the parts in `[[...]]` said or left out: * `a(n) before insert trigger on the {word} table will raise a(n) {sqlState} exception where:` * `a(n) before insert trigger on the {word} table on {dbService} will raise a(n) {sqlState} exception where:` * `a(n) {ordinal} ordered before insert trigger on the {word} table will raise a(n) {sqlState} exception where:` * `a(n) {ordinal} ordered before insert trigger on the {word} table on {dbService} will raise a(n) {sqlState} exception where:` **Example:** ```gherkin Given a before insert trigger on the parcels.parcels table will raise a 40001 exception where: | reference | PX-DBF-3001 | ``` ### `sql.trigger.raise.times` [Section titled “sql.trigger.raise.times”](#sqltriggerraisetimes) ```gherkin Given a(n)[[ {ordinal} ordered]] before insert trigger on the {word} table[[ on {dbService}]] will raise a(n) {sqlState} exception {int} time(s) where: | column | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Make inserts of matching rows into the table fail with that SQLSTATE, to test how the service handles database errors. * A BEFORE INSERT trigger raises the error, so the database must be PostgreSQL. * Only that many matching inserts fail; the ones after them succeed. * Triggers are numbered in the order they are created, whatever ordinal the step says. * The trigger is dropped when the scenario ends. | Parameter | Takes | For example | | ------------- | -------------------------------------------------------------------------------------------------------- | -------------------- | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{dbService}` | the name of a database registered in the scenario | `parcels-db` | | `{sqlState}` | a five-character SQLSTATE error code, like `40001` (serialization failure) or `23505` (unique violation) | `40001`, `23505` | | `{int}` | a whole number | `200`, `3` | Each row names a column and the value the inserted row holds in it, or `null` for none (`IS NULL`); values are escaped. **Variants**, the parts in `[[...]]` said or left out: * `a(n) before insert trigger on the {word} table will raise a(n) {sqlState} exception {int} time(s) where:` * `a(n) before insert trigger on the {word} table on {dbService} will raise a(n) {sqlState} exception {int} time(s) where:` * `a(n) {ordinal} ordered before insert trigger on the {word} table will raise a(n) {sqlState} exception {int} time(s) where:` * `a(n) {ordinal} ordered before insert trigger on the {word} table on {dbService} will raise a(n) {sqlState} exception {int} time(s) where:` **Example:** ```gherkin Given a before insert trigger on the parcels.parcels table will raise a 40001 exception 1 time where: | reference | PX-DBF-3001 | ``` ### `sql.trigger.insertRaise` [Section titled “sql.trigger.insertRaise”](#sqltriggerinsertraise) ```gherkin Given a(n)[[ {ordinal} ordered]] before insert trigger on the {word} table[[ on {dbService}]] will insert and raise a(n) {sqlState} exception where: | column | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Make inserts of matching rows into the table store the row and still fail with that SQLSTATE, like a write that succeeded but reported a failure. * A BEFORE INSERT trigger raises the error, so the database must be PostgreSQL. * The row is committed through a second connection before the error, with the `dblink` extension, which the step creates when the database lacks it. * Triggers are numbered in the order they are created, whatever ordinal the step says. * The trigger is dropped when the scenario ends. | Parameter | Takes | For example | | ------------- | -------------------------------------------------------------------------------------------------------- | -------------------- | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{dbService}` | the name of a database registered in the scenario | `parcels-db` | | `{sqlState}` | a five-character SQLSTATE error code, like `40001` (serialization failure) or `23505` (unique violation) | `40001`, `23505` | Each row names a column and the value the inserted row holds in it, or `null` for none (`IS NULL`); values are escaped. **Variants**, the parts in `[[...]]` said or left out: * `a(n) before insert trigger on the {word} table will insert and raise a(n) {sqlState} exception where:` * `a(n) before insert trigger on the {word} table on {dbService} will insert and raise a(n) {sqlState} exception where:` * `a(n) {ordinal} ordered before insert trigger on the {word} table will insert and raise a(n) {sqlState} exception where:` * `a(n) {ordinal} ordered before insert trigger on the {word} table on {dbService} will insert and raise a(n) {sqlState} exception where:` **Example:** ```gherkin Given a before insert trigger on the parcels.parcels table will insert and raise a 40001 exception where: | reference | PX-DBF-3001 | ``` ### `sql.trigger.insertRaise.times` [Section titled “sql.trigger.insertRaise.times”](#sqltriggerinsertraisetimes) ```gherkin Given a(n)[[ {ordinal} ordered]] before insert trigger on the {word} table[[ on {dbService}]] will insert and raise a(n) {sqlState} exception {int} time(s) where: | column | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Make inserts of matching rows into the table store the row and still fail with that SQLSTATE, like a write that succeeded but reported a failure. * A BEFORE INSERT trigger raises the error, so the database must be PostgreSQL. * The row is committed through a second connection before the error, with the `dblink` extension, which the step creates when the database lacks it. * Only that many matching inserts fail; the ones after them succeed. * Triggers are numbered in the order they are created, whatever ordinal the step says. * The trigger is dropped when the scenario ends. | Parameter | Takes | For example | | ------------- | -------------------------------------------------------------------------------------------------------- | -------------------- | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{dbService}` | the name of a database registered in the scenario | `parcels-db` | | `{sqlState}` | a five-character SQLSTATE error code, like `40001` (serialization failure) or `23505` (unique violation) | `40001`, `23505` | | `{int}` | a whole number | `200`, `3` | Each row names a column and the value the inserted row holds in it, or `null` for none (`IS NULL`); values are escaped. **Variants**, the parts in `[[...]]` said or left out: * `a(n) before insert trigger on the {word} table will insert and raise a(n) {sqlState} exception {int} time(s) where:` * `a(n) before insert trigger on the {word} table on {dbService} will insert and raise a(n) {sqlState} exception {int} time(s) where:` * `a(n) {ordinal} ordered before insert trigger on the {word} table will insert and raise a(n) {sqlState} exception {int} time(s) where:` * `a(n) {ordinal} ordered before insert trigger on the {word} table on {dbService} will insert and raise a(n) {sqlState} exception {int} time(s) where:` **Example:** ```gherkin Given a before insert trigger on the parcels.parcels table will insert and raise a 40001 exception 1 time where: | reference | PX-DBF-3001 | ``` ### `sql.trigger.raised` [Section titled “sql.trigger.raised”](#sqltriggerraised) ```gherkin Then the[[ {ordinal} ordered]] before insert trigger on the {word} table[[ on {dbService}]] was raised {int} time(s) ``` Check how many times a before insert trigger raised its error. Triggers are numbered in the order they are created; a table other than the trigger’s only logs a warning. | Parameter | Takes | For example | | ------------- | ---------------------------------------------------------------------- | -------------------- | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | `1st`, `2nd` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{dbService}` | the name of a database registered in the scenario | `parcels-db` | | `{int}` | a whole number | `200`, `3` | **Variants**, the parts in `[[...]]` said or left out: * `the before insert trigger on the {word} table was raised {int} time(s)` * `the before insert trigger on the {word} table on {dbService} was raised {int} time(s)` * `the {ordinal} ordered before insert trigger on the {word} table was raised {int} time(s)` * `the {ordinal} ordered before insert trigger on the {word} table on {dbService} was raised {int} time(s)` **Example:** ```gherkin Then the before insert trigger on the parcels.parcels table was raised 1 time Then the 2nd ordered before insert trigger on the parcels.parcels table on parcels-db was raised 3 times ``` ## Parameter types [Section titled “Parameter types”](#parameter-types) | Parameter | Takes | Values | For example | | ------------- | -------------------------------------------------------------------------------------------------------- | ------ | ---------------- | | `{dbService}` | the name of a database registered in the scenario | | `parcels-db` | | `{sqlState}` | a five-character SQLSTATE error code, like `40001` (serialization failure) or `23505` (unique violation) | | `40001`, `23505` | # Server-sent events > Open server-sent event streams (`text/event-stream`) of your services and check the events they send. Open server-sent event streams (`text/event-stream`) of your services and check the events they send. Register a stream with `the {word} event stream with the following properties:`: it opens there and then, with the headers the table gives, and belongs to the scenario, which closes it when it ends. A stream that does not answer 200 with `text/event-stream` fails that step and shows what it answered. * **Events are checked like messages:** `has an event where:` takes a table of paths into the event’s data, as JSON, and the values they have, with `event type` (the event’s `event:` field, or `message` without one) and `event id` rows; `has an event containing {string}` looks for text in any event’s data. The checks wait, 10 seconds unless `within {duration}` says otherwise. * **A stream the server ends** fails a check that is still waiting at once, after looking at every event it sent. * **Secrets stay secret:** `${env:..}` values in the URL and the headers are masked in logs and failures. Add it to a project with `axx pack add sse`. ## Steps [Section titled “Steps”](#steps) ### `sse.stream` [Section titled “sse.stream”](#ssestream) ```gherkin Given the {word} event stream with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Open a server-sent event stream under a name. It belongs to the scenario, which closes it when it ends. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Default | | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `url` | the stream’s address | *required* | | `header.` | a header of the request, such as `header.Authorization` | | | `asyncapi` | an AsyncAPI document, a file of the project or a URL, that the events follow: every message a scenario sends there, and every message a check finds, is checked against it (with the asyncapi pack) | | Values are expanded (`${env:..}`, `${sys:..}`); what `${env:..}` references expand to is masked everywhere. **Example:** ```gherkin Given the tracking event stream with the following properties: | url | http://localhost:8400/api/parcels/PX-LIV-5702/events | ``` *Since 0.1.5.* ### `sse.event` [Section titled “sse.event”](#sseevent) ```gherkin Then [[within {duration} ]]the {word} event stream has an event where: | path | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the stream sent an event with those values. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a path into the event’s data, as JSON (a field name, a dotted path or a JSONPath), and the value it has, compared as text: `null` for null and `undefined` for absent. The rows `event type` and `event id` are the event’s type (its `event:` field, or `message`) and id. **Variants**, the parts in `[[...]]` said or left out: * `the {word} event stream has an event where:` * `within {duration} the {word} event stream has an event where:` **Example:** ```gherkin Then within 20s the tracking event stream has an event where: | event type | delivered | | reference | PX-LIV-5702 | | status | DELIVERED | ``` *Since 0.1.5.* ### `sse.event.text` [Section titled “sse.event.text”](#sseeventtext) ```gherkin Then [[within {duration} ]]the {word} event stream has an event containing {string} ``` Check that the stream sent an event whose data contains the text. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} event stream has an event containing {string}` * `within {duration} the {word} event stream has an event containing {string}` **Example:** ```gherkin Then the tracking event stream has an event containing 'OUT_FOR_DELIVERY' ``` *Since 0.1.5.* # Accessibility > Check that pages are usable whatever the abilities of the people who use them. It builds on the `web-core` pack. Check that pages are usable whatever the abilities of the people who use them. It builds on the `web-core` pack. * **Audits.** `the page has no accessibility violations` audits the page with [axe-core](https://github.com/dequelabs/axe-core), the accessibility engine of Deque that Lighthouse runs too, against the WCAG 2.1 AA rules by default: fields without labels, text without enough contrast, images without alternative text… The pack downloads axe-core 4.13.0 the first time. A violation fails the step, which says which rule, how serious, and which elements. * **Structure.** `the page's accessible structure is:` checks what assistive technology reads: the headings, fields, buttons and texts, and their names, as Playwright’s ARIA snapshots describe them. List what matters; the rest of the page may be anything. ```yaml packs: web-a11y: standard: wcag21aa # wcag2a, wcag2aa, wcag21a, wcag21aa or wcag22aa ignore: [region] # rules to leave out, by their axe-core id ``` Add it to a project with `axx pack add web-a11y`; it builds on [`web-core`](/next/references/packs/web-core/), which comes with it. ## Settings [Section titled “Settings”](#settings) In `axx.yaml`, under `packs.web-a11y`: | Setting | Takes | Values | Default | | ---------- | ---------------------------------------------------------------------- | ------------------------------------------------------ | ---------- | | `ignore` | Rules the audits leave out, by their axe-core id, like color-contrast. | a list of text | | | `standard` | The WCAG level pages are audited against. | `wcag2a`, `wcag2aa`, `wcag21a`, `wcag21aa`, `wcag22aa` | `wcag21aa` | ## Steps [Section titled “Steps”](#steps) ### `web-a11y.page` [Section titled “web-a11y.page”](#web-a11ypage) ```gherkin Then the page has no accessibility violations ``` Audit the page, in every frame, with axe-core against the WCAG rules of `packs.web-a11y.standard` (2.1 AA by default): the step fails with each rule violated, how serious it is, and the elements that violate it. **Example:** ```gherkin Then the page has no accessibility violations ``` *Since 0.1.1.* ### `web-a11y.element` [Section titled “web-a11y.element”](#web-a11yelement) ```gherkin Then the {string} {element} has no accessibility violations ``` Audit one part of the page, by its name or a selector, with axe-core, as `the page has no accessibility violations` does. | Parameter | Takes | Values | For example | | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | | `"Get a quote"`, `'Express'` | | `{element}` | a kind of element on the page, plural after a number (`buttons`): a `field` by its label or placeholder, an `option` is a radio button, an `element` is anything, by its text, label, alternative text or title | `button`, `field`, `checkbox`, `option`, `link`, `tab`, `menu item`, `element` | `button`, `field` | **Example:** ```gherkin Then the "css=form" element has no accessibility violations ``` *Since 0.1.1.* ### `web-a11y.structure` [Section titled “web-a11y.structure”](#web-a11ystructure) ```gherkin Then [[within {duration} ]]the page's accessible structure is: """ ... """ ``` Check what assistive technology reads on the page, in Playwright’s ARIA snapshot format. * A line is a role and a name, like `- heading "Get a quote" [level=1]` or `- button "Get a quote"`, nested for what is inside. * Only what is listed is checked, in its order. * A name between slashes is a regular expression. * The check waits for the page to match: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | ----------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | **Variants**, the parts in `[[...]]` said or left out: * `the page's accessible structure is:` * `within {duration} the page's accessible structure is:` **Example:** ```gherkin Then the page's accessible structure is: ``` *Since 0.1.1.* ### `web-a11y.element.structure` [Section titled “web-a11y.element.structure”](#web-a11yelementstructure) ```gherkin Then [[within {duration} ]]the {string} {element}'s accessible structure is: """ ... """ ``` Check what assistive technology reads in one part of the page, by its name or a selector, as `the page's accessible structure is:` does. | Parameter | Takes | Values | For example | | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | | `"Get a quote"`, `'Express'` | | `{element}` | a kind of element on the page, plural after a number (`buttons`): a `field` by its label or placeholder, an `option` is a radio button, an `element` is anything, by its text, label, alternative text or title | `button`, `field`, `checkbox`, `option`, `link`, `tab`, `menu item`, `element` | `button`, `field` | **Variants**, the parts in `[[...]]` said or left out: * `the {string} {element}'s accessible structure is:` * `within {duration} the {string} {element}'s accessible structure is:` **Example:** ```gherkin Then the "css=form" element's accessible structure is: ``` *Since 0.1.1.* # Core > Use web apps in real browsers, the way the people who use them do: open pages, fill in forms, click buttons and links, answer dialogs, download files, and check what the page shows. Use web apps in real browsers, the way the people who use them do: open pages, fill in forms, click buttons and links, answer dialogs, download files, and check what the page shows. Each web app is registered with its address: ```gherkin Given the parcels web app with the following properties: | url | http://localhost:8400 | ``` The browsers run on the machine that runs Axx. The pack drives Playwright 1.62.1, and downloads its browsers the first time a scenario uses each one. The web app can use another `engine` (`firefox`, `webkit`, or the Google Chrome or Microsoft Edge installed on the machine), and present the browser as a `device` (like `iPhone 15`), with a `viewport`, a `locale` and a `timezone`. * **Pages.** `the {string} page is opened` opens a page of the app; the other steps act on it. A link that opens a new browser tab makes it the current tab, until it is closed. * **Finding things.** Fields are found by their label (or, without one, their placeholder), buttons, links, tabs and menu items by their name, and text anywhere on the page, in every frame: the names people see. Names must match exactly. Where a name is not enough, a step takes a selector in its place: `css=…`, `xpath=…` or `testid=…`. * **Waiting.** Every step waits for what it needs: an action for its element to be ready, a check for the page to show what it expects, 10 seconds unless `within {duration}` says otherwise. There is no step to sleep. * **Dialogs.** A dialog the page opens (an alert, a confirmation, a question) waits for `the dialog is accepted`, dismissed or answered; other steps fail while it is open. * **Isolation.** Each scenario has its own browser context: its own cookies and storage, so scenarios run in parallel. * **Failures.** A failed step attaches a screenshot of the page. A failed scenario keeps a Playwright trace in `.axx/web/traces`, step by step, with the feature file: open it with `npx playwright show-trace` or at trace.playwright.dev. * **Watching.** With `packs.web-core.watch: true` in axx.yaml, the browsers open in windows on your screen: watch the scenarios, and open DevTools. `slowdown` slows them down. * **Pausing.** `axx run --pause-at :` pauses before a step, and `packs.web-core.pauseOnFailure: true` where a scenario fails, in Playwright’s Inspector: resume, step through the browser’s actions, pick an element, which the Inspector names as steps do (`the "Register" button`), or record what you do on the page as steps. * **Sessions.** A web app can start signed in: `cookie.`, `header.`, `local storage.`, `session storage.`, and `username` and `password` for HTTP authentication, for the app’s address only. * **Secrets.** Take passwords and tokens from the environment, as `${env:…}`: whatever a step takes from the environment is shown as `********` in failures and traces. Values in the steps and the table expand `${env:..}` and `${sys:..}`. Add it to a project with `axx pack add web-core`. ## Settings [Section titled “Settings”](#settings) In `axx.yaml`, under `packs.web-core`: | Setting | Takes | Values | Default | | ----------------- | ---------------------------------------------------------------------------------------------------------------------- | --------------------------- | ------------- | | `pauseOnFailure` | Pause a failed scenario where it failed, in Playwright’s Inspector, until you resume it; the browsers show. | `true`, `false` | | | `scriptErrors` | What errors in the pages’ scripts do: report lists them with a failed scenario (the default), fail fails the scenario. | `report`, `fail` | | | `slowdown` | Wait this long after each browser action, to follow what happens (like 500ms). | text | | | `testIdAttribute` | The attribute testid= selectors look at. | text | `data-testid` | | `traces` | Which scenarios keep a Playwright trace in .axx/web/traces. | `failed`, `always`, `never` | `failed` | | `videos` | Which scenarios keep a video in .axx/web/videos. | `failed`, `always`, `never` | `never` | | `watch` | Show the browsers in windows on your screen, instead of running them headless. | `true`, `false` | | ## Steps [Section titled “Steps”](#steps) ### `web-core.app` [Section titled “web-core.app”](#web-coreapp) ```gherkin Given the {word} web app with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Register a web app, and how the browser presents itself to it. The first web app registered is the default. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Values | Default | | ----------------------- | ------------------------------------------------------------------------------------------ | --------------------------------------------------- | ---------- | | `url` | where the browser finds the app; page paths are below it | | *required* | | `engine` | the browser; `chrome` and `msedge` are those installed on the machine | `chromium`, `firefox`, `webkit`, `chrome`, `msedge` | `chromium` | | `device` | a device Playwright knows, like `iPhone 15` or `Pixel 7`: its screen, user agent and touch | | | | `viewport` | the size of the page, like `1280x720` | | `1280x720` | | `locale` | the browser’s language, like `de-DE` | | `en-US` | | `timezone` | the browser’s time zone, like `Europe/Berlin` | | `UTC` | | `color scheme` | what `prefers-color-scheme` says | `light`, `dark`, `no-preference` | | | `reduced motion` | what `prefers-reduced-motion` says | `reduce`, `no-preference` | | | `media` | the media the page’s styles are for | `screen`, `print` | `screen` | | `user agent` | the user agent the browser sends | | | | `location` | where the browser says it is: a latitude and a longitude, like `52.5200, 13.4050` | | | | `permissions` | the permissions the app has, like `notifications, clipboard-read` | | | | `tls.verify` | `false` accepts the app’s certificate, whatever it is | | `true` | | `username` | the username for the app’s HTTP authentication, with `password` | | | | `password` | the password for the app’s HTTP authentication, with `username` | | | | `cookie.` | a cookie the browser starts with, for the app’s address | | | | `header.` | a header the browser sends the app | | | | `local storage.` | an item of the app’s local storage, from the start | | | | `session storage.` | an item of the app’s session storage, from the start | | | **Example:** ```gherkin Given the parcels web app with the following properties: | url | http://localhost:8400/portal | | engine | webkit | ``` *Since 0.1.1.* ### `web-core.open` [Section titled “web-core.open”](#web-coreopen) ```gherkin When the {string} page[[ of the {word} web app]] is opened ``` Open a page of the web app: a path below its `url`, or a whole URL. The steps that follow act on this page. A page that answers with an error status (404, 500…) fails the step. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Variants**, the parts in `[[...]]` said or left out: * `the {string} page is opened` * `the {string} page of the {word} web app is opened` **Example:** ```gherkin When the "/quotes" page is opened When the "/parcels/PX-4101" page of the portal web app is opened ``` *Since 0.1.1.* ### `web-core.fill` [Section titled “web-core.fill”](#web-corefill) ```gherkin When the {string} field is filled with {string} ``` Type a value into the field with that label (or, without a label, that placeholder), replacing what it holds. * Date fields take dates as `2026-10-01`. * A value from `${env:…}`, such as a password, is a secret: failures and traces show it as `********`. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Example:** ```gherkin When the "Weight (grams)" field is filled with "1200" When the "Password" field is filled with "${env:SHOP_PASSWORD}" ``` *Since 0.1.1.* ### `web-core.select` [Section titled “web-core.select”](#web-coreselect) ```gherkin When {string} is chosen in the {string} field ``` Choose the option with that text in a drop-down field. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Example:** ```gherkin When "Germany" is chosen in the "Destination country" field ``` *Since 0.1.1.* ### `web-core.option` [Section titled “web-core.option”](#web-coreoption) ```gherkin When the {string} option is chosen ``` Choose the radio button with that label. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Example:** ```gherkin When the "Express" option is chosen ``` *Since 0.1.1.* ### `web-core.check` [Section titled “web-core.check”](#web-corecheck) ```gherkin When the {string} checkbox is checked ``` Tick the checkbox with that label (a checkbox already ticked stays so). | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Example:** ```gherkin When the "Insure this parcel" checkbox is checked ``` *Since 0.1.1.* ### `web-core.uncheck` [Section titled “web-core.uncheck”](#web-coreuncheck) ```gherkin When the {string} checkbox is unchecked ``` Clear the checkbox with that label (a checkbox already clear stays so). | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Example:** ```gherkin When the "Leave with a neighbour" checkbox is unchecked ``` *Since 0.1.1.* ### `web-core.upload` [Section titled “web-core.upload”](#web-coreupload) ```gherkin When the {filepath} file is uploaded in the {string} field ``` Choose a file of the project for the file field with that label, as if picked in the file dialog; or for a button with that name that opens the file dialog. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Example:** ```gherkin When the invoices/INV-2041.pdf file is uploaded in the "Customs invoice" field ``` *Since 0.1.1.* ### `web-core.button` [Section titled “web-core.button”](#web-corebutton) ```gherkin When the {string} button is clicked ``` Click the button with that name: its text, or its label. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Example:** ```gherkin When the "Get a quote" button is clicked ``` *Since 0.1.1.* ### `web-core.link` [Section titled “web-core.link”](#web-corelink) ```gherkin When the {string} link is clicked ``` Click the link with that text. A link that opens a new browser tab makes it the current tab, as a browser shows it. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Example:** ```gherkin When the "Track a parcel" link is clicked ``` *Since 0.1.1.* ### `web-core.tab` [Section titled “web-core.tab”](#web-coretab) ```gherkin When the {string} tab is clicked ``` Click the tab with that name, in a row of tabs on the page. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Example:** ```gherkin When the "History" tab is clicked ``` *Since 0.1.1.* ### `web-core.menuitem` [Section titled “web-core.menuitem”](#web-coremenuitem) ```gherkin When the {string} menu item is clicked ``` Click the item with that name in an open menu: a plain one, one that ticks on and off, or one that picks one of a group (`menuitem`, `menuitemcheckbox`, `menuitemradio`). | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Example:** ```gherkin When the "Cancel the parcel" menu item is clicked ``` *Since 0.1.1.* ### `web-core.element.click` [Section titled “web-core.element.click”](#web-coreelementclick) ```gherkin When the {string} element is clicked ``` Click an element that is none of the above: by its text, its label, an image’s alternative text or its title, or by a selector (`css=…`, `xpath=…` or `testid=…`). | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Example:** ```gherkin When the "css=.parcel-card" element is clicked ``` *Since 0.1.1.* ### `web-core.key` [Section titled “web-core.key”](#web-corekey) ```gherkin When the {word} key is pressed[[ in the {string} field]] ``` Press a key: `Enter`, `Escape`, `Tab`, `ArrowDown`, a letter, or a combination like `Control+A`. The key goes to the field with that label, or to whatever has the focus. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} key is pressed` * `the {word} key is pressed in the {string} field` **Example:** ```gherkin When the Enter key is pressed in the "Find a parcel" field When the Escape key is pressed ``` *Since 0.1.1.* ### `web-core.hover` [Section titled “web-core.hover”](#web-corehover) ```gherkin When the pointer is moved over {string} ``` Move the pointer over the button, link, tab or menu item with that name, or else over that text, so that what opens on hover opens: a menu, a tip. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Example:** ```gherkin When the pointer is moved over "About Express" ``` *Since 0.1.1.* ### `web-core.reload` [Section titled “web-core.reload”](#web-corereload) ```gherkin When the page is reloaded ``` Reload the current page. A page that answers with an error status fails the step. **Example:** ```gherkin When the page is reloaded ``` *Since 0.1.1.* ### `web-core.back` [Section titled “web-core.back”](#web-coreback) ```gherkin When the browser's back button is clicked ``` Go back to the page before, as the browser’s back button does. **Example:** ```gherkin When the browser's back button is clicked ``` *Since 0.1.1.* ### `web-core.closetab` [Section titled “web-core.closetab”](#web-coreclosetab) ```gherkin When the browser tab is closed ``` Close the current browser tab, such as one a link opened; the tab it was opened from becomes the current tab again. **Example:** ```gherkin When the browser tab is closed ``` *Since 0.1.1.* ### `web-core.accept` [Section titled “web-core.accept”](#web-coreaccept) ```gherkin When the dialog is accepted ``` Accept the dialog the page shows (its OK): a message, a confirmation or a question. The page waits for an answer, so every other step fails while a dialog is open. The step waits for the dialog: 10 seconds. **Example:** ```gherkin When the dialog is accepted ``` *Since 0.1.1.* ### `web-core.dismiss` [Section titled “web-core.dismiss”](#web-coredismiss) ```gherkin When the dialog is dismissed ``` Dismiss the dialog the page shows (its Cancel). The step waits for the dialog: 10 seconds. **Example:** ```gherkin When the dialog is dismissed ``` *Since 0.1.1.* ### `web-core.answer` [Section titled “web-core.answer”](#web-coreanswer) ```gherkin When the dialog is answered with {string} ``` Type an answer to the question the dialog asks, and accept it. The step waits for the dialog: 10 seconds. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Example:** ```gherkin When the dialog is answered with "The parcel arrived damaged" ``` *Since 0.1.1.* ### `web-core.dialog` [Section titled “web-core.dialog”](#web-coredialog) ```gherkin Then [[within {duration} ]]the dialog shows {string} ``` Check that the page shows a dialog with the text, as part of its message (case matters). The check waits for a dialog: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the dialog shows {string}` * `within {duration} the dialog shows {string}` **Example:** ```gherkin Then the dialog shows "Cancel parcel PX-4101?" ``` *Since 0.1.1.* ### `web-core.shows` [Section titled “web-core.shows”](#web-coreshows) ```gherkin Then [[within {duration} ]]the page shows {string} ``` Check that the page shows the text, anywhere, as part of what it shows (case matters). The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the page shows {string}` * `within {duration} the page shows {string}` **Example:** ```gherkin Then the page shows "6.90 EUR" Then within 30s the page shows "Parcel PX-4101 registered" ``` *Since 0.1.1.* ### `web-core.hides` [Section titled “web-core.hides”](#web-corehides) ```gherkin Then [[within {duration} ]]the page does not show {string} ``` Check that the page does not show the text anywhere (case matters). The check waits for the text to go: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the page does not show {string}` * `within {duration} the page does not show {string}` **Example:** ```gherkin Then the page does not show "Delivery not possible" ``` *Since 0.1.1.* ### `web-core.url` [Section titled “web-core.url”](#web-coreurl) ```gherkin Then [[within {duration} ]]the {string} page is shown ``` Check which page the browser is on: its path below the web app’s `url` (with the query, when the step gives one), or its whole URL. The check waits for the browser to get there: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {string} page is shown` * `within {duration} the {string} page is shown` **Example:** ```gherkin Then the "/quotes/Q-3317" page is shown ``` *Since 0.1.1.* ### `web-core.value` [Section titled “web-core.value”](#web-corevalue) ```gherkin Then [[within {duration} ]]the {string} field has the value {string} ``` Check the value a field shows: what it holds, or for a drop-down, the text of the option chosen. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {string} field has the value {string}` * `within {duration} the {string} field has the value {string}` **Example:** ```gherkin Then the "Weight (grams)" field has the value "1200" ``` *Since 0.1.1.* ### `web-core.disabled` [Section titled “web-core.disabled”](#web-coredisabled) ```gherkin Then [[within {duration} ]]the {string} button is disabled ``` Check that the button with that name is disabled. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {string} button is disabled` * `within {duration} the {string} button is disabled` **Example:** ```gherkin Then the "Register the parcel" button is disabled ``` *Since 0.1.1.* ### `web-core.enabled` [Section titled “web-core.enabled”](#web-coreenabled) ```gherkin Then [[within {duration} ]]the {string} button is enabled ``` Check that the button with that name is enabled. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {string} button is enabled` * `within {duration} the {string} button is enabled` **Example:** ```gherkin Then within 5s the "Register the parcel" button is enabled ``` *Since 0.1.1.* ### `web-core.row` [Section titled “web-core.row”](#web-corerow) ```gherkin Then [[within {duration} ]]the page shows a table row where: | column | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that a table on the page has a row with those values in those columns. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | ----------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | Each row names a column, by its header, and the value the table’s row has in it. **Variants**, the parts in `[[...]]` said or left out: * `the page shows a table row where:` * `within {duration} the page shows a table row where:` **Example:** ```gherkin Then the page shows a table row where: | Parcel | PX-WEB-5116 | | Status | IN_TRANSIT | ``` *Since 0.1.1.* ### `web-core.downloaded` [Section titled “web-core.downloaded”](#web-coredownloaded) ```gherkin Then [[within {duration} ]]the {string} file is downloaded ``` Check that the browser downloaded a file with that name, as the browser names it. The check waits for it: 10 seconds, or `within {duration}`. A failed scenario keeps the downloads its steps checked in `.axx/web/downloads`. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {string} file is downloaded` * `within {duration} the {string} file is downloaded` **Example:** ```gherkin Then the "PX-4101-label.zpl" file is downloaded ``` *Since 0.1.1.* ### `web-core.download.contains` [Section titled “web-core.download.contains”](#web-coredownloadcontains) ```gherkin Then the downloaded {string} file contains {string} ``` Check that a file the browser downloaded holds the text. * Case matters; runs of spaces and line breaks count as one space. * The text is read by the file’s type: the text of a PDF’s pages, the paragraphs and tables of a Word document (.docx), the cells of an Excel workbook (.xlsx), or the text itself. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Example:** ```gherkin Then the downloaded "parcels.csv" file contains "PX-4101,Anna Weber,IN_TRANSIT" ``` *Since 0.1.1.* ### `web-core.download.row` [Section titled “web-core.download.row”](#web-coredownloadrow) ```gherkin Then the downloaded {string} file has a row where: | column | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the table of a file the browser downloaded has a row with those values in those columns: a CSV or TSV file, or an Excel workbook (.xlsx, its first sheet), whose first row names the columns. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | Each row names a column, as the file’s first row names it, and the value the file’s row has in it. **Example:** ```gherkin Then the downloaded "parcels-birch-and-bloom.csv" file has a row where: | Parcel | PX-WEB-5401 | | Status | REGISTERED | ``` *Since 0.1.1.* ### `web-core.download.identical` [Section titled “web-core.download.identical”](#web-coredownloadidentical) ```gherkin Then the downloaded {string} file is identical to the {filepath} file ``` Check that a file the browser downloaded has exactly the content of a file of the project. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | **Example:** ```gherkin Then the downloaded "PX-4101-label.zpl" file is identical to the labels/PX-4101.zpl file ``` *Since 0.1.1.* ### `web-core.element.shown` [Section titled “web-core.element.shown”](#web-coreelementshown) ```gherkin Then [[within {duration} ]]the {string} {element} is shown ``` Check that the page shows the element: by the name people see, or by a selector (`css=…`, `xpath=…` or `testid=…`). The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | Values | For example | | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | | `"Get a quote"`, `'Express'` | | `{element}` | a kind of element on the page, plural after a number (`buttons`): a `field` by its label or placeholder, an `option` is a radio button, an `element` is anything, by its text, label, alternative text or title | `button`, `field`, `checkbox`, `option`, `link`, `tab`, `menu item`, `element` | `button`, `field` | **Variants**, the parts in `[[...]]` said or left out: * `the {string} {element} is shown` * `within {duration} the {string} {element} is shown` **Example:** ```gherkin Then the "Cancel the parcel" button is shown Then within 5s the "css=.parcel-card" element is shown ``` *Since 0.1.1.* ### `web-core.element.hidden` [Section titled “web-core.element.hidden”](#web-coreelementhidden) ```gherkin Then [[within {duration} ]]the {string} {element} is not shown ``` Check that the page does not show the element, by its name or a selector. The check waits for it to go: 10 seconds, or `within {duration}`. | Parameter | Takes | Values | For example | | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | | `"Get a quote"`, `'Express'` | | `{element}` | a kind of element on the page, plural after a number (`buttons`): a `field` by its label or placeholder, an `option` is a radio button, an `element` is anything, by its text, label, alternative text or title | `button`, `field`, `checkbox`, `option`, `link`, `tab`, `menu item`, `element` | `button`, `field` | **Variants**, the parts in `[[...]]` said or left out: * `the {string} {element} is not shown` * `within {duration} the {string} {element} is not shown` **Example:** ```gherkin Then the "Cancel the parcel" menu item is not shown ``` *Since 0.1.1.* ### `web-core.element.text` [Section titled “web-core.element.text”](#web-coreelementtext) ```gherkin Then [[within {duration} ]]the {string} {element} shows {string} ``` Check that the element shows the text, as part of what it shows (case matters). The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | Values | For example | | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | | `"Get a quote"`, `'Express'` | | `{element}` | a kind of element on the page, plural after a number (`buttons`): a `field` by its label or placeholder, an `option` is a radio button, an `element` is anything, by its text, label, alternative text or title | `button`, `field`, `checkbox`, `option`, `link`, `tab`, `menu item`, `element` | `button`, `field` | **Variants**, the parts in `[[...]]` said or left out: * `the {string} {element} shows {string}` * `within {duration} the {string} {element} shows {string}` **Example:** ```gherkin Then the "css=#price" element shows "6.90 EUR" Then the "Details" tab shows "3" ``` *Since 0.1.1.* ### `web-core.element.attribute` [Section titled “web-core.element.attribute”](#web-coreelementattribute) ```gherkin Then [[within {duration} ]]the {string} {element} has the {word} attribute {string} ``` Check the value of one of the element’s attributes, such as `href` or `aria-expanded`. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | Values | For example | | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | | `"Get a quote"`, `'Express'` | | `{element}` | a kind of element on the page, plural after a number (`buttons`): a `field` by its label or placeholder, an `option` is a radio button, an `element` is anything, by its text, label, alternative text or title | `button`, `field`, `checkbox`, `option`, `link`, `tab`, `menu item`, `element` | `button`, `field` | | `{word}` | one word, with no spaces | | `parcels`, `PX-4101` | **Variants**, the parts in `[[...]]` said or left out: * `the {string} {element} has the {word} attribute {string}` * `within {duration} the {string} {element} has the {word} attribute {string}` **Example:** ```gherkin Then the "Actions" button has the aria-expanded attribute "true" ``` *Since 0.1.1.* ### `web-core.element.focus` [Section titled “web-core.element.focus”](#web-coreelementfocus) ```gherkin Then [[within {duration} ]]the {string} {element} has the focus ``` Check that the element has the keyboard’s focus. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | Values | For example | | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | | `"Get a quote"`, `'Express'` | | `{element}` | a kind of element on the page, plural after a number (`buttons`): a `field` by its label or placeholder, an `option` is a radio button, an `element` is anything, by its text, label, alternative text or title | `button`, `field`, `checkbox`, `option`, `link`, `tab`, `menu item`, `element` | `button`, `field` | **Variants**, the parts in `[[...]]` said or left out: * `the {string} {element} has the focus` * `within {duration} the {string} {element} has the focus` **Example:** ```gherkin Then the "Find a parcel" field has the focus ``` *Since 0.1.1.* ### `web-core.checkbox.ticked` [Section titled “web-core.checkbox.ticked”](#web-corecheckboxticked) ```gherkin Then [[within {duration} ]]the {string} checkbox is ticked ``` Check that the checkbox with that name is ticked. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {string} checkbox is ticked` * `within {duration} the {string} checkbox is ticked` **Example:** ```gherkin Then the "Signature on delivery" checkbox is ticked ``` *Since 0.1.1.* ### `web-core.checkbox.unticked` [Section titled “web-core.checkbox.unticked”](#web-corecheckboxunticked) ```gherkin Then [[within {duration} ]]the {string} checkbox is not ticked ``` Check that the checkbox with that name is not ticked. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {string} checkbox is not ticked` * `within {duration} the {string} checkbox is not ticked` **Example:** ```gherkin Then the "Leave with a neighbour" checkbox is not ticked ``` *Since 0.1.1.* ### `web-core.option.selected` [Section titled “web-core.option.selected”](#web-coreoptionselected) ```gherkin Then [[within {duration} ]]the {string} option is selected ``` Check that the option with that name is selected. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {string} option is selected` * `within {duration} the {string} option is selected` **Example:** ```gherkin Then the "Express" option is selected ``` *Since 0.1.1.* ### `web-core.option.unselected` [Section titled “web-core.option.unselected”](#web-coreoptionunselected) ```gherkin Then [[within {duration} ]]the {string} option is not selected ``` Check that the option with that name is not selected. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {string} option is not selected` * `within {duration} the {string} option is not selected` **Example:** ```gherkin Then the "Standard" option is not selected ``` *Since 0.1.1.* ### `web-core.field.disabled` [Section titled “web-core.field.disabled”](#web-corefielddisabled) ```gherkin Then [[within {duration} ]]the {string} field is disabled ``` Check that the field with that name is disabled. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {string} field is disabled` * `within {duration} the {string} field is disabled` **Example:** ```gherkin Then the "Parcel reference" field is disabled ``` *Since 0.1.1.* ### `web-core.field.enabled` [Section titled “web-core.field.enabled”](#web-corefieldenabled) ```gherkin Then [[within {duration} ]]the {string} field is enabled ``` Check that the field with that name is enabled. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {string} field is enabled` * `within {duration} the {string} field is enabled` **Example:** ```gherkin Then the "Weight (grams)" field is enabled ``` *Since 0.1.1.* ### `web-core.element.count` [Section titled “web-core.element.count”](#web-coreelementcount) ```gherkin Then [[within {duration} ]]the page shows {int} {string} {element} ``` Check how many of the elements the page shows, by their name or a selector. The check waits for the number: 10 seconds, or `within {duration}`. | Parameter | Takes | Values | For example | | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | | `5s`, `2m` | | `{int}` | a whole number | | `200`, `3` | | `{string}` | text in double or single quotes, which the step leaves out | | `"Get a quote"`, `'Express'` | | `{element}` | a kind of element on the page, plural after a number (`buttons`): a `field` by its label or placeholder, an `option` is a radio button, an `element` is anything, by its text, label, alternative text or title | `button`, `field`, `checkbox`, `option`, `link`, `tab`, `menu item`, `element` | `button`, `field` | **Variants**, the parts in `[[...]]` said or left out: * `the page shows {int} {string} {element}` * `within {duration} the page shows {int} {string} {element}` **Example:** ```gherkin Then the page shows 2 "Remove" buttons Then within 5s the page shows 3 "css=.parcel-card" elements ``` *Since 0.1.1.* ### `web-core.row.count` [Section titled “web-core.row.count”](#web-corerowcount) ```gherkin Then [[within {duration} ]]the page shows {int} table row(s) where: | column | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check how many rows of the page’s tables have those values in those columns. The check waits for the number: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | ----------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{int}` | a whole number | `200`, `3` | Each row names a column, by its header, and the value the tables’ rows have in it. **Variants**, the parts in `[[...]]` said or left out: * `the page shows {int} table row(s) where:` * `within {duration} the page shows {int} table row(s) where:` **Example:** ```gherkin Then the page shows 2 table rows where: | Status | REGISTERED | ``` *Since 0.1.1.* ### `web-core.title` [Section titled “web-core.title”](#web-coretitle) ```gherkin Then [[within {duration} ]]the page title is {string} ``` Check the page’s title, as the browser tab shows it. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the page title is {string}` * `within {duration} the page title is {string}` **Example:** ```gherkin Then the page title is "Get a quote · Parcels" ``` *Since 0.1.1.* ### `web-core.errors.none` [Section titled “web-core.errors.none”](#web-coreerrorsnone) ```gherkin Then the page has no script errors ``` Check that the web app’s pages have had no errors in their scripts since the scenario opened them: no uncaught errors, and none logged to the console. **Example:** ```gherkin Then the page has no script errors ``` *Since 0.1.1.* ### `web-core.request.sent` [Section titled “web-core.request.sent”](#web-corerequestsent) ```gherkin Then [[within {duration} ]]the browser sent a {word} request to {string} ``` Check that the web app’s pages sent a request with that method to that address. * The address is a path below the web app’s `url`, or a whole URL. The query counts only when the step gives one. * The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the browser sent a {word} request to {string}` * `within {duration} the browser sent a {word} request to {string}` **Example:** ```gherkin Then the browser sent a POST request to "/api/parcels" ``` *Since 0.1.1.* ### `web-core.element.dblclick` [Section titled “web-core.element.dblclick”](#web-coreelementdblclick) ```gherkin When the {string} {element} is double-clicked ``` Double-click the element, by its name or a selector. | Parameter | Takes | Values | For example | | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | | `"Get a quote"`, `'Express'` | | `{element}` | a kind of element on the page, plural after a number (`buttons`): a `field` by its label or placeholder, an `option` is a radio button, an `element` is anything, by its text, label, alternative text or title | `button`, `field`, `checkbox`, `option`, `link`, `tab`, `menu item`, `element` | `button`, `field` | **Example:** ```gherkin When the "PX-4101" element is double-clicked ``` *Since 0.1.1.* ### `web-core.element.rightclick` [Section titled “web-core.element.rightclick”](#web-coreelementrightclick) ```gherkin When the {string} {element} is right-clicked ``` Right-click the element, by its name or a selector: a menu of the page’s own may open. | Parameter | Takes | Values | For example | | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | | `"Get a quote"`, `'Express'` | | `{element}` | a kind of element on the page, plural after a number (`buttons`): a `field` by its label or placeholder, an `option` is a radio button, an `element` is anything, by its text, label, alternative text or title | `button`, `field`, `checkbox`, `option`, `link`, `tab`, `menu item`, `element` | `button`, `field` | **Example:** ```gherkin When the "PX-4101" link is right-clicked ``` *Since 0.1.1.* ### `web-core.element.tap` [Section titled “web-core.element.tap”](#web-coreelementtap) ```gherkin When the {string} {element} is tapped ``` Tap the element, by its name or a selector, on a touch screen: the web app’s `device` must have one (like `iPhone 15`). | Parameter | Takes | Values | For example | | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | | `"Get a quote"`, `'Express'` | | `{element}` | a kind of element on the page, plural after a number (`buttons`): a `field` by its label or placeholder, an `option` is a radio button, an `element` is anything, by its text, label, alternative text or title | `button`, `field`, `checkbox`, `option`, `link`, `tab`, `menu item`, `element` | `button`, `field` | **Example:** ```gherkin When the "Register a parcel" link is tapped ``` *Since 0.1.1.* ### `web-core.element.scroll` [Section titled “web-core.element.scroll”](#web-coreelementscroll) ```gherkin When the {string} {element} is scrolled into view ``` Scroll the page (or the list it is in) until the element shows. | Parameter | Takes | Values | For example | | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | | `"Get a quote"`, `'Express'` | | `{element}` | a kind of element on the page, plural after a number (`buttons`): a `field` by its label or placeholder, an `option` is a radio button, an `element` is anything, by its text, label, alternative text or title | `button`, `field`, `checkbox`, `option`, `link`, `tab`, `menu item`, `element` | `button`, `field` | **Example:** ```gherkin When the "css=.parcel-card:last-child" element is scrolled into view ``` *Since 0.1.1.* ### `web-core.element.drag` [Section titled “web-core.element.drag”](#web-coreelementdrag) ```gherkin When the {string} {element} is dragged onto the {string} {element} ``` Drag an element onto another, each by its name or a selector, as with a mouse. | Parameter | Takes | Values | For example | | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | | `"Get a quote"`, `'Express'` | | `{element}` | a kind of element on the page, plural after a number (`buttons`): a `field` by its label or placeholder, an `option` is a radio button, an `element` is anything, by its text, label, alternative text or title | `button`, `field`, `checkbox`, `option`, `link`, `tab`, `menu item`, `element` | `button`, `field` | **Example:** ```gherkin When the "PX-4101" element is dragged onto the "Out for delivery" element ``` *Since 0.1.1.* ### `web-core.scroll.bottom` [Section titled “web-core.scroll.bottom”](#web-corescrollbottom) ```gherkin When the page is scrolled to the bottom ``` Scroll the page to its bottom, as far as it goes: a page that shows more as it scrolls shows it. **Example:** ```gherkin When the page is scrolled to the bottom ``` *Since 0.1.1.* ### `web-core.scroll.top` [Section titled “web-core.scroll.top”](#web-corescrolltop) ```gherkin When the page is scrolled to the top ``` Scroll the page to its top, as far as it goes: a page that shows more as it scrolls shows it. **Example:** ```gherkin When the page is scrolled to the top ``` *Since 0.1.1.* ### `web-core.select.many` [Section titled “web-core.select.many”](#web-coreselectmany) ```gherkin When the following options are chosen in the {string} field: | option | ``` Choose several options in a drop-down field that takes more than one. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | An option a row, by its text. **Example:** ```gherkin When the following options are chosen in the "Pickup days" field: | Monday | | Thursday | ``` *Since 0.1.1.* ### `web-core.clock.set` [Section titled “web-core.clock.set”](#web-coreclockset) ```gherkin Given the browser's clock is set to {string} ``` Set the time the web apps’ pages see, from then on. * The time is like `2026-09-25T14:00:00+02:00`, or `2026-09-25 14:00` in UTC. * Time goes on from there, so the pages’ timers still run. * Set it before opening the pages whose timers a step moves forward. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Example:** ```gherkin Given the browser's clock is set to "2026-09-25T14:00:00+02:00" ``` *Since 0.1.1.* ### `web-core.clock.forward` [Section titled “web-core.clock.forward”](#web-coreclockforward) ```gherkin When the browser's clock is moved forward by {duration} ``` Move the time the web apps’ pages see forward, as if the computer slept that long. * The timers due in between fire once, and what expires, such as a session, expires. * The pages’ timers are the browser’s own from when its clock is set: set it before opening them. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | ----------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | **Example:** ```gherkin When the browser's clock is moved forward by 31m ``` *Since 0.1.1.* ### `web-core.connection.offline` [Section titled “web-core.connection.offline”](#web-coreconnectionoffline) ```gherkin When the browser is offline ``` Take the browser offline: the web apps’ pages cannot reach anything until it is online again. **Example:** ```gherkin When the browser is offline ``` *Since 0.1.1.* ### `web-core.connection.online` [Section titled “web-core.connection.online”](#web-coreconnectiononline) ```gherkin When the browser is online ``` Bring the browser back online. **Example:** ```gherkin When the browser is online ``` *Since 0.1.1.* ## Parameter types [Section titled “Parameter types”](#parameter-types) | Parameter | Takes | Values | For example | | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | ----------------- | | `{element}` | a kind of element on the page, plural after a number (`buttons`): a `field` by its label or placeholder, an `option` is a radio button, an `element` is anything, by its text, label, alternative text or title | `button`, `field`, `checkbox`, `option`, `link`, `tab`, `menu item`, `element` | `button`, `field` | ## Tools for agents [Section titled “Tools for agents”](#tools-for-agents) What the pack lets coding agents do through `axx mcp`, in the scenario they keep open with `steps_try`. | Tool | | | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `web_page` | The page the web app of the agent’s session is on, once steps\_try opened it: its address and title, the elements steps can name, as they name them (the “Get a quote” button, the “Weight (grams)” field), what a screen reader reads (an ARIA snapshot), its script errors and a screenshot. Try steps with steps\_try, look with web\_page, then write the steps into the feature. | # Coverage > Find out how much of the web apps' JavaScript the scenarios run. The pack collects the coverage of the scripts the pages load, in Chromium, Chrome and Edge, as the scenarios run, and writes it when the run ends: `lcov.info`, for editors and coverage services, and Istanbul's `coverage-final.json` and `coverage-summary.json`, in `.axx/web/coverage` (`folder`). It has no steps: add it to the project, and every scenario's pages count. It builds on the `web-core` pack. Find out how much of the web apps’ JavaScript the scenarios run. The pack collects the coverage of the scripts the pages load, in Chromium, Chrome and Edge, as the scenarios run, and writes it when the run ends: `lcov.info`, for editors and coverage services, and Istanbul’s `coverage-final.json` and `coverage-summary.json`, in `.axx/web/coverage` (`folder`). It has no steps: add it to the project, and every scenario’s pages count. It builds on the `web-core` pack. * **What counts.** The scripts the pages load from their web app’s origin, named by their URL path, like `js/track.js`; for a script with a source map, its original files instead, TypeScript and all. Scripts of other origins, inline scripts and event handlers do not count. With `sources`, only the scripts under those URL paths count, named as the files of the project they are. * **How it counts.** Chromium’s own coverage (V8’s): the lines, functions and branches that ran, and how many times, over every page of every scenario, taken after each step and before a page leaves its document. The reports are those Node’s tools (c8) write from the same coverage. A tab a page opens counts from when it opens. In Firefox and WebKit, the pages run without coverage. ```yaml packs: web-coverage: sources: /portal/js/: ../app/web/js # the scripts under /portal/js/, in the project /portal/src/: ../app/web/src # the original files of their source maps ``` A source map without its original files’ content finds them through `sources` too. Add it to a project with `axx pack add web-coverage`; it builds on [`web-core`](/next/references/packs/web-core/), which comes with it. ## Settings [Section titled “Settings”](#settings) In `axx.yaml`, under `packs.web-coverage`: | Setting | Takes | Values | Default | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | ------------------- | | `folder` | The folder the reports go to, in the project: lcov.info, coverage-final.json and coverage-summary.json. | text | `.axx/web/coverage` | | `sources` | Where the scripts are in the project, by the start of their URL path: {“/portal/js/”: “../app/web/js”}, a folder relative to the project. Only the scripts (or, through their source maps, the original files) whose URL path starts with one count, named by its folder and the rest of the path. Without it, every script the pages load from their web app’s origin counts, named by its URL path without the leading slash. | names and values | | # Lighthouse > Audit a web app's pages with [Lighthouse](https://developer.chrome.com/docs/lighthouse), Google's tool for the quality of web pages. It builds on the `web-core` pack. Audit a web app’s pages with [Lighthouse](https://developer.chrome.com/docs/lighthouse), Google’s tool for the quality of web pages. It builds on the `web-core` pack. * **Scores.** `the {string} page scores at least:` checks the page’s Lighthouse scores, from 0 to 100: performance, accessibility, best practices and SEO. A score below its minimum fails the step, which says which audits cost the page the most. * **Loading.** `the {string} page loads within:` checks how fast the page loads, by the Core Web Vitals and Lighthouse’s other metrics: largest contentful paint, first contentful paint, total blocking time, speed index and cumulative layout shift. Lighthouse loads the page afresh in a tab of its own, beside the scenario’s, with the web app’s cookies and storage: signed-in pages are audited signed in. As for a first visit, it clears the browser’s cache and the site’s service workers first, for the scenario’s tabs too. It measures the page as a mid-range phone on a slow 4G connection loads it, or a desktop computer (`device`), simulating the device and the connection rather than slowing the browser down: an audit takes about 5 seconds. Performance varies with the machine’s load, so the pack audits it three times and judges the median run, as Lighthouse recommends; still, leave some room. Each audit attaches Lighthouse’s report, which the pack also keeps in `.axx/web/lighthouse`. Lighthouse audits pages in Chromium, Chrome and Edge. The pack downloads Lighthouse 13.5.0 the first time, about 18 MB from npm’s registry (`npm_config_registry` points it at a mirror), and runs it with the web-core pack’s Node.js. ```yaml packs: web-lighthouse: device: desktop # mobile (the default) or desktop ``` Add it to a project with `axx pack add web-lighthouse`; it builds on [`web-core`](/next/references/packs/web-core/), which comes with it. ## Settings [Section titled “Settings”](#settings) In `axx.yaml`, under `packs.web-lighthouse`: | Setting | Takes | Values | | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------- | | `device` | The device Lighthouse audits pages as: mobile, a mid-range phone on a slow 4G connection (the default), or desktop, Lighthouse’s desktop preset. | `mobile`, `desktop` | ## Steps [Section titled “Steps”](#steps) ### `web-lighthouse.scores` [Section titled “web-lighthouse.scores”](#web-lighthousescores) ```gherkin Then the {string} page scores at least: | category | score | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Audit a page of the web app with Lighthouse, a path below its `url` or a whole URL, and check its scores. * Lighthouse audits the categories listed only, as the `device` of `packs.web-lighthouse` (a phone by default). * With performance among the categories, Lighthouse audits the page three times, and the step judges the median run. * A score below its minimum fails the step, which lists the audits that cost the page the most. * Lighthouse’s report is attached. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | | Category | Takes | | ---------------- | ------------------------------------------- | | `performance` | the lowest score it may have, from 0 to 100 | | `accessibility` | the lowest score it may have, from 0 to 100 | | `best practices` | the lowest score it may have, from 0 to 100 | | `seo` | the lowest score it may have, from 0 to 100 | **Example:** ```gherkin Then the "/quote" page scores at least: | performance | 90 | | accessibility | 100 | | best practices | 100 | | seo | 100 | ``` *Since 0.1.1.* ### `web-lighthouse.loads` [Section titled “web-lighthouse.loads”](#web-lighthouseloads) ```gherkin Then the {string} page loads within: | metric | limit | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Audit how fast a page of the web app loads with Lighthouse, a path below its `url` or a whole URL, and check its metrics. * Lighthouse measures the page as the `device` of `packs.web-lighthouse` would load it (a phone on a slow 4G connection by default). * Lighthouse audits the page three times, and the step judges the median run: a load’s metrics vary with the machine’s load. * A metric above its limit fails the step, which lists what would bring it down. * Lighthouse’s report is attached. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | | Metric | Takes | | -------------------------- | ---------------------------------------------------------- | | `largest contentful paint` | the longest it may take, like `2.5s` (a Core Web Vital) | | `cumulative layout shift` | the most it may be, a number like `0.1` (a Core Web Vital) | | `first contentful paint` | the longest it may take, like `1.8s` | | `total blocking time` | the longest it may be, like `200ms` | | `speed index` | the longest it may take, like `3.4s` | **Example:** ```gherkin Then the "/quote" page loads within: | largest contentful paint | 2.5s | | total blocking time | 200ms | | cumulative layout shift | 0.1 | ``` *Since 0.1.1.* # Network > See and change what a web app's pages fetch, from the browser: requests that fail, answer with an error or a file, or take their time, a slow connection, a recording of the pages' traffic to answer from, and the messages of their websockets. It builds on the `web-core` pack. See and change what a web app’s pages fetch, from the browser: requests that fail, answer with an error or a file, or take their time, a slow connection, a recording of the pages’ traffic to answer from, and the messages of their websockets. It builds on the `web-core` pack. A request is named by its address, a path below the web app’s `url` (`/api/prices`) or a whole URL (`https://maps.example.com/tiles/**`): `*` stands for any part of a path, `**` for any parts; the query does not count unless the address has one. What a step sets holds for every web app of the scenario, from the pages they open next: set it before the page opens. The pages’ own requests only: for the services your app calls from its server, mock them instead (the `mock` pack). `record: true` in `axx.yaml` records what each web app’s pages fetch, for a scenario to answer from later. Add it to a project with `axx pack add web-network`; it builds on [`web-core`](/next/references/packs/web-core/), which comes with it. ## Settings [Section titled “Settings”](#settings) In `axx.yaml`, under `packs.web-network`: | Setting | Takes | Values | Default | | -------- | -------------------------------------------------------------------------------------------------------------------------- | --------------- | ------- | | `record` | Record what each scenario’s pages fetch, a HAR file for each web app in .axx/web/recordings, for scenarios to answer from. | `true`, `false` | `false` | ## Steps [Section titled “Steps”](#steps) ### `web-network.fail` [Section titled “web-network.fail”](#web-networkfail) ```gherkin Given the page's requests to {string} fail ``` Make the pages’ requests to an address fail, as when a service is down or out of reach: a path below the web app’s url, or a whole URL, with `*` and `**`. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Example:** ```gherkin Given the page's requests to "https://maps.example.com/**" fail ``` *Since 0.1.1.* ### `web-network.status` [Section titled “web-network.status”](#web-networkstatus) ```gherkin Given the page's requests to {string} answer with status {int} ``` Answer the pages’ requests to an address with a status and nothing else, like 503. | Parameter | Takes | For example | | ---------- | ---------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | | `{int}` | a whole number | `200`, `3` | **Example:** ```gherkin Given the page's requests to "/api/prices" answer with status 503 ``` *Since 0.1.1.* ### `web-network.file` [Section titled “web-network.file”](#web-networkfile) ```gherkin Given the page's requests to {string} answer with the {filepath} file ``` Answer the pages’ requests to an address with a file of the project, with the type of its extension. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | **Example:** ```gherkin Given the page's requests to "/api/prices" answer with the network/prices-berlin.json file ``` *Since 0.1.1.* ### `web-network.slow` [Section titled “web-network.slow”](#web-networkslow) ```gherkin Given the page's requests to {string} take {duration} ``` Make the pages’ requests to an address take that long before they go out, as a busy service does. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | **Example:** ```gherkin Given the page's requests to "/api/prices" take 3s ``` *Since 0.1.1.* ### `web-network.connection.slow` [Section titled “web-network.connection.slow”](#web-networkconnectionslow) ```gherkin Given the browser's connection is slow ``` Make the browser’s connection slow, as a phone’s in a bad spot: 400 ms of latency and 500 kbit/s each way. Chromium, Chrome and Edge only. **Example:** ```gherkin Given the browser's connection is slow ``` *Since 0.1.1.* ### `web-network.recording` [Section titled “web-network.recording”](#web-networkrecording) ```gherkin Given the page's requests are answered from the {filepath} recording ``` Answer the pages’ requests from a recording of what they fetched, a HAR file (`record: true` makes them): the requests it has, it answers; the others go out, and so do websockets. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | **Example:** ```gherkin Given the page's requests are answered from the network/quote.har recording ``` *Since 0.1.1.* ### `web-network.ws.sent` [Section titled “web-network.ws.sent”](#web-networkwssent) ```gherkin Then [[within {duration} ]]the page sent a websocket message containing {string} ``` Check that a page of the scenario sent a websocket message with that text in it. The check waits: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the page sent a websocket message containing {string}` * `within {duration} the page sent a websocket message containing {string}` **Example:** ```gherkin Then the page sent a websocket message containing "subscribe:PX-4101" ``` *Since 0.1.1.* ### `web-network.ws.received` [Section titled “web-network.ws.received”](#web-networkwsreceived) ```gherkin Then [[within {duration} ]]the page received a websocket message containing {string} ``` Check that a page of the scenario received a websocket message with that text in it. The check waits: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the page received a websocket message containing {string}` * `within {duration} the page received a websocket message containing {string}` **Example:** ```gherkin Then the page received a websocket message containing "IN_TRANSIT" ``` *Since 0.1.1.* # Screenshots > Check that pages look as designed: the page, or one element of it, compared with a screenshot taken before, pixel by pixel, as Playwright compares them (small differences of color and anti-aliased edges aside). It builds on the `web-core` pack. Check that pages look as designed: the page, or one element of it, compared with a screenshot taken before, pixel by pixel, as Playwright compares them (small differences of color and anti-aliased edges aside). It builds on the `web-core` pack. The screenshots are files of the project, `screenshots/.-.png`: browsers draw pages differently on each operating system, so each engine and platform has its own, compared on that platform only. The first time, a step takes its screenshot and fails: look at it, and keep it. ```yaml packs: web-screenshots: platforms: [linux, darwin] # the platforms whose screenshots you keep ``` With `platforms`, the steps compare on those platforms (`linux`, `darwin`, `windows`), each with its own screenshots, and pass elsewhere, saying so. `update: true` takes the screenshots of the platform it runs on again. Add it to a project with `axx pack add web-screenshots`; it builds on [`web-core`](/next/references/packs/web-core/), which comes with it. ## Settings [Section titled “Settings”](#settings) In `axx.yaml`, under `packs.web-screenshots`: | Setting | Takes | Values | Default | | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- | ------- | | `folder` | The folder of the screenshots, in the project. Unset, each feature keeps its screenshots in a screenshots folder next to its file. | text | | | `platforms` | The platforms whose screenshots the project keeps: each compares its own; elsewhere the steps pass without comparing (default: every platform). | a list of `linux`, `darwin`, `windows` | | | `tolerance` | The share of a screenshot’s pixels that may differ. | a number, 0 to 1 | `0` | | `update` | Take every screenshot again, as the page looks now, instead of comparing. | `true`, `false` | | ## Steps [Section titled “Steps”](#steps) ### `web-screenshots.page` [Section titled “web-screenshots.page”](#web-screenshotspage) ```gherkin Then [[within {duration} ]]the page looks like the {string} screenshot ``` Check that the page looks like its screenshot, pixel by pixel (anti-aliasing aside). * The screenshot is `.-.png`, in a `screenshots` folder next to the feature file. Without one, the step takes it and fails: look at it, and keep it. * When the page looks different, the step attaches the screenshot, the page and their difference. * The check waits for the page to settle and look right: 10 seconds, or `within {duration}`. * On a platform `platforms` leaves out, the step passes without comparing. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the page looks like the {string} screenshot` * `within {duration} the page looks like the {string} screenshot` **Example:** ```gherkin Then the page looks like the "quote" screenshot ``` *Since 0.1.1.* ### `web-screenshots.page.masked` [Section titled “web-screenshots.page.masked”](#web-screenshotspagemasked) ```gherkin Then [[within {duration} ]]the page looks like the {string} screenshot, apart from: | element | ``` Check that the page looks like its screenshot, apart from what changes from run to run, such as a date, painted over in both. * The screenshot is `.-.png`, in a `screenshots` folder next to the feature file. Without one, the step takes it and fails: look at it, and keep it. * When the page looks different, the step attaches the screenshot, the page and their difference. * The check waits for the page to settle and look right: 10 seconds, or `within {duration}`. * On a platform `platforms` leaves out, the step passes without comparing. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | An element a row: by its text or label, or a selector like `css=time`. **Variants**, the parts in `[[...]]` said or left out: * `the page looks like the {string} screenshot, apart from:` * `within {duration} the page looks like the {string} screenshot, apart from:` **Example:** ```gherkin Then the page looks like the "parcel" screenshot, apart from: | css=time | ``` *Since 0.1.1.* ### `web-screenshots.element` [Section titled “web-screenshots.element”](#web-screenshotselement) ```gherkin Then [[within {duration} ]]the {string} {element} looks like the {string} screenshot ``` Check that one element, by its name or a selector, looks like its screenshot. * The screenshot is `.-.png`, in a `screenshots` folder next to the feature file. Without one, the step takes it and fails: look at it, and keep it. * When the page looks different, the step attaches the screenshot, the page and their difference. * The check waits for the page to settle and look right: 10 seconds, or `within {duration}`. * On a platform `platforms` leaves out, the step passes without comparing. | Parameter | Takes | Values | For example | | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | | `5s`, `2m` | | `{string}` | text in double or single quotes, which the step leaves out | | `"Get a quote"`, `'Express'` | | `{element}` | a kind of element on the page, plural after a number (`buttons`): a `field` by its label or placeholder, an `option` is a radio button, an `element` is anything, by its text, label, alternative text or title | `button`, `field`, `checkbox`, `option`, `link`, `tab`, `menu item`, `element` | `button`, `field` | **Variants**, the parts in `[[...]]` said or left out: * `the {string} {element} looks like the {string} screenshot` * `within {duration} the {string} {element} looks like the {string} screenshot` **Example:** ```gherkin Then the "css=.quote-card" element looks like the "quote card" screenshot ``` *Since 0.1.1.* # WebSockets > Open WebSocket connections to your services, send messages on them, and check the messages they receive and how they close. Open WebSocket connections to your services, send messages on them, and check the messages they receive and how they close. Register a connection with `the {word} websocket with the following properties:`: it opens there and then, with the headers and subprotocol the table gives, and belongs to the scenario, which closes it when it ends. * **What is received** is checked like the messages of queues and topics: `received a message where:` takes a table of paths into a JSON message and the values they have, and `received a message containing {string}` looks for text in any message. The checks wait, 10 seconds unless `within {duration}` says otherwise, and look at every message the connection received. * **A connection the server closes** fails a check that is still waiting at once, saying why; `was closed with code {int}` checks the code it closed with. * **Secrets stay secret:** `${env:..}` values in the URL and the headers, such as a token, are masked in logs and failures. Add it to a project with `axx pack add websocket`. ## Steps [Section titled “Steps”](#steps) ### `websocket.connect` [Section titled “websocket.connect”](#websocketconnect) ```gherkin Given the {word} websocket with the following properties: | property | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Open a WebSocket connection under a name. It belongs to the scenario, which closes it when it ends. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | Property | Takes | Default | | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `url` | the address, `ws://` or `wss://` | *required* | | `header.` | a header of the opening request, such as `header.Authorization` | | | `subprotocol` | the subprotocols to offer, separated by commas | | | `asyncapi` | an AsyncAPI document, a file of the project or a URL, that the messages both ways follow: every message a scenario sends there, and every message a check finds, is checked against it (with the asyncapi pack) | | Values are expanded (`${env:..}`, `${sys:..}`); what `${env:..}` references expand to is masked everywhere. **Example:** ```gherkin Given the tracking websocket with the following properties: | url | ws://localhost:8400/portal/track/PX-LIV-5701/live | ``` *Since 0.1.5.* ### `websocket.send` [Section titled “websocket.send”](#websocketsend) ```gherkin When a message is sent to the {word} websocket: """ ... """ ``` Send the doc string as a text message on the connection. | Parameter | Takes | For example | | --------- | ------------------------ | -------------------- | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When a message is sent to the tracking websocket: """ {"follow": "PX-LIV-5701"} """ ``` *Since 0.1.5.* ### `websocket.send.file` [Section titled “websocket.send.file”](#websocketsendfile) ```gherkin When the {filepath} message is sent to the {word} websocket ``` Send the file of the project as a text message on the connection. | Parameter | Takes | For example | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | `seeds/parcels.yaml`, `kafka/scan-delivered.json` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | **Example:** ```gherkin When the tracking/follow-PX-LIV-5701.json message is sent to the tracking websocket ``` *Since 0.1.5.* ### `websocket.received` [Section titled “websocket.received”](#websocketreceived) ```gherkin Then [[within {duration} ]]the {word} websocket received a message where: | path | value | ``` The table may start with the row naming its columns, as above, or go straight to its rows. Check that the connection received a JSON message with those values. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | Each row is a path into the message (a field name, a dotted path or a JSONPath) and the value it has, compared as text: `null` for null and `undefined` for absent. **Variants**, the parts in `[[...]]` said or left out: * `the {word} websocket received a message where:` * `within {duration} the {word} websocket received a message where:` **Example:** ```gherkin Then within 20s the tracking websocket received a message where: | reference | PX-LIV-5701 | | status | OUT_FOR_DELIVERY | ``` *Since 0.1.5.* ### `websocket.received.text` [Section titled “websocket.received.text”](#websocketreceivedtext) ```gherkin Then [[within {duration} ]]the {word} websocket received a message containing {string} ``` Check that the connection received a message containing the text. The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | ---------------------------------------------------------- | ---------------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{string}` | text in double or single quotes, which the step leaves out | `"Get a quote"`, `'Express'` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} websocket received a message containing {string}` * `within {duration} the {word} websocket received a message containing {string}` **Example:** ```gherkin Then the tracking websocket received a message containing 'PX-LIV-5701' ``` *Since 0.1.5.* ### `websocket.closed` [Section titled “websocket.closed”](#websocketclosed) ```gherkin Then [[within {duration} ]]the {word} websocket was closed with code {int} ``` Check that the server closed the connection with that close code, such as 1000 (normal) or 1008 (a policy it breaks). The check waits for it: 10 seconds, or `within {duration}`. | Parameter | Takes | For example | | ------------ | -------------------------------------------- | -------------------- | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | `5s`, `2m` | | `{word}` | one word, with no spaces | `parcels`, `PX-4101` | | `{int}` | a whole number | `200`, `3` | **Variants**, the parts in `[[...]]` said or left out: * `the {word} websocket was closed with code {int}` * `within {duration} the {word} websocket was closed with code {int}` **Example:** ```gherkin Then the tracking websocket was closed with code 1008 ``` *Since 0.1.5.* # All steps > Every step Axx understands, one per line: pack, id and expression. Search it before writing a step. Every step Axx understands, one per line: `pack | id | expression`. `{name}` is a parameter; see [parameter types](/next/references/steps/#parameter-types). Search this file instead of inventing step text. ```text rest | rest.service | the {word} service with the following properties: [+table] rest | rest.openapi.levels | the OpenAPI validation levels are: [+table] rest | rest.openapi.levels | the OpenAPI validation levels on {service} are: [+table] rest | rest.request | a(n) {word} request to {word} rest | rest.request | a(n) {word} request to {word} on {service} rest | rest.request.ordered | a {ordinal} ordered {word} request to {word} rest | rest.request.ordered | a {ordinal} ordered {word} request to {word} on {service} rest | rest.request.header | the request header {word} is {string} rest | rest.request.header | the request header {word} is {string} for {ordinal} ordered request rest | rest.request.header.on | the request header {word} is {string} for request on {service} rest | rest.request.header.on | the request header {word} is {string} for {ordinal} ordered request on {service} rest | rest.request.headers | the request headers are: [+table] rest | rest.request.headers | the request headers for {ordinal} ordered request are: [+table] rest | rest.request.headers.on | the request headers for request on {service} are: [+table] rest | rest.request.headers.on | the request headers for {ordinal} ordered request on {service} are: [+table] rest | rest.request.payload.empty | a request payload using a(n) {mimeType} empty content template rest | rest.request.payload.empty | a request payload using a(n) {mimeType} empty content template for {ordinal} ordered request rest | rest.request.payload.empty.on | a request payload using a(n) {mimeType} empty content template for request on {service} rest | rest.request.payload.empty.on | a request payload using a(n) {mimeType} empty content template for {ordinal} ordered request on {service} rest | rest.request.payload.example | a request payload using a(n) {mimeType} content example rest | rest.request.payload.example | a request payload using a(n) {mimeType} content example named {string} rest | rest.request.payload.example | a request payload using a(n) {mimeType} content example for {ordinal} ordered request rest | rest.request.payload.example | a request payload using a(n) {mimeType} content example named {string} for {ordinal} ordered request rest | rest.request.payload.example.on | a request payload using a(n) {mimeType} content example for request on {service} rest | rest.request.payload.example.on | a request payload using a(n) {mimeType} content example named {string} for request on {service} rest | rest.request.payload.example.on | a request payload using a(n) {mimeType} content example for {ordinal} ordered request on {service} rest | rest.request.payload.example.on | a request payload using a(n) {mimeType} content example named {string} for {ordinal} ordered request on {service} rest | rest.request.payload.resource | a request payload using a(n) {mimeType} {filepath} resource rest | rest.request.payload.resource | a request payload using a(n) {mimeType} {filepath} resource for {ordinal} ordered request rest | rest.request.payload.resource.on | a request payload using a(n) {mimeType} {filepath} resource for request on {service} rest | rest.request.payload.resource.on | a request payload using a(n) {mimeType} {filepath} resource for {ordinal} ordered request on {service} rest | rest.request.property | the request payload property {word} is {string} rest | rest.request.property | the request payload property {word} is {string} for {ordinal} ordered request rest | rest.request.property.on | the request payload property {word} is {string} for request on {service} rest | rest.request.property.on | the request payload property {word} is {string} for {ordinal} ordered request on {service} rest | rest.request.properties | the request payload properties are: [+table] rest | rest.request.properties | the request payload properties for {ordinal} ordered request are: [+table] rest | rest.request.properties.on | the request payload properties for request on {service} are: [+table] rest | rest.request.properties.on | the request payload properties for {ordinal} ordered request on {service} are: [+table] rest | rest.request.property.null | the request payload property {word} is null rest | rest.request.property.null | the request payload property {word} is null for {ordinal} ordered request rest | rest.request.property.null.on | the request payload property {word} is null for request on {service} rest | rest.request.property.null.on | the request payload property {word} is null for {ordinal} ordered request on {service} rest | rest.execute | the request is executed rest | rest.execute | the {ordinal} ordered request is executed rest | rest.execute | the request is executed on {service} rest | rest.execute | the {ordinal} ordered request is executed on {service} rest | rest.response.status | the response status code is {int} rest | rest.response.status | the {ordinal} ordered response status code is {int} rest | rest.response.status | the response status code is {int} on {service} rest | rest.response.status | the {ordinal} ordered response status code is {int} on {service} rest | rest.response.body.contains | the response body contains {string} rest | rest.response.body.contains | the response body contains {string} for {ordinal} ordered response rest | rest.response.body.contains.on | the response body contains {string} for response on {service} rest | rest.response.body.contains.on | the response body contains {string} for {ordinal} ordered response on {service} rest | rest.response.header.is | the response header {word} is {string} rest | rest.response.header.is | the response header {word} is {string} for {ordinal} ordered response rest | rest.response.header.is.on | the response header {word} is {string} for response on {service} rest | rest.response.header.is.on | the response header {word} is {string} for {ordinal} ordered response on {service} rest | rest.response.header.matches | the response header {word} matches {pattern} rest | rest.response.header.matches | the response header {word} matches {pattern} for {ordinal} ordered response rest | rest.response.header.matches.on | the response header {word} matches {pattern} for response on {service} rest | rest.response.header.matches.on | the response header {word} matches {pattern} for {ordinal} ordered response on {service} rest | rest.response.header.missing | the response header {word} is missing rest | rest.response.header.missing | the response header {word} is missing for {ordinal} ordered response rest | rest.response.header.missing.on | the response header {word} is missing for response on {service} rest | rest.response.header.missing.on | the response header {word} is missing for {ordinal} ordered response on {service} rest | rest.response.headers.are | the response headers are: [+table] rest | rest.response.headers.are | the response headers for {ordinal} ordered response are: [+table] rest | rest.response.headers.are.on | the response headers for response on {service} are: [+table] rest | rest.response.headers.are.on | the response headers for {ordinal} ordered response on {service} are: [+table] rest | rest.response.headers.match | the response headers match: [+table] rest | rest.response.headers.match | the response headers for {ordinal} ordered response match: [+table] rest | rest.response.headers.match.on | the response headers for response on {service} match: [+table] rest | rest.response.headers.match.on | the response headers for {ordinal} ordered response on {service} match: [+table] rest | rest.response.headers.missing | the response headers are missing: [+table] rest | rest.response.headers.missing | the response headers for {ordinal} ordered response are missing: [+table] rest | rest.response.headers.missing.on | the response headers for response on {service} are missing: [+table] rest | rest.response.headers.missing.on | the response headers for {ordinal} ordered response on {service} are missing: [+table] rest | rest.response.property.is | the response payload property {word} is {string} rest | rest.response.property.is | the response payload property {word} is {string} for {ordinal} ordered response rest | rest.response.property.is.on | the response payload property {word} is {string} for response on {service} rest | rest.response.property.is.on | the response payload property {word} is {string} for {ordinal} ordered response on {service} rest | rest.response.property.null | the response payload property {word} is null rest | rest.response.property.null | the response payload property {word} is null for {ordinal} ordered response rest | rest.response.property.null.on | the response payload property {word} is null for response on {service} rest | rest.response.property.null.on | the response payload property {word} is null for {ordinal} ordered response on {service} rest | rest.response.property.undefined | the response payload property {word} is undefined rest | rest.response.property.undefined | the response payload property {word} is undefined for {ordinal} ordered response rest | rest.response.property.undefined.on | the response payload property {word} is undefined for response on {service} rest | rest.response.property.undefined.on | the response payload property {word} is undefined for {ordinal} ordered response on {service} rest | rest.response.property.matches | the response payload property {word} matches {pattern} rest | rest.response.property.matches | the response payload property {word} matches {pattern} for {ordinal} ordered response rest | rest.response.property.matches.on | the response payload property {word} matches {pattern} for response on {service} rest | rest.response.property.matches.on | the response payload property {word} matches {pattern} for {ordinal} ordered response on {service} rest | rest.response.properties.are | the response payload properties are: [+table] rest | rest.response.properties.are | the response payload properties for {ordinal} ordered response are: [+table] rest | rest.response.properties.are.on | the response payload properties for response on {service} are: [+table] rest | rest.response.properties.are.on | the response payload properties for {ordinal} ordered response on {service} are: [+table] rest | rest.response.properties.match | the response payload properties match: [+table] rest | rest.response.properties.match | the response payload properties for {ordinal} ordered response match: [+table] rest | rest.response.properties.match.on | the response payload properties for response on {service} match: [+table] rest | rest.response.properties.match.on | the response payload properties for {ordinal} ordered response on {service} match: [+table] rest | rest.token | the {word} token with the following properties: [+table] rest | rest.request.token | the request is authorized with the {word} token rest | rest.request.token | the request is authorized with the {word} token for {ordinal} ordered request rest | rest.request.token.on | the request is authorized with the {word} token for request on {service} rest | rest.request.token.on | the request is authorized with the {word} token for {ordinal} ordered request on {service} rest | rest.request.signed | the request is signed in the {word} header with the following properties: [+table] rest | rest.request.signed | the request is signed in the {word} header for {ordinal} ordered request with the following properties: [+table] rest | rest.request.signed.on | the request is signed in the {word} header for request on {service} with the following properties: [+table] rest | rest.request.signed.on | the request is signed in the {word} header for {ordinal} ordered request on {service} with the following properties: [+table] rest | rest.request.webhook | the request is signed as a standard webhook with the key {string} rest | rest.request.webhook | the request is signed as a standard webhook with the key {string} for {ordinal} ordered request rest | rest.request.webhook.on | the request is signed as a standard webhook with the key {string} for request on {service} rest | rest.request.webhook.on | the request is signed as a standard webhook with the key {string} for {ordinal} ordered request on {service} mock | mock.service | the mocked {word} service with the following properties: [+table] mock | mock.received | the mocked {word} request to {word} named {word} was received by {mockedService} mock | mock.received.path | the mocked {word} request to path {word} named {word} was received by {mockedService} mock | mock.openapi.levels | the OpenAPI validation levels for the mocked {mockedService} service are: [+table] mock | mock.count.exactly | the mocked request named {word} was received exactly {int} time(s) mock | mock.count.exactly | the mocked request named {word} on {mockedService} was received exactly {int} time(s) mock | mock.count.atLeast | the mocked request named {word} was received at least {int} time(s) mock | mock.count.atLeast | the mocked request named {word} on {mockedService} was received at least {int} time(s) mock | mock.count.atMost | the mocked request named {word} was received at most {int} time(s) mock | mock.count.atMost | the mocked request named {word} on {mockedService} was received at most {int} time(s) mock | mock.notReceived | the mocked {word} request to {word} named {word} was not received mock | mock.notReceived | the mocked {word} request to {word} named {word} on {mockedService} was not received mock | mock.notReceived.path | the mocked {word} request to path {word} named {word} was not received mock | mock.notReceived.path | the mocked {word} request to path {word} named {word} on {mockedService} was not received mock | mock.header.is | the header {word} for mocked request named {word} is {string} mock | mock.header.is | the header {word} for mocked request named {word} on {mockedService} is {string} mock | mock.headers.are | the headers for mocked request named {word} on {mockedService} are: [+table] mock | mock.header.matches | the header {word} for mocked request named {word} on {mockedService} matches {pattern} mock | mock.headers.match | the headers for mocked request named {word} on {mockedService} match: [+table] mock | mock.header.missing | the header {word} for mocked request named {word} on {mockedService} is missing mock | mock.headers.missing | the headers for mocked request named {word} on {mockedService} are missing: [+table] mock | mock.properties.are | the payload properties for mocked request named {word} are: [+table] mock | mock.properties.are | the payload properties for mocked request named {word} on {mockedService} are: [+table] mock | mock.query.are | the query parameters for mocked request named {word} are: [+table] mock | mock.query.are | the query parameters for mocked request named {word} on {mockedService} are: [+table] mock | mock.form.are | the form fields for mocked request named {word} are: [+table] mock | mock.form.are | the form fields for mocked request named {word} on {mockedService} are: [+table] mock | mock.query.none | none of the mocked {word} requests to path {word} have the query parameters: [+table] mock | mock.query.none | none of the mocked {word} requests to path {word} on {mockedService} have the query parameters: [+table] mock | mock.properties.none | none of the mocked {word} requests to path {word} have the payload properties: [+table] mock | mock.properties.none | none of the mocked {word} requests to path {word} on {mockedService} have the payload properties: [+table] mock | mock.form.none | none of the mocked {word} requests to path {word} have the form fields: [+table] mock | mock.form.none | none of the mocked {word} requests to path {word} on {mockedService} have the form fields: [+table] mock | mock.signed | the mocked request named {word} is signed in the {word} header with the following properties: [+table] mock | mock.signed | the mocked request named {word} on {mockedService} is signed in the {word} header with the following properties: [+table] mock | mock.webhook | the mocked request named {word} is signed as a standard webhook with the key {string} mock | mock.webhook | the mocked request named {word} on {mockedService} is signed as a standard webhook with the key {string} mock | mock.model.asked | the mocked {mockedService} model was asked about {string} mock | mock.model.asked | the mocked {mockedService} model was asked about {string} {int} time(s) mock | mock.model.contains | the mocked {mockedService} model's request about {string} contains {string} mock | mock.model.notContains | the mocked {mockedService} model's request about {string} does not contain {string} mock | mock.model.tool | the mocked {mockedService} model was offered the {word} tool in the request about {string} mock | mock.model.schema | the mocked {mockedService} model was asked for the {filepath} schema in the request about {string} mock | mock.mcp.called | the mocked {mockedService} mcp server's {word} tool was called with the following arguments: [+table] mock | mock.mcp.called | the mocked {mockedService} mcp server's {word} tool was called {int} time(s) with the following arguments: [+table] mock | mock.a2a.sent | the mocked {mockedService} a2a agent was sent a message containing {string} mock | mock.a2a.sent | the mocked {mockedService} a2a agent was sent a message containing {string} {int} time(s) grpc | grpc.service | the {word} grpc service with the following properties: [+table] grpc | grpc.call | the {word} method is called on the {word} grpc service grpc | grpc.call.fields | the {word} method is called on the {word} grpc service with the following fields: [+table] grpc | grpc.call.file | the {word} method is called on the {word} grpc service with the {filepath} message grpc | grpc.call.file | the {word} method is called on the {word} grpc service with the {filepath} message and the following fields: grpc | grpc.status | the {word} grpc service answered {word} grpc | grpc.status | within {duration} the {word} grpc service answered {word} grpc | grpc.status | the {word} grpc service answered {word} with a message containing {string} grpc | grpc.status | within {duration} the {word} grpc service answered {word} with a message containing {string} grpc | grpc.answer | the {word} grpc service's answer has the following fields: [+table] grpc | grpc.metadata | the {word} grpc service's answer has the following metadata: [+table] grpc | grpc.streamed | the {word} grpc service streamed a message where: [+table] grpc | grpc.streamed | within {duration} the {word} grpc service streamed a message where: [+table] jsonrpc | jsonrpc.service | the {word} jsonrpc service with the following properties: [+table] jsonrpc | jsonrpc.call | the {word} method is called on the {word} jsonrpc service jsonrpc | jsonrpc.call.params | the {word} method is called on the {word} jsonrpc service with the following params: [+table] jsonrpc | jsonrpc.call.doc | the {word} method is called on the {word} jsonrpc service with the params: [+docstring] jsonrpc | jsonrpc.result | the {word} jsonrpc service's result has the following properties: [+table] jsonrpc | jsonrpc.result.is | the {word} jsonrpc service's result is {string} jsonrpc | jsonrpc.error | the {word} jsonrpc service answered the error {int} jsonrpc | jsonrpc.error | the {word} jsonrpc service answered the error {int} with a message containing {string} jsonrpc | jsonrpc.error.properties | the {word} jsonrpc service's error has the following properties: [+table] jsonrpc | jsonrpc.openrpc.levels | the OpenRPC validation levels are: [+table] graphql | graphql.service | the {word} graphql service with the following properties: [+table] graphql | graphql.send.file | the {filepath} query/mutation is sent to the {word} graphql service graphql | graphql.send.file | the {filepath} query/mutation is sent to the {word} graphql service with the following variables: graphql | graphql.send.doc | a query/mutation is sent to the {word} graphql service: [+docstring] graphql | graphql.subscribe | the {filepath} subscription is started on the {word} graphql service graphql | graphql.subscribe | the {filepath} subscription is started on the {word} graphql service with the following variables: graphql | graphql.no.errors | the {word} graphql service answered without errors graphql | graphql.data | the {word} graphql service's data has the following properties: [+table] graphql | graphql.error | the {word} graphql service answered an error where: [+table] graphql | graphql.received | the {word} graphql service's subscription received a message where: [+table] graphql | graphql.received | within {duration} the {word} graphql service's subscription received a message where: [+table] graphql | graphql.levels | the GraphQL validation levels are: [+table] mcp | mcp.server | the {word} mcp server with the following properties: [+table] mcp | mcp.tool | the {word} mcp server has the {word} tool mcp | mcp.tool | the {word} mcp server has the {word} tool with the following properties: mcp | mcp.noTool | the {word} mcp server does not have the {word} tool mcp | mcp.call | the {word} tool is called on the {word} mcp server mcp | mcp.call | the {word} tool is called on the {word} mcp server with the following arguments: mcp | mcp.call.file | the {word} tool is called on the {word} mcp server with the {filepath} arguments mcp | mcp.notError | the {word} tool's result is not an error mcp | mcp.notError | the {word} tool's result on the {word} mcp server is not an error mcp | mcp.error | the {word} tool's result is an error mcp | mcp.error | the {word} tool's result on the {word} mcp server is an error mcp | mcp.contains | the {word} tool's result contains {string} mcp | mcp.contains | the {word} tool's result on the {word} mcp server contains {string} mcp | mcp.properties | the {word} tool's result has the following properties: [+table] mcp | mcp.properties | the {word} tool's result on the {word} mcp server has the following properties: [+table] mcp | mcp.refused | the {word} tool's call failed with the error code {int} mcp | mcp.refused | the {word} tool's call on the {word} mcp server failed with the error code {int} mcp | mcp.resource.read | the {word} resource is read from the {word} mcp server mcp | mcp.resource.contains | the {word} resource contains {string} mcp | mcp.resource.properties | the {word} resource has the following properties: [+table] mcp | mcp.prompt | the {word} prompt is requested from the {word} mcp server mcp | mcp.prompt | the {word} prompt is requested from the {word} mcp server with the following arguments: mcp | mcp.prompt.contains | the {word} prompt contains {string} mcp | mcp.levels | the MCP validation levels are: [+table] a2a | a2a.agent | the {word} a2a agent with the following properties: [+table] a2a | a2a.card | the {word} a2a agent's card has the following properties: [+table] a2a | a2a.skill | the {word} a2a agent has the {word} skill a2a | a2a.send | a message is sent to the {word} a2a agent: [+docstring] a2a | a2a.send.file | the {filepath} message is sent to the {word} a2a agent a2a | a2a.stream | a message is streamed to the {word} a2a agent: [+docstring] a2a | a2a.reply | a reply is sent to the {word} a2a agent: [+docstring] a2a | a2a.cancel | the {word} a2a agent is asked to cancel its task a2a | a2a.state | the {word} a2a agent's task is {word} a2a | a2a.state | within {duration} the {word} a2a agent's task is {word} a2a | a2a.answer | the {word} a2a agent's answer contains {string} a2a | a2a.artifact | the {word} a2a agent's task has an artifact where: [+table] a2a | a2a.update | the {word} a2a agent's stream received an update where: [+table] a2a | a2a.update | within {duration} the {word} a2a agent's stream received an update where: [+table] sql | sql.service | a(n) {word} database with the following properties: [+table] sql | sql.seed | a {filepath} db seed sql | sql.seed | a {filepath} db seed on {dbService} sql | sql.lock | the rows in the {word} table are locked where: [+table] sql | sql.lock | the rows in the {word} table on {dbService} are locked where: [+table] sql | sql.unlock | the row locks are released sql | sql.unlock | the row locks on {dbService} are released sql | sql.select | a selection of rows is retrieved from the {word} table where: [+table] sql | sql.select | a {ordinal} selection of rows is retrieved from the {word} table where: [+table] sql | sql.select | a selection of rows is retrieved from the {word} table on {dbService} where: [+table] sql | sql.select | a {ordinal} selection of rows is retrieved from the {word} table on {dbService} where: [+table] sql | sql.select.poll | within {duration} a selection of at least {int} row(s) is retrieved from the {word} table where: [+table] sql | sql.select.poll | within {duration} a {ordinal} selection of at least {int} row(s) is retrieved from the {word} table where: [+table] sql | sql.select.poll | within {duration} a selection of at least {int} row(s) is retrieved from the {word} table on {dbService} where: [+table] sql | sql.select.poll | within {duration} a {ordinal} selection of at least {int} row(s) is retrieved from the {word} table on {dbService} where: [+table] sql | sql.select.jsonb | a selection of rows is retrieved from the {word} table where the {word} jsonb column contains: [+table] sql | sql.select.jsonb | a {ordinal} selection of rows is retrieved from the {word} table where the {word} jsonb column contains: [+table] sql | sql.select.jsonb | a selection of rows is retrieved from the {word} table on {dbService} where the {word} jsonb column contains: [+table] sql | sql.select.jsonb | a {ordinal} selection of rows is retrieved from the {word} table on {dbService} where the {word} jsonb column contains: [+table] sql | sql.json.are | the {ordinal} row {word} property for the selection json properties are: [+table] sql | sql.json.are | the {ordinal} row {word} property for the {ordinal} selection json properties are: [+table] sql | sql.json.are | the {ordinal} row {word} property for the selection on {dbService} json properties are: [+table] sql | sql.json.are | the {ordinal} row {word} property for the {ordinal} selection on {dbService} json properties are: [+table] sql | sql.json.match | the {ordinal} row {word} property for the selection json properties match: [+table] sql | sql.json.match | the {ordinal} row {word} property for the {ordinal} selection json properties match: [+table] sql | sql.json.match | the {ordinal} row {word} property for the selection on {dbService} json properties match: [+table] sql | sql.json.match | the {ordinal} row {word} property for the {ordinal} selection on {dbService} json properties match: [+table] sql | sql.rows.eq | the selection has {int} row(s) sql | sql.rows.eq | the {ordinal} selection has {int} row(s) sql | sql.rows.eq | the selection on {dbService} has {int} row(s) sql | sql.rows.eq | the {ordinal} selection on {dbService} has {int} row(s) sql | sql.rows.gt | the selection has more than {int} row(s) sql | sql.rows.gt | the {ordinal} selection has more than {int} row(s) sql | sql.rows.gt | the selection on {dbService} has more than {int} row(s) sql | sql.rows.gt | the {ordinal} selection on {dbService} has more than {int} row(s) sql | sql.rows.lt | the selection has fewer than {int} row(s) sql | sql.rows.lt | the {ordinal} selection has fewer than {int} row(s) sql | sql.rows.lt | the selection on {dbService} has fewer than {int} row(s) sql | sql.rows.lt | the {ordinal} selection on {dbService} has fewer than {int} row(s) sql | sql.trigger.raise | a(n) before insert trigger on the {word} table will raise a(n) {sqlState} exception where: [+table] sql | sql.trigger.raise | a(n) {ordinal} ordered before insert trigger on the {word} table will raise a(n) {sqlState} exception where: [+table] sql | sql.trigger.raise | a(n) before insert trigger on the {word} table on {dbService} will raise a(n) {sqlState} exception where: [+table] sql | sql.trigger.raise | a(n) {ordinal} ordered before insert trigger on the {word} table on {dbService} will raise a(n) {sqlState} exception where: [+table] sql | sql.trigger.raise.times | a(n) before insert trigger on the {word} table will raise a(n) {sqlState} exception {int} time(s) where: [+table] sql | sql.trigger.raise.times | a(n) {ordinal} ordered before insert trigger on the {word} table will raise a(n) {sqlState} exception {int} time(s) where: [+table] sql | sql.trigger.raise.times | a(n) before insert trigger on the {word} table on {dbService} will raise a(n) {sqlState} exception {int} time(s) where: [+table] sql | sql.trigger.raise.times | a(n) {ordinal} ordered before insert trigger on the {word} table on {dbService} will raise a(n) {sqlState} exception {int} time(s) where: [+table] sql | sql.trigger.insertRaise | a(n) before insert trigger on the {word} table will insert and raise a(n) {sqlState} exception where: [+table] sql | sql.trigger.insertRaise | a(n) {ordinal} ordered before insert trigger on the {word} table will insert and raise a(n) {sqlState} exception where: [+table] sql | sql.trigger.insertRaise | a(n) before insert trigger on the {word} table on {dbService} will insert and raise a(n) {sqlState} exception where: [+table] sql | sql.trigger.insertRaise | a(n) {ordinal} ordered before insert trigger on the {word} table on {dbService} will insert and raise a(n) {sqlState} exception where: [+table] sql | sql.trigger.insertRaise.times | a(n) before insert trigger on the {word} table will insert and raise a(n) {sqlState} exception {int} time(s) where: [+table] sql | sql.trigger.insertRaise.times | a(n) {ordinal} ordered before insert trigger on the {word} table will insert and raise a(n) {sqlState} exception {int} time(s) where: [+table] sql | sql.trigger.insertRaise.times | a(n) before insert trigger on the {word} table on {dbService} will insert and raise a(n) {sqlState} exception {int} time(s) where: [+table] sql | sql.trigger.insertRaise.times | a(n) {ordinal} ordered before insert trigger on the {word} table on {dbService} will insert and raise a(n) {sqlState} exception {int} time(s) where: [+table] sql | sql.trigger.raised | the before insert trigger on the {word} table was raised {int} time(s) sql | sql.trigger.raised | the {ordinal} ordered before insert trigger on the {word} table was raised {int} time(s) sql | sql.trigger.raised | the before insert trigger on the {word} table on {dbService} was raised {int} time(s) sql | sql.trigger.raised | the {ordinal} ordered before insert trigger on the {word} table on {dbService} was raised {int} time(s) mongo | mongo.service | a(n) {word} mongo database with the following properties: [+table] mongo | mongo.seed | a {filepath} mongo db seed mongo | mongo.seed.named | a {filepath} MongoDB seed for {word} mongo | mongo.seed.named.alt | a {filepath} mongo db seed for {word} mongo | mongo.find | a selection of documents is retrieved from the {word} collection where: [+table] mongo | mongo.find | a {ordinal} selection of documents is retrieved from the {word} collection where: [+table] mongo | mongo.find | a selection of documents is retrieved from the {word} collection on {mongoService} where: [+table] mongo | mongo.find | a {ordinal} selection of documents is retrieved from the {word} collection on {mongoService} where: [+table] mongo | mongo.find.poll | within {duration} a selection of at least {int} document(s) is retrieved from the {word} collection where: [+table] mongo | mongo.find.poll | within {duration} a {ordinal} selection of at least {int} document(s) is retrieved from the {word} collection where: [+table] mongo | mongo.find.poll | within {duration} a selection of at least {int} document(s) is retrieved from the {word} collection on {mongoService} where: [+table] mongo | mongo.find.poll | within {duration} a {ordinal} selection of at least {int} document(s) is retrieved from the {word} collection on {mongoService} where: [+table] mongo | mongo.docs.eq | the selection has {int} document(s) mongo | mongo.docs.eq | the {ordinal} selection has {int} document(s) mongo | mongo.docs.eq | the selection on {mongoService} has {int} document(s) mongo | mongo.docs.eq | the {ordinal} selection on {mongoService} has {int} document(s) mongo | mongo.docs.gt | the selection has more than {int} document(s) mongo | mongo.docs.gt | the {ordinal} selection has more than {int} document(s) mongo | mongo.docs.gt | the selection on {mongoService} has more than {int} document(s) mongo | mongo.docs.gt | the {ordinal} selection on {mongoService} has more than {int} document(s) mongo | mongo.docs.lt | the selection has fewer than {int} document(s) mongo | mongo.docs.lt | the {ordinal} selection has fewer than {int} document(s) mongo | mongo.docs.lt | the selection on {mongoService} has fewer than {int} document(s) mongo | mongo.docs.lt | the {ordinal} selection on {mongoService} has fewer than {int} document(s) mongo | mongo.doc.are | the {ordinal} document for the selection properties are: [+table] mongo | mongo.doc.are | the {ordinal} document for the {ordinal} selection properties are: [+table] mongo | mongo.doc.are | the {ordinal} document for the selection on {mongoService} properties are: [+table] mongo | mongo.doc.are | the {ordinal} document for the {ordinal} selection on {mongoService} properties are: [+table] mongo | mongo.doc.match | the {ordinal} document for the selection properties match: [+table] mongo | mongo.doc.match | the {ordinal} document for the {ordinal} selection properties match: [+table] mongo | mongo.doc.match | the {ordinal} document for the selection on {mongoService} properties match: [+table] mongo | mongo.doc.match | the {ordinal} document for the {ordinal} selection on {mongoService} properties match: [+table] redis | redis.server | the {word} redis server with the following properties: [+table] redis | redis.seed | a {filepath} redis seed redis | redis.value | the {word} redis key has the value {string} redis | redis.value | within {duration} the {word} redis key has the value {string} redis | redis.properties | the {word} redis key has the following properties: [+table] redis | redis.properties | within {duration} the {word} redis key has the following properties: [+table] redis | redis.absent | the {word} redis key does not exist redis | redis.absent | within {duration} the {word} redis key does not exist kafka | kafka.service | the {word} kafka service with the following properties: [+table] kafka | kafka.client | the {word} kafka topic client kafka | kafka.client.props | a(n) {word} kafka topic client with the following properties: [+table] kafka | kafka.client.props | a(n) {word} kafka topic client on the {word} kafka service with the following properties: [+table] kafka | kafka.event | a(n) {word} kafka event kafka | kafka.event | a(n) {ordinal} ordered {word} kafka event kafka | kafka.event | a(n) {word} kafka event on {word} kafka service kafka | kafka.event | a(n) {ordinal} ordered {word} kafka event on {word} kafka service kafka | kafka.event.key | the {word} kafka event key is {word} kafka | kafka.event.key | the {ordinal} ordered {word} kafka event key is {word} kafka | kafka.event.key | the {word} kafka event key is {word} on the {word} kafka service kafka | kafka.event.key | the {ordinal} ordered {word} kafka event key is {word} on the {word} kafka service kafka | kafka.event.headers | the {word} kafka event headers are: [+table] kafka | kafka.event.headers | the {ordinal} ordered {word} kafka event headers are: [+table] kafka | kafka.event.headers | the {word} kafka event headers on the {word} kafka service are: [+table] kafka | kafka.event.headers | the {ordinal} ordered {word} kafka event headers on the {word} kafka service are: [+table] kafka | kafka.event.payload.resource | the {word} kafka event payload is a(n) {filepath} resource kafka | kafka.event.payload.resource | the {ordinal} ordered {word} kafka event payload is a(n) {filepath} resource kafka | kafka.event.payload.resource | the {word} kafka event payload is a(n) {filepath} resource on the {word} kafka service kafka | kafka.event.payload.resource | the {ordinal} ordered {word} kafka event payload is a(n) {filepath} resource on the {word} kafka service kafka | kafka.event.properties.first | the kafka event payload properties are: [+table] kafka | kafka.event.properties.first | the {ordinal} ordered kafka event payload properties are: [+table] kafka | kafka.event.properties.first | the kafka event payload properties on the {word} kafka service are: [+table] kafka | kafka.event.properties.first | the {ordinal} ordered kafka event payload properties on the {word} kafka service are: [+table] kafka | kafka.event.properties | the {word} kafka event payload properties are: [+table] kafka | kafka.event.properties | the {word} kafka event payload properties on the {word} kafka service are: [+table] kafka | kafka.event.properties.ordinal | the {ordinal} ordered {word} kafka event payload properties are: [+table] kafka | kafka.event.properties.ordinal | the {ordinal} ordered {word} kafka event payload properties on the {word} kafka service are: [+table] kafka | kafka.event.property.null | the {word} kafka event payload property {word} is null kafka | kafka.event.property.null | the {ordinal} ordered {word} kafka event payload property {word} is null kafka | kafka.event.property.null | the {word} kafka event payload property {word} is null on the {word} kafka service kafka | kafka.event.property.null | the {ordinal} ordered {word} kafka event payload property {word} is null on the {word} kafka service kafka | kafka.event.publish.schema | the {word} kafka event is published using schema {filepath} kafka | kafka.event.publish.schema | the {ordinal} ordered {word} kafka event is published using schema {filepath} kafka | kafka.event.publish.schema | the {word} kafka event is published using schema {filepath} on the {word} kafka service kafka | kafka.event.publish.schema | the {ordinal} ordered {word} kafka event is published using schema {filepath} on the {word} kafka service kafka | kafka.event.publish | the {word} kafka event is published kafka | kafka.event.publish | the {ordinal} ordered {word} kafka event is published kafka | kafka.event.publish | the {word} kafka event is published on the {word} kafka service kafka | kafka.event.publish | the {ordinal} ordered {word} kafka event is published on the {word} kafka service kafka | kafka.consumed.key | the {word} kafka event named {word} key is {word} kafka | kafka.consumed.key | the {word} kafka event named {word} key is {word} on the {word} kafka service kafka | kafka.consumed.properties | the {word} kafka event named {word} payload properties are: [+table] kafka | kafka.consumed.properties | the {word} kafka event named {word} payload properties on the {word} kafka service are: [+table] kafka | kafka.consumed.headers | the {word} kafka event named {word} headers are: [+table] kafka | kafka.consumed.headers | the {word} kafka event named {word} headers on the {word} kafka service are: [+table] kafka | kafka.consumed.headers.match | the {word} kafka event named {word} headers match: [+table] kafka | kafka.consumed.headers.match.service | the {word} kafka event named {word} headers on the {word} kafka service match: [+table] amqp | amqp.broker | the {word} amqp broker with the following properties: [+table] amqp | amqp.queue.send | a message is sent to the {word} amqp queue: [+docstring] amqp | amqp.queue.send.file | the {filepath} message is sent to the {word} amqp queue amqp | amqp.queue.send.file | the {filepath} message is sent to the {word} amqp queue with the following properties: amqp | amqp.queue.received | the {word} amqp queue has a message where: [+table] amqp | amqp.queue.received | within {duration} the {word} amqp queue has a message where: [+table] amqp | amqp.exchange.send | a message is published to the {word} amqp exchange: [+docstring] amqp | amqp.exchange.send | a message is published to the {word} amqp exchange with the routing key {string}: [+docstring] amqp | amqp.exchange.send.file | the {filepath} message is published to the {word} amqp exchange amqp | amqp.exchange.send.file | the {filepath} message is published to the {word} amqp exchange with the following properties: amqp | amqp.exchange.received | the {word} amqp exchange has a message where: [+table] amqp | amqp.exchange.received | within {duration} the {word} amqp exchange has a message where: [+table] mqtt | mqtt.broker | the {word} mqtt broker with the following properties: [+table] mqtt | mqtt.send | a message is published to the {word} mqtt topic: [+docstring] mqtt | mqtt.send.file | the {filepath} message is published to the {word} mqtt topic mqtt | mqtt.send.file | the {filepath} message is published to the {word} mqtt topic with the following properties: mqtt | mqtt.received | the {word} mqtt topic has a message where: [+table] mqtt | mqtt.received | within {duration} the {word} mqtt topic has a message where: [+table] nats | nats.server | the {word} nats server with the following properties: [+table] nats | nats.subject.send | a message is published to the {word} nats subject: [+docstring] nats | nats.subject.send.file | the {filepath} message is published to the {word} nats subject nats | nats.subject.send.file | the {filepath} message is published to the {word} nats subject with the following headers: nats | nats.subject.received | the {word} nats subject has a message where: [+table] nats | nats.subject.received | within {duration} the {word} nats subject has a message where: [+table] nats | nats.stream.received | the {word} nats stream has a message where: [+table] nats | nats.stream.received | within {duration} the {word} nats stream has a message where: [+table] logs | logs.log | the {word} log with the following properties: [+table] logs | logs.entry | the {word} log has an entry matching {string} logs | logs.entry | within {duration} the {word} log has an entry matching {string} logs | logs.entries | the {word} log has entries matching: [+table] logs | logs.entries | within {duration} the {word} log has entries matching: [+table] logs | logs.count | the {word} log has {int} entry/entries matching {string} logs | logs.count | within {duration} the {word} log has {int} entry/entries matching {string} logs | logs.across | the logs have entries matching: [+table] logs | logs.across | within {duration} the logs have entries matching: [+table] files | files.folder | the {word} folder with the following properties: [+table] files | files.copy | the {path} file in the {word} folder is a copy of the {filepath} file files | files.content | the {path} file in the {word} folder has the content: [+docstring] files | files.emptied | the {word} folder is emptied files | files.has | the {word} folder has a(n) file named {path} files | files.has | within {duration} the {word} folder has a(n) file named {path} files | files.identical | the {path} file in the {word} folder is identical to the {filepath} file files | files.identical | within {duration} the {path} file in the {word} folder is identical to the {filepath} file files | files.properties | the {path} file in the {word} folder has the following properties: [+table] files | files.properties | within {duration} the {path} file in the {word} folder has the following properties: [+table] files | files.contains | the {path} file in the {word} folder contains {string} files | files.contains | within {duration} the {path} file in the {word} folder contains {string} files | files.row | the {path} file in the {word} folder has a row where: [+table] files | files.row | within {duration} the {path} file in the {word} folder has a row where: [+table] files | files.absent | the {word} folder has no file named {path} files | files.absent | within {duration} the {word} folder has no file named {path} files | files.empty | the {word} folder is empty files | files.empty | within {duration} the {word} folder is empty files | files.screenshot | the {path} file in the {word} folder looks like the {string} screenshot files | files.screenshot | within {duration} the {path} file in the {word} folder looks like the {string} screenshot mail | mail.mailbox | the {word} mailbox with the following properties: [+table] mail | mail.received | the {word} mailbox has an email where: [+table] mail | mail.received | within {duration} the {word} mailbox has an email where: [+table] cli | cli.command | the {word} command with the following properties: [+table] cli | cli.run | the {word} command is run cli | cli.run | the {word} command is run with {string} cli | cli.run.input | the {word} command is run with the input: [+docstring] cli | cli.run.input | the {word} command is run with {string} and the input: [+docstring] cli | cli.exit | the {word} command's exit code is {int} cli | cli.output.contains | the {word} command's output contains {string} cli | cli.output.contains | the {word} command's error output contains {string} cli | cli.output.line | the {word} command's output has a line matching {string} cli | cli.output.line | the {word} command's error output has a line matching {string} cli | cli.output.is | the {word} command's output is: [+docstring] cli | cli.output.is | the {word} command's error output is: [+docstring] cli | cli.output.empty | the {word} command's output is empty cli | cli.output.empty | the {word} command's error output is empty cli | cli.output.properties | the {word} command's output has the following properties: [+table] cli | cli.output.identical | the {word} command's output is identical to the {filepath} file websocket | websocket.connect | the {word} websocket with the following properties: [+table] websocket | websocket.send | a message is sent to the {word} websocket: [+docstring] websocket | websocket.send.file | the {filepath} message is sent to the {word} websocket websocket | websocket.received | the {word} websocket received a message where: [+table] websocket | websocket.received | within {duration} the {word} websocket received a message where: [+table] websocket | websocket.received.text | the {word} websocket received a message containing {string} websocket | websocket.received.text | within {duration} the {word} websocket received a message containing {string} websocket | websocket.closed | the {word} websocket was closed with code {int} websocket | websocket.closed | within {duration} the {word} websocket was closed with code {int} sse | sse.stream | the {word} event stream with the following properties: [+table] sse | sse.event | the {word} event stream has an event where: [+table] sse | sse.event | within {duration} the {word} event stream has an event where: [+table] sse | sse.event.text | the {word} event stream has an event containing {string} sse | sse.event.text | within {duration} the {word} event stream has an event containing {string} asyncapi | asyncapi.levels | the AsyncAPI validation levels are: [+table] web-core | web-core.app | the {word} web app with the following properties: [+table] web-core | web-core.open | the {string} page is opened web-core | web-core.open | the {string} page of the {word} web app is opened web-core | web-core.fill | the {string} field is filled with {string} web-core | web-core.select | {string} is chosen in the {string} field web-core | web-core.option | the {string} option is chosen web-core | web-core.check | the {string} checkbox is checked web-core | web-core.uncheck | the {string} checkbox is unchecked web-core | web-core.upload | the {filepath} file is uploaded in the {string} field web-core | web-core.button | the {string} button is clicked web-core | web-core.link | the {string} link is clicked web-core | web-core.tab | the {string} tab is clicked web-core | web-core.menuitem | the {string} menu item is clicked web-core | web-core.element.click | the {string} element is clicked web-core | web-core.key | the {word} key is pressed web-core | web-core.key | the {word} key is pressed in the {string} field web-core | web-core.hover | the pointer is moved over {string} web-core | web-core.reload | the page is reloaded web-core | web-core.back | the browser's back button is clicked web-core | web-core.closetab | the browser tab is closed web-core | web-core.accept | the dialog is accepted web-core | web-core.dismiss | the dialog is dismissed web-core | web-core.answer | the dialog is answered with {string} web-core | web-core.dialog | the dialog shows {string} web-core | web-core.dialog | within {duration} the dialog shows {string} web-core | web-core.shows | the page shows {string} web-core | web-core.shows | within {duration} the page shows {string} web-core | web-core.hides | the page does not show {string} web-core | web-core.hides | within {duration} the page does not show {string} web-core | web-core.url | the {string} page is shown web-core | web-core.url | within {duration} the {string} page is shown web-core | web-core.value | the {string} field has the value {string} web-core | web-core.value | within {duration} the {string} field has the value {string} web-core | web-core.disabled | the {string} button is disabled web-core | web-core.disabled | within {duration} the {string} button is disabled web-core | web-core.enabled | the {string} button is enabled web-core | web-core.enabled | within {duration} the {string} button is enabled web-core | web-core.row | the page shows a table row where: [+table] web-core | web-core.row | within {duration} the page shows a table row where: [+table] web-core | web-core.downloaded | the {string} file is downloaded web-core | web-core.downloaded | within {duration} the {string} file is downloaded web-core | web-core.download.contains | the downloaded {string} file contains {string} web-core | web-core.download.row | the downloaded {string} file has a row where: [+table] web-core | web-core.download.identical | the downloaded {string} file is identical to the {filepath} file web-core | web-core.element.shown | the {string} {element} is shown web-core | web-core.element.shown | within {duration} the {string} {element} is shown web-core | web-core.element.hidden | the {string} {element} is not shown web-core | web-core.element.hidden | within {duration} the {string} {element} is not shown web-core | web-core.element.text | the {string} {element} shows {string} web-core | web-core.element.text | within {duration} the {string} {element} shows {string} web-core | web-core.element.attribute | the {string} {element} has the {word} attribute {string} web-core | web-core.element.attribute | within {duration} the {string} {element} has the {word} attribute {string} web-core | web-core.element.focus | the {string} {element} has the focus web-core | web-core.element.focus | within {duration} the {string} {element} has the focus web-core | web-core.checkbox.ticked | the {string} checkbox is ticked web-core | web-core.checkbox.ticked | within {duration} the {string} checkbox is ticked web-core | web-core.checkbox.unticked | the {string} checkbox is not ticked web-core | web-core.checkbox.unticked | within {duration} the {string} checkbox is not ticked web-core | web-core.option.selected | the {string} option is selected web-core | web-core.option.selected | within {duration} the {string} option is selected web-core | web-core.option.unselected | the {string} option is not selected web-core | web-core.option.unselected | within {duration} the {string} option is not selected web-core | web-core.field.disabled | the {string} field is disabled web-core | web-core.field.disabled | within {duration} the {string} field is disabled web-core | web-core.field.enabled | the {string} field is enabled web-core | web-core.field.enabled | within {duration} the {string} field is enabled web-core | web-core.element.count | the page shows {int} {string} {element} web-core | web-core.element.count | within {duration} the page shows {int} {string} {element} web-core | web-core.row.count | the page shows {int} table row(s) where: [+table] web-core | web-core.row.count | within {duration} the page shows {int} table row(s) where: [+table] web-core | web-core.title | the page title is {string} web-core | web-core.title | within {duration} the page title is {string} web-core | web-core.errors.none | the page has no script errors web-core | web-core.request.sent | the browser sent a {word} request to {string} web-core | web-core.request.sent | within {duration} the browser sent a {word} request to {string} web-core | web-core.element.dblclick | the {string} {element} is double-clicked web-core | web-core.element.rightclick | the {string} {element} is right-clicked web-core | web-core.element.tap | the {string} {element} is tapped web-core | web-core.element.scroll | the {string} {element} is scrolled into view web-core | web-core.element.drag | the {string} {element} is dragged onto the {string} {element} web-core | web-core.scroll.bottom | the page is scrolled to the bottom web-core | web-core.scroll.top | the page is scrolled to the top web-core | web-core.select.many | the following options are chosen in the {string} field: [+table] web-core | web-core.clock.set | the browser's clock is set to {string} web-core | web-core.clock.forward | the browser's clock is moved forward by {duration} web-core | web-core.connection.offline | the browser is offline web-core | web-core.connection.online | the browser is online web-screenshots | web-screenshots.page | the page looks like the {string} screenshot web-screenshots | web-screenshots.page | within {duration} the page looks like the {string} screenshot web-screenshots | web-screenshots.page.masked | the page looks like the {string} screenshot, apart from: [+table] web-screenshots | web-screenshots.page.masked | within {duration} the page looks like the {string} screenshot, apart from: [+table] web-screenshots | web-screenshots.element | the {string} {element} looks like the {string} screenshot web-screenshots | web-screenshots.element | within {duration} the {string} {element} looks like the {string} screenshot web-a11y | web-a11y.page | the page has no accessibility violations web-a11y | web-a11y.element | the {string} {element} has no accessibility violations web-a11y | web-a11y.structure | the page's accessible structure is: [+docstring] web-a11y | web-a11y.structure | within {duration} the page's accessible structure is: [+docstring] web-a11y | web-a11y.element.structure | the {string} {element}'s accessible structure is: [+docstring] web-a11y | web-a11y.element.structure | within {duration} the {string} {element}'s accessible structure is: [+docstring] web-network | web-network.fail | the page's requests to {string} fail web-network | web-network.status | the page's requests to {string} answer with status {int} web-network | web-network.file | the page's requests to {string} answer with the {filepath} file web-network | web-network.slow | the page's requests to {string} take {duration} web-network | web-network.connection.slow | the browser's connection is slow web-network | web-network.recording | the page's requests are answered from the {filepath} recording web-network | web-network.ws.sent | the page sent a websocket message containing {string} web-network | web-network.ws.sent | within {duration} the page sent a websocket message containing {string} web-network | web-network.ws.received | the page received a websocket message containing {string} web-network | web-network.ws.received | within {duration} the page received a websocket message containing {string} web-lighthouse | web-lighthouse.scores | the {string} page scores at least: [+table] web-lighthouse | web-lighthouse.loads | the {string} page loads within: [+table] app-core | app-core.launch | the {word} app is launched app-core | app-core.restart | the {word} app is restarted app-core | app-core.tapped | the {string} {control} is tapped in the {word} app app-core | app-core.clicked | the {string} {control} is clicked in the {word} app app-core | app-core.fill | the {string} field in the {word} app is filled with {string} app-core | app-core.scroll | the {string} {control} is scrolled into view in the {word} app app-core | app-core.shows | the {word} app shows {string} app-core | app-core.shows | within {duration} the {word} app shows {string} app-core | app-core.hides | the {word} app does not show {string} app-core | app-core.hides | within {duration} the {word} app does not show {string} app-core | app-core.control.shown | the {string} {control} is shown in the {word} app app-core | app-core.control.shown | within {duration} the {string} {control} is shown in the {word} app app-core | app-core.control.enabled | the {string} {control} is enabled in the {word} app app-core | app-core.control.enabled | within {duration} the {string} {control} is enabled in the {word} app app-core | app-core.control.disabled | the {string} {control} is disabled in the {word} app app-core | app-core.control.disabled | within {duration} the {string} {control} is disabled in the {word} app app-core | app-core.field.value | the {string} field in the {word} app has the value {string} app-core | app-core.field.value | within {duration} the {string} field in the {word} app has the value {string} app-core | app-core.screenshot | the {word} app looks like the {string} screenshot app-core | app-core.screenshot | within {duration} the {word} app looks like the {string} screenshot mobile-core | mobile-core.link | the {word} app is opened with the {string} link mobile-core | mobile-core.background | the {word} app is sent to the background mobile-core | mobile-core.foreground | the {word} app is brought back mobile-core | mobile-core.swipe | the {word} app is swiped {direction} mobile-core | mobile-core.dialog.accepted | the {word} app's dialog is accepted mobile-core | mobile-core.dialog.dismissed | the {word} app's dialog is dismissed mobile-core | mobile-core.dialog.shows | the {word} app's dialog shows {string} mobile-core | mobile-core.dialog.shows | within {duration} the {word} app's dialog shows {string} mobile-core | mobile-core.camera-sees | the {word} app's camera sees {filepath} mobile-core | mobile-core.phone-to-camera | the {word} app's phone moves to the scene's {string} camera mobile-core | mobile-core.phone-path | the {word} app's phone {travel} the scene's {string} path mobile-core | mobile-core.phone-path | the {word} app's phone {travel} the scene's {string} path at {speed} mobile-core | mobile-core.phone-move | the {word} app's phone moves {float} m {direction} mobile-core | mobile-core.phone-turn | the {word} app's phone turns {int} degrees {direction} mobile-core | mobile-core.camera-tap | the {string} in the {word} app's camera view is tapped mobile-core | mobile-core.notification | the {word} app shows a notification {string} mobile-core | mobile-core.notification | within {duration} the {word} app shows a notification {string} mobile-core | mobile-core.notification-action | the {string} action of the {string} notification is tapped in the {word} app mobile-core | mobile-core.background-task | the {string} background task is run in the {word} app mobile-core | mobile-core.travel-through | the {word} app's phone {travel} at {speed} through: [+table] mobile-core | mobile-core.travel-route | the {word} app's phone {travel} the {filepath} route mobile-core | mobile-core.travel-route | the {word} app's phone {travel} the {filepath} route at {speed} mobile-core | mobile-core.route-end | the {word} app's phone reaches the end of its route mobile-core | mobile-core.route-end | within {duration} the {word} app's phone reaches the end of its route mobile-android | mobile-android.app | the {word} android app with the following properties: [+table] mobile-android | mobile-android.back | the {word} app's back button is pressed mobile-ios | mobile-ios.app | the {word} ios app with the following properties: [+table] desktop-core | desktop-core.key | the {word} key is pressed in the {word} app desktop-core | desktop-core.click-at | the {string} {control} in the {word} app is clicked at {int}, {int} desktop-core | desktop-core.click-at-anchor | the {string} {control} in the {word} app is clicked at {int}, {int} from its {anchor} desktop-core | desktop-core.drag | the pointer is dragged from {int}, {int} to {int}, {int} on the {string} {control} in the {word} app desktop-core | desktop-core.drag-from-anchor | the pointer is dragged from {int}, {int} to {int}, {int} from the {anchor} of the {string} {control} in the {word} app desktop-macos | desktop-macos.app | the {word} macos app with the following properties: [+table] desktop-windows | desktop-windows.app | the {word} windows app with the following properties: [+table] desktop-linux | desktop-linux.app | the {word} linux app with the following properties: [+table] models-core | models-core.microphone | the {word} app's microphone hears the {string} file models-core | models-core.speaker | the {word} app's speaker plays the {string} file models-core | models-core.speaker | within {duration} the {word} app's speaker plays the {string} file aws-core | aws-core.account | the {word} aws account with the following properties: [+table] aws-s3 | aws-s3.upload | the {filepath} file is uploaded to the {word} s3 bucket aws-s3 | aws-s3.upload | the {filepath} file is uploaded to the {word} s3 bucket as {path} aws-s3 | aws-s3.has | the {word} s3 bucket has a(n) object named {path} aws-s3 | aws-s3.has | within {duration} the {word} s3 bucket has a(n) object named {path} aws-s3 | aws-s3.identical | the {path} object in the {word} s3 bucket is identical to the {filepath} file aws-s3 | aws-s3.identical | within {duration} the {path} object in the {word} s3 bucket is identical to the {filepath} file aws-s3 | aws-s3.properties | the {path} object in the {word} s3 bucket has the following properties: [+table] aws-s3 | aws-s3.properties | within {duration} the {path} object in the {word} s3 bucket has the following properties: [+table] aws-s3 | aws-s3.contains | the {path} object in the {word} s3 bucket contains {string} aws-s3 | aws-s3.contains | within {duration} the {path} object in the {word} s3 bucket contains {string} aws-s3 | aws-s3.row | the {path} object in the {word} s3 bucket has a row where: [+table] aws-s3 | aws-s3.row | within {duration} the {path} object in the {word} s3 bucket has a row where: [+table] aws-sqs | aws-sqs.send | a message is sent to the {word} sqs queue: [+docstring] aws-sqs | aws-sqs.send.file | the {filepath} message is sent to the {word} sqs queue aws-sqs | aws-sqs.send.file | the {filepath} message is sent to the {word} sqs queue with the following attributes: aws-sqs | aws-sqs.received | the {word} sqs queue has a message where: [+table] aws-sqs | aws-sqs.received | within {duration} the {word} sqs queue has a message where: [+table] aws-sns | aws-sns.send | a message is published to the {word} sns topic: [+docstring] aws-sns | aws-sns.send.file | the {filepath} message is published to the {word} sns topic aws-sns | aws-sns.send.file | the {filepath} message is published to the {word} sns topic with the following attributes: aws-sns | aws-sns.received | the {word} sns topic has a message where: [+table] aws-sns | aws-sns.received | within {duration} the {word} sns topic has a message where: [+table] aws-eventbridge | aws-eventbridge.put | a(n) {string} event from {word} is put on the {word} eventbridge bus: [+docstring] aws-eventbridge | aws-eventbridge.received | the {word} eventbridge bus has an event where: [+table] aws-eventbridge | aws-eventbridge.received | within {duration} the {word} eventbridge bus has an event where: [+table] aws-dynamodb | aws-dynamodb.seed | a {filepath} dynamodb seed aws-dynamodb | aws-dynamodb.item | the {word} dynamodb table has an item where: [+table] aws-dynamodb | aws-dynamodb.item | within {duration} the {word} dynamodb table has an item where: [+table] aws-dynamodb | aws-dynamodb.items | the {word} dynamodb table has {int} item(s) where: [+table] aws-dynamodb | aws-dynamodb.items | within {duration} the {word} dynamodb table has {int} item(s) where: [+table] gcp-core | gcp-core.project | the {word} gcp project with the following properties: [+table] gcp-storage | gcp-storage.upload | the {filepath} file is uploaded to the {word} gcs bucket gcp-storage | gcp-storage.upload | the {filepath} file is uploaded to the {word} gcs bucket as {path} gcp-storage | gcp-storage.has | the {word} gcs bucket has a(n) object named {path} gcp-storage | gcp-storage.has | within {duration} the {word} gcs bucket has a(n) object named {path} gcp-storage | gcp-storage.identical | the {path} object in the {word} gcs bucket is identical to the {filepath} file gcp-storage | gcp-storage.identical | within {duration} the {path} object in the {word} gcs bucket is identical to the {filepath} file gcp-storage | gcp-storage.properties | the {path} object in the {word} gcs bucket has the following properties: [+table] gcp-storage | gcp-storage.properties | within {duration} the {path} object in the {word} gcs bucket has the following properties: [+table] gcp-storage | gcp-storage.contains | the {path} object in the {word} gcs bucket contains {string} gcp-storage | gcp-storage.contains | within {duration} the {path} object in the {word} gcs bucket contains {string} gcp-storage | gcp-storage.row | the {path} object in the {word} gcs bucket has a row where: [+table] gcp-storage | gcp-storage.row | within {duration} the {path} object in the {word} gcs bucket has a row where: [+table] gcp-pubsub | gcp-pubsub.send | a message is published to the {word} pubsub topic: [+docstring] gcp-pubsub | gcp-pubsub.send.file | the {filepath} message is published to the {word} pubsub topic gcp-pubsub | gcp-pubsub.send.file | the {filepath} message is published to the {word} pubsub topic with the following attributes: gcp-pubsub | gcp-pubsub.received | the {word} pubsub topic has a message where: [+table] gcp-pubsub | gcp-pubsub.received | within {duration} the {word} pubsub topic has a message where: [+table] gcp-bigquery | gcp-bigquery.seed | a {filepath} bigquery seed gcp-bigquery | gcp-bigquery.row | the {word} bigquery table has a row where: [+table] gcp-bigquery | gcp-bigquery.row | within {duration} the {word} bigquery table has a row where: [+table] gcp-bigquery | gcp-bigquery.rows | the {word} bigquery table has {int} row(s) where: [+table] gcp-bigquery | gcp-bigquery.rows | within {duration} the {word} bigquery table has {int} row(s) where: [+table] gcp-firestore | gcp-firestore.seed | a {filepath} firestore seed gcp-firestore | gcp-firestore.document | the {word} firestore document has the following properties: [+table] gcp-firestore | gcp-firestore.document | within {duration} the {word} firestore document has the following properties: [+table] gcp-firestore | gcp-firestore.collection | the {word} firestore collection has a document where: [+table] gcp-firestore | gcp-firestore.collection | within {duration} the {word} firestore collection has a document where: [+table] azure-blob | azure-blob.account | the {word} azure storage account with the following properties: [+table] azure-blob | azure-blob.upload | the {filepath} file is uploaded to the {word} blob container azure-blob | azure-blob.upload | the {filepath} file is uploaded to the {word} blob container as {path} azure-blob | azure-blob.has | the {word} blob container has a(n) blob named {path} azure-blob | azure-blob.has | within {duration} the {word} blob container has a(n) blob named {path} azure-blob | azure-blob.identical | the {path} blob in the {word} blob container is identical to the {filepath} file azure-blob | azure-blob.identical | within {duration} the {path} blob in the {word} blob container is identical to the {filepath} file azure-blob | azure-blob.properties | the {path} blob in the {word} blob container has the following properties: [+table] azure-blob | azure-blob.properties | within {duration} the {path} blob in the {word} blob container has the following properties: [+table] azure-blob | azure-blob.contains | the {path} blob in the {word} blob container contains {string} azure-blob | azure-blob.contains | within {duration} the {path} blob in the {word} blob container contains {string} azure-blob | azure-blob.row | the {path} blob in the {word} blob container has a row where: [+table] azure-blob | azure-blob.row | within {duration} the {path} blob in the {word} blob container has a row where: [+table] azure-servicebus | azure-servicebus.namespace | the {word} service bus namespace with the following properties: [+table] azure-servicebus | azure-servicebus.send | a message is sent to the {word} service bus queue:/topic: [+docstring] azure-servicebus | azure-servicebus.send.file | the {filepath} message is sent to the {word} service bus queue/topic azure-servicebus | azure-servicebus.send.file | the {filepath} message is sent to the {word} service bus queue/topic with the following properties: azure-servicebus | azure-servicebus.received | the {word} service bus queue/topic has a message where: [+table] azure-servicebus | azure-servicebus.received | within {duration} the {word} service bus queue/topic has a message where: [+table] ``` # How steps read > The grammar every step of Axx's packs follows (optional parts, services, ordinals, arguments) and the parameter types. Every step of Axx’s packs follows the same grammar. The [pack pages](/next/references/packs/) list each step with its parameters, the rows its table takes, its variants and an example. ## Optional parts [Section titled “Optional parts”](#optional-parts) An expression writes optional parts as `[[ ... ]]`. A step matches with or without each of them. `the[[ {ordinal} ordered]] response status code is {int}[[ on {service}]]` matches: * `the response status code is 200` * `the 2nd ordered response status code is 201` * `the response status code is 200 on parcels` * `the 2nd ordered response status code is 201 on parcels` `axx steps show` prints every variant of a step, given its id, its expression or a line that uses it. ## Services [Section titled “Services”](#services) A step without a service name uses the default service: the first one of its kind registered in the scenario. The named form (`on parcels`, `on parcels-db`, `on the events kafka service`) picks another. ## Ordinals [Section titled “Ordinals”](#ordinals) `1st`, `2nd`, `3rd`, `4th`, … count requests, selections and events within a scenario, starting at 1. A step without an ordinal uses the first. ## Arguments [Section titled “Arguments”](#arguments) * `{string}` takes single or double quotes; the quotes are removed. * `a(n)`, `time(s)`, `row(s)`, `document(s)` and `item(s)` are optional text: `1 time` and `2 times` both match. * A step that ends with `:` takes a data table, or a doc string, on the following lines. Its entry shows the table’s columns and the rows it knows. * How a value is typed (string, number, `null`, `undefined`) depends on the step; each step’s entry says how it reads its values. ## Parameter types [Section titled “Parameter types”](#parameter-types) | Parameter | Takes | Values | For example | Pack | | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | ------------ | | `{anchor}` | the point of a control that places on it are measured from: its `middle`, or a corner. Places run right and down from it, so one left of or above it is negative | `middle`, `top left`, `top right`, `bottom left`, `bottom right` | `middle`, `bottom right` | desktop-core | | `{bigdecimal}` | a decimal number of any precision | | `0.1000000000000000055` | cucumber | | `{biginteger}` | a whole number of any size | | `123456789012345678901234567890` | cucumber | | `{byte}` | a whole number, from -128 to 127 | | `12` | cucumber | | `{control}` | a kind of control an app shows, found by the name people see: a `button` by its text, a `field` by its label, a `list item` or a `row` by one of its texts, a `menu item` in its open menu, an `element` is anything, by its text or accessibility label; a phone’s `notification` is tapped, by a text in its title or its text. A kind an app’s platform does not have (a phone has no menu bar) fails the step | `button`, `field`, `checkbox`, `radio button`, `switch`, `tab`, `menu`, `menu item`, `list item`, `row`, `link`, `image`, `text`, `element`, `notification` | `button`, `list item` | app-core | | `{dbService}` | the name of a database registered in the scenario | | `parcels-db` | sql | | `{direction}` | where a swipe goes, as the finger moves (`down` from the top pulls a list to refresh); or which way a phone moves or turns in the scene its camera sees, where it also moves `forward` and `back` | `up`, `down`, `left`, `right`, `forward`, `back` | `down`, `left`, `forward` | mobile-core | | `{double}` | a decimal number | | `19.90` | cucumber | | `{duration}` | a duration in seconds (`s`) or minutes (`m`) | | `5s`, `2m` | core | | `{element}` | a kind of element on the page, plural after a number (`buttons`): a `field` by its label or placeholder, an `option` is a radio button, an `element` is anything, by its text, label, alternative text or title | `button`, `field`, `checkbox`, `option`, `link`, `tab`, `menu item`, `element` | `button`, `field` | web-core | | `{filepath}` | a file of the project: a path relative to the feature file’s directory, the `resources` directories or axx.yaml’s directory (looked for in that order), or an absolute path, quoted when it has a space | | `seeds/parcels.yaml`, `kafka/scan-delivered.json`, `"seeds/day one.yaml"` | core | | `{float}` | a decimal number | | `2.5`, `0.1` | cucumber | | `{int}` | a whole number | | `200`, `3` | cucumber | | `{long}` | a whole number, however large | | `9007199254740993` | cucumber | | `{mimeType}` | a content type | `application/json`, `text/json`, `application/problem+json`, `application/x-www-form-urlencoded` | `application/json` | core | | `{mockedService}` | the name of a mocked service the scenario registered | | `addresses` | mock | | `{mongoService}` | the name of a MongoDB database registered in the scenario | | `tracking-db` | mongo | | `{ordinal}` | a position, counting from 1; an optional ordinal left out is the first | | `1st`, `2nd`, `3rd` | core | | `{path}` | a path in a folder or a store, like `manifests/M-1/report.csv`, quoted when it has a space. A `*` stands for any characters but a slash, for a name a check cannot know (`snap-*.json`, named by the time): the path then names the one file it matches | | `manifests/M-KESTREL-0412/report.csv`, `"Parcels/Depot desk/arrivals.json"`, `snap-*.json` | core | | `{pattern}` | a regular expression (Java syntax) with no spaces, which matches the whole value | | `PX-\d{4}`, `[A-Z]{2}-\d+` | core | | `{service}` | the name of a REST service the scenario registered | | `parcels` | rest | | `{short}` | a whole number, from -32768 to 32767 | | `1200` | cucumber | | `{speed}` | how fast a phone travels: `25 mph`, `40 km/h` or `11 m/s`; or, for a GPX file with times, how many times its own pace: `4x` | | `25 mph`, `40 km/h`, `4x` | mobile-core | | `{sqlState}` | a five-character SQLSTATE error code, like `40001` (serialization failure) or `23505` (unique violation) | | `40001`, `23505` | sql | | `{string}` | text in double or single quotes, which the step leaves out | | `"Get a quote"`, `'Express'` | cucumber | | `{travel}` | whether the step waits for the phone to arrive (`travels`), or lets the scenario go on while it moves (`starts traveling`) | `travels`, `starts traveling` | `travels`, `starts traveling` | mobile-core | | `{word}` | one word, with no spaces | | `parcels`, `PX-4101` | cucumber | # FAQ > Common questions about Axx - the name, how it relates to Cucumber, what it needs, and how it fits with other testing tools. ### Why “axxeptance”? [Section titled “Why “axxeptance”?”](#why-axxeptance) It is *acceptance* testing, with a name of its own. Axx is the short form, and the command is `axx`. ### Do I need Java, Node or Python? [Section titled “Do I need Java, Node or Python?”](#do-i-need-java-node-or-python) No. Axx is a single static binary, and your service can be written in anything. What Axx needs, it downloads itself and keeps in its cache: the Go toolchain it uses to build itself with a project’s [packs](/next/guides/use-packs/), custom ones included, and the Node.js and browsers the web packs drive. Docker is only needed if your services start with Docker Compose. ### Is Axx Cucumber? [Section titled “Is Axx Cucumber?”](#is-axx-cucumber) Axx runs Gherkin, the language Cucumber defined, with its own executor built on the official Cucumber libraries for Go (the Gherkin parser, Cucumber Expressions, tag expressions and Cucumber Messages). Feature files are standard Gherkin, and reports use Cucumber formats. You do not write step definitions for what Axx’s packs cover: their steps come with Axx. ### How is this different from Postman, Karate or REST Assured? [Section titled “How is this different from Postman, Karate or REST Assured?”](#how-is-this-different-from-postman-karate-or-rest-assured) Axx tests a whole service from the outside, not only its HTTP API: it seeds databases, publishes and consumes events, verifies calls to mocked dependencies, and validates everything against your OpenAPI document. It also manages the service’s lifecycle (start, wait, stop, clean up) and runs scenarios in parallel. Scenarios are plain Gherkin, readable by the people who wrote the acceptance criteria. ### Can I test services that are not HTTP? [Section titled “Can I test services that are not HTTP?”](#can-i-test-services-that-are-not-http) Yes. Axx’s packs cover SQL databases, MongoDB, Kafka, the logs and files your services write, and cloud services on AWS, Google Cloud and Azure. Anything else can be reached with a [custom step](/next/guides/write-custom-steps/), written as a Go pack. ### Does Axx run my unit tests? [Section titled “Does Axx run my unit tests?”](#does-axx-run-my-unit-tests) No. Keep unit and integration tests in your language’s test framework. Axx tests the running system through its public interfaces. ### Can Axx test a deployed environment? [Section titled “Can Axx test a deployed environment?”](#can-axx-test-a-deployed-environment) Yes. Write the part of the service URLs that differs as a property (`http://${sys:parcels.host}:8400`), set it for the environment in a profile (say `--profile staging`), and run with `--no-start` so Axx does not try to start services. Keep data isolation in mind: the environment is shared with everyone else. ### Why do my scenarios pass alone and fail together? [Section titled “Why do my scenarios pass alone and fail together?”](#why-do-my-scenarios-pass-alone-and-fail-together) They share data. Scenarios run in parallel against shared infrastructure, so each one needs its own ids, names and keys. See [Scenario isolation](/next/explanations/scenario-isolation/) and [`axx lint`](/next/guides/isolate-test-data/). ### Does Axx collect telemetry? [Section titled “Does Axx collect telemetry?”](#does-axx-collect-telemetry) No. Axx sends no telemetry and does not check for updates. The only downloads it makes are those it needs, once, into its cache: the Go toolchain and the packs’ Go modules when it builds itself with a project’s packs, and the Node.js, Playwright, browsers, axe-core and Lighthouse the web packs use. Otherwise it talks only to the services, databases, brokers and clouds your scenarios name. ### What license is Axx under? [Section titled “What license is Axx under?”](#what-license-is-axx-under) Apache-2.0. Contributions are welcome with a DCO sign-off (`git commit -s`); see [CONTRIBUTING.md](https://github.com/nimbusxr/axx/blob/main/CONTRIBUTING.md). ### Where do I report a bug or a security issue? [Section titled “Where do I report a bug or a security issue?”](#where-do-i-report-a-bug-or-a-security-issue) Bugs and questions: [GitHub issues](https://github.com/nimbusxr/axx/issues). Security issues privately, as described in [SECURITY.md](https://github.com/nimbusxr/axx/blob/main/SECURITY.md).