The verification layer for agentic software factories
Axx is an open-source acceptance testing framework that verifies real software against executable behavioral expectations, independent of the language, architecture or agent that built it.
Define what correct means. Let implementations evolve. Verify the behavior that matters.
Coding agents are changing how quickly software can be created and changed. But generating an implementation and knowing that it meets its requirements are different problems.
Code review, unit tests and static analysis remain valuable. None of them shows that the running system delivers the behavior expected of it. As software factories become more autonomous, verifying that behavior independently matters more, not less.
The challenge isn’t generating more software. It’s changing software continuously without losing confidence in what it does.
Software exists to meet requirements, not to run code. Acceptance criteria state the behaviors a system must provide, as scenarios that engineers, product owners and agents can all review, and that stay meaningful while the implementation changes.
Refactor a service, switch frameworks, replace a component or rewrite it in another language. While the external contract holds, the acceptance criteria still apply: axx never loads your code.
Axx runs each scenario against the running system, through its APIs, events, data stores and user interfaces. An agent can use axx to check its own work, but the code it wrote isn’t what defines success. The acceptance criteria are, and your team decides who approves them.
Write the acceptance criteria: what the system must accomplish, as scenarios made of real steps.
Engineers and coding agents build, change, refactor or replace the implementation.
Axx starts the system, exercises it through its observable interfaces and checks the expected outcomes.
Failures come back as structured data, expected against actual, so the next change fixes the code, not the requirement.
Axx is the verification layer of a software factory, not the factory: it doesn’t orchestrate agents or write code. It runs wherever the software is built, from a developer’s terminal to a CI pipeline to an agent’s tool calls.
Scenarios exercise running systems through the boundaries their clients use: HTTP and gRPC, events, databases, files, browsers and apps. They never import a class or depend on a framework, so they outlive refactors and rewrites.
Send a request, check the call your service made to a mocked dependency, then the rows it stored and the event it published, all in one scenario, with steps from axx’s packs and no glue code.
Agents search the steps actually installed, validate scenarios without starting anything, work through axx’s MCP server, and read failures as structured data, instead of guessing at a test API or parsing logs.
axx run starts your services, waits until they’re healthy, runs scenarios in parallel and cleans up. Generated fixtures, isolation checks, polling steps and reruns of what failed keep each cycle quick and reliable.
Test AI features, MCP servers, A2A agents, mobile and desktop apps, and the models apps run on the device, replaced by stubs axx builds in the model’s own format, whose answers are your mappings.
This scenario registers a parcel through a service’s REST API. In one run, axx checks the request and the response against the service’s OpenAPI document, checks that the service asked its address service (a WireMock mock) about the postcode, and checks what it stored in PostgreSQL, down to a JSON column.
Feature: Register parcels
Background: Given the parcels service with the following properties: | url | http://localhost:8400 | | openapi | http://localhost:8400/openapi.json | And the mocked addresses service with the following properties: | url | http://localhost:8081 | And a parcels-db database with the following properties: | url | postgres://localhost:5432/parcels | | user | parcels | | password | parcels |
Scenario: A shop registers a parcel Given a POST request to /api/parcels And a request payload using an application/json content example And the request payload properties are: | reference | PX-REG-1001 | | recipient.postcode | "53111" | When the request is executed Then the response status code is 201 And the mocked GET request to /v1/postcodes/DE/53111 named postcode-check was received by addresses And a selection of rows is retrieved from the parcels.parcels table where: | reference | PX-REG-1001 | And the 1st row details property for the selection json properties are: | source | api | | zone | DE-1 |Nothing in it names a class, a framework or a language. It describes what the service does at its boundaries, so the same scenario verifies a refactor, a new framework or a rewrite. Every line is a step from one of axx’s packs: the step index lists them all, and the parcels example runs a full suite like it.
Mock the model APIs your service calls (OpenAI’s and the servers that speak it, Anthropic’s, Gemini’s) and check what it sent them. For the models an app runs itself, axx builds stubs in their native formats, from ONNX and Core ML to TFLite and GGUF, and stands in for the platform’s own: Apple’s Vision, Android’s ML Kit, a browser’s built-in AI.
Call an MCP server’s tools as an AI assistant does, over stdio or HTTP, with the tools’ own schemas as the contract. Message A2A agents over JSON-RPC, HTTP+JSON or gRPC and check their tasks and artifacts. Mock the servers and agents your service depends on.
Drive Android and iOS apps on emulators and simulators, and macOS, Windows and Linux apps, by the names people see. Move the phone along a route, tap its notifications, and show a simulator’s or an emulator’s camera a photo, a video or a 3D scene, which the app’s AR (ARKit, ARCore) finds where it is.
Publish and consume Kafka events with Avro and Schema Registry, seed and query SQL and MongoDB, mock dependencies, check cloud services like S3, Pub/Sub and Service Bus, and validate against OpenAPI and AsyncAPI contracts, in the same scenario.
Axx comes from years of maintaining acceptance suites for complex enterprise systems, where executable acceptance criteria shaped how teams built and shipped software. Its architecture reflects those lessons: black-box scenarios, step text that never changes, and test data that can’t collide between parallel scenarios. It wasn’t built to make Gherkin work with AI.
Engineers use it directly: one binary, feature files, axx run. Coding agents use the same tool, with interfaces made for them.
Axx publishes how coding agents do with it. In its agent evaluation, an agent writes the tests for a parcels service. A suite succeeds only if it passes on the working service and fails on every bug planted in it.
It’s a small sample, run by the project itself: one model (openai/gpt-6-luna), six tasks, three tries each. The tasks, planted bugs, checks and transcripts are all public.
Whether software is written by an engineer, generated by an agent or rebuilt from scratch, its required behavior still matters. Axx verifies that behavior against the running system.
Axx is open source (Apache-2.0) and in beta (v0.x): interfaces may change before v1.0.0.
For coding agents: in a project, the axx command line answers faster than this site: axx steps lists every step of the project’s packs, a line each, and axx steps show <id> gives one step’s documentation and examples. On this site, every page has a Markdown twin (add .md to its path), llms.txt lists them, and a pack’s steps are at /references/packs/<pack>.md.