# Behavioral verification for agentic software factories

> Axx is an open-source black-box acceptance testing framework for agentic software factories. Verify running systems across APIs, events, data, native apps, and AI.

Axx is an open-source acceptance testing framework that verifies real software against executable behavioral expectations, independent of the language, architecture or agent that built it.

Define what correct means. Let implementations evolve. Verify the behavior that matters.

- [Get started](https://axx.nimbusxr.us/next/tutorials/quickstart/)
- [Why axx](https://axx.nimbusxr.us/next/explanations/why-axx/)
- [GitHub](https://github.com/nimbusxr/axx)

Source: https://axx.nimbusxr.us/next/

## Generating software is getting easier. Trusting it isn't.

Coding agents are changing how quickly software can be created and changed. But generating an implementation and knowing that it meets its requirements are different problems.

Code review, unit tests and static analysis remain valuable. None of them shows that the running system delivers the behavior expected of it. As software factories become more autonomous, verifying that behavior independently matters more, not less.

**The challenge isn't generating more software. It's changing software continuously without losing confidence in what it does.**

## Keep what correct means apart from how it's built.

### [Intent](https://axx.nimbusxr.us/next/explanations/step-design/)
Software exists to meet requirements, not to run code. Acceptance criteria state the behaviors a system must provide, as scenarios that engineers, product owners and agents can all review, and that stay meaningful while the implementation changes.

### [Freedom of implementation](https://axx.nimbusxr.us/next/explanations/black-box-testing/)
Refactor a service, switch frameworks, replace a component or rewrite it in another language. While the external contract holds, the acceptance criteria still apply: axx never loads your code.

### [Independent verification](https://axx.nimbusxr.us/next/explanations/how-axx-works/)
Axx runs each scenario against the running system, through its APIs, events, data stores and user interfaces. An agent can use axx to check its own work, but the code it wrote isn't what defines success. The acceptance criteria are, and your team decides who approves them.

## Give agents freedom to build. Keep control of what correct means.

### Define behavior
Write the acceptance criteria: what the system must accomplish, as scenarios made of real steps.

### Implement freely
Engineers and coding agents build, change, refactor or replace the implementation.

### Verify independently
Axx starts the system, exercises it through its observable interfaces and checks the expected outcomes.

### Iterate with evidence
Failures come back as structured data, expected against actual, so the next change fixes the code, not the requirement.

Axx is the verification layer of a software factory, not the factory: it doesn't orchestrate agents or write code. It runs wherever the software is built, from a developer's terminal to a CI pipeline to an agent's tool calls.

## Built for real systems. Built for autonomous engineering.

### [Black-box by architecture](https://axx.nimbusxr.us/next/explanations/black-box-testing/)
Scenarios exercise running systems through the boundaries their clients use: HTTP and gRPC, events, databases, files, browsers and apps. They never import a class or depend on a framework, so they outlive refactors and rewrites.

### [One scenario across the system](https://axx.nimbusxr.us/next/explanations/why-axx/)
Send a request, check the call your service made to a mocked dependency, then the rows it stored and the event it published, all in one scenario, with steps from axx's packs and no glue code.

### [Native interfaces for coding agents](https://axx.nimbusxr.us/next/explanations/axx-for-agents/)
Agents search the steps actually installed, validate scenarios without starting anything, work through axx's MCP server, and read failures as structured data, instead of guessing at a test API or parsing logs.

### [Fast, repeatable feedback](https://axx.nimbusxr.us/next/guides/parallel-runs/)
`axx run` starts your services, waits until they're healthy, runs scenarios in parallel and cleans up. Generated fixtures, isolation checks, polling steps and reruns of what failed keep each cycle quick and reliable.

### [Verification beyond traditional applications](https://axx.nimbusxr.us/next/guides/stub-embedded-models/)
Test AI features, MCP servers, A2A agents, mobile and desktop apps, and the models apps run on the device, replaced by stubs axx builds in the model's own format, whose answers are your mappings.

## Acceptance criteria you can execute.

This scenario registers a parcel through a service's REST API. In one run, axx checks the request and the response against the service's OpenAPI document, checks that the service asked its address service (a WireMock mock) about the postcode, and checks what it stored in PostgreSQL, down to a JSON column.

```gherkin title="features/register-parcels.feature"
Feature: Register parcels

  Background:
    Given the parcels service with the following properties:
      | url     | http://localhost:8400              |
      | openapi | http://localhost:8400/openapi.json |
    And the mocked addresses service with the following properties:
      | url | http://localhost:8081 |
    And a parcels-db database with the following properties:
      | url      | postgres://localhost:5432/parcels |
      | user     | parcels                           |
      | password | parcels                           |

  Scenario: A shop registers a parcel
    Given a POST request to /api/parcels
    And a request payload using an application/json content example
    And the request payload properties are:
      | reference          | PX-REG-1001 |
      | recipient.postcode | "53111"     |
    When the request is executed
    Then the response status code is 201
    And the mocked GET request to /v1/postcodes/DE/53111 named postcode-check was received by addresses
    And a selection of rows is retrieved from the parcels.parcels table where:
      | reference | PX-REG-1001 |
    And the 1st row details property for the selection json properties are:
      | source | api  |
      | zone   | DE-1 |
```

Nothing in it names a class, a framework or a language. It describes what the service does at its boundaries, so the same scenario verifies a refactor, a new framework or a rewrite. Every line is a step from one of axx's packs: the [step index](https://axx.nimbusxr.us/next/references/step-index/) lists them all, and the [parcels example](https://github.com/nimbusxr/axx/tree/main/examples/parcels) runs a full suite like it.

## Go beyond conventional acceptance testing.

### [Deterministic AI testing](https://axx.nimbusxr.us/next/guides/test-ai-features/)
Mock the model APIs your service calls (OpenAI's and the servers that speak it, Anthropic's, Gemini's) and check what it sent them. For the models an app runs itself, axx builds stubs in their native formats, from ONNX and Core ML to TFLite and GGUF, and stands in for the platform's own: Apple's Vision, Android's ML Kit, a browser's built-in AI.

### [Agent protocol testing](https://axx.nimbusxr.us/next/guides/test-mcp/)
Call an MCP server's tools as an AI assistant does, over stdio or HTTP, with the tools' own schemas as the contract. Message [A2A agents](https://axx.nimbusxr.us/next/guides/test-a2a/) over JSON-RPC, HTTP+JSON or gRPC and check their tasks and artifacts. Mock the servers and agents your service depends on.

### [Native application verification](https://axx.nimbusxr.us/next/guides/test-mobile-apps/)
Drive Android and iOS apps on emulators and simulators, and [macOS, Windows and Linux apps](https://axx.nimbusxr.us/next/guides/test-desktop-apps/), by the names people see. Move the phone along a route, tap its notifications, and show a simulator's or an emulator's camera a photo, a video or a 3D scene, which the app's AR (ARKit, ARCore) finds where it is.

### [Distributed workflows](https://axx.nimbusxr.us/next/guides/test-kafka-avro/)
Publish and consume Kafka events with Avro and Schema Registry, seed and query SQL and MongoDB, mock dependencies, check [cloud services](https://axx.nimbusxr.us/next/guides/test-cloud-services/) like S3, Pub/Sub and Service Bus, and validate against OpenAPI and AsyncAPI contracts, in the same scenario.

## Designed for engineers. Ready for agents.

Axx comes from years of maintaining acceptance suites for complex enterprise systems, where executable acceptance criteria shaped how teams built and shipped software. Its architecture reflects those lessons: black-box scenarios, step text that never changes, and test data that can't collide between parallel scenarios. It wasn't built to make Gherkin work with AI.

Engineers use it directly: one binary, feature files, `axx run`. Coding agents use the same tool, with [interfaces made for them](https://axx.nimbusxr.us/next/tutorials/with-an-agent/).

Axx publishes how coding agents do with it. In its [agent evaluation](https://axx.nimbusxr.us/next/explanations/agent-evals/), an agent writes the tests for a parcels service. A suite succeeds only if it passes on the working service and fails on every bug planted in it.

- **With axx:** all 72 suites succeeded, across four setups (with its skills, its MCP server, both or neither).
- **Without axx:** 14 of 18 did.

It's a small sample, run by the project itself: one model (`openai/gpt-6-luna`), six tasks, three tries each. The tasks, planted bugs, checks and transcripts are all [public](https://github.com/nimbusxr/axx-evals).

## Build faster. Keep the contract.

Whether software is written by an engineer, generated by an agent or rebuilt from scratch, its required behavior still matters. Axx verifies that behavior against the running system.

- [Start verifying](https://axx.nimbusxr.us/next/tutorials/quickstart/)
- [Explore the documentation](https://axx.nimbusxr.us/next/explanations/why-axx/)

Axx is open source (Apache-2.0) and in beta (`v0.x`): interfaces may change before `v1.0.0`.

**For coding agents:** in a project, the axx command line answers faster than this site: `axx steps` lists every step of the project's packs, a line each, and `axx steps show <id>` gives one step's documentation and examples. On this site, every page has a Markdown twin (add `.md` to its path), [llms.txt](https://axx.nimbusxr.us/next/llms.txt) lists them, and a pack's steps are at `/references/packs/<pack>.md`.