# Isolate test data

> Use axx lint to guarantee that ids, keys and names in seeds, payloads and features never collide, so scenarios can share infrastructure and run in parallel.

Source: https://axx.nimbusxr.us/next/guides/isolate-test-data/

Scenarios run in parallel against one database, one broker and one set of mocks. They stay independent only if each one owns its data. Two seed files that both insert the parcel `PX-KES-1001`, or two scenarios that both publish events keyed `PX-TRK-3002`, produce failures that come and go with scheduling. `axx lint` finds those collisions before a run does.

## Write rules

Rules live under `lint:` in `axx.yaml`. Each one extracts values from files and says how unique they must be:

```yaml title="axx.yaml"
lint:
  config:
    baseDir: .          # patterns are relative to this directory
    mode: error         # error fails the lint; warn only reports

  rules:
    - name: Parcel references in seeds
      filePatterns: ["seeds/*.yaml"]
      regex: '^\s+-?\s*reference:\s*"([^"]+)"'
      description: Every seeded parcel and manifest line has its own reference
      validation: cross-file-unique

    - name: Manifest line ids
      filePatterns: ["seeds/*.yaml"]
      regex: '^\s+-?\s*id:\s*"([^"]+)"'
      validation: cross-file-unique

    - name: Depot scan ids
      filePatterns: ["seeds/*.json"]
      type: jsonpath
      jsonPath: "scans[*].scanId"
      validation: cross-file-unique
```

A `jsonpath` rule reads each file by its extension, as JSON, YAML, XML or CSV, so it needs no pattern for the file's layout. A seed's rows read the same in YAML, flat XML and CSV: `$['parcels.manifest_lines'][*]['id']` is the `id` of every manifest line, whichever format the seed is in.

Each rule key is described in the [configuration reference](https://axx.nimbusxr.us/next/references/config/#lint). A rule that scans files but extracts no value from them checks nothing, so `axx lint` warns about it instead of reporting ok: most often its pattern misses the files' format, like a regex anchored at `reference:` for YAML list items, whose lines start with `- ` (the rules above allow it with `-?`).

Generated rules can be merged in with `include`:

```yaml title="axx.yaml"
lint:
  include: [axx-lint.generated.yaml]   # written by `axx fixtures generate`
```

## One scenario, several files

A `cross-file-unique` rule compares file with file: a value two files share is a collision. That holds while each scenario's data is in a file of its own. Some scenarios need more than one file. The couriers' app's list refresh signs a courier in with one parcel, then gives the same courier a second parcel while the scenario runs, from a second seed file. Both files name the courier, and one scenario owns it.

Such a scenario keeps its files in a folder of its own, named after it, and the rule names that folder with a pattern that ends in `/`. All the files below a folder that pattern matches count as one file: they may share values with each other, and with nothing else.

```
features/courier-app/seeds/
  courier-app-today.yaml        one scenario's seed: one file
  courier-app-refresh/          one scenario's seeds: one folder, counted as one file
    at-sign-in.yaml
    given-since.yaml
```

```yaml title="axx.yaml"
lint:
  rules:
    - name: Couriers in seeds
      filePatterns:
        - "features/*/seeds/*.yaml"   # each scenario's file
        - "features/*/seeds/*/"       # each scenario's folder, as one file
      regex: '"courier": "([^"]+)"'
      description: Every courier's parcels are one scenario's
      validation: cross-file-unique
```

```gherkin
Scenario: A courier pulls the list down for a parcel given to them since
  Given a seeds/courier-app-refresh/at-sign-in.yaml db seed
  # ...
  When a seeds/courier-app-refresh/given-since.yaml db seed
```

A courier in `courier-app-refresh/` and in `courier-app-today.yaml` is reported as two files sharing a value (`a folder's files count as one`), and so is a courier two folders share.

- **One folder, one scenario.** A folder says its files are one scenario's, and the rule takes its word: two scenarios' files in one folder may share values unnoticed. A folder the pattern matches that is no scenario's, like one that keeps the fixtures a [factory](https://axx.nimbusxr.us/next/guides/fixture-factories/) generates, goes in `excludePatterns` (`"features/manifest-import/seeds/manifests/**"`); its own rules check it.
- **A folder pattern takes every file below the folder**, whatever its kind; `excludePatterns` remove the ones the rule should not read.
- **Every rule over the seeds names the folders.** A rule without the folder pattern checks the folder's files one by one, or not at all when its file pattern does not reach them (`features/*/seeds/*.yaml` does not).
- **A folder claims its files.** A file a file pattern matches too (`seeds/*.yaml` reaches into subfolders) still counts as its folder's; folders inside a matched folder are part of it.
- `cross-file-unique` and `file-unique` count the folder as one file; `global-unique` counts every occurrence, as always. `excludePatterns` still remove files one by one.

## Run it

```sh
axx lint
```

```console
FAIL Parcel references in seeds (cross-file-unique, 52 files): 1 duplicate value
     Every seeded parcel and manifest line has its own reference
     value "PX-KES-1001" appears in 2 files (cross-file-unique) [AXX-E0820]
       seeds/manifest-kestrel-resend.yaml:4:17  reference: "PX-KES-1001"
       seeds/manifest-kestrel.yaml:4:17         reference: "PX-KES-1001"
ok   Manifest line ids (cross-file-unique, 52 files)
ok   Depot scan ids (cross-file-unique, 1 file)
ok   kafka/depot-scans.factory.yaml: scanId uniqueness (cross-file-unique, 2 files)
ok   requests/registrations.factory.yaml: reference uniqueness (cross-file-unique, 2 files)
ok   Services registered before use (13 files)
ok   SQL selection and trigger ordinals (13 files)
ok   Stale selections (13 files)
ok   REST request ordinals (13 files)
ok   REST payload values (13 files)
hint: seeds/ has 39 hand-written .yaml files of one shape (parcels.parcels) that no fixture factory generates; a factory would keep what they share in one place (optional; `axx fixtures adopt --help`)
hint: seeds/ has 5 hand-written .yaml files of one shape (parcels.manifest_lines) that no fixture factory generates; a factory would keep what they share in one place (optional; `axx fixtures adopt --help`)
axx lint: 10 rules, 70 files: 1 error, 0 warnings
```

A new seed file reused a reference that `seeds/manifest-kestrel.yaml` already inserts. The rules from `axx-lint.generated.yaml` run too, and so do the built-in checks of the features. They catch:

- a When after a Then: a scenario that does something and checks it, then does something else and checks that, which is several scenarios in one (write one per behavior, with what an earlier round did as Given steps);
- a REST, SQL, MongoDB or Kafka step that comes before the scenario registers a service of its pack;
- SQL selections and triggers, and REST requests, addressed by ordinals they cannot have (a `3rd ordered` request after only one, or a second request added without its ordinal);
- a step without an ordinal that checks the first selection while a later one goes unchecked (`the selection has 1 row` after a second retrieval);
- request payload values in single quotes, which in a table are part of the value.

They are warnings, and `axx validate` reports them as well.

The `hint:` lines are suggestions, never errors or warnings. When the scenarios read three or more hand-written `.json` or `.yaml` files from one directory, whose top-level keys are the same and that no [fixture factory](https://axx.nimbusxr.us/next/guides/fixture-factories/) generates, `axx lint` suggests one: it would keep what the files share in one place and each file's differences in its own. Factories are optional; `axx fixtures adopt` turns the files into one without changing a value.

`axx lint`, `axx validate` and a passing `axx run` also hint at scenarios whose checks prove little: those that check only a success status (a 2xx response, or a command's exit code 0), which says it was accepted, not what it did, and those that check only that something did not happen, which also passes when the action never ran. A hint names the scenarios; whether one needs another check is for its author to judge.

`axx lint` exits with `3` when an `error`-mode rule finds a duplicate, like `axx validate` does for undefined steps. Run it in CI next to `axx validate`, and start new rules in `warn` mode while you clean up existing data. `axx lint --mode error` makes every rule an error, the built-in checks of the features too, so CI fails on an ordinal that would fail every run, before any service starts.

## Tips

- **Prove a rule bites.** After writing a rule, plant a duplicate value and check that `axx lint` fails. A pattern that matches no files still passes (its line only says `no files matched`).
- **Name values after the scenario.** `PX-DBF-3001` tells a reader which feature owns it; `test` tells nobody anything.
- **Declare identities once.** With [fixture factories](https://axx.nimbusxr.us/next/guides/fixture-factories/), an `identity:` in the factory spec generates the matching lint rule for you.
- **Some scenarios cannot share.** A scenario that changes a table's triggers affects everyone. Tag it with a tag from `run.exclusive` so it runs alone ([Run in parallel](https://axx.nimbusxr.us/next/guides/parallel-runs/)).

[Scenario isolation](https://axx.nimbusxr.us/next/explanations/scenario-isolation/) explains the reasoning behind shared infrastructure with isolated data.