# Faults

> Check what your services do when a connection goes wrong: cut off, slow, hanging, reset, or closing halfway through an answer.

Source: https://axx.nimbusxr.us/next/references/packs/faults/

Check what your services do when a connection goes wrong: cut off, slow, hanging, reset, or closing halfway through an answer.

Your service reaches the dependency through [Toxiproxy](https://github.com/Shopify/toxiproxy), which runs in your test environment like any other service, and Axx tells it what to do to a connection. Declare the connections in Toxiproxy's own config, since your service needs their ports before it starts, and register Toxiproxy once, usually in the `Background`:

```gherkin
Given the parcels fault proxy with the following properties:
  | url | http://localhost:8474 |

Scenario: A cached tracking view answers while the database is away
  Given the parcels-db connection is cut off
  ...
```

- **It works on any TCP connection:** PostgreSQL, Oracle, Kafka, Redis, an HTTP API. A connection is the name of a proxy in Toxiproxy's config.
- **It sees bytes, not requests,** so it cannot pick a URL, a query or a row. To fail one request, stub it with the mock pack's WireMock mappings; to fail one row's insert, use the sql pack's triggers.
- **Connections already open are hit too,** which is what a service's connection pool holds. A fault that counts bytes counts theirs from when its step runs.
- **Each fault can say more,** at its end, in this order:
  - `coming back` (the default) or `going out`: the direction it applies to. With `coming back`, what the service sends still arrives and only the answer is lost, so a write can be made though the service never hears so; a driver that checks a connection before it uses one waits on that check instead. With `going out`, nothing the service sends arrives.
  - `on 30% of connections`: each new connection has the fault or does not, by chance.
  - `for 5s`: it ends by itself. A request step waits for its answer, so no restore step can run during it: this is how a scenario tests a service retrying through an outage.
- **A connection cut off proves what did not use it:** an answer that still comes, with the database cut off, came from somewhere else, like a cache.
- **A scenario undoes only what it did,** when it ends or at `is restored`. Scenarios running at the same time share the connection and its faults: `run.exclusiveUses` in axx.yaml can run the scenarios that use this pack alone.

Add it to a project with `axx pack add faults`.

## Steps

Each step's expression and what it does. Its documentation, parameters and examples: `axx steps show <id>` (offline, for the axx installed), or this page in full: https://axx.nimbusxr.us/next/references/packs/faults.full.md. The pack's steps, whether the project uses it or not: `axx pack show faults`.

- `faults.proxy`: `Given the {word} fault proxy with the following properties: | property | value |` Register the Toxiproxy that holds the connections the steps name.
- `faults.cutoff`: `Given the {connection} connection is cut off[[ for {duration}]]` Take the connection away: connections open now close, and new ones are refused.
- `faults.slowed`: `Given the {connection} connection is slowed by {duration}[[ {dataDirection}]][[ on {int}% of connections]][[ for {duration}]]` Delay the connection's data by that long, each way it is applied. A new connection waits several times over, for each exchange it starts with.
- `faults.limited`: `Given the {connection} connection is limited to {int} KB\/s[[ {dataDirection}]][[ on {int}% of connections]][[ for {duration}]]` Let the connection's data through at that rate, in thousands of bytes a second.
- `faults.hangs`: `Given the {connection} connection hangs[[ {dataDirection}]][[ on {int}% of connections]][[ for {duration}]]` Hold the connection's data, and answer nothing, until the fault ends: the service waits as long as its own timeout lets it.
- `faults.hangs.closes`: `Given the {connection} connection hangs for {duration}, then closes[[ {dataDirection}]][[ on {int}% of connections]][[ for {duration}]]` Hold the connection's data for that long, then close the connection.
- `faults.reset`: `Given the {connection} connection is reset[[ {dataDirection}]][[ on {int}% of connections]][[ for {duration}]]` Reset the connection as data arrives, which the service sees as a connection reset by its peer.
- `faults.reset.after`: `Given the {connection} connection is reset after {duration}[[ {dataDirection}]][[ on {int}% of connections]][[ for {duration}]]` Reset the connection that long after data arrives.
- `faults.slowclose`: `Given the {connection} connection takes {duration} to close[[ {dataDirection}]][[ on {int}% of connections]][[ for {duration}]]` Delay the close of the connection by that long. What the connection carries before it closes is not delayed.
- `faults.pieces`: `Given the {connection} connection breaks into {int}-byte pieces, {duration} apart[[ {dataDirection}]][[ on {int}% of connections]][[ for {duration}]]` Send the connection's data in pieces of that many bytes, that long apart, which tests a service that reads a whole answer in one read.
- `faults.closes.after`: `Given the {connection} connection closes after {int} KB[[ {dataDirection}]][[ on {int}% of connections]][[ for {duration}]]` Close the connection once that many thousands of bytes have passed through it, which leaves an answer cut off partway. A connection already open counts them from when the step runs.
- `faults.closes.after.bytes`: `Given the {connection} connection closes after {int} bytes[[ {dataDirection}]][[ on {int}% of connections]][[ for {duration}]]` Close the connection once that many bytes have passed through it. A connection already open counts them from when the step runs, so a few hundred cut off even a short answer on a connection a service's pool holds.
- `faults.loses`: `Given the {connection} connection loses {int}% of its data[[ {dataDirection}]][[ on {int}% of connections]][[ for {duration}]]` Drop that share of the connection's data, by chance. The service gets no error: what it was waiting for just does not come.
- `faults.toxic`: `Given the {connection} connection has a(n) {word} toxic with the following attributes: | attribute | value |` Give the connection any toxic Toxiproxy has by that type, one compiled into your own Toxiproxy included, with the attributes Toxiproxy's documentation gives it.
- `faults.restored`: `When the {connection} connection is restored` End the faults this scenario gave the connection. A scenario's faults end by themselves when it does; another scenario's are left alone.

## Parameter types

| Parameter | Takes | Values | For example |
|---|---|---|---|
| `{connection}` | a connection: the name of a proxy in Toxiproxy's config |  | `parcels-db`, `cache` |
| `{dataDirection}` | the direction of a connection's data a fault applies to: `coming back` to the service, or `going out` from it |  | `coming back`, `going out` |