Faults

Check what your services do when a connection goes wrong: cut off, slow, hanging, reset, or closing halfway through an answer.

Your service reaches the dependency through Toxiproxy, which runs in your test environment like any other service, and Axx tells it what to do to a connection. Declare the connections in Toxiproxy’s own config, since your service needs their ports before it starts, and register Toxiproxy once, usually in the Background:

Given the parcels fault proxy with the following properties:
  | url | http://localhost:8474 |

Scenario: A cached tracking view answers while the database is away
  Given the parcels-db connection is cut off
  ...

• It works on any TCP connection: PostgreSQL, Oracle, Kafka, Redis, an HTTP API. A connection is the name of a proxy in Toxiproxy’s config.
• It sees bytes, not requests, so it cannot pick a URL, a query or a row. To fail one request, stub it with the mock pack’s WireMock mappings; to fail one row’s insert, use the sql pack’s triggers.
• Connections already open are hit too, which is what a service’s connection pool holds. A fault that counts bytes counts theirs from when its step runs.
• Each fault can say more, at its end, in this order:
  • coming back (the default) or going out: the direction it applies to. With coming back, what the service sends still arrives and only the answer is lost, so a write can be made though the service never hears so; a driver that checks a connection before it uses one waits on that check instead. With going out, nothing the service sends arrives.
  • on 30% of connections: each new connection has the fault or does not, by chance.
  • for 5s: it ends by itself. A request step waits for its answer, so no restore step can run during it: this is how a scenario tests a service retrying through an outage.
• A connection cut off proves what did not use it: an answer that still comes, with the database cut off, came from somewhere else, like a cache.
• A scenario undoes only what it did, when it ends or at is restored. Scenarios running at the same time share the connection and its faults: run.exclusiveUses in axx.yaml can run the scenarios that use this pack alone.

Add it to a project with axx pack add faults.

Steps

faults.proxy

Given the {word} fault proxy with the following properties:
  | property | value |

The table may start with the row naming its columns, as above, or go straight to its rows.

Register the Toxiproxy that holds the connections the steps name.

• A connection is found on whichever registered Toxiproxy has it.
• Values expand ${env:..} and ${sys:..}; the token is masked.

Parameter	Takes	For example
{word}	one word, with no spaces	parcels, PX-4101

Property	Takes	Default
url	where Toxiproxy’s API is, like http://localhost:8474	required
token	the token of a Toxiproxy started with TOXIPROXY_AUTH_TOKEN, like ${env:TOXIPROXY_TOKEN}

Example:

Given the parcels fault proxy with the following properties:
  | url | http://localhost:8474 |

Since 0.3.1.

faults.cutoff

Given the {connection} connection is cut off[[ for {duration}]]

Take the connection away: connections open now close, and new ones are refused.

• With for {duration} it comes back by itself, which is how a scenario tests a service retrying through an outage.

Parameter	Takes	For example
{connection}	a connection: the name of a proxy in Toxiproxy’s config	parcels-db, cache
{duration}	a duration in milliseconds (ms), seconds (s) or minutes (m)	5s, 2m

Variants, the parts in [[...]] said or left out:

• the {connection} connection is cut off
• the {connection} connection is cut off for {duration}

Examples:

Given the parcels-db connection is cut off

Given the parcels-db connection is cut off for 5s

Since 0.3.1.

faults.slowed

Given the {connection} connection is slowed by {duration}[[ {dataDirection}]][[ on {int}% of connections]][[ for {duration}]]

Delay the connection’s data by that long, each way it is applied. A new connection waits several times over, for each exchange it starts with.

• coming back (the default) or going out is the direction it applies to.
• on 30% of connections applies it to that share of connections, by chance.
• for 5s ends it by itself after that long.
• It also hits the connections open now, and ends when the scenario does.

Parameter	Takes	For example
{connection}	a connection: the name of a proxy in Toxiproxy’s config	parcels-db, cache
{duration}	a duration in milliseconds (ms), seconds (s) or minutes (m)	5s, 2m
{dataDirection}	the direction of a connection’s data a fault applies to: coming back to the service, or going out from it	coming back, going out
{int}	a whole number	200, 3

Variants, the parts in [[...]] said or left out:

• the {connection} connection is slowed by {duration}
• the {connection} connection is slowed by {duration} for {duration}
• the {connection} connection is slowed by {duration} {dataDirection}
• the {connection} connection is slowed by {duration} on {int}% of connections
• the {connection} connection is slowed by {duration} {dataDirection} for {duration}
• the {connection} connection is slowed by {duration} on {int}% of connections for {duration}
• the {connection} connection is slowed by {duration} {dataDirection} on {int}% of connections
• the {connection} connection is slowed by {duration} {dataDirection} on {int}% of connections for {duration}

Examples:

Given the parcels-db connection is slowed by 2s

Given the cache connection is slowed by 200ms going out on 30% of connections

Since 0.3.1.

faults.limited

Given the {connection} connection is limited to {int} KB\/s[[ {dataDirection}]][[ on {int}% of connections]][[ for {duration}]]

Let the connection’s data through at that rate, in thousands of bytes a second.

• coming back (the default) or going out is the direction it applies to.
• on 30% of connections applies it to that share of connections, by chance.
• for 5s ends it by itself after that long.
• It also hits the connections open now, and ends when the scenario does.

Parameter	Takes	For example
{connection}	a connection: the name of a proxy in Toxiproxy’s config	parcels-db, cache
{int}	a whole number	200, 3
{dataDirection}	the direction of a connection’s data a fault applies to: coming back to the service, or going out from it	coming back, going out
{duration}	a duration in milliseconds (ms), seconds (s) or minutes (m)	5s, 2m

Variants, the parts in [[...]] said or left out:

• the {connection} connection is limited to {int} KB\/s
• the {connection} connection is limited to {int} KB\/s for {duration}
• the {connection} connection is limited to {int} KB\/s {dataDirection}
• the {connection} connection is limited to {int} KB\/s on {int}% of connections
• the {connection} connection is limited to {int} KB\/s {dataDirection} for {duration}
• the {connection} connection is limited to {int} KB\/s on {int}% of connections for {duration}
• the {connection} connection is limited to {int} KB\/s {dataDirection} on {int}% of connections
• the {connection} connection is limited to {int} KB\/s {dataDirection} on {int}% of connections for {duration}

Example:

Given the parcels-db connection is limited to 100 KB/s

Since 0.3.1.

faults.hangs

Given the {connection} connection hangs[[ {dataDirection}]][[ on {int}% of connections]][[ for {duration}]]

Hold the connection’s data, and answer nothing, until the fault ends: the service waits as long as its own timeout lets it.

• coming back (the default) or going out is the direction it applies to.
• on 30% of connections applies it to that share of connections, by chance.
• for 5s ends it by itself after that long.
• It also hits the connections open now, and ends when the scenario does.

Parameter	Takes	For example
{connection}	a connection: the name of a proxy in Toxiproxy’s config	parcels-db, cache
{dataDirection}	the direction of a connection’s data a fault applies to: coming back to the service, or going out from it	coming back, going out
{int}	a whole number	200, 3
{duration}	a duration in milliseconds (ms), seconds (s) or minutes (m)	5s, 2m

Variants, the parts in [[...]] said or left out:

• the {connection} connection hangs
• the {connection} connection hangs for {duration}
• the {connection} connection hangs {dataDirection}
• the {connection} connection hangs on {int}% of connections
• the {connection} connection hangs {dataDirection} for {duration}
• the {connection} connection hangs on {int}% of connections for {duration}
• the {connection} connection hangs {dataDirection} on {int}% of connections
• the {connection} connection hangs {dataDirection} on {int}% of connections for {duration}

Examples:

Given the parcels-db connection hangs

Given the parcels-db connection hangs coming back

Given the shops connection hangs for 10s

Since 0.3.1.

faults.hangs.closes

Given the {connection} connection hangs for {duration}, then closes[[ {dataDirection}]][[ on {int}% of connections]][[ for {duration}]]

Hold the connection’s data for that long, then close the connection.

• coming back (the default) or going out is the direction it applies to.
• on 30% of connections applies it to that share of connections, by chance.
• for 5s ends it by itself after that long.
• It also hits the connections open now, and ends when the scenario does.

Parameter	Takes	For example
{connection}	a connection: the name of a proxy in Toxiproxy’s config	parcels-db, cache
{duration}	a duration in milliseconds (ms), seconds (s) or minutes (m)	5s, 2m
{dataDirection}	the direction of a connection’s data a fault applies to: coming back to the service, or going out from it	coming back, going out
{int}	a whole number	200, 3

Variants, the parts in [[...]] said or left out:

• the {connection} connection hangs for {duration}, then closes
• the {connection} connection hangs for {duration}, then closes for {duration}
• the {connection} connection hangs for {duration}, then closes {dataDirection}
• the {connection} connection hangs for {duration}, then closes on {int}% of connections
• the {connection} connection hangs for {duration}, then closes {dataDirection} for {duration}
• the {connection} connection hangs for {duration}, then closes on {int}% of connections for {duration}
• the {connection} connection hangs for {duration}, then closes {dataDirection} on {int}% of connections
• the {connection} connection hangs for {duration}, then closes {dataDirection} on {int}% of connections for {duration}

Example:

Given the parcels-db connection hangs for 5s, then closes

Since 0.3.1.

faults.reset

Given the {connection} connection is reset[[ {dataDirection}]][[ on {int}% of connections]][[ for {duration}]]

Reset the connection as data arrives, which the service sees as a connection reset by its peer.

• coming back (the default) or going out is the direction it applies to.
• on 30% of connections applies it to that share of connections, by chance.
• for 5s ends it by itself after that long.
• It also hits the connections open now, and ends when the scenario does.

Parameter	Takes	For example
{connection}	a connection: the name of a proxy in Toxiproxy’s config	parcels-db, cache
{dataDirection}	the direction of a connection’s data a fault applies to: coming back to the service, or going out from it	coming back, going out
{int}	a whole number	200, 3
{duration}	a duration in milliseconds (ms), seconds (s) or minutes (m)	5s, 2m

Variants, the parts in [[...]] said or left out:

• the {connection} connection is reset
• the {connection} connection is reset for {duration}
• the {connection} connection is reset {dataDirection}
• the {connection} connection is reset on {int}% of connections
• the {connection} connection is reset {dataDirection} for {duration}
• the {connection} connection is reset on {int}% of connections for {duration}
• the {connection} connection is reset {dataDirection} on {int}% of connections
• the {connection} connection is reset {dataDirection} on {int}% of connections for {duration}

Example:

Given the parcels-db connection is reset

Since 0.3.1.

faults.reset.after

Given the {connection} connection is reset after {duration}[[ {dataDirection}]][[ on {int}% of connections]][[ for {duration}]]

Reset the connection that long after data arrives.

• coming back (the default) or going out is the direction it applies to.
• on 30% of connections applies it to that share of connections, by chance.
• for 5s ends it by itself after that long.
• It also hits the connections open now, and ends when the scenario does.

Parameter	Takes	For example
{connection}	a connection: the name of a proxy in Toxiproxy’s config	parcels-db, cache
{duration}	a duration in milliseconds (ms), seconds (s) or minutes (m)	5s, 2m
{dataDirection}	the direction of a connection’s data a fault applies to: coming back to the service, or going out from it	coming back, going out
{int}	a whole number	200, 3

Variants, the parts in [[...]] said or left out:

• the {connection} connection is reset after {duration}
• the {connection} connection is reset after {duration} for {duration}
• the {connection} connection is reset after {duration} {dataDirection}
• the {connection} connection is reset after {duration} on {int}% of connections
• the {connection} connection is reset after {duration} {dataDirection} for {duration}
• the {connection} connection is reset after {duration} on {int}% of connections for {duration}
• the {connection} connection is reset after {duration} {dataDirection} on {int}% of connections
• the {connection} connection is reset after {duration} {dataDirection} on {int}% of connections for {duration}

Example:

Given the parcels-db connection is reset after 1s

Since 0.3.1.

faults.slowclose

Given the {connection} connection takes {duration} to close[[ {dataDirection}]][[ on {int}% of connections]][[ for {duration}]]

Delay the close of the connection by that long. What the connection carries before it closes is not delayed.

• coming back (the default) or going out is the direction it applies to.
• on 30% of connections applies it to that share of connections, by chance.
• for 5s ends it by itself after that long.
• It also hits the connections open now, and ends when the scenario does.

Parameter	Takes	For example
{connection}	a connection: the name of a proxy in Toxiproxy’s config	parcels-db, cache
{duration}	a duration in milliseconds (ms), seconds (s) or minutes (m)	5s, 2m
{dataDirection}	the direction of a connection’s data a fault applies to: coming back to the service, or going out from it	coming back, going out
{int}	a whole number	200, 3

Variants, the parts in [[...]] said or left out:

• the {connection} connection takes {duration} to close
• the {connection} connection takes {duration} to close for {duration}
• the {connection} connection takes {duration} to close {dataDirection}
• the {connection} connection takes {duration} to close on {int}% of connections
• the {connection} connection takes {duration} to close {dataDirection} for {duration}
• the {connection} connection takes {duration} to close on {int}% of connections for {duration}
• the {connection} connection takes {duration} to close {dataDirection} on {int}% of connections
• the {connection} connection takes {duration} to close {dataDirection} on {int}% of connections for {duration}

Example:

Given the parcels-db connection takes 2s to close

Since 0.3.1.

faults.pieces

Given the {connection} connection breaks into {int}-byte pieces, {duration} apart[[ {dataDirection}]][[ on {int}% of connections]][[ for {duration}]]

Send the connection’s data in pieces of that many bytes, that long apart, which tests a service that reads a whole answer in one read.

• coming back (the default) or going out is the direction it applies to.
• on 30% of connections applies it to that share of connections, by chance.
• for 5s ends it by itself after that long.
• It also hits the connections open now, and ends when the scenario does.

Parameter	Takes	For example
{connection}	a connection: the name of a proxy in Toxiproxy’s config	parcels-db, cache
{int}	a whole number	200, 3
{duration}	a duration in milliseconds (ms), seconds (s) or minutes (m)	5s, 2m
{dataDirection}	the direction of a connection’s data a fault applies to: coming back to the service, or going out from it	coming back, going out

Variants, the parts in [[...]] said or left out:

• the {connection} connection breaks into {int}-byte pieces, {duration} apart
• the {connection} connection breaks into {int}-byte pieces, {duration} apart for {duration}
• the {connection} connection breaks into {int}-byte pieces, {duration} apart {dataDirection}
• the {connection} connection breaks into {int}-byte pieces, {duration} apart on {int}% of connections
• the {connection} connection breaks into {int}-byte pieces, {duration} apart {dataDirection} for {duration}
• the {connection} connection breaks into {int}-byte pieces, {duration} apart on {int}% of connections for {duration}
• the {connection} connection breaks into {int}-byte pieces, {duration} apart {dataDirection} on {int}% of connections
• the {connection} connection breaks into {int}-byte pieces, {duration} apart {dataDirection} on {int}% of connections for {duration}

Example:

Given the parcels-db connection breaks into 100-byte pieces, 1ms apart

Since 0.3.1.

faults.closes.after

Given the {connection} connection closes after {int} KB[[ {dataDirection}]][[ on {int}% of connections]][[ for {duration}]]

Close the connection once that many thousands of bytes have passed through it, which leaves an answer cut off partway. A connection already open counts them from when the step runs.

• coming back (the default) or going out is the direction it applies to.
• on 30% of connections applies it to that share of connections, by chance.
• for 5s ends it by itself after that long.
• It also hits the connections open now, and ends when the scenario does.

Parameter	Takes	For example
{connection}	a connection: the name of a proxy in Toxiproxy’s config	parcels-db, cache
{int}	a whole number	200, 3
{dataDirection}	the direction of a connection’s data a fault applies to: coming back to the service, or going out from it	coming back, going out
{duration}	a duration in milliseconds (ms), seconds (s) or minutes (m)	5s, 2m

Variants, the parts in [[...]] said or left out:

• the {connection} connection closes after {int} KB
• the {connection} connection closes after {int} KB for {duration}
• the {connection} connection closes after {int} KB {dataDirection}
• the {connection} connection closes after {int} KB on {int}% of connections
• the {connection} connection closes after {int} KB {dataDirection} for {duration}
• the {connection} connection closes after {int} KB on {int}% of connections for {duration}
• the {connection} connection closes after {int} KB {dataDirection} on {int}% of connections
• the {connection} connection closes after {int} KB {dataDirection} on {int}% of connections for {duration}

Example:

Given the parcels-db connection closes after 50 KB

Since 0.3.1.

faults.closes.after.bytes

Given the {connection} connection closes after {int} bytes[[ {dataDirection}]][[ on {int}% of connections]][[ for {duration}]]

Close the connection once that many bytes have passed through it. A connection already open counts them from when the step runs, so a few hundred cut off even a short answer on a connection a service’s pool holds.

• coming back (the default) or going out is the direction it applies to.
• on 30% of connections applies it to that share of connections, by chance.
• for 5s ends it by itself after that long.
• It also hits the connections open now, and ends when the scenario does.

Parameter	Takes	For example
{connection}	a connection: the name of a proxy in Toxiproxy’s config	parcels-db, cache
{int}	a whole number	200, 3
{dataDirection}	the direction of a connection’s data a fault applies to: coming back to the service, or going out from it	coming back, going out
{duration}	a duration in milliseconds (ms), seconds (s) or minutes (m)	5s, 2m

Variants, the parts in [[...]] said or left out:

• the {connection} connection closes after {int} bytes
• the {connection} connection closes after {int} bytes for {duration}
• the {connection} connection closes after {int} bytes {dataDirection}
• the {connection} connection closes after {int} bytes on {int}% of connections
• the {connection} connection closes after {int} bytes {dataDirection} for {duration}
• the {connection} connection closes after {int} bytes on {int}% of connections for {duration}
• the {connection} connection closes after {int} bytes {dataDirection} on {int}% of connections
• the {connection} connection closes after {int} bytes {dataDirection} on {int}% of connections for {duration}

Example:

Given the parcels-db connection closes after 100 bytes

Since 0.3.1.

faults.loses

Given the {connection} connection loses {int}% of its data[[ {dataDirection}]][[ on {int}% of connections]][[ for {duration}]]

Drop that share of the connection’s data, by chance. The service gets no error: what it was waiting for just does not come.

• coming back (the default) or going out is the direction it applies to.
• on 30% of connections applies it to that share of connections, by chance.
• for 5s ends it by itself after that long.
• It also hits the connections open now, and ends when the scenario does.

Parameter	Takes	For example
{connection}	a connection: the name of a proxy in Toxiproxy’s config	parcels-db, cache
{int}	a whole number	200, 3
{dataDirection}	the direction of a connection’s data a fault applies to: coming back to the service, or going out from it	coming back, going out
{duration}	a duration in milliseconds (ms), seconds (s) or minutes (m)	5s, 2m

Variants, the parts in [[...]] said or left out:

• the {connection} connection loses {int}% of its data
• the {connection} connection loses {int}% of its data for {duration}
• the {connection} connection loses {int}% of its data {dataDirection}
• the {connection} connection loses {int}% of its data on {int}% of connections
• the {connection} connection loses {int}% of its data {dataDirection} for {duration}
• the {connection} connection loses {int}% of its data on {int}% of connections for {duration}
• the {connection} connection loses {int}% of its data {dataDirection} on {int}% of connections
• the {connection} connection loses {int}% of its data {dataDirection} on {int}% of connections for {duration}

Example:

Given the parcels-db connection loses 10% of its data

Since 0.3.1.

faults.toxic

Given the {connection} connection has a(n) {word} toxic with the following attributes:
  | attribute | value |

The table may start with the row naming its columns, as above, or go straight to its rows.

Give the connection any toxic Toxiproxy has by that type, one compiled into your own Toxiproxy included, with the attributes Toxiproxy’s documentation gives it.

• stream and toxicity are the toxic’s own: upstream or downstream, and the share of connections it applies to, from 0.0 to 1.0.
• Any other row is one of its attributes, of the type Toxiproxy reads it as.

Parameter	Takes	For example
{connection}	a connection: the name of a proxy in Toxiproxy’s config	parcels-db, cache
{word}	one word, with no spaces	parcels, PX-4101

Each row names an attribute of the toxic and its value. Values are typed: 42 an integer, 1.5 a decimal, true and false booleans, null null, {...} and [...] JSON, "42" in double quotes the string, and other text a string; 42 is not "42", nor 2 2.0.

Example:

Given the parcels-db connection has a latency toxic with the following attributes:
  | latency  | 2000       |
  | jitter   | 500        |
  | stream   | "upstream" |
  | toxicity | 0.5        |

Since 0.3.1.

faults.restored

When the {connection} connection is restored

End the faults this scenario gave the connection. A scenario’s faults end by themselves when it does; another scenario’s are left alone.

Parameter	Takes	For example
{connection}	a connection: the name of a proxy in Toxiproxy’s config	parcels-db, cache

Example:

When the parcels-db connection is restored

Since 0.3.1.

Parameter types

Parameter	Takes	Values	For example
{connection}	a connection: the name of a proxy in Toxiproxy’s config		parcels-db, cache
{dataDirection}	the direction of a connection’s data a fault applies to: coming back to the service, or going out from it		coming back, going out