# Warehouse Cell Test Plan and Scenario Catalogue

## 1. Objective

This plan verifies the POC from deterministic low-level behavior through the user-visible order-to-pallet workflow. It emphasizes inventory integrity, legal state changes, vehicle routing, picker/pallet behavior, fault/safety handling, ROS contract health, Gazebo observability, deterministic replay, and actionable failure evidence.

This is software verification of a simulation. It is not machinery qualification, safety certification, performance benchmarking of a physical cell, or validation of a commercial warehouse-control system.

## 2. Test levels

| Suite/marker | Test object | Main technique | Simulator |
|---|---|---|---|
| `unit` | Domain classes, routing, pallet pattern, scenarios, report conversion | Direct Python calls and parametrized boundary tests | Not used |
| `integration` | Generated ROS interfaces, running controller contract, graph/process behavior | Type construction, `rclpy` probe, service calls, state observation | Required for tests also marked `gazebo` |
| `system` | A test-owned deployed process set and its failure/recovery boundary | Exact PID/process-group signals, isolated DDS domain, replacement process, external state/diagnostic assertions | Not required by the current pause/restart cases |
| `e2e` | Complete cell workflows across controller, ROS, visualizer, and world | Black-box service/action drive plus asynchronous state/event assertions | Required; visual by default |
| `deployment` | Container image, orchestration manifest, HTTP probe gateway, and opt-in disposable-cluster recovery | Static parsing/CLI checks plus explicitly authorized kind Pod replacement | Gazebo intentionally excluded |
| `slow` | Scenarios unsuitable for the short developer loop | Longer timeout/repetition | Usually required |

Additional markers are `ros`, `gazebo`, `performance`, `characterization`, `concurrency`, `recovery`, `schema`, `observability`, `deployment`, `k8s_runtime`, and `packet_fault`. Performance/characterization mean that a test records measurements and a non-gating baseline; they do not turn a POC measurement into a production acceptance threshold. `k8s_runtime` is destructive local-infrastructure coverage and must be opted into explicitly. `packet_fault` requires Linux user/network namespaces plus `ip`/`tc` and impairs only its disposable namespace. Marker definitions are strict in `test/pytest.ini`; unknown markers fail configuration rather than silently weakening selection.

Coverage status in this document is evidence-based:

- **Executable:** a collected test contains the scenario ID. It counts as coverage for a run only when it executes rather than skips.
- **Planned:** acceptance criteria are defined, but a deterministic seam, fixture, implementation, or test is still absent.
- **Characterization:** the test records current behavior/measurements without asserting a production requirement that has not been established.

Use `pytest --collect-only` as the definitive status check at the current revision; prose must not turn a planned case into claimed coverage.

### 2.1 Final validation record

The recorded 2026-08-13 UTC headless `all` qualification contains **105 results: 104 passed, 0 failed, 0 errored, and 1 skipped** in 88.739 seconds. The sole skip is `WH-K8S-001`, whose test is collected but deliberately does not create a kind cluster without `--k8s-runtime`. All other collected unit, integration, system, end-to-end, deployment-static/gateway, packet-fault, and observability cases executed and passed. The evidence is [test-report.md](../artifacts/final-validation-wave2/test-report.md), [summary.json](../artifacts/final-validation-wave2/summary.json), and [junit.xml](../artifacts/final-validation-wave2/junit.xml).

This is not Kubernetes-runtime evidence. An explicit deployment-prerequisite run fails before cluster creation because `kind` and `kubectl` are absent, and this login also lacks Docker daemon/socket access. Once `--k8s-runtime` is requested, a missing prerequisite is a failure, not a skip.

## 3. Entry and environment controls

Before a ROS/Gazebo test:

- build and source ROS 2 Humble and the workspace overlay;
- record Gazebo Sim 8, Python, RMW, OS, commit, and dirty-tree state;
- allow the runner to select an isolated `ROS_DOMAIN_ID`/`GZ_PARTITION`, or pass an assigned domain explicitly;
- ensure no stale warehouse/Gazebo process is publishing in that domain;
- establish node, topic, service/action, controller-state, world, and entity readiness;
- create a unique artifact directory;
- record scenario, seed, launch arguments, timeout, and visual/headless mode.

Use the repository wrapper for lifecycle management:

```bash
./scripts/run_tests.sh --suite unit --no-sim
./scripts/run_tests.sh --suite integration
./scripts/run_tests.sh --suite system
./scripts/run_tests.sh --suite e2e
./scripts/run_tests.sh --suite performance
./scripts/run_tests.sh --suite recovery
./scripts/run_tests.sh --suite schema
./scripts/run_tests.sh --suite observability
./scripts/run_tests.sh --suite deployment
./scripts/run_tests.sh --suite deployment --k8s-runtime
./scripts/run_tests.sh --headless --suite all
```

`all`, `unit`, `integration`, `system`, and `e2e` select test directories. `performance`, `recovery`, `schema`, `observability`, and `deployment` select strict markers across `test/`; these focused suites do not start managed Gazebo by default. `--suite integration` selects only `test/integration/` and does not silently include `test/system/`. The deployment suite skips `WH-K8S-001` unless `--k8s-runtime` sets the explicit opt-in; after opt-in, missing Docker/kind/kubectl/curl prerequisites fail the requested qualification.

Visual mode is the default. `--headless` still runs Gazebo without its GUI; `--no-sim` prevents the runner from starting Gazebo and is appropriate for a pure suite or an already-running cell.

Managed ROS/Gazebo suites start from `scenario=empty`, `auto_start_orders=false`, seed `7`, and logical speed `8.0` so each test owns its stimulus and completes promptly. Individual tests immediately reset to their documented scenario/seed. These runner settings differ deliberately from the visual demo defaults (`mixed_wave`, automatic orders, `3.0×`).

## 4. Test data and repeatability

Named cell definitions provide stable initial conditions:

| Scenario | Purpose |
|---|---|
| `empty` | Populated, idle inventory and fleet; the test submits orders explicitly. |
| `mixed_wave` | Two mixed-SKU orders distributed across both picker cells. |
| `surge_wave` | Concurrent demand for queueing and fleet utilization. |
| `fault_recovery` | Controlled order used for an AGV or picker fault. |

Blocked-aisle geometry is covered at the A* routing unit layer. A dedicated blocked-world end-to-end scenario remains an extension; `WH-ROUTE-001` currently uses `surge_wave` to exercise contention and deadlock resistance.

The Python scenario implementation is the executable source of truth; the YAML catalogue is the operator-facing description. A contract test should keep the two representations aligned where both define the same data.

Every scenario accepts a seed. The same scenario, seed, and configuration must create the same entity IDs, initial poses, inventory, obstacles, and default orders. A failure report must contain the seed. Tests use explicit order IDs rather than relying on a previous run's state.

## 5. Common functional invariants

Assert these throughout, not only at the end:

1. Entity IDs are non-empty and unique within their type.
2. A case is stored, reserved, in transit, at a picker, or on a pallet—never owned by two locations.
3. At most one AGV carries a case and one AGV has the corresponding active transport ownership.
4. No order completes before all requested cases are palletized.
5. A case is palletized once; completed count does not exceed requested count.
6. AGV battery remains in `[0, 100]`, speed is non-negative, and positions are finite.
7. The configured collision/separation envelope is not violated in the domain model.
8. Emergency stop prevents affected movement/progression while asserted.
9. Fault injection and clearance produce observable, correlated state/events.
10. Each accepted request reaches one terminal outcome or a bounded, explicit timeout/failure reason.
11. Reset removes prior order/progress/fault state and reproduces the selected seed.
12. No expected rejection partially mutates inventory or cell state.

## 6. Data-intensive systems test model

This model turns the 2026 second-edition DDIA themes into ROS 2 test decisions. The POC now has an optional local SQLite durability exercise; it still does not imply a message broker, replicated/high-availability database, or production capacity model.

### 6.1 Distributed failure and recovery matrix

| Fault point | Client knowledge | Required oracle | Scenario/status |
|---|---|---|---|
| Request rejected with response | Known application rejection | Response plus unchanged `/warehouse/state` | Existing negative/live tests |
| Application abandons response after order mutation | Outcome is unknown to that application even though another observer can see commit | Retry the same `request_id`: equivalent content yields `REPLAYED`, with one business effect | Unit `WH-DIST-UNIT-001..002`; live `WH-DIST-001` executable |
| Application relay delays before forwarding | Authority has not been invoked during the configured pre-forward interval | Monotonic lower/upper response bound plus exactly one eventual order | `WH-NET-001`, executable system/integration |
| Application relay delays a response after commit | Client deadline expires even though the authority has committed | Independent state reconciliation; identical-key retry returns `REPLAYED`; one acceptance event | `WH-NET-002`, executable system/integration |
| Application relay drops before forwarding or holds a logical partition | Client initially has no response; configured relay guarantees zero authoritative calls before recovery | Zero pre-recovery orders; old call fails explicitly; only a deliberate post-`PASS` retry commits | `WH-NET-003`, executable system/integration |
| Fast DDS UDP packets are delayed/lost on isolated loopback | The real request, response, state, and DDS reliability traffic traverse a Linux qdisc rather than a service proxy | Measured latency; commit reconciliation after response timeout; no response/order visible during the loss window; one effect after heal | `WH-NETEM-001`, executable/passed system integration |
| Controller process paused | Process may remain discovered while callbacks/state publication stop | Monotonic deadline, frozen state/revision, stale diagnostic, coherent resume | `WH-PAUSE-001`, executable system/integration |
| Controller exits mid-order with SQLite enabled | Outstanding clients/actions fail independently from restorable domain state | Exact process loss, replacement state/run/revision, once-only completion, durable event evidence | `WH-RESTART-001`, executable system/integration |
| Controller exits after outbox publish but before published-mark commit | A subscriber may receive the same durable event again after restore | Same `event_id` twice at transport/application ingress, consumer effect once, row eventually marked, order completes once | `WH-OUTBOX-CRASH-001`, executable system/integration |
| Controller restores from optional SQLite checkpoint | A local committed checkpoint can retain in-flight work and idempotency data | Same run/revision/work ownership restores and completes once | Unit `WH-RESTART-UNIT-001`; live replacement `WH-RESTART-001` executable; launch-managed respawn specifically remains planned |
| Controller restarts without `state_db_path` | Default live work is not durable | Report fresh scenario state and lost-work limitation; do not call reset “recovery” | Planned default-mode variant; persistent `WH-RESTART-001` does not cover it |
| Late state subscriber | Observer missed history but needs current state | Compatible subscriber receives retained full snapshot | `WH-QOS-001`, executable integration |
| Late event subscriber | Volatile history is not promised | Subscribe before stimulus or reconstruct from state/query; no false durable-log claim | Covered conceptually by `WH-STREAM-001`; dedicated late-event case is future work |
| Kubernetes deletes one ready Pod using PVC state | Replacement is orchestrator-managed, but only the PVC can retain controller state | Different Pod UID, readiness recovery, same run, non-regressing revision, retained order | `WH-K8S-001` script implemented; runtime host-blocked/unvalidated; static `WH-K8S-STATIC-001..002` executable |

Every injected failure must name the exact boundary and synchronization observation. “Kill it at some point” is not reproducible. Prefer an explicit controller test hook; where an OS signal is intentionally tested, resolve and record the exact managed PID, preserve cleanup ownership, and use monotonic wall-clock supervision. The relay scenarios are not packet-level tests; `WH-NETEM-001` is the separate same-host loopback packet case. Neither boundary proves an external, asymmetric, long-duration production partition.

### 6.2 Transaction, concurrency, and stream oracles

The controller's process-local lock is an implementation detail, not the oracle. For concurrent order tests, synchronize clients at a barrier, then assert responses and authoritative state together: one owner per case, one reservation, no partial rejected order, and bounded completion. Repeat enough interleavings/seeds to expose timing sensitivity.

For stream tests, distinguish:

- `/warehouse/state`, a periodic full snapshot with schema/run/revision and reliable/transient-local/keep-last-1 controller QoS;
- `/warehouse/events`, a reliable/volatile live transition stream with schema/event identity, within-run sequence, correlation/causation, and depth 100;
- `/warehouse/commands`, an asynchronous ingress whose submit-order `command_id` is an idempotency key, but which has no broker-style durable ACK queue;
- `/warehouse/metrics`, a versioned JSON observability projection rather than authoritative state;
- the optional SQLite event outbox, which republishes pending events in sequence and marks after publication, giving at-least-once rather than exactly-once delivery.

Subscribe before stimulus except when testing late join. Scope sequences to `run_id`; correlate by request/order/entity ID; check unique event IDs, monotonic sequence for the observed run, absence of duplicate terminal **effects**, and agreement with the final snapshot/revision. If an outbox replay repeats a delivered event after publish-before-mark, a consumer should deduplicate its effect by `event_id`; tests still cannot claim exactly-once transport or global ordering across publishers.

`WH-OUTBOX-CRASH-001` makes the publish-before-mark window deterministic with an SQA-only failpoint. It requires controller generation 1 to publish `ORDER_ACCEPTED` and hard-exit with code 86 while that row is still `published=0`; generation 2 must replay the same `event_id`, after which the row becomes `published=1`. The raw subscriber must see two deliveries, its `EventIdDeduplicator` must apply one logical acceptance, and both authoritative state and durable rows must show one completed order/one `ORDER_COMPLETE`. `WH-STREAM-UNIT-003..004` separately gate sequential and concurrent event-ID claims. Because this deduplicator is in-memory, the result is apply-once only for one live consumer; consumer-crash durability requires a stored claim committed with the business effect and remains outside this POC.

### 6.3 Nonfunctional measurement method

Performance asks how the system behaves at a stated workload; scalability asks how behavior changes as workload or resources change. Report wall-time p50, p95, p99, maximum, sample count, failures/timeouts, and the percentile method. With a small sample, tail percentiles are characterization only.

`WH-PERF-001` is a deliberately smaller first step: it records p50/p95/p99 and failure rate for ten pure-engine waves, but not maximum, RSS/CPU, ROS callback/API latency, or Gazebo real-time factor. Those are explicit expansion items under this broader method.

Record these controls with every measurement: host/OS, ROS distribution and RMW, Gazebo version, visual/headless mode, scenario/seed, `tick_hz`, `simulation_speed`, warm-up, request count/concurrency, test timeout, and enabled debug/tracing. Use a monotonic wall clock for client latency and record logical plant time separately.

| Test shape | Workload | Measures | Interpretation |
|---|---|---|---|
| `WH-PERF-001` fixed-workload characterization | Ten seeded pure-engine `surge_wave` runs (3 orders/12 cases/4 AGVs/2 pickers) | p50/p95/p99 wall runtime, logical cycle time, throughput, and failure rate | Non-gating trend observation; correctness and logical deadline remain gates. Not ROS service latency, physical cycle time, or production capacity. |
| Load (planned extension) | Expected concurrent order wave | API/action latency, completion latency, throughput, queue depth, state gaps | Verify the expected operating point with functional invariants enabled. |
| Stress/capacity (planned) | Ramp demand until rejection/degradation | Tail latency, errors/timeouts, first limit, invariant violations | Rejection must remain explicit and mutation-safe. |
| `WH-SOAK-001` (planned) | Fixed waves or duration | RSS/CPU, backlog, stale-state gaps, cleanup, ID reuse, latency trend | Detect resource/state degradation over time. |
| `WH-SCALE-001` fleet scalability | Same 3-order/12-case/two-picker surge wave with 1/2/4 AGVs, replayed five times per configuration | Deterministic logical cycle and items/min values, gain between fleet sizes, repeatability, failures/invariants | Deterministic engine characterization: useful 1→2 gain and diminishing 2→4 return; repeats are not independent workload samples, and this is not live ROS or physical capacity. |

Do not baseline performance in default visual mode, under GDB, or with verbose tracing. Those modes remain useful for functional observation and diagnosis, and their perturbation must be reported if used.

Current `WH-SCALE-001` observations are 217.25 s/3.314 items per minute for one AGV, 116.10 s/6.202 for two, and 115.60 s/6.228 for four. Median logical cycle falls 46.559% from one to two and only 0.431% from two to four. The test interprets the latter as approach to the fixed two-picker/dock bottleneck. Exact values are reportable baselines, while the gates are completion, no invariant violations, two faster than one, four no slower than two, and a smaller second incremental gain.

### 6.4 Encoding and schema evolution method

Version 1 of `/warehouse/metrics` requires `schema_version`, `scenario`, `simulation_time_sec`, `completed_orders`, `palletized_items`, `throughput_items_per_min`, and `emergency_stop` with stable value types. The current producer also emits `run_id`, `revision`, and `queue_depth`; an older version-1 document receives safe defaults for those three fields. The umbrella `WH-SCHEMA-001` scenario covers current round-trip, older/additive data, and actionable malformed/type errors.

ROS `.msg`, `.srv`, and `.action` evolution is a different problem. A future mixed-version qualification must build isolated old/new publishers and subscribers, list supported combinations, verify discovery/deserialization/semantics, and replay representative bags where supported. An additive IDL field is not assumed compatible without evidence. The JSON case does not claim ROS IDL compatibility.

## 7. Executable test index

### 7.1 Low-level tests

| Scenario ID / file | Coverage |
|---|---|
| `WH-UNIT-MODEL-001..004` — `test/unit/test_models.py` | Bounded pose movement, safe negative movement, yaw quaternion, derived order counts. |
| `WH-ENGINE-001..008`, `WH-SAFE-UNIT-001`, `WH-FAULT-UNIT-001..002`, `WH-SCALE-UNIT-001..002` — `test/unit/test_engine.py` | Fulfillment/inventory, parallel waves, validation atomicity, duplicate IDs, stop/resume, AGV/picker faults, deep snapshots, replay/event draining, and stable supported fleet-size selection/rejection. |
| `WH-UNIT-PALLET-001..003` — `test/unit/test_pallet_pattern.py` | Unique balanced first layer, interlocked second layer, invalid index rejection. |
| `WH-ROUTE-UNIT-001..006` — `test/unit/test_routing.py` | Exact endpoint, blocked-aisle detour, determinism, invalid endpoints, no-route condition, resolution validation. |
| `WH-UNIT-SCENARIO-001..003` — `test/unit/test_scenarios.py` | Populated topology, fresh independent builds, unknown-scenario diagnostics. |
| `WH-DIST-UNIT-001..002`, `WH-RESTART-UNIT-001`, `WH-STREAM-UNIT-001..002`, `WH-SCALE-UNIT-003` — `test/unit/test_state_store.py` | Idempotent replay/conflict, SQLite restoration of in-flight work, ordered at-least-once outbox behavior, correlation/causation across restore, and preservation of configured fleet size. |
| `WH-STREAM-UNIT-003..004` — `test/unit/test_event_dedup.py` | Duplicate `event_id` applies once, and one of 64 competing callback claims wins atomically. |
| `WH-SCHEMA-001` — `test/unit/test_metrics_schema.py` | Version-1 metrics round-trip, older/additive document compatibility, and actionable malformed/type errors. |
| `WH-K8S-STATIC-001..002` — `test/unit/test_deployment_assets.py` | Manifest wiring for Namespace/PV/PVC/Deployment/Service/probes, one-replica `Recreate`, non-root image contract, and bounded disposable-cluster smoke CLI syntax/help. |
| `test/unit/test_junit_to_jira.py` | JUnit parsing, summary totals, Jira-ready Markdown, JSON, invalid/missing input errors. |
| `test/unit/test_cli_scripts.py` | Runner help exposes visual/headless/debug selection; report CLI writes artifacts and propagates failed-test status on request. |

Recommended additions as the POC evolves are a complete state-transition table, inventory reservation rollback, exact/over pallet capacity, each supported fault timer, emergency-stop idempotence, and event-code/correlation tests.

### 7.2 Interface and live integration tests

| Scenario ID / file | Coverage |
|---|---|
| `WH-INT-001` — `test/integration/test_ros_interface_contract.py` | `CellState` composes every entity type. |
| `WH-INT-002` — same file | Service/action SKU and quantity arrays remain aligned. |
| `WH-INT-003` — same file | Fault and stop services return diagnostic success/message fields. |
| `WH-INT-LIVE-001` — `test/integration/test_live_cell_contract.py` | Running state stream contains four AGVs, two pickers, populated unique inventory, finite/valid telemetry. |
| `WH-INT-LIVE-002` — same file | Reset with the same seed reproduces state and clears progress. |
| `WH-INT-LIVE-003` — same file | Invalid order is rejected without cell mutation. |
| `WH-DIST-001` — `test/integration/test_idempotent_retry_contract.py` | A short-lived client abandons its successful response; independent state proves commit; a new client receives idempotent `REPLAYED` with one order/reservation/event effect. |
| `WH-TXN-001`, `WH-STREAM-001`, `WH-QOS-001` — `test/integration/test_distributed_semantics.py` | Concurrent scarce-inventory atomicity/idempotent replay, causal unique event-chain reconciliation, and transient-local state versus volatile-event late-join behavior. |
| `WH-PERF-001` — `test/integration/test_performance_characterization.py` | Ten seeded pure-engine surge waves emit non-gating p50/p95/p99 wall/cycle/throughput properties while correctness and logical deadline remain gates. |
| `WH-SCALE-001` — `test/integration/test_fleet_scalability.py` | Five identical deterministic replays at 1/2/4 AGVs establish repeatability and characterize logical cycle/throughput and diminishing returns for one fixed surge workload while gating completion/invariants. |
| `WH-K8S-GATEWAY-001` — `test/integration/test_health_gateway.py` | HTTP `/healthz` stays live independently, `/readyz` changes from waiting to ready to stale with ROS-state freshness, and `/state`/`/metrics` preserve run/revision projection. |
| `WH-GZ-CONVERGE-001` — `test/integration/test_gazebo_pose_convergence.py` | Native Gazebo model pose converges with a materially moved, stopped AGV's authoritative ROS state within tolerance. |

Interface construction tests prove generated types are available and structurally compatible. They do not prove that a live server implements the contract; that is why both static interface and black-box live tests exist.

### 7.3 Process/system tests

| Scenario ID / file | Coverage |
|---|---|
| `WH-PAUSE-001` — `test/system/test_controller_pause_recovery.py` | Exact controller `SIGSTOP` freezes state/revision, health monitor reports stale ERROR, and `SIGCONT` restores advancing coherent state and healthy diagnostics in an isolated DDS domain. |
| `WH-RESTART-001` — `test/system/test_controller_restart_recovery.py` | `SIGKILL` one persistent controller mid-order, start a replacement on the same temporary SQLite store, restore run/revision/order, and complete exactly once with one ROS and outbox completion event. |
| `WH-NET-001..003` — `test/system/test_network_fault_relay.py` | Deterministic pre-forward latency, post-commit response delay/client timeout/idempotent reconciliation, and pre-forward drop/partition with explicit recovery retry. |
| `WH-NETEM-001` — `test/system/test_netem_packet_faults.py` | Forces Fast DDS to UDPv4 and uses real `tc netem` loopback delay/loss in a disposable user/network namespace to prove latency, unknown-outcome retry, partition healing, and one effect. |
| `WH-OUTBOX-CRASH-001` — `test/system/test_outbox_duplicate_delivery.py` | Hard-exit after first publish/before mark, same-ID replay after SQLite restore, two raw deliveries but one live-consumer effect and one completion. |
| `WH-OBS-GDB-001` — `test/system/test_gdb_observability.py` | Runs the installed controller under GDB, breaks at `rcl_wait`, and asserts full-thread ROS/rclpy/CPython/RMW stack evidence. |

`WH-PAUSE-001` uses OS process signals intentionally and owns the exact child process groups it starts. It verifies pause/resume, not process exit, SQLite restore, respawn, action-client recovery, failover, or disaster recovery.

`test/deployment/test_kubernetes_restart.py` is the opt-in pytest wrapper around `scripts/k8s_smoke.sh` for `WH-K8S-001`. The final non-opted-in run collected and skipped it; the explicit opt-in prerequisite run failed before the runtime scenario because `kind`/`kubectl` are missing and Docker socket access is unavailable. Its passing static/gateway companion tests are not Pod-restart evidence.

### 7.4 End-to-end tests

| Scenario ID / file | Coverage |
|---|---|
| `WH-NOM-001` — `test/e2e/test_nominal_fulfillment.py` | Three-case mixed-SKU order, visible AGV motion, correct picker, palletization and aggregate counts. |
| `WH-NOM-002` — same file | Two independent orders flow through both picker stations. |
| `WH-ROUTE-001` — `test/e2e/test_routing_resilience.py` | Three surge orders route through multiple AGVs/pickers without collision, fault, or deadlock. |
| `WH-SAFE-001` — `test/e2e/test_safety_and_recovery.py` | Simulated emergency stop halts motion, holds position, clears, and resumes. |
| `WH-FAULT-001` — same file | Active loaded AGV stalls, exposes the fault, is cleared, and completes coherently. |
| `WH-FAULT-002` — same file | Timed picker jam becomes visible, self-clears, and preserves/palletizes its case. |
| `WH-ACTION-001` — `test/e2e/test_action_fulfillment.py` | Fulfill-order feedback/result and final authoritative cell state agree. |

Use collection as the definitive current index:

```bash
PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 \
  python3 -m pytest -c test/pytest.ini test --collect-only -q
```

A collected test that is skipped because ROS/Gazebo is unavailable is not functional coverage for that run. Review skip reasons in `pytest -ra` and the Markdown report.

## 8. Detailed functional scenarios

### `WH-NOM-001`: mixed-SKU order to pallet

**Purpose:** prove the principal goods-to-picker workflow.

**Preconditions:** reset `empty` with seed `1001`; four healthy AGVs; `picker_01` available; at least two RED and one BLUE cases.

**Stimulus:** submit `ORD-E2E-NOM-001` for RED ×2 and BLUE ×1 to `picker_01`.

**Required observations:**

- submission is accepted;
- order leaves its initial queued state;
- at least one AGV travels and case ownership becomes in-transit;
- each selected case arrives at the requested picker;
- three distinct cases become `PALLETIZED` once;
- requested SKU multiset is exactly `{RED: 2, BLUE: 1}`;
- order reaches `COMPLETE` with `3/3`;
- aggregate completed order and palletized counts change by one and three;
- no invariant violation, unexpected fault, or process failure occurs.

**Visual checks:** cases follow the assigned AGV and appear at `picker_01`/its pallet. These checks supplement ROS assertions.

### `WH-NOM-002`: parallel picker utilization

**Purpose:** prove two independent picker flows can overlap without state contamination.

**Preconditions:** reset `empty` with seed `1002`; both pickers healthy; sufficient GREEN and YELLOW stock.

**Stimulus:** submit two two-case orders targeted to `picker_01` and `picker_02`.

**Required observations:** both orders exist simultaneously, both picker stations show queued/active work, each order remains assigned to its target picker, and both reach `COMPLETE` with the correct case counts.

**Failure risks:** a global single-picker bottleneck, wrong order-to-picker correlation, shared mutable order list, duplicate case reservation, or premature aggregate completion.

### `WH-ROUTE-001`: surge-wave routing and contention

**Purpose:** demonstrate useful fleet concurrency without collision or deadlock at the single-dock picker cells.

**Preconditions:** reset `surge_wave` with seed `2001`; four healthy AGVs; both pickers available; the scenario's 12 requested cases are in stock.

**Stimulus:** submit `surge_2001` (RED ×2, PURPLE ×2) to `picker_01`, `surge_2002` (GREEN ×2, BLUE ×2) to `picker_02`, and `surge_2003` (YELLOW ×2, ORANGE ×2) to `picker_01`.

**Required observations:** all three correlated orders exist; at least two vehicles perform non-idle work; one-dock ownership remains coherent; all three orders reach `COMPLETE`; exactly 12 requested cases are completed; at least two AGVs travel more than one metre; no vehicle ends faulted; no separation invariant or deadlock occurs.

**Low-level companion:** `WH-ROUTE-UNIT-002` proves a route detours around a blocked aisle with clearance, while `WH-ROUTE-UNIT-005` proves a complete barrier produces `NoRouteError`. A future Gazebo-world blocked-aisle scenario can extend this coverage without overstating the current e2e case.

### `WH-SAFE-001`: emergency stop and controlled resume

**Purpose:** verify the POC's simulated stop-state contract.

**Preconditions:** reset with seed `3001`; submit a three-case RED order; observe an AGV moving.

**Stimulus:** assert `/warehouse/emergency_stop` with a correlated reason; observe; reset the stop.

**Required observations:** service succeeds, state exposes `emergency_stop=true`, all commanded speeds fall to the stopped tolerance, poses remain within tolerance during the hold, no case is palletized due to movement while held, clear is acknowledged, legal work resumes, and the original order completes once.

This verifies software behavior only. It is not a test of emergency-stop hardware, safe torque off, braking distance, safety PLCs, or a certified stop category.

### `WH-FAULT-001`: loaded vehicle stall and recovery

**Purpose:** verify that a fault does not lose or duplicate an in-transit case.

**Preconditions:** reset with seed `4001`; submit a three-case BLUE order to `picker_02`; identify a moving, loaded AGV.

**Stimulus:** inject the supported drive-stall fault on that AGV, observe the held state, then call `/warehouse/clear_fault`.

**Required observations:** the target shows a non-empty fault and near-zero speed; the order remains non-terminal while work is blocked; the carried case retains one owner; clear succeeds; fault becomes empty; work resumes or is coherently reassigned; order reaches `COMPLETE` with `3/3`; no case is duplicated.

Fault-name spelling is part of the service contract. Tests and documentation must use the canonical value accepted by the running controller and should assert that unsupported values are rejected with choices.

### `WH-FAULT-002`: timed picker jam

**Purpose:** verify timed picker-fault expiry without losing or duplicating the case assigned to the station.

**Preconditions:** reset `fault_recovery` with seed `4002`; `picker_01` available; one ORANGE case in stock.

**Stimulus:** submit a one-case order, then inject `PICKER_JAM` on `picker_01` for 20 simulated seconds.

**Required observations:** the service accepts the supported fault; picker state reports `PICKER_JAM`; the order remains non-terminal while jammed; the timer clears the fault; the order reaches `COMPLETE`; exactly one matching case exists and is `PALLETIZED`.

### `WH-ACTION-001`: fulfill-order feedback and result

**Purpose:** validate the long-running ROS action as an operations or SQA client experiences it, not only the underlying order service/state.

**Preconditions:** reset `empty` with seed `5001`; `picker_02` available; one RED and one BLUE case in stock.

**Stimulus:** send `/warehouse/fulfill_order` goal `ORD-E2E-ACTION-001` for RED ×1 and BLUE ×1 to `picker_02`.

**Required observations:** the goal succeeds; at least one feedback sample is received; every sample reports total `2`; completed counts are monotonic and within `{0,1,2}`; active vehicle IDs belong to the four known AGVs; result reports two palletized cases and positive logical duration; `/warehouse/state` contains the same order at `COMPLETE` with `2/2`.

Goal rejection, cancellation races, and server disappearance remain important negative action extensions. They are not implied by this nominal action case.

### `WH-TXN-001`: concurrent last-case reservation

**Status:** executable live integration.

**Purpose:** translate the lost-update problem into a ROS service race without treating the controller lock itself as proof.

**Preconditions:** reset `empty` with seed 6101 and verify exactly four RED cases are stored; prepare two independent clients with distinct request/order IDs, each requesting RED ×3 at a different picker.

**Stimulus:** release two `/warehouse/submit_order` calls from a synchronization barrier so their callbacks can overlap in the multithreaded executor.

**Required observations:** both futures complete within monotonic deadlines; exactly one request is `ACCEPTED`; the other is `REJECTED` for insufficient inventory rather than timing out; one order exists with three unique owned cases; no rejected-order record, negative inventory, duplicate reservation, or deadlock occurs. Retrying the winner's identical `request_id` returns `REPLAYED` and still leaves one order/three owned cases.

### `WH-STREAM-001`: state/event correlation and ordering

**Status:** executable live integration; the SQLite outbox and lineage across restore have separate unit coverage in `WH-STREAM-UNIT-001..002`.

**Purpose:** prove that the live event projection carries enough identity/order context to reconcile with authoritative snapshots.

**Preconditions:** reset `empty`; subscribe to `/warehouse/state` with the documented transient-local profile and `/warehouse/events` with the documented volatile profile before stimulus; capture the active `run_id` and initial revision/sequence.

**Stimulus:** submit one uniquely correlated order and allow it to reach its terminal state.

**Required observations:** state schema/run metadata is non-empty and revisions do not decrease; event schema version is supported; IDs are unique; sequences and simulation times do not decrease; every order event retains the request correlation; the first event has no causation and each later event cites the prior event ID; each case has the exact `TRANSPORT_ASSIGNED` → `CASE_LOADED` → `CASE_DELIVERED` → `PICK_STARTED` → `CASE_PALLETIZED` chain; one `ORDER_COMPLETE` and final authoritative state agree. Gaps must be reported with QoS/discovery context rather than silently converted into pass.

This test does not prove a durable audit history or exactly-once transport. With the optional outbox, publish-before-mark can redeliver an event and consumers must deduplicate by `event_id`.

### `WH-QOS-001`: transient-local late state subscriber

**Status:** executable live integration.

**Purpose:** verify the recovery promise made by the implemented `/warehouse/state` QoS, independently of periodic waiting assumptions.

**Preconditions:** allow the controller to publish a coherent state before the test creates its subscriber.

**Stimulus:** create a compatible reliable/transient-local subscriber after publication.

**Required observations:** the late subscriber receives a bounded, coherent current snapshot containing the pre-join order and a non-empty `run_id`; because `/warehouse/events` is volatile, it does not receive the pre-join `ORDER_ACCEPTED` event ID; after it injects a new picker fault, it does receive that post-join event with a later sequence. This proves the contrast between retained state and live-only event history, not an audit-log guarantee.

### `WH-PERF-001`: seeded surge-wave characterization

**Status:** executable pure-engine integration/characterization, not a live ROS test or production benchmark.

**Purpose:** make percentile reporting over a deterministic warehouse workload executable while preserving functional correctness.

**Preconditions:** record host/software configuration and run without GDB/tracing. The fixed workload is ten `surge_wave` engines with seeds 7101–7110; each wave contains three default orders, 12 cases, four AGVs, and two pickers.

**Stimulus:** start a monotonic high-resolution timer, advance each pure engine through `run_until` with `dt_sec=0.05`, and capture wall runtime, final logical cycle time, and throughput.

**Required observations:** all ten waves complete within the 180-second logical deadline with no invariant violations; JUnit `measurement.*` properties contain sample count, failure rate, p50/p95/p99 wall milliseconds, p95 logical cycle seconds, and p50 throughput; percentile order is coherent. The report renders these separately as non-SLO characterization. Timing values are explicitly non-gating and are not ROS service/action latency, simulator real-time factor, visual motion, or physical cycle time.

### `WH-SCALE-001`: 1/2/4-AGV diminishing-return characterization

**Status:** executable deterministic integration/characterization in `test/integration/test_fleet_scalability.py`; not a live ROS/Gazebo or host-performance benchmark.

**Preconditions:** use the same `surge_wave` definition for every run: three orders, 12 cases, and two fixed single-dock picker cells. Replay seeds 7201–7205 for each supported fleet prefix of one, two, and four AGVs. The current model has no seeded process noise, so these are deliberate repeatability checks—not five statistically independent samples.

**Stimulus:** submit each wave's default orders and advance the pure engine at `dt_sec=0.05` until every order completes or the 240-second logical deadline expires.

**Required observations:** all 15 waves complete without invariant violations; report five samples per fleet, median logical cycle, median logical items/minute, and incremental cycle reductions. The current baseline is 1 AGV = 217.25 s/3.314 items/min, 2 = 116.10/6.202, and 4 = 115.60/6.228. Two AGVs must be faster than one, four no slower than two, and the 2→4 gain must be smaller than the 1→2 gain (currently 0.431% versus 46.559%). This identifies a modeled fixed-picker/dock bottleneck for this workload; it does not state production capacity, physical cycle time, or universal saturation behavior.

### `WH-SCHEMA-001`: versioned metrics compatibility

**Status:** executable unit coverage in `test/unit/test_metrics_schema.py`.

**Purpose:** protect the JSON monitoring boundary without conflating it with ROS IDL compatibility.

**Required observations:** a current producer document round-trips with schema/run/revision/queue data; an older version-1 document receives documented defaults and preserves an additive unknown field; invalid JSON, non-object data, missing required fields, and wrong known-field types raise actionable errors. These cases do not prove that old and new generated ROS message types interoperate.

### `WH-PAUSE-001`: controller process pause and resume

**Status:** executable slow system/integration coverage in `test/system/test_controller_pause_recovery.py`.

**Purpose:** show that an alive/discovered authority can be non-progressing and that the independent monitor exposes the condition.

**Preconditions:** the test launches installed `cell_controller` and `health_monitor` itself in an isolated DDS domain, records their exact process groups, and waits for at least three coherent states plus healthy diagnostics.

**Stimulus:** send `SIGSTOP` to the exact controller process group, then `SIGCONT` after stale state is detected.

**Required observations:** monitor status becomes `ERROR` with a stale-state reason; state logical time and revision remain frozen during the observation window; both clients/processes stay bounded; after resume, simulation time/revision advance, diagnostics return healthy, the four-AGV/unique-inventory state remains coherent, and cleanup resumes a stopped child before terminating its group.

This is a process-pause test, not an in-flight-order, crash, persistence, failover, or disaster-recovery test.

### `WH-DIST-001`: abandoned response after commit

**Status:** executable live integration in `test/integration/test_idempotent_retry_contract.py`; unit replay/conflict behavior is covered by `WH-DIST-UNIT-001..002`.

**Stimulus:** a short-lived application node sends RED ×2 with a unique `request_id`/order ID but never spins or reads its service future. An independent subscribed test client proves the order was committed, after which the short-lived node is destroyed. A new client retries identical content under the same `request_id`.

**Required observations:** the retry is accepted with disposition `REPLAYED`; exactly one order with two owned/reserved items exists; exactly one correlated `ORDER_ACCEPTED` event is observed; its `event_id` is non-empty and sequence positive; no second business effect occurs.

This is a deterministic **client-side response abandonment** seam. DDS may have delivered the response to middleware; the application intentionally never observes it. The test does not inject packet loss, prove network-partition behavior, or measure a timeout path.

### `WH-NET-001..003`: deterministic application-boundary communication faults

**Status:** executable slow ROS system/integration coverage in `test/system/test_network_fault_relay.py`; all three scenarios use a test-owned isolated DDS domain.

**Test boundary:** the client calls `/warehouse_fault_proxy/submit_order`, while `/warehouse_fault_proxy/configure` selects a `ConfigureNetworkFault` mode. The relay forwards to the real `/warehouse/submit_order` authority. This is an application service proxy—not `netem`, packet loss, a DDS transport fault, discovery churn, an asymmetric link, or physical-network qualification.

- `WH-NET-001` configures `DELAY_BEFORE_FORWARD=0.35` seconds, submits one case, and requires `ACCEPTED`, exactly one authoritative order, and measured monotonic round-trip `>=0.30` and `<3.0` seconds.
- `WH-NET-002` configures `DELAY_AFTER_COMMIT=0.80` seconds and applies a real 0.20-second client deadline. The original future must still be incomplete at the deadline while authoritative state already contains exactly one order. After switching to `PASS`, the identical `request_id`/payload must return `REPLAYED`; the delayed original eventually completes; only one correlated `ORDER_ACCEPTED` exists.
- `WH-NET-003` proves `DROP_BEFORE_FORWARD` and `PARTITION` requests exceed short client observation deadlines but create zero orders. Each old call eventually fails with its explicit fault disposition and is never forwarded invisibly. After `PASS`, a new deliberate retry commits once; the earlier dropped order remains absent.

The strongest oracle is not “the client timed out”; it is the combination of configured fault position, monotonic deadline, proxy result, authoritative state, event count, and idempotent retry disposition.

### `WH-NETEM-001`: real Fast DDS UDP loopback impairment

**Status:** executable slow packet-fault/system/integration coverage in `test/system/test_netem_packet_faults.py`; passed in the final full run in 6.807 seconds.

**Preconditions:** Linux must provide `unshare`, `ip`, `tc`, and unprivileged user/network namespaces. The test creates a disposable namespace, enables its loopback device, chooses an isolated ROS domain, and sets `FASTDDS_BUILTIN_TRANSPORTS=UDPv4`. That last control is part of the oracle: Fast DDS shared-memory delivery would otherwise bypass the loopback qdisc and make the packet experiment invalid. Do not add `ROS_LOCALHOST_ONLY` to this Humble-specific fixture without re-proving the transport path; its override can defeat the forced transport selection.

**Stimulus and required observations:** apply 180 ms one-way `tc netem` delay and require at least 0.28 seconds observed round trip plus one order. Next apply 300 ms one-way delay: the authority must commit, but its response must miss a 0.45-second real client deadline; after removing impairment, same-key retry returns `REPLAYED` and state contains one effect. Finally apply 100% loopback packet loss: no response and zero commit may appear during the 0.40-second window; after healing, explicitly retry and require one order. Reliable DDS may retransmit the original after heal, so the winner can return `ACCEPTED` or `REPLAYED`; business effect count—not which racing response wins—is the invariant.

The test stores observed round-trip/deadline values and structured network evidence in JUnit properties. It proves real qdisc impairment of same-host Fast DDS UDPv4 traffic. It does not cover external interfaces or hosts, asymmetric paths, discovery scale, sustained congestion, reordering/corruption/MTU faults, clock skew, or production network/security configuration.

### `WH-OUTBOX-CRASH-001`: publish-before-mark duplicate replay

**Status:** executable slow ROS system/integration coverage in `test/system/test_outbox_duplicate_delivery.py`, with consumer primitives covered by `WH-STREAM-UNIT-003..004`.

**Preconditions:** own an isolated DDS domain and temporary SQLite database; subscribe before stimulus; run controller generation 1 with restore enabled and the SQA-only `ORDER_ACCEPTED` crash-after-publish failpoint armed.

**Stimulus:** submit one idempotent order. Generation 1 must publish its acceptance and exit via `os._exit(86)` before the outbox row mark. Verify that exact row remains `published=0`, then start generation 2 against the same database.

**Required observations:** before restart there is one `ORDER_ACCEPTED`; after restore the same non-empty `event_id` is delivered once more; raw deliveries for that ID equal two; one live `EventIdDeduplicator` applies the acceptance once and records a duplicate; the replay row becomes `published=1`; the order completes once; state reports `completed_orders=1`; and ROS/logical plus SQLite evidence contains one `ORDER_COMPLETE` effect. The test proves at-least-once producer replay and live-process consumer deduplication, not exactly-once transport or consumer dedup state that survives its own crash.

### `WH-RESTART-001`: persisted controller crash/restart

**Status:** executable slow system/integration coverage in `test/system/test_controller_restart_recovery.py`; pure restoration also has `WH-RESTART-UNIT-001` coverage.

**Preconditions:** the test owns an isolated DDS domain and unique temporary SQLite path, launches controller generation 1 with `restore_from_checkpoint:=true`, submits a two-case idempotent order, and waits until it is active/in flight with an observable checkpoint.

**Stimulus:** send `SIGKILL` to the exact first controller process group, then start controller generation 2 against the same SQLite path.

**Required observations:** generation 2 publishes the same non-empty `run_id`; its first restored revision does not regress and subsequent revision advances; the same order and in-flight ownership restore; the order reaches `COMPLETE` with `2/2` once; final aggregate completion is one; exactly one matching `ORDER_COMPLETE` is observed on ROS and stored in the event outbox; the final checkpoint has the same run and at least the final revision; both process groups receive bounded cleanup.

The executable test starts the replacement explicitly; it does not prove the launch file's `controller_respawn` mechanism. It also does not prove that an action future owned by the killed server reconnects, nor corrupt SQLite, host/disk loss, multi-controller access, backup/restore, RPO/RTO, or standby takeover. This is single-active-controller recovery, not failover or disaster recovery.

### `WH-OBS-GDB-001`: runtime native-stack evidence

**Status:** executable system/ROS/observability coverage in `test/system/test_gdb_observability.py`; validated artifact retained at `artifacts/observability/gdb-cell-controller-full-stack.txt`.

**Stimulus:** start the installed `warehouse_core/cell_controller` with `/usr/bin/python3` under batch GDB, break at native `rcl_wait`, and capture thread inventory, `thread apply all bt full`, and loaded shared libraries.

**Required observations:** the helper exits successfully and the artifact contains `rcl_wait`, `rclpy/_rclpy_pybind11`, CPython `_PyEval_EvalFrameDefault`, `rmw_fastrtps`, a `Thread 1` record, and the native-library section. The retained validation artifact is 744 lines and includes Fast DDS worker threads. This verifies useful native observability through a Python ROS node; it is not a Python-logic test, performance measurement, or proof that every crash preserves symbols/locals.

### `WH-K8S-STATIC-001..002`, `WH-K8S-GATEWAY-001`, and `WH-K8S-001`: container/Pod recovery

**Static/gateway status:** executable unit checks in `test/unit/test_deployment_assets.py` parse `deploy/k8s/warehouse-cell.yaml` and assert Namespace/PV/PVC/one-replica `Recreate` Deployment/controller-plus-gateway containers, PVC-backed SQLite parameter, `/readyz` and `/healthz` probes, matching ClusterIP Service selector/port, non-root ROS Humble image, and bounded smoke-script CLI syntax/help. `WH-K8S-GATEWAY-001` in `test/integration/test_health_gateway.py` separately proves liveness, waiting/ready/stale readiness, run/revision state projection, and Prometheus text without a cluster.

**Runtime status:** `test/deployment/test_kubernetes_restart.py` wraps `scripts/k8s_smoke.sh` when `--suite deployment --k8s-runtime` explicitly authorizes a new disposable cluster. It was the sole deliberate skip in the final non-opted-in `all` run. A separate explicit opt-in attempt failed its prerequisite check before cluster creation because `kind`/`kubectl` are absent; Docker daemon/socket access is also unavailable. Report Pod recovery as blocked/unvalidated, not skipped-as-pass or inferred from the static/gateway tests.

**Runtime stimulus/oracle:** create a uniquely named disposable kind cluster, build/load the image, apply the manifest, and wait for `/readyz`; capture `/state` with a non-empty durable order; record Pod UID, `run_id`, and revision; deliberately delete the whole Pod; require a different UID and readiness; then assert the same run, non-regressing revision, and retained order through the PVC. Preserve image-build, kind, apply, Pod, state, and all-container logs. This is local single-node Pod replacement with `hostPath` storage, not controller-only crash/hang, probe-triggered restart, node rescheduling, rolling HA, cloud-volume portability, or Gazebo coverage. `/healthz` covers only the gateway process; stale controller state makes `/readyz` fail but the manifest has no controller liveness probe.

## 9. Boundary and negative cases

These cases are important even when not yet represented by a dedicated e2e test:

| Area | Boundary/negative case | Expected behavior |
|---|---|---|
| Order input | Empty ID, empty lines, zero quantity, SKU/quantity length mismatch | Reject with diagnostic; no mutation. |
| Order identity | Duplicate ID | Reject/idempotently return prior result per documented policy; never create duplicate work. |
| Inventory | Exact availability, shortage by one, unknown SKU | Exact accepts; shortages reject without partial reservation. |
| Picker | Unknown picker, jam during active pick, future queue-capacity boundary | Explicit rejection/hold/recovery; one case owner. Queue limits require a future configured policy. |
| Vehicle | Low battery, localization loss, persistent stall, fault clear without fault | Excluded/held/recovered per policy; diagnostic event. |
| Routing | Start/goal obstructed, complete barrier, narrow clearance | Contextual no-route result; never cross obstacle. |
| Pallet | First/second layer, negative index, future upper capacity | Current pattern yields unique legal poses and rejects negative indices; upper capacity requires a new configured limit/test. |
| Stop | Assert twice, clear twice, stop during load/unload | Idempotent response; no duplicate transfer on resume. |
| Reset | Reset during work; different seed; unknown scenario | Defined cancellation/reset semantics and fresh state, or explicit rejection. |
| ROS action | Cancel before acceptance, during travel, during pick; server exit | One terminal action state and coherent domain ownership. |
| QoS | Late state subscriber, slow event subscriber, incompatible profile | Behavior matches documented QoS; loss/recovery is visible. |
| Time | Gazebo pause, logical reset, slow wall timer, future `/clock` mode | Current split-clock behavior is explicit and bounded; a future coupled mode cannot hang or duplicate/burst transitions. |
| Shutdown | SIGINT during work, Gazebo exit, controller crash | Bounded group cleanup and non-zero/diagnostic result where appropriate. |

## 10. Asynchronous test method

The `WarehouseRosClient` intentionally uses monotonic deadlines and bounded `rclpy` spinning. Test predicates should describe business observations and include a useful timeout description. On timeout the helper reports the latest scenario/time, stop state, aggregate counts, AGV state/pose/speed/fault, and order progress.

Do not replace readiness or completion predicates with fixed sleeps. A fixed sleep is simultaneously slower on fast hosts and flaky on slow hosts. If a state can be missed because a topic is not durable, subscribe before the stimulus or use the documented query interface.

For position assertions, use an explicit tolerance based on model update rate—not exact floating-point equality. For “no movement,” compare both reported speed and pose delta over a defined wall/sim interval.

## 11. Visual monitoring protocol

During an attended e2e run:

1. Keep test output and Gazebo visible side by side.
2. Identify the order and vehicle IDs printed by the scenario/log.
3. Observe case pickup, travel, picker arrival, and pallet placement.
4. For stop/fault tests, verify the affected model visibly halts.
5. If the visual scene conflicts with `/warehouse/state`, record both and classify it as visualizer/Gazebo-Transport divergence.

Headless runs execute the same ROS assertions. They do not provide rendering coverage, so at least the nominal scenario should be run visually for a POC acceptance demonstration.

## 12. Pass, fail, error, and skip

- **Pass:** all assertions and required process exit checks succeed.
- **Fail:** an observed product/test behavior violates an acceptance criterion.
- **Error:** setup, infrastructure, collection, fixture, simulator, or unexpected process failure prevents the intended assertions.
- **Skip:** declared prerequisite/capability is absent. A skip is visible debt, not a pass.

A flaky retry must not overwrite the original result. Report the first failure and the retry frequency. Do not configure unconditional automatic retries for safety, integrity, or state-transition tests.

## 13. Exit criteria

For the POC demonstration:

- all unit and interface-contract tests pass;
- live integration tests pass from a clean ROS domain;
- nominal e2e passes visually and headlessly;
- route, stop, and fault scenarios pass or are explicitly reported as known gaps;
- `WH-OBS-GDB-001` either produces its required native-stack artifact or reports a missing debugger as an observability prerequisite failure/skip;
- deployment static/gateway tests pass, and any requested `WH-K8S-001` Pod-restart qualification executes rather than being inferred from a host-prerequisite skip;
- no test leaves managed ROS/Gazebo processes behind;
- Markdown/JUnit artifacts are created and contain exact reproduction commands;
- unexpected skips, warnings, core dumps, and error-level warehouse events are reviewed.

For requirement mapping and broader SQA guidance, see [MASTER_SQA_GUIDE.md](MASTER_SQA_GUIDE.md). For ticket-quality output, see [REPORTING.md](REPORTING.md).