# Automated Warehouse Cell POC: Master SQA Guide ## 1. Purpose and scope This is the master engineering and software-quality guide for the automated warehouse cell proof of concept (POC). The POC models a small goods-to-person flow: autonomous guided vehicles (AGVs) move cases through a warehouse cell, picker stations consume those cases, and pallet state is updated as cases are palletized. ROS 2 provides the distributed control and observation layer; Gazebo Sim provides the visible world; Python and `pytest` provide the test harness. The intended baseline is: - Ubuntu 22.04 - ROS 2 Humble - Gazebo Sim 8 (Gazebo Harmonic) - Python 3.10 or the Python version supplied with the ROS installation - `colcon`, `pytest`, `launch_testing`, Gazebo Transport 13 Python bindings, and Gazebo Messages 10 Python bindings > **Simulation boundary:** vehicle travel, manipulation, palletization, inventory, collision/safety events, and timing in this repository are simulated behaviors. A rendered model moving in Gazebo is not evidence that the corresponding localization, motion-planning, manipulation, safety PLC, or machinery-control problem has been solved for a physical warehouse. This project is an educational test and integration fixture. It is inspired by the general automated-warehouse domain and is not affiliated with, endorsed by, or a representation of any named commercial vendor's implementation. ## 2. Quick orientation The normal workflow is: ```bash source /opt/ros/humble/setup.bash ./scripts/build.sh source install/setup.bash # Human-observable demo; Gazebo GUI is the default. ./scripts/run_demo.sh --no-build # Full automated suite; visual mode is also the default. ./scripts/run_tests.sh # CI or a machine without a display. ./scripts/run_tests.sh --headless ``` The command wrappers are the supported entry points because they establish ROS environment variables, launch the correct process set, capture artifacts, and stop child processes. Direct `ros2` commands are still valuable during investigation and are documented below. ### 2.1 Current validation snapshot The final headless `all` run recorded on 2026-08-13 UTC collected 105 cases: **104 passed, 0 failed, 0 errored, and 1 skipped** in 88.739 seconds. The only skip was opt-in `WH-K8S-001`; the ordinary full suite does not create a kind cluster without `--k8s-runtime`. The executed ROS, Gazebo, real UDP packet-fault, outbox/restart, scalability, GDB, deployment-static, and health-gateway checks all passed. See the [human-readable final report](../artifacts/final-validation-wave2/test-report.md) and [machine-readable summary](../artifacts/final-validation-wave2/summary.json). That overall `PASSED` result does not qualify Kubernetes Pod recovery. A separate explicit `--k8s-runtime` prerequisite run failed before cluster creation because `kind` and `kubectl` are absent; Docker daemon/socket access is also unavailable to this login. Missing prerequisites are a hard failure after the operator opts in, while the non-opted-in case remains the one deliberate skip in the 105-case run. See [QUICKSTART.md](QUICKSTART.md) for setup checks and common invocations, [ARCHITECTURE.md](ARCHITECTURE.md) for the control/data flow, [TEST_PLAN.md](TEST_PLAN.md) for scenarios and acceptance criteria, and [DEBUGGING.md](DEBUGGING.md) for GDB, core dumps, ROS tracing, and Python debugging. ## 3. System component inventory ### 3.1 Repository-level components | Component | Responsibility | Important evidence/outputs | |---|---|---| | `warehouse_interfaces` | Warehouse-specific ROS messages, services, and actions shared between packages. | Generated Python/C++ type support; interface definitions used as the system contract. | | `warehouse_core` | Domain simulation and supervisory control: order/task dispatch, vehicle state, routing, picker/pallet state, inventory, fault handling, state publication, idempotency, and an optional SQLite checkpoint/outbox seam. | Node logs, state/event topics, task outcomes, inventory/pallet assertions, optional checkpoint/outbox records. | | `warehouse_gazebo` | Gazebo world, simple models, ROS-to-Gazebo state visualization, and simulator launch. | `warehouse.launch.py`, `worlds/warehouse.sdf`, moving AGV/case visuals, Gazebo logs. | | `deploy/docker` and `deploy/k8s` | Non-root controller image and disposable single-replica/PVC Kubernetes recovery topology; Gazebo is intentionally excluded. | Dockerfile/entrypoint, manifest, HTTP probe/state evidence, Pod/PVC events. | | `test/` and package-local tests | Unit, interface/contract, ROS integration, Gazebo integration, and end-to-end scenarios. | Pytest/JUnit results, ROS logs, scenario timelines, assertions. | | `scripts/run_demo.sh` | Starts a populated interactive demonstration in visual Gazebo by default. | Terminal logs and an observable warehouse run. | | `scripts/run_tests.sh` | Selects suites, manages simulation, enables debug options, and creates reports. | JUnit XML, Markdown summary/Jira drafts, logs, optional debugger output. | | `scripts/build.sh` | Builds against ROS Humble's Ubuntu system-Python ABI even if Conda is active. | `build/`, `install/`, `log/`; incompatible cache is preserved below the colcon-ignored `build/_stale/` directory. | | `scripts/doctor.sh` | Performs read-only ROS, Gazebo, GDB, colcon, and Python ABI/import checks. | Actionable `OK`/`WARN`/`MISS` environment summary. | | `scripts/debug_node.sh` | Runs one installed Python ROS executable under GDB/batch GDB or `pdb`. | Interactive session or native full-thread backtrace. | | `scripts/capture_backtrace.sh` | Attaches batch GDB to an exact live PID and writes every thread's stack. | Timestamped ticket attachment below `artifacts/backtraces/` by default. | | `scripts/capture_gdb_ros_stack.sh` | Starts one installed Python ROS node under GDB, breaks at a native ROS wait boundary, and records all native thread stacks and loaded libraries. | Reproducible ROS/rclpy/CPython/RMW evidence; exercised by `WH-OBS-GDB-001`. | | `scripts/k8s_smoke.sh` | Builds and qualifies a newly created disposable kind cluster, including readiness and PVC-backed Pod replacement. | Docker/kind/kubectl logs, pre/post state, Pod UIDs, `WH-K8S-001` summary. | | `docs` | Operator, SQA, architecture, test, debug, and reporting guidance. | This guide and its focused companion documents. | The interface definitions and launch files are the executable source of truth. To enumerate what is installed after a build: ```bash ros2 pkg executables warehouse_core ros2 pkg executables warehouse_gazebo ros2 interface package warehouse_interfaces ros2 launch warehouse_gazebo warehouse.launch.py --show-args ``` ### 3.2 Runtime component roles The precise executable names are discoverable with `ros2 pkg executables`; the stable roles are: | Runtime role | Input | Output | Primary test oracle | |---|---|---|---| | Order/task source | Scenario definition or operator request | Work demand | Requested SKU/case counts and order identity | | Cell coordinator | Orders, fleet/picker state, inventory | Assignments and aggregate warehouse state | Legal state transitions and completion/failure reason | | AGV controller/simulator | Assignment and route | Vehicle pose/state and delivery event | Route validity, single-case ownership, travel progress, timeout behavior | | Picker/palletizer simulator | Delivered case and station availability | Pallet contents/state | Correct SKU/count, placement/layer, duplicate prevention | | Inventory model | Reservations, picks, returns/fault recovery | On-hand/reserved/in-transit counts | Conservation and non-negative quantities | | Safety/fault monitor | Proximity, blocked route, injected fault | Hold, stop, retry, or failed task | No work progresses through a safety hold; explicit recovery | | Warehouse state publisher | Domain state | `/warehouse/state` snapshot/updates | Coherent, monotonic state observed by tests and visualization | | Gazebo state visualizer | `/warehouse/state` | Entity poses via `/world/warehouse/set_pose` | Visual agreement with domain state; entity mapping diagnostics | | Gazebo server/client | World/model SDF and pose changes | Physics/rendered scene and Gazebo services | World loaded, expected entities exist, simulator remains healthy | | Test scenario driver | Test data and ROS graph | Commands and fault injections | Scenario-specific acceptance criteria | | Network-fault relay (test-only) | Proxied `SubmitOrder` request and configured application-boundary fault mode | Passed, delayed, or explicitly failed proxy response | Client deadline, authoritative state, idempotent retry, and one business effect | | Health gateway (deployment-only) | Latest `/warehouse/state` with receive age | HTTP liveness/readiness/state/Prometheus projection | Waiting/ready/stale status, run/revision, and HTTP code | | Evidence/report generator | JUnit, logs, metadata | Markdown/Jira-ready report | Reproducibility and links to raw evidence | ### 3.3 Gazebo assets and visible entities The reference world is `warehouse_gazebo/worlds/warehouse.sdf`. The populated world includes: - AGVs `agv_01` through `agv_04`; - picker stations `picker_01` and `picker_02`; - case entities beginning with `case_001`; - pallet, lane, rack, staging, and station geometry appropriate to the POC. The state visualizer consumes `/warehouse/state` and requests entity updates through Gazebo's `/world/warehouse/set_pose` service. This is deliberately a visualization adapter: the authoritative business state is in ROS, and Gazebo mirrors that state. If a future implementation makes physics authoritative, its ownership, feedback loop, and test oracles must be redesigned to prevent competing pose writers. ### 3.4 Nominal information flow ```text scenario/order | v cell coordinator ---> inventory reservation | +-- assignment/route --> AGV --> delivered case | v picker station | v pallet state | +---------------> /warehouse/state ----------------+ v state_visualizer | v Gazebo /set_pose + GUI pytest/launch_testing observes state/events, injects requests or faults, and asserts outcomes without using the rendered pixels as the sole oracle. ``` The visualizer is an observability aid, not the sole pass/fail mechanism. Automated assertions inspect ROS state, responses, results, and invariant violations; this permits the same test logic to run headlessly. ### 3.5 Container and Kubernetes recovery exercise `deploy/docker/Dockerfile` builds a non-root ROS 2 Humble image containing `warehouse_interfaces` and `warehouse_core`; Gazebo is intentionally excluded, so this is a controller/recovery topology rather than the visual demo. `deploy/k8s/warehouse-cell.yaml` defines a disposable SQA topology: - one `warehouse-cell` Deployment replica with `Recreate` strategy, preventing an intended rollout from overlapping two SQLite-writing controllers; - a controller container using `fleet_size:=2`, `state_db_path:=/var/lib/warehouse/state.sqlite3`, and checkpoint restore; - a same-Pod `health-gateway` sidecar that converts latest `/warehouse/state` freshness into HTTP without treating HTTP as a ROS/DDS replacement; - startup/readiness on `/readyz`, liveness on `/healthz`, diagnostic projections on `/state` and `/metrics`, and a ClusterIP Service exposing port 8080; - a 256 MiB `ReadWriteOnce` PVC/PV for controller state and an `emptyDir` for writable runtime files, with non-root/read-only-root-filesystem container security settings. The gateway's liveness answer proves only that its HTTP process responds. Readiness additionally requires a recent compatible ROS state, so a live sidecar with a dead/stale controller is not ready. There is no controller-specific liveness probe: a hung-but-running controller can leave the Pod unready without forcing a restart. The Service exposes only the HTTP gateway; it is not a DDS broker, discovery server, order API, or Gazebo endpoint. `ROS_LOCALHOST_ONLY=1` deliberately confines DDS to the shared Pod network namespace. `WH-K8S-STATIC-001..002` parse and assert the manifest/container/smoke-script contract without a cluster. `WH-K8S-GATEWAY-001` directly exercises the HTTP server/projection: liveness remains independent, readiness moves from waiting to ready and then stale as ROS state ages, and state/metrics preserve run/revision. The bounded runtime exercise is: ```bash ./scripts/k8s_smoke.sh --artifacts artifacts/k8s-smoke/manual ``` It requires Docker daemon access, `kind`, `kubectl`, and `curl`; creates and owns a new disposable cluster; builds/loads the image; waits for readiness; records a durable active order; deliberately deletes the whole Pod; waits for a different Pod UID; and compares pre/post `run_id`, non-regressing revision, and retained order state through the same PVC. It does not test controller-only container exit, liveness-triggered restart, node loss, or eviction. `test/deployment/test_kubernetes_restart.py` exposes it through `--suite deployment --k8s-runtime`. Without that opt-in, `WH-K8S-001` is the deliberate skip in the final full run. With the opt-in, missing `kind`/`kubectl` is reported as a failed prerequisite rather than another skip; Docker socket access is also unavailable on this host. Static/gateway test success is not a substitute for Pod-restart evidence. The checked-in `hostPath` PV is suitable only for a local disposable kind exercise. It is not portable cloud storage, replicated state, backup, multi-node rescheduling proof, zero-downtime rollout, or HA. One replica plus `Recreate` and local SQLite intentionally avoid—but do not solve—multi-writer/split-brain behavior. ## 4. Domain model and invariants ### 4.1 Implemented lifecycle vocabulary Order and case states are separate contracts: ```text order: QUEUED -> ACTIVE -> COMPLETE case: STORED -> RESERVED -> ASSIGNED -> IN_TRANSIT -> AT_PICKER -> PALLETIZED ``` Vehicles additionally expose operational states such as `IDLE`, `TO_PICKUP`, `LOADING`, `TO_PICKER`, `UNLOADING`, `RETURNING`, `YIELDING`, and `FAULTED`; pickers expose `IDLE`, `PICKING`, and `FAULTED`. The cell-wide emergency stop is an explicit Boolean/reason, not an invented order state. Tests should assert both the final state and legal transition history. A jump directly from `QUEUED` to `COMPLETE`, continued motion under stop, or a second terminal result is a defect even when the final item count appears correct. Future `CANCELLED`/`FAILED` order states require corresponding interface, action-result, and recovery tests before being treated as supported. ### 4.2 System invariants These invariants apply across scenarios: - A case has at most one authoritative location at a time: storage, on an AGV, at a picker, on a pallet, or an explicit exception location. - An AGV carries at most one case and is not assigned two incompatible active jobs. - Reserved stock plus available stock is consistent with total stock, accounting for completed picks and explicitly modeled loss/rework. - Inventory counts and pallet quantities never become negative. - A case is palletized at most once. - Pallet placement indices are non-negative, each case is assigned one placement/layer, and placements in the tested layers are unique. The POC does not impose an upper case/weight/volume capacity. - A task emits one terminal outcome. - A safety hold prevents affected motion or work progression until recovery is acknowledged. - Simulated time and ROS timestamps do not move backward within a test epoch. - Every request, task, case, vehicle, picker, pallet, and order has a stable identifier suitable for log correlation. - Failure and timeout outcomes include a machine-readable reason, not only prose in a log. Unit tests should protect local invariants. Integration and end-to-end tests should protect invariants whose truth depends on several nodes. ### 4.3 Fault catalogue The controller accepts these canonical fault strings through `/warehouse/inject_fault`: | Target | Faults | Behavior represented | |---|---|---| | AGV | `STALLED`, `LOW_BATTERY`, `LOCALIZATION_LOSS` | Vehicle cannot perform normal transport until policy/timer clears it; low battery also constrains dispatch eligibility. | | Picker | `PICKER_JAM`, `SENSOR_DROPOUT` | Picker work is held until policy/timer clears it. | A positive `duration_sec` creates a timed fault; zero creates a persistent fault that requires `/warehouse/clear_fault`. These are logical fault states, not physics/sensor failure models. Unsupported targets/types are rejected with a diagnostic response and must not mutate the cell. ## 5. Test paradigm and suite structure ### 5.1 Quality strategy The suite follows a layered strategy. A defect should be caught at the cheapest layer that can represent it, while a smaller number of end-to-end tests prove that the assembled system works. | Layer | Scope | ROS/Gazebo required | Typical duration | Examples | |---|---|---:|---:|---| | Static/build | Packaging, syntax, style, generated interfaces | Build environment | Seconds | `colcon build`, lint, importability | | Unit | One Python class/function/state machine | No | Milliseconds | route selection, pallet placement, transition guards, inventory arithmetic | | Contract | Interface shape and semantic assumptions | Usually no; type support may be built | Milliseconds-seconds | required fields, enum values, identifier/timestamp rules | | ROS integration | Several nodes and DDS communication | Yes | Seconds | topic QoS, service response, action feedback/result, startup/shutdown | | Gazebo integration | Bridge, world, entities, visual adapter | Yes, GUI optional | Seconds-minutes | world readiness, entity mapping, pose updates, simulator exit | | End-to-end functional | Complete order-to-pallet flow | Yes, Gazebo by default | Minutes | nominal wave, contention, safety hold, fault/recovery | | Deployment qualification | Container image, orchestration topology, probes, persistent restart | Docker/Kubernetes; Gazebo excluded | Seconds-minutes | static manifest contract, gateway freshness, opt-in kind Pod replacement | | Reliability/soak | Repetition and resource stability | Yes; normally headless | Tens of minutes+ | repeated waves, no orphan processes, bounded memory/event backlog | ### 5.2 Pytest markers and suite selection Tests use these markers to state infrastructure and scope explicitly: | Marker | Meaning | |---|---| | `unit` | Pure or near-pure low-level logic; no live ROS graph. | | `integration` | Multiple components or a live ROS graph. | | `system` | Owns and perturbs a deployed process set, such as controller pause/restart in an isolated domain. | | `e2e` | User-visible business flow through the assembled cell. | | `ros` | Requires an initialized ROS context/domain. | | `gazebo` | Requires Gazebo server/world; GUI availability depends on runner mode. | | `slow` | Excluded from short developer loops unless selected. | | `performance` | Records latency, throughput, or resource measurements. | | `characterization` | Records a non-gating baseline while retaining correctness/deadline gates. | | `concurrency` | Coordinates clients around one shared-mutation boundary. | | `recovery` | Exercises process pause, crash, checkpoint, or restore behavior. | | `schema` | Exercises an encoding/compatibility contract. | | `observability` | Validates debugger, logging, metric, trace, or other diagnostic evidence. | | `deployment` | Validates container/orchestrator packaging or runtime topology. | | `k8s_runtime` | Creates a disposable Kubernetes cluster and therefore requires explicit opt-in. | | `packet_fault` | Forces ROS/DDS user traffic through UDP and impairs real packets in an isolated Linux network namespace. | Supported runner selections are: ```bash ./scripts/run_tests.sh --suite unit ./scripts/run_tests.sh --suite integration ./scripts/run_tests.sh --suite system ./scripts/run_tests.sh --suite e2e ./scripts/run_tests.sh --suite performance ./scripts/run_tests.sh --suite recovery ./scripts/run_tests.sh --suite schema ./scripts/run_tests.sh --suite observability ./scripts/run_tests.sh --suite deployment # Explicitly authorize disposable kind cluster creation for WH-K8S-001. ./scripts/run_tests.sh --suite deployment --k8s-runtime ./scripts/run_tests.sh --suite all ``` Use `--no-sim` to prevent the runner from launching Gazebo (for a pure suite or an already-running cell), `--keep-sim` only for an attended investigation, and `--report-dir DIR` to place evidence in a known artifact directory. ROS-dependent suites incrementally rebuild by default so generated interfaces and console entry points cannot silently remain stale; `--no-build` deliberately trusts the existing overlay. `--debug` increases diagnostic logging and enables core dumps; `--gdb-node` accepts `none`, `cell_controller`, `health_monitor`, `state_visualizer`, or `all`; `--gdb-gazebo` targets the server and forces headless mode. `--build`, `--no-build`, `--timeout SEC`, `--ros-domain-id 0..232`, and passthrough pytest arguments after `--` support focused development runs. Confirm the exact options at any revision with: ```bash ./scripts/run_tests.sh --help ``` The runner exposes `all`, `unit`, `integration`, `system`, `e2e`, `performance`, `recovery`, `schema`, `observability`, and `deployment`. The first five select directories; the latter five select strict pytest markers across `test/`. System/performance/recovery/schema/observability/deployment runs do not start managed Gazebo by default. `--k8s-runtime` is a separate destructive-infrastructure opt-in; without it, collected `WH-K8S-001` skips visibly, and with it, absent infrastructure prerequisites fail the requested qualification. Marker names still describe properties independently of suite selection. ### 5.3 Test anatomy Every integration or end-to-end test should have these phases: 1. **Arrange:** choose a unique ROS domain where practical; start a known world; wait for required nodes, topics, services, and action servers; seed deterministic inventory and configuration. 2. **Act:** submit one correlated request or inject one named fault. Record the start time and IDs. 3. **Observe:** collect state/event messages from before the request through the terminal outcome. Store logs as artifacts. 4. **Assert:** check the terminal response, required intermediate transitions, time bounds, domain invariants, and absence of unexpected error events. 5. **Clean up:** cancel timers/goals, destroy test nodes, request orderly shutdown, and verify no managed child processes remain. Readiness must be established through graph/service/action checks, not fixed sleeps. Timeouts must be finite and should include the last observed system state in their failure messages. ### 5.4 Determinism and isolation - Fix random seeds and report them. A randomized failure must print a replayable seed. - Use unique task/order IDs per test, even if processes are reused. - Prefer one fresh ROS context and cell state per test scenario. - Let the runner choose a unique `ROS_DOMAIN_ID` and `GZ_PARTITION` for managed simulation, or pass `--ros-domain-id` when reproducibility/infrastructure assigns one. This avoids collisions with developer graphs and parallel workers. - Do not assume discovery is instantaneous. Wait for publishers/subscribers/services/actions with bounded polling or ROS APIs. - Distinguish the controller's accelerated logical `simulation_time_sec` from the harness's monotonic wall clock. Gazebo pause does not pause the controller in this POC because no ROS `/clock` bridge is used. Report both time bases and do not assume they share an epoch. - Treat tests that rely on GUI rendering or real-time factor as non-deterministic unless their tolerances and platform requirements are explicit. - Preserve the first failure's evidence; teardown errors should not overwrite it. ### 5.5 Oracles and evidence A strong functional oracle checks more than `COMPLETE`: - request accepted/rejected state; - correlation IDs; - assignment and route decisions; - state-transition sequence; - case identity and SKU; - inventory conservation; - pallet contents/placement layer; - safety/fault events; - elapsed simulated and wall time; - node/process health; - absence of duplicate terminal outcomes. Rendered motion is useful corroborating evidence but should not be parsed as the primary oracle. If screenshots or recordings are captured, link them to the ROS state timeline and test ID. ### 5.6 Functional SQA vocabulary applied to this cell These terms describe different questions; they are not interchangeable labels for the same run. | Technique or test type | Question it answers | Warehouse/ROS 2 example | |---|---|---| | Black-box | Does the deployed system honor its public contract without relying on internals? | Submit `/warehouse/fulfill_order`, observe feedback plus `/warehouse/state`, and assert the palletized result. | | Gray-box | Can architecture knowledge make the stimulus and evidence more incisive while still asserting public behavior? | Select a QoS profile, ROS domain, fault target, and callback timing deliberately, then assert only topics/services/actions and process health. | | White-box | Does a known branch, state transition, or data structure behave correctly? | Call the pure `WarehouseEngine` directly to exercise inventory rollback or an invalid transition. | | Directed | Does one predefined stimulus produce one specified outcome? | Inject `STALLED` into the loaded AGV and verify its documented recovery path. | | Exploratory/ad hoc | What credible risks emerge while a skilled tester varies timing, ordering, and observations? | Pause the controller at several transfer phases, vary cancellation timing, and preserve any new failure as a reproducible test. Exploratory evidence is valuable, but an undocumented ad-hoc observation is not regression coverage. | | Smoke | Is the build minimally testable? | Launch the assembled graph, establish world/node/interface readiness, receive one coherent state, and shut down cleanly. | | Integration | Do component boundaries exchange the right data and failure semantics? | Exercise a live service and verify the generated ROS type, QoS, callback result, state mutation, and event correlation. | | System/end-to-end | Does the assembled cell complete a user-visible workflow? | Fulfill a mixed-SKU order through dispatch, AGV travel, picker processing, and palletization. | | Acceptance | Does an agreed business-facing criterion pass in the intended environment? | Demonstrate the nominal wave visually and headlessly with the recorded build, seed, and acceptance thresholds. | | Verification | Did the implementation meet its specified contract? | Prove a duplicate order ID is rejected without another reservation. | | Validation | Is the modeled behavior suitable for its intended use? | Review whether the simulated workflow is a useful SQA fixture. This POC does not validate physical machinery or a commercial warehouse design. | | Performance/load | At a stated workload, what latency, throughput, backlog, and resource behavior occur? | Measure order/API latency percentiles and throughput for a fixed concurrent wave. | | Stress/capacity | Where does behavior degrade or reject work as demand rises? | Increase concurrent order demand until an explicit limit or failure is observed, while continuing to assert inventory invariants. | | Duration/soak | Does state or resource health degrade over time? | Repeat waves while tracking RSS, event backlog, stale state, identifier reuse, and orphan processes. | | Scalability | How does capacity or latency change as resources or demand change? | Compare the same workload with one, two, and four enabled AGVs; do not assume a linear improvement. | | Topology | Is the functional outcome stable across supported deployment shapes? | Compare isolated domains/namespaces or, in a future distributed deployment, alternate component placement. | | Failover | Does a redundant component take over within a defined recovery objective? | Not implemented here: there is no standby, leader election, or split-brain control. Starting one replacement from local SQLite is recovery, not failover. | | Disaster recovery | Can durable service be restored after wider loss within declared RPO/RTO targets? | Not implemented here: the optional local checkpoint is not a qualified backup/restore system and has no declared RPO/RTO or host-loss procedure. | A senior test plan names the layer, technique, workload, oracle, failure model, and evidence independently. For example, “gray-box ROS integration test under fixed load” conveys more than calling every multi-node test “end-to-end.” ## 6. Realistic scenario catalogue Scenario IDs remain stable even if test names change. Detailed steps and coverage are in [TEST_PLAN.md](TEST_PLAN.md). | ID | Scenario | Purpose | Expected outcome | Status | |---|---|---|---|---| | `WH-NOM-001` | Mixed-SKU order-to-pallet wave | Demonstrate one complete multi-case flow to its requested picker. | Requested cases reach `picker_01`, are palletized once, inventory balances, and the order completes. | Executable e2e | | `WH-NOM-002` | Parallel picker utilization | Demonstrate two independent orders overlapping across both picker stations. | Each order remains correlated to its target picker and both complete without cross-contamination. | Executable e2e | | `WH-ROUTE-001` | Surge-wave routing and contention | Exercise three concurrent orders across four AGVs and two single-dock picker cells. | At least two AGVs progress, all 12 cases complete without deadlock/collision/fault, and picker ownership stays coherent. | Executable e2e | | `WH-SAFE-001` | Simulated emergency stop and resume | Verify cell-wide software stop/hold semantics and controlled resume. | Motion/progression freezes within tolerance; reset resumes legal work without duplication. | Executable e2e | | `WH-FAULT-001` | Loaded vehicle stall and recovery | Verify a manually cleared AGV fault does not lose or duplicate its carried case. | Fault and halt are observable, ownership remains coherent, clear resumes work, and inventory is conserved. | Executable e2e | | `WH-FAULT-002` | Timed picker jam | Verify automatic fault expiry without losing a waiting case. | Jam is visible, order stays active, fault clears after its duration, and the case is palletized exactly once. | Executable e2e | | `WH-ACTION-001` | Action feedback and result | Validate the long-running action contract as an operations/SQA client sees it. | Feedback is present and monotonic; active IDs are valid; result and authoritative final state agree. | Executable e2e | | `WH-INV-001` | Insufficient stock | Validate atomic rejection semantics in the domain engine. | No phantom case is created; response names the shortage; unrelated inventory is unchanged. | Executable unit | | `WH-DIST-UNIT-001..002` | Idempotent replay and conflict | Protect repeated-request semantics below ROS transport. | Equivalent reuse returns `REPLAYED` without another effect; changed content under the same key returns conflict without mutation. | Executable unit | | `WH-TXN-001` | Concurrent last-item reservation | Expose a lost-update/double-allocation risk at a live ROS service boundary. | Exactly one competing request acquires the last case; the other rejects without mutation. | Executable integration | | `WH-STREAM-001` | State/event stream integrity | Correlate live transition events with the authoritative snapshot stream. | Required order events are ordered and unique, and terminal event/state agree. | Executable integration | | `WH-STREAM-UNIT-001` | Durable outbox replay | Protect ordered pending-event storage below ROS transport. | Pending events retain unique IDs/sequence order and are marked only after publication. | Executable unit | | `WH-STREAM-UNIT-002` | Causation across restore | Prove one order's event chain retains lineage through a checkpoint/restart boundary. | Correlation is stable; each later event points to the prior event ID; sequence and IDs remain valid. | Executable unit | | `WH-STREAM-UNIT-003..004` | Consumer event-ID deduplication | Prove duplicate delivery is applied once and competing callbacks atomically claim one event ID. | One callback applies the effect; duplicates are counted and suppressed. | Executable unit | | `WH-OUTBOX-CRASH-001` | Publish-before-mark controller crash | Exercise the durable outbox's intentional duplicate-delivery window and a live consumer's apply-once behavior. | The same `ORDER_ACCEPTED` `event_id` arrives twice, is applied once, and the recovered order completes once. | Executable system/integration | | `WH-QOS-001` | Late state/event subscriber | Verify transient-local state recovery versus volatile event history. | A late client receives current state, does not receive a pre-join event, and does receive a new post-join fault event. | Executable integration | | `WH-PERF-001` | Seeded surge-wave characterization | Run ten pure-engine surge waves and record wall runtime, logical cycle time, throughput, and failure rate without host-specific timing gates. | Every wave completes within its logical deadline/invariants; p50/p95/p99 observations are emitted to JUnit properties. | Executable integration/characterization | | `WH-SCALE-001` | 1/2/4-AGV scaling characterization | Hold the 12-case/two-picker surge workload fixed while changing modeled transport capacity. | Completion/invariants hold; two AGVs materially improve cycle time over one, while four expose diminishing returns near the picker/dock bottleneck. | Executable integration/characterization | | `WH-SCHEMA-001` | JSON metrics schema contract | Protect the versioned `/warehouse/metrics` document and tolerant-reader expectations. | Current data round-trips; older optional fields receive safe defaults; additive fields are tolerated; malformed/type errors are actionable. | Executable unit | | `WH-DIST-001` | Accepted response abandoned by client | Model an uncertain application outcome and verify idempotent reconciliation. | An independent observer proves commit; a new client retries the same request and receives `REPLAYED` with one order/reservation/event effect. | Executable integration | | `WH-NET-001` | Deterministic pre-forward latency | Exercise a bounded application-boundary delay before the authoritative service call. | The response respects the configured lower bound and exactly one order commits. | Executable system/integration | | `WH-NET-002` | Client deadline after authoritative commit | Exercise a real client timeout while the proxy delays a successful downstream response. | State proves one commit; same-key retry is `REPLAYED`; one `ORDER_ACCEPTED` exists. | Executable system/integration | | `WH-NET-003` | Application-boundary drop/partition and recovery | Prevent forwarding while the test double is in drop/partition mode, then restore it. | No pre-recovery commit occurs; the old call fails explicitly; a deliberate retry commits once. | Executable system/integration | | `WH-NETEM-001` | Real loopback UDP latency/timeout/partition | Disable Fast DDS shared memory, apply Linux `tc netem` to loopback packets in a disposable user/network namespace, and reconcile outcomes. | Latency is measurable, a delayed response creates an unknown outcome resolved by `REPLAYED`, and a 100%-loss window heals to one effect. | Executable system/integration | | `WH-PAUSE-001` | Controller process pause | Exercise stale-state detection and coherent resume across the controller/monitor boundary. | `SIGSTOP` freezes the state stream, the monitor reports staleness, and `SIGCONT` restores a coherent advancing snapshot. | Executable system/integration | | `WH-RESTART-001` | Persistent controller restart mid-order | Verify single-controller crash/restart through the optional SQLite seam. | A replacement restores the same run/order/revision and finishes once, with one observed and persisted completion event. | Executable system/integration | | `WH-RESTART-UNIT-001` | SQLite checkpoint restoration | Verify the opt-in persistence seam below a live process restart. | An in-flight route/job restores under the same run/revision and completes once without invariant violations. | Executable unit | | `WH-OBS-GDB-001` | Runtime native-stack observability | Start the installed controller under GDB and stop at `rcl_wait`. | Evidence includes ROS/rclpy/CPython frames, Fast DDS/RMW threads, and loaded native libraries. | Executable system/observability | | `WH-K8S-STATIC-001..002` | Container/Kubernetes static contract | Parse the checked-in Deployment/Service/storage/probe topology and inspect its container/smoke entry points. | One-replica `Recreate`, PVC, gateway probes, Service wiring, non-root image, and bounded disposable-cluster script remain connected. | Executable unit/static | | `WH-K8S-GATEWAY-001` | HTTP orchestration projection | Exercise liveness, readiness freshness, state, and metrics without Kubernetes. | Liveness responds; readiness transitions waiting→ready→stale; run/revision metrics match the supplied ROS projection. | Executable integration | | `WH-K8S-001` | Kubernetes Pod replacement with PVC recovery | Delete a ready Pod during durable work and let its Deployment create a replacement. | New Pod UID becomes ready and reports the same `run_id`, a non-regressing revision, and retained order state. | Collected/opt-in; skipped in final `all`, host-prerequisite-blocked when requested | | `WH-CAP-001` | Pallet capacity boundary | Add and exercise a future configured upper capacity. | Exact capacity is accepted; excess is routed/waited/rejected; no over-capacity placement occurs. | Planned | | `WH-TIME-001` | Clock-domain characterization | Make the POC's wall/logical/Gazebo clock split explicit and prepare a future `/clock`-coupled mode. | Current mode proves Gazebo pause does not pause logical work; any future coupled mode requires bounded pause/resume tests. | Planned | | `WH-SHUT-001` | Ordered shutdown during active work | Validate teardown behavior. | Processes terminate within bounds, no orphan simulator remains, and partial work is reported consistently. | Planned | | `WH-SOAK-001` | Repeated mixed waves | Detect leaks, stale state, and identifier reuse. | Each wave completes independently; RSS/event counts remain bounded; no cross-wave contamination. | Planned | The implementation may initially mark advanced scenarios as planned or skipped with an explicit reason. A skip is not a pass and should remain visible in the report and traceability matrix. ## 7. Data-intensive systems concepts applied to ROS 2 This section maps the most relevant parts of *Designing Data-Intensive Applications*, second edition (2026), to this repository. It is a study guide and test-design aid. The optional local SQLite checkpoint/outbox is intentionally inspectable, but it is not an external broker, replicated database, high-availability controller, or production recovery architecture. | 2026 chapter | Systems question | ROS 2 warehouse application | Primary scenarios | |---|---|---|---| | Chapter 2, nonfunctional requirements | How fast, reliable, scalable, operable, and maintainable is the system under a declared workload? | Measure bounded service/action latency, state-publication gaps, order throughput, backlog, resource use, recovery time, and resource-scaling efficiency rather than relying on one average. | `WH-PERF-001`, `WH-SCALE-001`, planned `WH-SOAK-001` | | Chapter 5, encoding and evolution | Can producers, consumers, recordings, and tools evolve their data contracts safely? | Treat ROS IDL and the JSON `/warehouse/metrics` payload as versioned contracts with types, units, defaults, and compatibility tests. | `WH-SCHEMA-001` | | Chapter 8, transactions | Can concurrent work mutate shared inventory without lost updates or partial state? | Define the order/reservation boundary, serialize mutation, assert invariants, and test two clients competing for the last case. | `WH-TXN-001`, `WH-INV-001` | | Chapter 9, distributed-system failures | What does a client know after a timeout, process pause, communication-boundary fault, or server/Pod restart? | Separate rejection from unknown outcome; retry with an idempotency key; compare an application relay with actual impaired UDP packets; state exactly which fault layer was injected. | `WH-DIST-001`, `WH-NET-001..003`, `WH-NETEM-001`, `WH-PAUSE-001`, `WH-RESTART-001`, host-blocked `WH-K8S-001` | | Chapter 12, stream processing | How do snapshots and events remain useful when consumers join late, lag, restart, or see only part of a history? | Document state versus event semantics, QoS, correlation, causation, reconstruction, duplicate delivery, consumer deduplication, and optional outbox replay rules. | `WH-STREAM-UNIT-001..004`, `WH-STREAM-001`, `WH-QOS-001`, `WH-OUTBOX-CRASH-001` | ### 7.1 Chapter 9: partial failure and uncertain outcomes A distributed request has more outcomes than “success” and “failure”: - **explicit acceptance/rejection:** a response arrived and its application meaning is known; - **discovery or connection failure before dispatch:** the request may not have reached a server, but the client must use evidence rather than assumption; - **timeout after dispatch:** the operation may have failed, may still be running, or may have committed while its response was delayed/lost; - **process pause:** a node can remain alive and discovered while callbacks make no progress; - **process exit/restart:** graph membership, in-memory state, and outstanding service/action futures can each fail differently; - **partition or severe delay:** some nodes may communicate while others cannot, producing a partial rather than cell-wide failure; - **clock disagreement:** a timeout measured with monotonic wall time must not be inferred from Gazebo or resettable logical time. For `/warehouse/submit_order`, the critical uncertain-outcome window is: ```text client sends ORD-42 | v controller validates and mutates in-memory order/inventory state | +---- queues an ORDER_ACCEPTED event for a later publish tick | v service response returns to client ``` If the client loses the response after mutation, blindly submitting a new order ID can create different work. `SubmitOrder` and `FulfillOrder` therefore carry a business `request_id` (falling back to `order_id` for older callers). The first valid request returns disposition `ACCEPTED`; an equivalent retry under the same key returns the original order with `REPLAYED` and creates no second order, reservation, or acceptance event. Reusing the key with different order content returns `CONFLICT` without mutation. `TaskCommand.command_id` serves the same idempotency-key role for asynchronous order submission. The processed-request ledger is in memory in the normal demo. Supplying `state_db_path` checkpoints it with engine state before a service/action response escapes; `restore_from_checkpoint:=true` restores it. `WH-DIST-UNIT-001..002` prove replay/conflict logic. Live `WH-DIST-001` sends through a short-lived application node that never spins or reads its response, uses an independent state observer to prove commit, destroys that application node, and retries through a new client; the retry returns `REPLAYED` with one effect. This models an abandoned/unknown application response, not network packet loss or DDS fault injection. `WH-RESTART-UNIT-001` proves engine checkpoint reconstruction, and live `WH-RESTART-001` kills one controller and starts a replacement against the same local SQLite store. Reliable DDS transport still must not be described as exactly-once warehouse processing: one business effect comes from `request_id`, fingerprint comparison, and retained outcome state. `WH-NET-001..003` add a deterministic **application-boundary relay** for `/warehouse/submit_order`. The test client calls `/warehouse_fault_proxy/submit_order`; `/warehouse_fault_proxy/configure` selects `PASS`, `DELAY_BEFORE_FORWARD`, `DELAY_AFTER_COMMIT`, `DROP_BEFORE_FORWARD`, or `PARTITION`. The relay is a controlled ROS service proxy, not Linux `netem`, a DDS transport shim, packet loss, discovery failure, or a physical link partition: - `WH-NET-001` applies 0.35 seconds before forwarding and requires an observed round trip of at least 0.30 and less than 3.0 seconds, followed by one authoritative order; - `WH-NET-002` forwards and commits, withholds the downstream success for 0.80 seconds, and proves a real 0.20-second client deadline expires. State then proves one commit, and an identical retry returns `REPLAYED` with one `ORDER_ACCEPTED`; - `WH-NET-003` keeps drop and partition requests from reaching the authority. Both exceed the client observation deadline, create zero orders, and fail explicitly; after `PASS`, only a new deliberate retry commits once. The relay seam makes boundary placement and outcome knowledge deterministic, but by itself does not establish behavior under impaired packets or DDS retransmission. Dedicated `WH-NETEM-001` covers a narrower real-packet case. It creates a disposable unprivileged Linux user/network namespace, brings up loopback, sets `FASTDDS_BUILTIN_TRANSPORTS=UDPv4` so Fast DDS shared memory cannot bypass the impairment, and applies `tc netem` to the loopback qdisc: - 180 ms one-way delay must produce a measured request/response round trip of at least 0.28 seconds and one committed order; - 300 ms one-way delay must keep the response beyond a 0.45-second client deadline even though the authority commits; after removing the qdisc, same-key retry returns `REPLAYED` and state contains one order; - 100% packet loss must yield no response and zero observed commit during the 0.40-second partition window; after healing, reliable DDS may deliver/retransmit the original or the retry first, so the recovered response may be `ACCEPTED` or `REPLAYED`, but authoritative state must contain one effect. The test records the observed packet-latency/deadline measurements and JSON evidence in JUnit properties. It passed in the final full run in 6.807 seconds. This is real Fast DDS UDP traffic and Linux queue impairment, but it remains same-host, loopback, symmetric, short-duration, and single-controller. It does not prove behavior across physical interfaces/hosts, asymmetric routing, MTU/corruption/reordering, sustained congestion, discovery storms, clock skew, or production network policy. Shared-memory or another transport must not be silently enabled in a packet-fault test, or a green latency assertion can miss the intended boundary entirely. Test implication: inject a fault at a named boundary, use finite monotonic deadlines, retain the same business ID, and reconcile the service response, `/warehouse/state`, `/warehouse/events`, action state, logs, and process lifecycle. A timeout alone is not a product verdict. ### 7.2 Chapter 12: snapshots, event streams, and delivery semantics The public topics have different recovery roles: | Endpoint | Data model | Implemented delivery/recovery meaning | Do not infer | |---|---|---|---| | `/warehouse/state` | Periodic full aggregate snapshot | Reliable, transient-local, keep-last 1 from the controller; a compatible late subscriber can obtain the retained current sample. | That every intermediate state was observed, or that DDS retention itself survives publisher restart. Optional SQLite restoration is a separate mechanism. | | `/warehouse/events` | Live transition/fault/safety event stream | Reliable, volatile, keep-last 100; useful to observers already subscribed. With optional SQLite enabled, unpublished events are stored in an ordered outbox and retried. | An audit-complete broker log or at-most-once delivery; publish-before-mark can intentionally redeliver an event. | | `/warehouse/commands` | Asynchronous command ingress | Default depth-10 topic; order submission uses `command_id` as its idempotency key. | Broker-style ACKs, durable command queueing, or persistence when `state_db_path` is empty. | | `/warehouse/metrics` | Periodic JSON operational projection | Observability summary derived from snapshots. | Authoritative state or a stable database record. | | `/diagnostics` | Health projection | Current component warnings/status for operators and tests. | A complete incident timeline. | Delivery terms should be applied at the correct layer: - **at-most-once processing** permits loss but prevents repeat application; - **at-least-once processing** retries toward delivery but requires the consumer to tolerate duplicates; - **exactly-once effect** means one effective business mutation, normally requiring idempotency and transactional state—not merely a reliable topic; - **idempotent operation** produces the same effective state when the same logical request is applied again; - **ordering** must name a scope, such as events for one order, not imply one global order across all publishers; - **eventual consistency** is acceptable only when the convergence rule and maximum observation window are explicit. Current `WarehouseEvent` carries `schema_version`, `event_id`, process-run `sequence`, `correlation_id`, `causation_id`, timestamps, code/source, and entity IDs; `CellState` similarly carries `schema_version`, `run_id`, and monotonic `revision` within a run. `WH-STREAM-001` asserts unique/ordered observed events, stable request correlation, an empty initial causation, each subsequent event caused by the prior event ID, complete five-event case chains, and agreement with final state. With optional SQLite, `WH-STREAM-UNIT-001` proves pending outbox rows are ordered/marked after publication, while `WH-STREAM-UNIT-002` proves correlation/causation lineage survives checkpoint restore. The executable duplicate-delivery proof is `WH-OUTBOX-CRASH-001`. Controller generation 1 publishes `ORDER_ACCEPTED`, then an SQA-only failpoint performs `os._exit(86)` before its SQLite outbox row can be marked `published=1`. Generation 2 restores the same database and publishes the pending row again with the **same** `event_id`. The test observes two raw deliveries, while one live `EventIdDeduplicator` applies that logical acceptance once; the restored order and durable outbox contain one logical `ORDER_COMPLETE`, and the replayed row is eventually marked published. `WH-STREAM-UNIT-003` proves sequential duplicate suppression, and `WH-STREAM-UNIT-004` proves only one of 64 competing callback claims wins atomically. That helper stores seen IDs only in one process's memory. A consumer whose own restart must preserve apply-once effects needs a durable deduplication ledger committed atomically with its business-side effect. The POC proves at-least-once producer delivery plus live-process consumer deduplication; it does not prove exactly-once transport or a crash-safe exactly-once consumer. None of the stream metadata creates an audit-complete broker history or a global order across publishers. For stream tests, subscribe before the stimulus unless late joining is the behavior under test. Compare both streams: an event with no compatible state transition may expose a dual-write/serialization defect; a state change with no observed volatile event may be a subscription window or loss and must be triaged using QoS, bags, logs, and the query/snapshot path. ### 7.3 Chapter 8: transaction and consistency boundary ACID is useful vocabulary only when its scope is stated: | Property | What the POC provides | Boundary/limitation | |---|---|---| | Atomicity | `submit_order` validates identity, lines, picker, and complete SKU availability before creating the order and reserving selected cases. Expected validation rejection does not partially reserve. When SQLite is enabled, a checkpoint and new outbox rows are written in one local SQLite transaction at the publish tick. | There is no general rollback if an unexpected exception occurs after in-memory mutation, and ROS publication plus response cannot be one SQLite/DDS transaction. | | Consistency | Domain invariants constrain ownership, counts, completion, collision envelope, and legal modeled state. | Invariants are application rules, not a database constraint system; tests and the health monitor must observe them. | | Isolation | The live controller uses a reentrant lock around engine mutation while a four-thread executor can invoke reentrant callbacks. | This is process-local serialization, not distributed transaction isolation. A blocking callback can still starve progress. | | Durability | Optional: a non-empty `state_db_path` checkpoints the full engine, processed-request ledger, in-flight routes/jobs, pending events, and ordered outbox with SQLite WAL/`synchronous=FULL`; startup can restore it. | The default path is empty/in-memory. `WH-RESTART-001` verifies a live crash and explicit replacement; launch-managed respawn itself, replicated storage, broker durability, backup/restore, and high availability remain outside the proof. | `WH-TXN-001` translates a classic lost-update race into ROS 2: two service clients are released together to request the last available case. One request must accept and reserve it; the other must reject without mutation. The test should assert results plus the authoritative snapshot, not infer correctness only from the controller's use of a lock. The state/event/response sequence remains a **dual-publication boundary**, not one distributed commit. The engine mutates state and appends an event during the callback. With persistence enabled, mutation is checkpointed before a service/action response escapes. On a timer tick, the updated checkpoint and drained events are atomically written to SQLite/outbox; pending rows are published and then marked. This closes useful local crash windows but deliberately permits duplicate publication after publish-before-mark, which is why stable `event_id` matters. `WH-OUTBOX-CRASH-001` now exercises that exact interval rather than merely reasoning about it. DDS delivery and SQLite commit are not atomic, and the design has no multi-controller consensus or external broker transaction. ### 7.4 Language-independent concurrency model Concurrency is possible work overlap; parallelism is simultaneous execution. A process owns an address space; threads share process memory. Asynchronous I/O allows a task to wait without necessarily creating a thread. In this POC, a `MultiThreadedExecutor`, a reentrant callback group, timers, services, subscriptions, and the action callback can overlap, while an `RLock` guards the engine. | Risk | Meaning | ROS 2 symptom/test idea | |---|---|---| | Race condition | Outcome depends on uncontrolled interleaving. | Two order callbacks reserve the same case unless mutation is serialized; exercise with a barrier, many iterations, and varied seeds. | | Critical section/atomicity | Related reads and writes must appear indivisible within a stated scope. | Validate and reserve the complete order under one engine mutation boundary. | | Deadlock | Participants wait forever for resources held by one another. | A callback holds the engine lock while synchronously waiting on work that needs the same executor/lock. Capture all thread stacks. | | Livelock | Callbacks run and retry but useful state never advances. | AGVs continually yield/replan without delivery; time advances while order progress does not. | | Starvation | One callback or job is indefinitely denied service. | State timers or one picker's queue stop progressing under repeated service/action traffic. | | Thread safety | Shared state remains correct for all allowed interleavings. | Repeated concurrent service/action/fault calls preserve invariants and yield one terminal outcome. | | Backpressure | Producers outpace bounded consumers/queues. | State/events arrive faster than a slow subscriber processes them; define whether samples drop, queue, or are reconstructed. | Concurrency tests need synchronized starts and repeated interleavings, not just two requests submitted sequentially. Preserve time bounds and stack/trace evidence so a hang can be distinguished from slow simulation. ### 7.5 Chapter 2: nonfunctional requirements and percentiles **Performance** asks how the system behaves at workload X. **Scalability** asks how behavior changes as X or available resources change. One quick response does not establish either, and average latency hides operationally important tails. For a finite sample sorted by duration, report at least p50, p95, p99, maximum, sample count, warm-up policy, and failures/timeouts. State the percentile convention used by the tool; small samples make p99 effectively a maximum and support characterization, not a strong capacity claim. Measure wall-clock latency with a monotonic clock and retain logical plant time as separate domain evidence. Useful ROS/warehouse measures include: - service round-trip latency and action goal-to-result latency; - submission-to-first-progress and submission-to-pallet completion latency; - `/warehouse/state` inter-arrival gaps and stale-state intervals; - completed orders/items per wall minute and per logical plant minute; - active/queued orders, picker queues, event backlog or dropped-sample diagnostics; - process RSS/CPU, callback/executor latency, simulator real-time factor, and cleanup time; - recovery time after a fault/pause, plus invariant and explicit-rejection counts. Use distinct test shapes: | Shape | Controlled variable | Required interpretation | |---|---|---| | Fixed-workload performance (`WH-PERF-001`) | Ten seeded pure-engine `surge_wave` runs | Emits non-gating p50/p95/p99 wall runtime, logical cycle time, throughput, and failure rate while gating correctness/deadline. It is not live ROS latency, physical cycle time, or production capacity. | | Load | Representative expected workload | Demonstrates behavior at that load with functional invariants still enabled. | | Stress/capacity | Increasing demand until a limit/degradation | Finds the knee/failure mode and proves rejection is explicit and state-safe. | | Soak (`WH-SOAK-001`, planned) | Fixed workload over many waves or a fixed duration | Looks for leaks, backlog growth, identifier reuse, stale state, and degradation over time. | | Scalability (`WH-SCALE-001`) | Same 3-order/12-case/two-picker surge wave with 1/2/4 AGVs, replayed five times per fleet to prove repeatability | Compares deterministic logical cycle and throughput values while retaining completion/invariant gates; the repeats are not independent workload samples and make no linear-scaling assumption. | | Topology/failover/DR | Changed placement, component loss, standby takeover, or restore | Requires architecture that the POC does not currently implement. Restart is not failover; deterministic reset is not disaster recovery. | `WH-SCALE-001` currently records this deterministic logical baseline: | Modeled fleet | Median logical cycle | Median logical throughput | Incremental cycle reduction | |---:|---:|---:|---:| | 1 AGV | 217.25 s | 3.314 items/min | baseline | | 2 AGVs | 116.10 s | 6.202 items/min | 46.559% versus 1 AGV | | 4 AGVs | 115.60 s | 6.228 items/min | 0.431% versus 2 AGVs | The first additional vehicle removes substantial transport pressure; two more vehicles add almost no benefit for this fixed two-picker/single-dock-per-picker workload, so the modeled bottleneck has moved toward picker/dock service. That is an interpretation of this deterministic engine configuration, not a physical capacity claim or proof that every workload saturates at two AGVs. The test gates completion, invariant safety, and ordering of the medians; the exact values are characterization properties for trend review. Performance evidence must record host/OS, ROS/RMW/Gazebo versions, visual/headless mode, scenario/seed, rates, workload, sample count, and instrumentation. Do not use GUI, GDB, verbose tracing, or debug-log timing as a baseline unless their perturbation is the subject under test. ### 7.6 Chapter 5: encoding and evolution There are two important encoding boundaries: 1. `warehouse_interfaces` `.msg`, `.srv`, and `.action` definitions generate strongly typed ROS support. Field names, types, array alignment, units, state vocabulary, and QoS together form the semantic contract. 2. `/warehouse/metrics` is a `std_msgs/String` containing schema-versioned JSON. Version 1 requires `schema_version`, `scenario`, `simulation_time_sec`, `completed_orders`, `palletized_items`, `throughput_items_per_min`, and `emergency_stop`; the current producer also supplies `run_id`, `revision`, and `queue_depth`, which the parser safely defaults for older version-1 documents. Do not assume a ROS interface change is wire-compatible merely because it “only adds a field.” Generated publishers/subscribers may have been built from different IDL, recordings embed type metadata, and compatibility depends on the ROS/RMW/type-evolution capabilities actually deployed. A safe change defines supported old/new combinations, rebuilds isolated producer/consumer fixtures, checks discovery and deserialization, and replays representative bags where supported. If no mixed-version support is promised, reject it clearly and coordinate deployment. For the JSON projection, prefer a tolerant reader that requires known fields/types, ignores additive unknown fields, and supplies documented defaults only for fields declared optional. Malformed JSON, a type change such as `completed_orders: "3"`, renamed/removed fields, and semantic/unit changes must be visible failures. The umbrella `WH-SCHEMA-001` scenario protects current round-trip, older/additive document behavior, and malformed/type errors; it does not prove arbitrary ROS IDL compatibility. ROS interfaces that lack an explicit schema field still need a release compatibility declaration. A version number does not replace semantic documentation: `throughput_items_per_min`, for example, must state whether its denominator is logical or wall time. Treat stored bags and test fixtures as versioned consumers during upgrade testing. ### 7.7 Senior functional SQA reasoning checklist For a scenario such as “the scheduler/controller stops after assigning a case,” reason aloud in this order: 1. What is the authoritative state, and which copies are projections or visualizations? 2. What was the last operation that could have committed, and where is its transaction boundary? 3. Which evidence may be stale, volatile, delayed, duplicated, missing, or out of order? 4. Does the client know failure, success, or only an uncertain outcome? 5. Is a retry safe under the same business/idempotency key? What prevents duplicate physical work? 6. What survives a process pause, process exit, host loss, or restart? What does not? 7. How will software state be reconciled with the simulated/physical case and AGV location? 8. Which invariant is safety- or inventory-critical, and what is the first invalid transition? 9. What workload and timing distribution expose the risk, including the tail rather than only the mean? 10. How will the test synchronize the fault, remain bounded, reproduce with seed/configuration, and preserve logs/state/events/bag/stack evidence? 11. Is this verification of a stated contract, validation of intended use, or exploratory discovery? 12. What limitation prevents the result from being generalized to a physical or production warehouse? 13. At which layer was a communication fault injected—application proxy, DDS/RMW, kernel/network, or infrastructure—and which lower layers remain untested? 14. In an orchestrated deployment, what distinguishes process liveness, application readiness, durable state, Pod replacement, rescheduling, and true failover? A strong answer connects architecture, failure model, oracle, automation, and recovery. It does not stop at “kill the node and see whether the test passes.” ## 8. ROS 2 concepts a functional SQA engineer needs ### 8.1 ROS graph and discovery ROS 2 processes host nodes; nodes communicate through a DDS-backed graph. Discovery is asynchronous, so a process existing does not prove its publisher, subscriber, service, or action is ready. Useful inspection commands: ```bash ros2 node list ros2 node info /node_name ros2 topic list -t ros2 service list -t ros2 action list -t ros2 param list /node_name ros2 doctor --report ``` Test implication: assert the graph contract at startup and produce a graph snapshot on failure. Avoid tests that pass merely because a process remained alive. ### 8.2 Topics Topics are asynchronous streams. They suit state, telemetry, and events. Publishing does not imply that a subscriber received a sample. A topic test needs to account for discovery, QoS compatibility, publication rate, and whether a message is a full snapshot or a delta. ```bash ros2 topic info /warehouse/state --verbose ros2 topic echo /warehouse/state --once ros2 topic hz /warehouse/state ``` Functional questions include: Is state complete and internally consistent? Are event IDs unique? Are timestamps meaningful? Can a slow or late subscriber recover? ### 8.3 Services Services are bounded request/response interactions. They are appropriate for queries and quick commands, not long-running warehouse work. Tests should distinguish service discovery timeout, transport timeout, explicit rejection, and server exception. `/warehouse/submit_order` also demonstrates application idempotency: clients send a stable `request_id`; `disposition` distinguishes `ACCEPTED`, equivalent `REPLAYED`, content `CONFLICT`, and validation `REJECTED`. The response field is business meaning layered above ROS service transport. A future completing only proves that some response arrived; an abandoned/timed-out future requires reconciliation and same-key retry. Gazebo's native Transport service `/world/warehouse/set_pose` is used by the visualizer. It is not a ROS service and is not exposed through a bridge in this POC. A successful call means Gazebo accepted the pose request; it does not by itself prove the domain task succeeded. ```bash gz service -l | rg '^/world/warehouse/set_pose$' gz service -i -s /world/warehouse/set_pose ``` The adapter uses `gz.transport13` with `gz.msgs10.Pose` and `gz.msgs10.Boolean`. ROS-side service commands will not list this native Gazebo endpoint. ### 8.4 Actions Actions model work with goal acceptance, feedback, cancellation, and a terminal result. They are a natural fit for move, retrieve, or fulfill operations. Tests must cover: - accepted and rejected goals; - feedback order and correlation; - successful, aborted, and cancelled terminal states; - cancellation races; - server disappearance and client timeout; - exactly one result per accepted goal. An action client's future completing is not the same as a successful domain result—assert both the action terminal code and result payload. `/warehouse/fulfill_order` carries the same `request_id` and result `disposition` as service submission. Idempotent replay protects the order/reservation effect, but a controller restart still replaces the action server: restored domain work does not reconnect a goal future owned by the dead server. ### 8.5 Parameters Parameters configure nodes. Record non-default parameters in every test artifact because scenario, seed, time scale, automatic-order behavior, update periods, `state_db_path`, and `restore_from_checkpoint` can change outcomes. `controller_respawn` is a launch argument controlling process orchestration rather than a node parameter. ```bash ros2 param dump /node_name ros2 param describe /node_name parameter_name ros2 param get /node_name use_sim_time ``` Dynamic parameter tests should verify rejection of invalid values and whether a change applies atomically. Avoid changing configuration mid-test unless that is the behavior under test. ### 8.6 QoS ROS 2 delivery depends on compatible Quality of Service policies: - **Reliability:** reliable retries delivery; best-effort may drop samples. - **Durability:** transient-local can supply retained samples to late joiners; volatile cannot. - **History/depth:** controls queued samples and overflow behavior. - **Deadline/liveliness/lifespan:** enable health contracts when explicitly configured. The implemented profiles are: | Endpoint | History/depth | Reliability | Durability | Rationale | |---|---|---|---|---| | Controller `/warehouse/state` publisher | keep-last 1 | reliable | transient-local | Current aggregate snapshot is recoverable by late joiners. | | Controller `/warehouse/events` publisher | keep-last 100 | reliable | volatile | Ordered live operational evidence without treating old events as new after subscription. | | State visualizer subscriber | keep-last 1 | best-effort | volatile | Rendering keeps only the newest state and cannot backlog the controller. | A reliable publisher is compatible with a best-effort subscriber, but the visualizer intentionally gives up retained late-joiner behavior and relies on the periodic state stream. Critical events must not silently use a profile that allows loss unless a queryable state model repairs it. The actual discovered endpoints remain the contract; inspect them with `ros2 topic info --verbose` and test incompatible-profile diagnostics. ### 8.7 ROS time, simulation time, and rates The implementation has three distinct notions of time: - **logical plant time:** `CellState.simulation_time_sec`, advanced by `simulation_speed / tick_hz` on each controller wall-timer callback and reset by `/warehouse/reset`; - **ROS/wall timestamps:** message headers and node timers use the ROS system clock because `use_sim_time` is false; - **Gazebo time/real-time factor:** maintained inside Gazebo but not bridged to ROS or used to drive the controller. The unified launch defaults to `tick_hz=20` and `simulation_speed=3.0`. Pausing Gazebo only pauses its own simulation; logical warehouse work continues and the native-Transport visualizer continues to request poses on wall time. Tests must report logical and monotonic wall time and must not infer domain pause from the GUI pause button. For a future physics-coupled design, bridge `/clock`, set every participating node's `use_sim_time` consistently, and add bounded pause, resume, time-jump, and timer-burst tests. A harness should still supervise with a monotonic wall deadline because a ROS-time wait can hang while `/clock` is paused. ### 8.8 Executors, callbacks, and concurrency `rclpy` callbacks run through an executor. Single-threaded execution is easier to reason about; multithreaded execution exposes races and requires appropriate callback groups and synchronization. Blocking inside a callback can starve subscriptions, services, action feedback, and timers. Test symptoms of executor starvation include a healthy process with frozen state, service calls that never complete, and teardown hangs. Capture thread stacks and callback-related logs before killing the process. ### 8.9 Launch and `launch_testing` ROS launch files describe processes, parameters, remappings, namespaces, environment, and shutdown. `launch_testing` adds active tests while processes run and post-shutdown assertions after they exit. A robust launch test should: - start only the required processes; - use a readiness action/event or bounded graph probe; - assert process liveness while testing; - request shutdown; - assert exit codes and inspect stderr after shutdown; - isolate its ROS domain if parallel execution is possible. Use ordinary `pytest` for pure domain logic. Use `launch_testing` when process boundaries, launch arguments, DDS discovery, signals, or exit codes are part of the contract. The current live suite uses `scripts/run_tests.sh` to own the unified launch and a black-box `rclpy` pytest client to drive it. That is appropriate for reusable end-to-end scenarios. Add `launch_testing` cases when an assertion specifically depends on launch events, an individual process's exit code/stderr, required-process shutdown propagation, or alternate launch topology; do not claim such coverage from a functional state assertion alone. ### 8.10 Node composition Composable C++ nodes can share a process and optionally use intra-process communication. Composition lowers overhead but changes the failure and debug boundary: one crash can remove several components, process-level GDB cannot isolate a single component, and intra-process delivery can differ from DDS delivery. For SQA: - run at least one non-composed configuration when debugging an individual component; - test the deployment configuration actually shipped; - retain component/node names and identifiers in logs; - understand that `ros2 node list` shows nodes, while the OS process list may show one container; - disable intra-process communication temporarily if transport behavior is under investigation, documenting the diagnostic change. Python `rclpy` nodes in this POC are normally separate processes unless the launch implementation states otherwise; do not claim composition coverage based solely on several nodes started by one launch file. ### 8.11 Namespaces, remapping, and ROS domains Names determine graph connectivity. A typo or namespace mismatch can look like an idle but healthy system. Capture fully qualified names and remappings in artifacts. `ROS_DOMAIN_ID` isolates DDS discovery, but it is not a security control and does not replace per-test state isolation. ```bash export ROS_DOMAIN_ID=71 ros2 node list ``` Choose domain IDs through the test runner rather than hard-coding one shared value when tests can run concurrently. ### 8.12 Bags, tracing, and introspection `ros2 bag` records selected topics for replay and forensic analysis. It does not capture service calls or action semantics completely unless their underlying topics are included and interpreted. Avoid recording secrets or unbounded high-rate streams. ```bash ros2 bag record -o artifacts/bags/WH-FAULT-001 \ /warehouse/state /warehouse/events /warehouse/metrics /diagnostics /rosout ros2 bag info artifacts/bags/WH-FAULT-001 ``` ROS 2 tracing (`ros2 trace`) can expose callback and executor latency when `ros-humble-ros2trace` and its LTTng prerequisites are installed. They are absent on the current validation host, so no runtime ROS trace is claimed in this revision. Treat that as an unavailable diagnostic capability, not as a passing trace test. When installed elsewhere, use it selectively because trace volume and instrumentation overhead can perturb timing. ## 9. Python, framework, and testing concepts used ### 9.1 Package and build model ROS Python packages use `ament_python`; interface packages normally use `ament_cmake` with ROS interface generators. `colcon` builds a workspace in dependency order into `build/`, `install/`, and `log/`. After every build, source the workspace overlay in the shell that runs nodes or tests: ```bash source /opt/ros/humble/setup.bash ./scripts/build.sh source install/setup.bash ``` The helper runs a `--symlink-install`/`RelWithDebInfo` build with ROS Humble's `/usr/bin/python3` ABI. This shortens the Python/data edit loop and avoids Conda poisoning generated interfaces. Rebuild after changing interfaces, package metadata, entry points, CMake, or installed launch/config/assets. ### 9.2 `rclpy` `rclpy` supplies Python nodes, publishers, subscribers, clients, services, actions, timers, parameters, clocks, callback groups, and executors. Important testing practices are: - initialize and shut down each ROS context cleanly; - destroy nodes so DDS participants do not leak across tests; - spin futures with finite deadlines; - avoid module-global mutable cell state; - keep domain logic independent of ROS callbacks where possible, enabling fast unit tests; - log structured identifiers with messages. ### 9.3 `pytest` The Python suite uses fixtures for setup/teardown, markers for selection, parametrization for boundary tables, and assertion introspection for useful failures. Prefer: - small fixtures with explicit scope; - `yield` fixtures that guarantee teardown; - `pytest.approx` for meaningful floating-point tolerances; - `monkeypatch` for pure seams, not as a substitute for ROS integration tests; - deterministic fake clocks in unit tests; - bounded eventual assertions for asynchronous observations; - JUnit XML plus Markdown reporting for automation and human review. Avoid raw `time.sleep()` as synchronization. A short polling interval within a bounded, diagnostic `wait_until` helper is acceptable when no event/future interface exists. ### 9.4 State machines and data models Use enums for states, dataclasses or validated structures for domain records, and explicit transition functions. Test transition tables parametrically: valid transitions succeed, invalid transitions are rejected without partial mutation, terminal states are terminal, and retry counters/causes are preserved. Serialization boundaries deserve separate tests. A valid Python object is not proof that the equivalent ROS message contains every field or uses correct units. ### 9.5 Error handling and observability Expected domain failures should be results with stable reason codes. Unexpected programmer/process failures should fail loudly and preserve a traceback or core. Logs should identify scenario, order, task, AGV, case, picker, pallet, state, and cause where applicable. Assertions should explain expected versus observed data and include the last known state. Do not reduce errors to a generic `False` or timeout message. ## 10. Gazebo visual and headless workflows ### 10.1 Default visual run Visual mode is the default so an SQA engineer can watch assignments, AGV motion, case delivery, and pallet growth while assertions execute. ```bash ./scripts/run_demo.sh ./scripts/run_tests.sh --suite e2e ``` The Gazebo-specific launch can be run directly for isolation: ```bash ros2 launch warehouse_gazebo warehouse.launch.py \ headless:=false scenario:=mixed_wave seed:=7 \ auto_start_orders:=true simulation_speed:=3.0 \ paused:=false state_visualizer:=true verbosity:=2 ``` The unified launch also exposes `controller`, `health_monitor`, `auto_start_delay_sec`, `fleet_size`, `tick_hz`, `world`, `world_name`, `state_topic`, `visualizer_rate`, `transport_timeout_ms`, recording, `debug`, `gdb_node`, `gdb_gazebo`, and shutdown arguments. `gdb_gazebo:=true` forces headless mode and captures Gazebo's native backtrace in batch GDB. Query the installed launch description with `--show-args` before assuming defaults at a different revision. The ordinary demo is deliberately in-memory. For an attended opt-in recovery exercise, use an explicit database outside source-controlled paths: ```bash # First run creates/updates the SQLite checkpoint and event outbox. ./scripts/run_demo.sh --state-db /tmp/warehouse-recovery.sqlite3 \ --respawn-controller # Start a deliberately fresh persisted exercise, replacing prior DB state. ./scripts/run_demo.sh --state-db /tmp/warehouse-recovery.sqlite3 \ --fresh-state --respawn-controller ``` `--respawn-controller` requires `--state-db`; without `--state-db`, restart creates normal fresh in-memory scenario state. Use a unique file per test/run and preserve or remove it according to artifact policy. The equivalent launch arguments are `state_db_path`, `restore_from_checkpoint`, and `controller_respawn`. ### 10.2 Headless run Use headless mode for CI, remote machines without display forwarding, and long/repeated runs: ```bash ./scripts/run_demo.sh --headless ./scripts/run_tests.sh --headless --suite all ros2 launch warehouse_gazebo warehouse.launch.py headless:=true ``` Headless does not mean “without simulation”; it means the Gazebo server runs without the GUI client. `--no-sim` is a distinct test-runner option that prevents a managed simulator launch; use it for a pure suite or when targeting a separately managed running cell. ### 10.3 What to watch During a visual functional run, compare the scene to ROS evidence: - Is the expected AGV moving, and only while its state permits motion? - Does the case stay associated with one vehicle/location? - Does arrival occur at the assigned picker? - Does pallet content change exactly once per completed pick? - Does a held/faulted vehicle stop changing pose? - Do terminal task/order events agree with the visible state? Visual disagreement is a defect even if the domain assertions pass: it may indicate a visualizer mapping, entity-name, Gazebo Transport, or coordinate-frame error. Conversely, attractive movement is not evidence that business invariants pass. ### 10.4 Simulator readiness and failures Before scenario traffic, tests should confirm: - Gazebo server process is alive; - the Gazebo world and native Transport service are responsive; - the `warehouse` world and expected entities exist; - Gazebo Transport reports `/world/warehouse/set_pose` when visualization is enabled; - `/warehouse/state` has a publisher and the visualizer has subscribed; - initial pose/state synchronization completed. On failure, preserve the Gazebo server/client logs, ROS logs, launch output, world path, launch arguments, and graphics/session details. A GUI crash should not erase the domain test result; report simulator/rendering failure separately. ## 11. Debugging and diagnostic runs The runner's supported debug surface is: ```bash ./scripts/run_tests.sh --debug --suite integration ./scripts/run_tests.sh --debug --gdb-node state_visualizer --suite e2e ``` `--debug` enables richer diagnostics. The runner accepts `cell_controller`, `health_monitor`, `state_visualizer`, or `all` for `--gdb-node` and uses batch GDB to preserve native thread backtraces. These executables are Python, so GDB is primarily useful for native Gazebo Transport/RMW/interpreter crashes; use `pdb`, `debugpy`, `faulthandler`, or a Python traceback for Python logic. ### 11.1 Build for native debugging ```bash source /opt/ros/humble/setup.bash ./scripts/build.sh source install/setup.bash ``` `RelWithDebInfo` preserves symbols with more representative timing than a fully unoptimized build. Reproduce with `Debug` if variables were optimized out. ### 11.2 Run one executable under GDB ```bash ./scripts/debug_node.sh warehouse_core cell_controller ./scripts/debug_node.sh --batch warehouse_core cell_controller # Reproducible all-thread evidence at a native ROS wait boundary. ./scripts/capture_gdb_ros_stack.sh \ --output artifacts/observability/gdb-cell-controller-full-stack.txt ``` The helper explicitly runs Python console scripts through `/usr/bin/python3`, which avoids the executable-format error that GDB can produce when pointed directly at a shebang script. For a native ELF ROS executable, `ros2 run --prefix 'gdb -q -ex run --args' PACKAGE EXECUTABLE` is also appropriate. For a launch file, apply a `prefix`/`launch-prefix` to only the target node in the Python launch description or use the wrapper's `--gdb-node`. Avoid placing the entire multi-process launch under one GDB session; it obscures which child is controlled. `WH-OBS-GDB-001` has runtime-validated this path on the current host. It starts the installed controller through `/usr/bin/python3`, breaks at `rcl_wait`, runs `thread apply all bt full`, lists shared libraries, and asserts a native call chain through `rcl_wait`, `rclpy/_rclpy_pybind11`, and CPython's `_PyEval_EvalFrameDefault`, plus `rmw_fastrtps`/Fast DDS worker evidence. The retained 744-line artifact is [gdb-cell-controller-full-stack.txt](../artifacts/observability/gdb-cell-controller-full-stack.txt). This proves that native stacks are collectible and attributable; it does not prove domain correctness, Python exception diagnostics, or useful source locals in every optimized dependency. To investigate a Gazebo server crash itself: ```bash ros2 launch warehouse_gazebo warehouse.launch.py \ headless:=true debug:=true gdb_gazebo:=true ``` This launch path runs the Gazebo Ruby launcher/server chain under batch GDB and prints `thread apply all bt full`; `gdb` must be installed. Use node-level `gdb_node` separately for the Python ROS processes/native extensions. In GDB: ```text set pagination off run thread apply all bt full info sharedlibrary ``` Attach to an already running native process only during an attended debug session: ```bash pgrep -af 'target_executable' gdb -q -p PID ``` Attaching stops the process and can cause ROS deadlines, action timeouts, or safety monitors to fire. Those secondary failures must be distinguished from the original defect. ### 11.3 Core dumps Before reproduction in the same shell: ```bash ulimit -c unlimited ``` On a systemd-based host, inspect captured dumps with: ```bash coredumpctl list coredumpctl info PID_OR_EXE coredumpctl gdb PID_OR_EXE ``` Core storage policy is host-specific and may be disabled or handled by Ubuntu crash tooling. Do not change a shared machine's kernel/core policy without authorization. Core files can contain credentials, message payloads, and inventory/order data; handle them as sensitive artifacts. ### 11.4 Python debugging For an isolated Python executable: ```bash python3 -m pdb "$(ros2 pkg prefix PACKAGE)/lib/PACKAGE/EXECUTABLE" ``` For hangs or CI-safe stack evidence, enable Python fault handling: ```bash export PYTHONFAULTHANDLER=1 ./scripts/run_tests.sh --suite integration ``` Add temporary `faulthandler.dump_traceback_later(...)` or `debugpy` hooks only on an investigation branch or behind an explicit debug parameter. Never leave a public debug listener enabled in a normal launch. ### 11.5 Recommended evidence order For an intermittent failure, collect in this order: 1. Markdown/JUnit report and seed; 2. relevant node and launch logs; 3. ROS graph, parameters, QoS, and environment snapshot; 4. state/event timeline or small rosbag; 5. Python traceback or native core/backtrace; 6. ROS trace/performance profile if the earlier evidence suggests latency or starvation. See [DEBUGGING.md](DEBUGGING.md) for detailed recipes and cautions. ## 12. Reporting and Jira workflow ### 12.1 Artifact set Each run should produce or reference: - run ID, UTC timestamp, scenario IDs, seed, commit SHA and dirty-tree indicator; - OS, ROS distribution, RMW implementation, Gazebo version, Python version; - launch/test command with non-secret environment overrides; - pass/fail/error/skip counts and durations; - JUnit XML for machines; - Markdown summary for humans; - failing assertion and expected/observed state; - relevant logs and, when captured, bag/trace/core/screenshot paths; - cleanup status and list of any remaining processes. JUnit properties prefixed `measurement.` are rendered as a separate characterization table and retained in the JSON summary. They are observations for comparison, not production SLOs or pass thresholds; correctness/deadline assertions remain ordinary test results. Never paste secrets, access tokens, full environment dumps, or sensitive core contents into Jira. ### 12.2 Jira-ready defect template The report generator should produce one draft per actionable failure in Markdown that can be pasted into a Jira description: ```markdown ## [WH-FAULT-001] Task is duplicated after AGV recovery **Type:** Bug **Priority suggestion:** High **Environment:** Ubuntu 22.04; ROS 2 Humble; Gazebo Sim 8; RMW implementation **Build:** commit `` (`clean|dirty`) **Test:** `path/to/test_file.py::test_name` **Run ID / seed:** `` / `` ### Preconditions - Inventory/configuration summary - Visual or headless mode and relevant launch arguments ### Steps to reproduce 1. Build and source the workspace. 2. Run the exact sanitized command. 3. Observe the correlated order/task/vehicle IDs. ### Expected One terminal result; case ownership and inventory remain coherent. ### Actual Observed terminal results, state sequence, timestamps, and error text. ### Impact Business/safety/test impact without speculation. ### Evidence - `relative/path/to/junit.xml` - `relative/path/to/log` - `relative/path/to/bag-or-backtrace` ### Triage notes First bad state, suspected component boundary, reproducibility, and known workaround. ``` ### 12.3 Triage rules - Create one ticket per root-cause hypothesis, not one per repeated parametrized failure, unless evidence shows distinct defects. - Preserve the first failing transition and its identifiers. - Separate product defects, test defects, environment failures, and unsupported/skipped coverage. - Do not label a timeout as “performance” until graph readiness, QoS mismatch, controller timer progress, logical versus wall time, deadlock/starvation, and Gazebo health have been checked. - Attach or link raw artifacts; keep the ticket narrative short enough to scan. - Rerun with the same seed before changing configuration. If it is intermittent, report frequency and sample size. ## 13. Requirements-to-test traceability Status meanings: **Covered** has an executable test; **Partial** has some executable evidence but not every stated behavior; **Planned** describes intended coverage and must not be reported as passing. | Requirement | Acceptance summary | Primary layer/scenario | Evidence | Coverage status | |---|---|---|---|---| | `REQ-SYS-001` | Populated automated warehouse cell can be launched. | Gazebo smoke, `WH-NOM-001` | world/entity readiness and launch logs | Covered by POC smoke path | | `REQ-SYS-002` | Autonomous vehicles move items to picker destinations. | Integration and `WH-NOM-001` | vehicle/case state history; visible Gazebo motion | Covered by nominal path | | `REQ-SYS-003` | Pickers palletize delivered cases. | Unit pallet-pattern tests and `WH-NOM-001` | pallet contents, placement layer, and inventory deltas | Covered by nominal path | | `REQ-VIS-001` | Tests are visible in Gazebo by default. | Runner smoke/e2e | default command, Gazebo client process/log | Covered by runner contract | | `REQ-VIS-002` | Headless option is available. | Headless smoke/e2e | `--headless` result | Covered by runner contract | | `REQ-TST-001` | Low-level tests cover domain logic. | `unit` | pytest/JUnit | Covered | | `REQ-TST-002` | Integration tests cover ROS/process boundaries. | `integration`, `system`, `ros` | final JUnit, launch/process logs, ROS observations | Covered for the POC interfaces and failure seams | | `REQ-TST-003` | Top-level realistic functional tests exist. | `e2e`, `WH-NOM-001..002`, `WH-ROUTE-001`, `WH-SAFE-001`, `WH-FAULT-001..002`, `WH-ACTION-001` | scenario timelines and final report | Covered for the simulated POC workflows; all passed in final validation | | `REQ-TST-004` | Suite is implemented in Python. | All pytest layers | collected tests and source | Covered | | `REQ-DBG-001` | Debug runs support useful GDB traces. | `WH-OBS-GDB-001` plus runner/manual recipes | 744-line controller stack with ROS/rclpy/CPython/Fast DDS frames | Covered for native-stack collection around a Python ROS node; Python logic still uses Python tools | | `REQ-REP-001` | Human-readable, Jira-friendly reporting is generated. | Reporting self-tests/full run | Markdown report and JUnit | Covered by reporting path | | `REQ-DATA-001` | An uncertain application response can be reconciled without duplicate work. | `WH-DIST-001`, `WH-NET-002`, `WH-NETEM-001` | client deadline/abandonment, UDP impairment, authoritative state, `REPLAYED`, one effect | Covered for deterministic application and same-host Fast DDS UDP-loopback boundaries; external/asymmetric/long-duration networks remain untested | | `REQ-DATA-002` | Optional local state/outbox recovers one in-flight order after controller loss. | `WH-RESTART-UNIT-001`, `WH-RESTART-001` | run/revision/order plus ROS/SQLite completion evidence | Covered for explicit single-controller replacement; not HA/failover/DR | | `REQ-DATA-003` | State/events expose version, identity, ordering, correlation, and duplicate-handling contracts. | `WH-STREAM-UNIT-001..004`, `WH-STREAM-001`, `WH-QOS-001`, `WH-OUTBOX-CRASH-001` | message metadata, observed stream, retained state, repeated `event_id`, apply-once consumer effect | Covered for one publisher and one live in-memory deduplicating consumer; no durable consumer ledger/global broker order | | `REQ-DATA-004` | Concurrent last-stock requests cannot double reserve inventory. | `WH-TXN-001` | service results and authoritative snapshot/invariants | Covered; live scenario passed in final validation | | `REQ-NFR-001` | A fixed warehouse workload reports controlled percentile observations. | `WH-PERF-001` | sample count, p50/p95/p99 wall/cycle/throughput, failure rate | Partial characterization; no max/resource/live-ROS measure or production threshold/capacity claim | | `REQ-NFR-002` | Fixed demand can be compared across supported modeled fleet sizes. | `WH-SCALE-UNIT-001..003`, `WH-SCALE-001` | 1/2/4-AGV logical cycle/throughput medians and invariant results | Covered as deterministic engine characterization; not physical or live ROS capacity | | `REQ-DEP-001` | Container/Kubernetes assets express one persistent, readiness-gated controller topology. | `WH-K8S-STATIC-001..002`, `WH-K8S-GATEWAY-001`, host-blocked runtime `WH-K8S-001` | parsed manifest, image/script contract, live gateway freshness, future Pod UID and pre/post state evidence | Partial: static/gateway covered; Pod runtime blocked on current host prerequisites | | `REQ-SCH-001` | Version-1 JSON metrics remain parseable across documented additive/older variants. | `WH-SCHEMA-001` | metrics unit results | Covered for JSON; ROS IDL mixed-version compatibility planned | | `REQ-DOC-001` | System components are inventoried. | Documentation review | Section 3 | Covered | | `REQ-DOC-002` | Test paradigm/components are documented. | Documentation review | Sections 5-6 | Covered | | `REQ-DOC-003` | ROS/debug concepts relevant to functional SQA are documented. | Documentation review | Sections 7-11 | Covered | | `REQ-DOC-004` | ROS framework, Python, and testing concepts are documented. | Documentation review | Sections 8-9 | Covered | The test report should replace broad “Covered” labels with concrete node IDs such as `test_file.py::test_name`. Keep planned rows visible so a green run cannot be mistaken for complete requirement coverage. ## 14. Entry, exit, and release criteria ### 14.1 Test entry criteria - Workspace builds successfully from a clean shell. - Required ROS/Gazebo versions are recorded. - No unrelated ROS graph is visible in the selected domain. - World and required nodes reach readiness within the configured timeout. - Test inventory/configuration and seed are known. - Artifact directory is writable and has enough space. ### 14.2 POC acceptance criteria - Unit and contract suites pass. - Required integration tests pass in a clean ROS domain. - `WH-NOM-001` passes in default visual mode and headless mode. - Safety/fault scenarios either pass their defined criteria or are explicitly reported as planned/unsupported; they are never silently omitted. - No unexpected node crash, Python exception, simulator exit, or orphan process occurs. - Markdown and JUnit reports are generated and contain reproducible commands. - Known limitations and environment-specific skips are reviewed. This is POC acceptance, not production release qualification. ## 15. Known limitations and safety disclaimer - The POC uses simplified kinematics and/or state-driven pose visualization. It does not establish localization accuracy, traction, braking distance, load stability, collision avoidance, manipulator reach, gripper behavior, or physical cycle time. - Gazebo geometry and contacts are illustrative. Visual non-intersection is not a validated safety clearance. - The “safety” scenarios test software state/hold behavior only. They do not implement or certify safety-rated hardware, PLC logic, emergency-stop circuits, guarded zones, risk reduction, or compliance with industrial robotics/vehicle standards. - Network, DDS, clock, CPU, GPU, and simulator behavior vary by host. Timing thresholds require platform characterization before they can support performance claims. - The inventory/order model is intentionally small and does not cover a commercial warehouse-management system's reconciliation, audit, exception, cybersecurity, or high-availability requirements. - Optional SQLite persistence is a single-controller SQA seam. It does not provide replication, leader election, split-brain protection, an external durable broker, backup/restore qualification, declared RPO/RTO, or disaster recovery. The normal demo remains in-memory unless `--state-db`/`state_db_path` is supplied. - The network-fault relay is an application-level service test double; do not present its modes as packets. `WH-NETEM-001` separately proves short same-host Fast DDS UDP loopback delay/loss with `tc netem`, but not external hosts, asymmetric/long-duration partitions, arbitrary network faults, or physical-network behavior. - The Kubernetes manifest is a local kind exercise with one replica, `Recreate`, local `hostPath` storage, and an HTTP projection sidecar. Static/gateway checks passed; runtime `WH-K8S-001` was the final run's deliberate non-opted-in skip, and an explicit attempt failed its missing-prerequisite gate. It is not a portable or highly available production deployment. - The pallet pattern creates additional layers without a configured maximum case count, mass, volume, stability, or load envelope. Capacity claims and over-capacity scenarios remain planned. - GDB, tracing, debug logging, and GUI rendering perturb scheduling. Do not use debug-run timing as a performance baseline. - GDB is installed on the validation host and `WH-OBS-GDB-001` produced the retained full-thread controller artifact. Other machines still need the `gdb` prerequisite; a missing debugger must skip/fail observability qualification rather than being reported as runtime evidence. - `ros2 trace` is unavailable on the validation host because `ros-humble-ros2trace`/LTTng support is not installed. No callback/executor trace result is claimed. - Headless success does not prove GUI/rendering health; visual success does not prove headless/CI isolation. - ROS domain separation is not authentication or authorization. - Core dumps, bags, reports, and logs may contain operational data. Apply retention and access controls appropriate to the environment. Do not connect this POC to physical actuators, production warehouse systems, or safety-related equipment without a separate system design, hazard/risk analysis, validated controls, site acceptance process, and qualified engineering review. ## 16. Maintaining this guide When code changes, update documentation in the same change when it affects: - a package, node, interface, topic/service/action name; - QoS, parameters, coordinate frames, state transitions, or units; - a launch argument or script option; - a scenario's preconditions or acceptance criteria; - an artifact/report path or schema; - an implemented versus planned traceability status; - a known safety, timing, simulation, or compatibility limitation. Before publishing the guide, validate commands against the installed overlay and compare the traceability table with `pytest --collect-only` output. Documentation is part of the test contract, not a substitute for executable evidence.