# Debugging ROS 2 and Gazebo Test Runs

## Start with a reproducible run

Record the scenario, seed, exact command, commit/dirty state, ROS domain, RMW, visual/headless mode, and timeout before changing anything. Then reproduce through the wrapper so logs and reports are retained:

```bash
export PYTHONFAULTHANDLER=1
./scripts/run_tests.sh --debug --suite integration \
  --report-dir artifacts/debug-run
```

For a failed e2e scenario, first retry that exact test with its original seed. Avoid “fixing” a timeout by only increasing the timeout; first determine whether discovery, QoS, controller timer progress, the logical/wall/Gazebo clock split, executor starvation, routing, or a crashed dependency stopped progress.

## Debug option map

| Symptom | Best first tool | Why |
|---|---|---|
| Python exception/assertion | Pytest traceback and captured logs | Preserves Python frames and local values. |
| Python node hangs | `faulthandler`, process/thread inspection | Reveals blocked callbacks/threads without a native-only view. |
| Native crash (`gz`, RMW, C/C++ component) | Core dump or GDB | Captures native threads, symbols, and shared libraries. |
| Healthy process but missing messages | ROS graph/QoS introspection | Commonly discovery, name, namespace, or incompatible QoS. |
| Frozen logical cell or scene | `CellState.simulation_time_sec`, Gazebo server log, pause state | Separates controller timer failure from a paused/stale Gazebo scene. |
| Latency/starvation | ROS trace after basic evidence | Shows callback/executor scheduling at higher artifact cost. |
| Correct ROS state, wrong visuals | Visualizer log and Gazebo entity/service checks | Isolates the state-to-scene adapter. |
| Several composed nodes disappear | Container core/backtrace | A composed process is the shared crash boundary. |

## Build with symbols

```bash
source /opt/ros/humble/setup.bash
./scripts/build.sh
source install/setup.bash
```

The helper uses `RelWithDebInfo`, the default diagnostic compromise, and pins the ROS Humble system-Python ABI. If GDB reports optimized-out locals, reproduce through a controlled manual build with `-DCMAKE_BUILD_TYPE=Debug`. Record the build type in the ticket because timing can change.

## Runner-controlled GDB

The supported workflow is:

```bash
./scripts/run_tests.sh --debug --gdb-node state_visualizer --suite e2e
```

The current runner accepts `cell_controller`, `health_monitor`, `state_visualizer`, or `all`. Target one process first. These nodes are Python and require Python-level tooling unless the suspected fault is in a native extension, DDS library, Gazebo binding, or interpreter runtime.

GDB is installed on the current validation host, and `WH-OBS-GDB-001` has exercised a real installed controller through the native ROS wait boundary. Other hosts must still satisfy the `gdb` prerequisite; `scripts/doctor.sh` reports it. Do not convert a prerequisite skip into observability evidence.

The repository helper avoids debugging the intermediate `ros2` CLI process and resolves the installed node script directly:

```bash
./scripts/debug_node.sh --batch warehouse_core cell_controller
./scripts/debug_node.sh --pdb warehouse_core cell_controller
```

Omit `--batch` for an interactive GDB session. The helper uses `/usr/bin/python3`, enables fault handling/core limits, and reports an actionable error when GDB is not installed.

For a deterministic ticket-ready ROS/Python/RMW stack capture:

```bash
./scripts/capture_gdb_ros_stack.sh \
  --output artifacts/observability/gdb-cell-controller-full-stack.txt
```

The helper starts the installed controller through `/usr/bin/python3`, stops at `rcl_wait`, records thread inventory, executes `thread apply all bt full`, and lists loaded shared libraries. `WH-OBS-GDB-001` asserts the resulting chain contains `rcl_wait`, `rclpy/_rclpy_pybind11`, CPython `_PyEval_EvalFrameDefault`, and `rmw_fastrtps`. The retained validation artifact is [gdb-cell-controller-full-stack.txt](../artifacts/observability/gdb-cell-controller-full-stack.txt): 744 lines including Fast DDS workers. This establishes native-stack observability, not Python-domain correctness or a timing baseline.

GDB is interactive and stops its target at breakpoints/signals. A visually monitored test may time out while the target is stopped; that is expected debugger perturbation. Increase only the attended debug timeout and do not treat debug timing as a performance result.

## Run one executable under GDB

For an installed native ROS executable:

```bash
ros2 run --prefix 'gdb -q -ex run --args' PACKAGE EXECUTABLE
```

Useful GDB commands:

```text
set pagination off
set print thread-events off
handle SIGINT nostop noprint pass
run
bt
thread apply all bt full
info threads
info sharedlibrary
info registers
```

Do not use `handle SIGSEGV nostop`: a segmentation violation is the evidence being investigated.

For an existing native process:

```bash
pgrep -af 'exact_executable_name'
gdb -q -p PID
```

Or capture a ticket-ready all-thread backtrace with the guarded helper:

```bash
./scripts/capture_backtrace.sh PID artifacts/backtraces/issue-key.txt
```

It validates that the explicit PID exists, records the command line/timestamp, and attaches batch GDB. It still pauses the target while attached.

Resolve the exact PID first. Attaching pauses the process and can trigger ROS deadline/liveliness events or harness timeouts in other processes. Note those as debugger side effects unless they preceded attachment.

## Apply GDB to one launched node

ROS launch starts several processes, so applying GDB to the whole launch command is usually ineffective. Add a debug-only `prefix` to the target `launch_ros.actions.Node`, or use the runner's `--gdb-node` support. A typical temporary launch fragment is:

```python
Node(
    package="native_package",
    executable="native_executable",
    name="target_node",
    prefix=["gdb -q -ex run --args"],
    output="screen",
    emulate_tty=True,
)
```

Keep this behind an explicit debug launch argument; normal automated runs must not wait for an interactive debugger.

## Run Gazebo itself under GDB

The unified launch has a dedicated batch-GDB path for the simulator:

```bash
ros2 launch warehouse_gazebo warehouse.launch.py \
  headless:=true debug:=true gdb_gazebo:=true
```

`gdb_gazebo:=true` forces headless mode, runs the Gazebo launcher/server under GDB, and prints `thread apply all bt full` on failure; `gdb` must be installed. The demo wrapper exposes the same intent as:

```bash
./scripts/run_demo.sh --headless --gdb-gazebo
```

This is separate from `gdb_node`, which targets `cell_controller`, `health_monitor`, `state_visualizer`, or all three. Do not use a batch-GDB run as a performance baseline.

## Core dumps

Enable core creation for processes launched from the current shell:

```bash
ulimit -c unlimited
./scripts/run_tests.sh --debug --suite e2e
```

On a systemd-based Ubuntu host:

```bash
coredumpctl list
coredumpctl info PID_OR_EXE
coredumpctl gdb PID_OR_EXE
```

Inside the resulting GDB session, run `thread apply all bt full`. Save a text backtrace alongside the test report; retain the original core only under an appropriate artifact policy.

Core handling varies by host and may be intercepted or disabled. Do not change a shared machine's `kernel.core_pattern`, systemd storage policy, or Ubuntu crash handler without authorization. A core can contain message data, credentials, environment values, and operational identifiers; treat it as sensitive.

## Python nodes and tests

### Preserve tracebacks and hang stacks

```bash
export PYTHONFAULTHANDLER=1
python3 -X faulthandler -m pytest -c test/pytest.ini test/unit -vv
```

For a suspected hang, a temporary debug-only hook can call:

```python
import faulthandler
faulthandler.dump_traceback_later(30.0, repeat=True)
```

Cancel the timer during clean shutdown. Keep the hook behind a debug parameter/environment flag so routine runs do not emit misleading stacks.

### `pdb`

Pure test:

```bash
PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 python3 -m pytest \
  -c test/pytest.ini test/unit/test_routing.py \
  --pdb -x -vv
```

`--pdb` enters at the failing assertion; `--trace` enters before test execution. For an installed Python ROS console script:

```bash
python3 -m pdb "$(ros2 pkg prefix warehouse_core)/lib/warehouse_core/cell_controller"
```

Source ROS and the workspace first so imports and generated interfaces resolve.

### `debugpy`

`debugpy` is optional and should not be a runtime dependency unless explicitly selected. A debug-only launch path can run:

```bash
python3 -m debugpy --listen 127.0.0.1:5678 --wait-for-client \
  "$(ros2 pkg prefix warehouse_core)/lib/warehouse_core/cell_controller"
```

Bind to loopback, use it only in a trusted attended environment, and never enable a waiting/public listener in default demo or CI launch.

### GDB around Python

Putting `python3` under GDB is helpful for a segfault in `rclpy`, the RMW/DDS layer, or another native extension. It generally does not replace a Python traceback for exceptions or domain logic. Collect both when the interpreter crashes.

## ROS graph and QoS triage

Capture a graph snapshot close to failure:

```bash
ros2 node list
ros2 node info /warehouse_cell_controller
ros2 topic list -t
ros2 topic info /warehouse/state --verbose
ros2 topic info /warehouse/events --verbose
ros2 service list -t
ros2 action list -t
ros2 doctor --report
```

Interpretation:

- no node: launch/process/startup failure or wrong ROS domain;
- node but no endpoint: callback setup, namespace/remapping, lifecycle, or initialization failure;
- publisher and subscriber but no data: QoS mismatch, executor starvation, stopped controller timer, or application state;
- transient startup miss: test did not wait for discovery/readiness;
- wrong fully qualified name: namespace/remapping contract defect.

`ROS_DOMAIN_ID` isolates discovery but is not a security mechanism. Include it in evidence and use a fresh value when stale external nodes are suspected.

## Clock and rate triage

The POC does not bridge Gazebo `/clock` into ROS. The controller and visualizer use wall timers; the controller advances a separate logical plant clock. Check it directly:

```bash
ros2 topic echo /warehouse/state --once
ros2 topic hz /warehouse/state
ros2 param get /warehouse_cell_controller simulation_speed
ros2 param get /warehouse_cell_controller tick_hz
gz topic -e -t /clock
```

`gz topic` shows Gazebo's own clock for comparison; the controller does not consume it. At default launch settings, a 20 Hz wall timer advances logical time at 3× wall time. Gazebo `paused:=true` does not pause domain work. Record the last `CellState.simulation_time_sec`, wall-clock elapsed time, and whether Gazebo time moves before classifying a freeze.

`/warehouse/reset` returns logical `simulation_time_sec` to zero but does not rewind Gazebo. This split is expected in the POC. If a future physics-coupled configuration bridges `/clock`, add consistent `use_sim_time` checks and retain a monotonic harness deadline so pause cannot hang evidence collection.

## ROS logging

Run with unbuffered Python and debug logs when supported:

```bash
export PYTHONUNBUFFERED=1
./scripts/run_tests.sh --debug --suite integration
```

ROS logs are normally under `~/.ros/log`; use the artifact copies produced by the runner for tickets because `latest` can change on the next process start. Correlate by scenario/order/task/vehicle/case/picker IDs rather than wall time alone.

Do not dump an entire environment or all logs into Jira without review. Redact secrets and keep high-volume artifacts as attachments/linked artifacts.

## Record a small rosbag

Record only the topics needed to reconstruct the defect:

```bash
mkdir -p artifacts/bags
ros2 bag record -o artifacts/bags/WH-FAULT-001 \
  /warehouse/state /warehouse/events /warehouse/metrics /diagnostics /rosout
```

Inspect before sharing:

```bash
ros2 bag info artifacts/bags/WH-FAULT-001
```

A bag does not automatically capture service request/response semantics or action meaning. Include related event/state messages and the exact stimulus in the report. Avoid recording unbounded high-rate streams.

## ROS tracing

When `ros-humble-ros2trace` and LTTng host support are installed, tracing can expose callback/executor delay:

```bash
ros2 trace -s warehouse_debug
```

Start a bounded scenario in another shell and stop tracing after the failure window. Trace only after ordinary logs, graph state, and stacks suggest scheduling latency; tracing produces large artifacts and changes timing.

Those tracing prerequisites are absent on the current validation host, so this repository revision has no runtime `ros2 trace` artifact. Record the capability as unavailable/blocked rather than claiming that a command example was executed. GDB evidence answers native stack-location questions; it is not a substitute for callback scheduling traces.

Questions tracing can answer include:

- Did the subscription callback run?
- Was a timer delayed behind a blocking callback?
- Is a multithreaded executor actually providing concurrency?
- Did callback duration grow across repeated waves?

## Node composition

Multiple nodes in one launch file are not necessarily composed. True composition puts components in one OS process and may use intra-process communication.

Debug consequences:

- attach GDB to the component container process;
- one native crash can remove every hosted component;
- a process backtrace needs node/component names in logs to attribute work;
- intra-process communication can bypass DDS behavior under test;
- isolate a component in its own process for diagnosis, then reproduce once in the deployed composition.

The Python POC normally runs separate `rclpy` processes. Do not claim composition coverage without an actual component container.

## Gazebo-specific triage

### World/server readiness

```bash
gz service -l | rg '/world/warehouse'
gz service -l | rg '^/world/warehouse/set_pose$'
gz service -i -s /world/warehouse/set_pose
gz topic -e -t /clock
```

If the world does not load, inspect the SDF parse error and Gazebo resource path. If the server is healthy but the GUI fails, compare a headless run and inspect display/GPU/client logs.

### State/visual mismatch

1. Capture one `/warehouse/state` sample.
2. Confirm the expected entity ID exists in the world.
3. Confirm `state_visualizer` is running and subscribed.
4. Check its entity-name mapping and coordinate conversion.
5. Inspect `/world/warehouse/set_pose` availability and response diagnostics.
6. Decide whether only the visual entity is stale or the ROS domain state is also wrong.

Screenshots are useful evidence but never the only oracle.

## Deadlock/hang checklist

When a process is alive but work stops:

1. Record the last state/event and both time bases.
2. Check whether logical time advances; inspect Gazebo time separately if the scene is stale.
3. Check graph endpoints and QoS.
4. Capture Python `faulthandler` output or native thread stacks.
5. Look for a callback waiting synchronously on a service/action handled by the same single-threaded executor.
6. Inspect callback groups and locks for nested acquisition.
7. Check if a debugger, paused GUI, or stdout pipe backpressure stopped a process.
8. Let the runner perform bounded cleanup and record any process that fails to exit.

## Evidence checklist for a ticket

- Exact sanitized reproduction command and seed
- Test node ID and scenario ID
- Commit/dirty state, ROS/Gazebo/Python/RMW versions, `ROS_DOMAIN_ID`
- Last coherent state and first invalid transition/event
- Relevant node/Gazebo logs
- Graph/QoS/parameter snapshot
- JUnit and Markdown report paths
- Bag/trace/backtrace/core reference if captured
- Visual/headless mode and whether the other mode reproduces
- Debugger-induced timing effects

See [REPORTING.md](REPORTING.md) for the Jira-ready workflow and [MASTER_SQA_GUIDE.md](MASTER_SQA_GUIDE.md) for the broader SQA model.