# Test Reporting and Jira Workflow

## Outputs

The automated run produces complementary evidence:

| Artifact | Audience | Purpose |
|---|---|---|
| JUnit XML | CI and tooling | Canonical per-test status, duration, message, captured output. |
| Markdown report | SQA, engineering, product | Scannable run summary and Jira-ready failure drafts. |
| JSON summary, when enabled | Dashboards/automation | Stable aggregate and normalized test-result data. |
| ROS/Gazebo/process logs | Triage | Timeline and component diagnostics. |
| Optional bag/trace/backtrace/core/screenshot | Deep diagnosis | Evidence for intermittent, timing, native-crash, or visual defects. |

The Markdown file does not replace the raw JUnit or logs. Keep them together under the run's artifact directory.

## Generate a report

The normal test wrapper creates reports automatically (under `artifacts/test-runs/<UTC>/` unless overridden):

```bash
./scripts/run_tests.sh --suite all --report-dir artifacts/warehouse-run
```

To convert one or more existing JUnit files/directories/globs:

```bash
python3 scripts/generate_report.py artifacts/results.xml \
  --output artifacts/warehouse-report.md \
  --json-output artifacts/warehouse-summary.json \
  --title 'Warehouse Cell Qualification Run' \
  --command './scripts/run_tests.sh --suite all' \
  --mode visual \
  --suite all \
  --fail-on-test-failures
```

Important behavior:

- invalid/missing JUnit returns an actionable report-tool error;
- `--fail-on-test-failures` makes a parsed failed/error result return non-zero;
- without that option, successful conversion does not mean the tests passed—read `Overall result`;
- empty JUnit produces `NO RESULTS`, which must not be treated as a pass;
- a run in which every collected case is skipped produces `NOT RUN`, also not functional evidence; the repository test runner returns non-zero for that condition;
- failed and errored cases receive individual Jira ticket drafts;
- passed-only reports explicitly state that no failure draft was produced.
- JUnit properties named `measurement.*` are rendered in a separate characterization table and retained in JSON; they are observations, not production SLOs.
- JUnit properties named `evidence.*` are rendered as an attachment table. The runner exports `WAREHOUSE_REPORT_DIR`, so scenarios such as `WH-OBS-GDB-001` and `WH-NETEM-001` place their raw stack/packet artifacts beside the report and record the exact path.

## Report structure

The generated Markdown includes:

1. overall result;
2. totals for passed, failed, error, skipped, and duration;
3. environment and exact reproduction command;
4. per-test table;
5. optional characterization measurements from `measurement.*` properties;
6. optional raw-artifact paths from `evidence.*` properties;
7. failure details and captured output;
8. one suggested Jira ticket per failed/error case;
9. attachment checklist.

Default environment collection is intentionally limited to useful, non-secret metadata: generation time, host, OS, Python executable/version, ROS distribution, RMW selection, Git commit, ROS domain, and Gazebo partition where available. Add scenario seed, Gazebo version, launch arguments, and dirty-tree status through the runner/artifact metadata where available.

## Jira creation workflow

1. Open the Markdown report and confirm the failure is not an environment error or expected skip.
2. Reproduce once with the reported command and seed before changing settings.
3. Group repeated parametrized failures only if the first invalid transition/evidence supports one root cause.
4. Copy the relevant `Ticket N` draft into Jira.
5. Replace generic expected text with the scenario's precise acceptance criterion.
6. Add business/safety/test impact without speculation.
7. Attach or link JUnit, focused logs, and diagnostic artifacts.
8. Record reproducibility (`N/M` runs), first bad state/event, and any safe workaround.
9. Redact secrets and sensitive operational data.

Suggested labels are `warehouse-cell`, `automated-test`, the scenario ID, and the outcome (`failed` or `error`). Choose priority based on actual impact: a software emergency-stop violation or inventory duplication is more serious than a Gazebo color/rendering defect, while a test-environment outage should not be filed as a product failure.

## What a high-quality ticket contains

```markdown
## [WH-SAFE-001] AGV pose changes while simulated emergency stop is asserted

**Test:** `test.e2e.test_safety_and_recovery.test_emergency_stop_halts_motion_then_allows_order_to_resume`
**Build / environment:** commit, dirty state, Ubuntu, ROS 2 Humble, Gazebo Sim 8, RMW
**Mode / domain:** visual or headless; ROS_DOMAIN_ID
**Scenario / seed:** `empty`; `3001`

### Preconditions
Three-case order accepted and a loaded AGV observed moving.

### Steps to reproduce
1. Build/source the recorded workspace.
2. Run the exact sanitized command from the report.
3. Correlate the stated order and AGV IDs.

### Expected
All AGV speeds reach the stopped tolerance and poses remain within the allowed
delta until the stop is reset.

### Actual
Include the first changing pose, timestamps, speed, event, and assertion.

### Impact
State the simulated software impact; do not claim physical safety certification.

### Evidence
- Markdown report and JUnit
- Focused controller/visualizer/Gazebo logs
- State/event bag or screenshot if captured
- Python traceback or GDB backtrace if relevant
```

## Classification guidance

| Classification | Examples | Ticket routing |
|---|---|---|
| Product/domain defect | Duplicate palletization, illegal transition, route crosses obstacle | Owning core/controller component. |
| ROS integration defect | QoS mismatch, wrong service field mapping, callback starvation | ROS adapter/unified-launch owner. |
| Gazebo/visual defect | Entity mapping or pose differs from correct ROS state | Gazebo visualizer/world owner. |
| Test defect | Wrong oracle, stale fixture data, unbounded wait | Test-suite owner; do not present as product failure. |
| Environment/infrastructure | Missing Harmonic Transport bindings, display/GPU failure, exhausted disk | Environment/CI owner; preserve setup diagnostics. |
| Unsupported/planned coverage | Explicit documented skip | Backlog/coverage item, not a passed or failed product test. |

Timeouts require triage before classification. Check graph readiness, QoS, controller timer/logical-time progress, executor starvation, process exit, Gazebo health, and the last state before describing a timeout as a performance defect.

## Artifact naming and retention

Use stable, sortable paths where the runner permits it:

```text
artifacts/<UTC-run-id>/
  test-report.md
  summary.json
  junit.xml
  pytest.log
  warehouse-launch.log
  colcon-build.log          # only when the runner builds
  metadata/                 # optional diagnostic expansion
  logs/                     # optional focused log copies
  bags/
  traces/
  backtraces/
  gdb-cell-controller-full-stack.txt
  netem-evidence.json
  screenshots/
```

Include scenario/test IDs in diagnostic filenames. Avoid spaces and machine-specific absolute paths inside tickets when a relative artifact link is sufficient.

Retention and access must account for the contents:

- core dumps can contain the complete process address space;
- bags/logs can contain orders, inventory, hostnames, and message payloads;
- screenshots can expose workstation or operational information;
- environment dumps can expose secrets and tokens.

Keep only the focused evidence required by policy. Never paste credentials or an unreviewed full environment into a ticket.

## Report self-tests

The report converter has pure unit coverage and can be validated without ROS/Gazebo:

```bash
PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 \
  python3 -m pytest -c test/pytest.ini test/unit/test_junit_to_jira.py -vv
```

This covers JUnit variants, totals, Markdown escaping/ticket drafts, JSON output, and actionable invalid-input errors. Any schema change should update these tests and this document together.

For scenario acceptance details, see [TEST_PLAN.md](TEST_PLAN.md). For diagnostic capture, see [DEBUGGING.md](DEBUGGING.md).