Bug: failure detail lost when test.init name differs from test word name #424

New issue

Closed

opened 2026-04-24 18:32:42 +00:00 by navicore · 1 comment

navicore commented

2026-04-24 18:32:42 +00:00

(Migrated from github.com)

Summary

When a test word calls test.init with a descriptive name that differs from the word name itself, the at line N: expected X, got Y detail produced by #422 is emitted by the runtime but dropped by the test runner — it never reaches the TEST FAILURES: section of the output.

Reproducer

# file: /tmp/test-name-mismatch.seq
: test-foo ( -- )
    "My Descriptive Test" test.init       # init name ≠ word name
    5 8 test.assert-eq
    test.finish
;

Run:

seqc test /tmp/test-name-mismatch.seq

Actual

Running tests in /tmp/test-name-mismatch.seq...
  test-foo ... FAILED

========================================
Results: 0 passed, 1 failed

TEST FAILURES:

/tmp/test-name-mismatch.seq::test-foo

Note the empty TEST FAILURES section — no at line N: expected X, got Y line.

Expected

Running tests in /tmp/test-name-mismatch.seq...
  test-foo ... FAILED

========================================
Results: 0 passed, 1 failed

TEST FAILURES:

/tmp/test-name-mismatch.seq::test-foo
  test-foo ... FAILED
    at line 3: expected 5, got 8

Confirmed by changing "My Descriptive Test" to "test-foo" (matching the word name) — the detail reappears.

Root cause (likely)

collect_failure_block in crates/compiler/src/test_runner.rs searches captured stdout for an exact header match constructed from the discovered word name:

let header = format!("{} ... FAILED", test_name);
...
if line == header {
    // collect following indented lines
}

But the runtime (patch_seq_test_finish in crates/runtime/src/test.rs) prints the header using the friendly name passed to test.init:

let test_name = ctx.current_test.as_deref().unwrap_or("unknown");
...
println!("{} ... FAILED", test_name);

When those differ, the exact-match lookup fails, collect_failure_block returns None, and the detail lines that follow on stdout are orphaned.

Suggested fixes (pick one)

Runtime side: have patch_seq_test_finish print the word name (the stable identifier the runner discovered) rather than the friendly init name. The friendly name could still appear as a prefix in the TEST FAILURES section for human readability.
Runner side: relax collect_failure_block to match any header ending in ... FAILED, or to scan by position rather than exact word name. (Slightly more fragile if multiple tests interleave output.)
Both: give the runner a way to know the init-given name for each test word (e.g. emit a machine-parseable marker like TEST_BEGIN test-foo "My Descriptive Test" on stdout), and match on either.

Option 1 is the simplest and matches the rest of the pipeline, which keys on word name.

Impact

Seqlings is unaffected (exercises don't use test.init), but any Seq test suite that uses descriptive init names will silently lose the new failure-detail benefit of #422.

## Summary When a test word calls `test.init` with a descriptive name that differs from the word name itself, the `at line N: expected X, got Y` detail produced by #422 is emitted by the runtime but dropped by the test runner — it never reaches the `TEST FAILURES:` section of the output. ## Reproducer ```seq # file: /tmp/test-name-mismatch.seq : test-foo ( -- ) "My Descriptive Test" test.init # init name ≠ word name 5 8 test.assert-eq test.finish ; ``` Run: ``` seqc test /tmp/test-name-mismatch.seq ``` ### Actual ``` Running tests in /tmp/test-name-mismatch.seq... test-foo ... FAILED ======================================== Results: 0 passed, 1 failed TEST FAILURES: /tmp/test-name-mismatch.seq::test-foo ``` Note the empty TEST FAILURES section — no `at line N: expected X, got Y` line. ### Expected ``` Running tests in /tmp/test-name-mismatch.seq... test-foo ... FAILED ======================================== Results: 0 passed, 1 failed TEST FAILURES: /tmp/test-name-mismatch.seq::test-foo test-foo ... FAILED at line 3: expected 5, got 8 ``` Confirmed by changing `"My Descriptive Test"` to `"test-foo"` (matching the word name) — the detail reappears. ## Root cause (likely) `collect_failure_block` in `crates/compiler/src/test_runner.rs` searches captured stdout for an exact header match constructed from the **discovered word name**: ```rust let header = format!("{} ... FAILED", test_name); ... if line == header { // collect following indented lines } ``` But the runtime (`patch_seq_test_finish` in `crates/runtime/src/test.rs`) prints the header using the **friendly name passed to `test.init`**: ```rust let test_name = ctx.current_test.as_deref().unwrap_or("unknown"); ... println!("{} ... FAILED", test_name); ``` When those differ, the exact-match lookup fails, `collect_failure_block` returns `None`, and the detail lines that follow on stdout are orphaned. ## Suggested fixes (pick one) 1. **Runtime side:** have `patch_seq_test_finish` print the word name (the stable identifier the runner discovered) rather than the friendly init name. The friendly name could still appear as a prefix in the TEST FAILURES section for human readability. 2. **Runner side:** relax `collect_failure_block` to match any header ending in `... FAILED`, or to scan by position rather than exact word name. (Slightly more fragile if multiple tests interleave output.) 3. **Both:** give the runner a way to know the init-given name for each test word (e.g. emit a machine-parseable marker like `TEST_BEGIN test-foo "My Descriptive Test"` on stdout), and match on either. Option 1 is the simplest and matches the rest of the pipeline, which keys on word name. ## Impact - **Seqlings** is unaffected (exercises don't use `test.init`), but any Seq test suite that uses descriptive init names will silently lose the new failure-detail benefit of #422.

navicore commented

2026-04-24 19:06:35 +00:00

(Migrated from github.com)

https://github.com/navicore/patch-seq/pull/425