Test discovery overloads the 'test-' prefix and conflicts with user word names #435

New issue

Closed

opened 2026-04-27 01:48:43 +00:00 by navicore · 1 comment

navicore commented

2026-04-27 01:48:43 +00:00

(Migrated from github.com)

Observation

The test runner appears to treat any word whose name starts with test- as a test entry point, attempting to invoke it with an empty stack. That collides with user code that wants to use the natural English prefix test- for non-test words (predicates like test-mode-active?, validators like test-input, helpers like test-config-loaded?).

Reproducer

A user defines a helper called test-flag with stack effect ( Int Int -- Bool ) in a non-test context, then runs the file through the test runner:

: test-flag ( Int Int -- Bool )
    band 0 i.<>
;

: my-actual-test ( -- )
    "My test" test.init
    5 4 test-flag test.assert
    test.finish
;

Result:

Compilation error: In 'main': test-flag: stack underflow - requires 2 value(s), only 0 provided

The runner is generating a synthetic main that calls test-flag as if it were a ( -- ) test entry point. Renaming test-flag to flag-set? makes the file compile.

Why this is more than a documentation issue

The test- prefix is one of the most natural English prefixes for predicates, validators, and harness helpers. A language imposing a convention that this prefix is "magic" forces users to invent unnatural vocabulary (flag-set? instead of test-flag, is-mode-active instead of test-mode-active?). It also breaks silently in a way that's hard to diagnose without prior knowledge — the error talks about main and stack underflow, not about test discovery.

The closer analogue in other ecosystems isn't test_* (Rust's #[test] is explicit; Python's pytest discovers by test_ but only inside files that match a pattern, not in arbitrary user code). The current Seq behavior is closer to "any function named test_* in any file is a test," which Rust deliberately moved away from.

Possible directions (non-prescriptive)

Explicit registration via an attribute or marker word at definition time, e.g. a @test annotation or a test-export form. Test discovery only finds words explicitly marked. Most invasive, most precise.
Discovery by signature, not name. A test entry point is any ( -- ) word in a file invoked with seqc test. No name-based magic; users with test- predicates just keep them. The test.init/test.finish framing already gives the runner enough handle to count results.
Restrict discovery to files matching a pattern (e.g., test-*.seq only). Inside a test file, test- words are tests. Outside, they're plain words. This is pytest's model and the convention seqlings already uses (it copies exercise files to test-<name>.seq before running).
Reserve a less-natural prefix for the magic, e.g. seqtest- or t/ namespace. Frees test- for user code.

Option 3 is probably least disruptive — most existing tests already live in test-*.seq files, so the discovery rule could be tightened to "only words named test-* in files named test-*.seq are entry points" without breaking anything.

Surfacing context

This came up while building the seqlings chapter 18 (bitwise ops). The natural name for a "is this bit set?" predicate was test-flag, since it parallels Unix test/stat etc. Had to rename it to flag-set? to dodge the test runner — a reasonable rename in this case, but the next user who hits this on a non-renamable word (or a public API) will be more annoyed.

What does NOT need to change

The test.init / test.finish / test.assert* words are fine — those are the framework, not the discovery mechanism.
Test files that genuinely live in test-*.seq and contain only test entry points keep working under any of the proposed designs.

## Observation The test runner appears to treat **any word whose name starts with `test-`** as a test entry point, attempting to invoke it with an empty stack. That collides with user code that wants to use the natural English prefix `test-` for non-test words (predicates like `test-mode-active?`, validators like `test-input`, helpers like `test-config-loaded?`). ## Reproducer A user defines a helper called `test-flag` with stack effect `( Int Int -- Bool )` in a non-test context, then runs the file through the test runner: ```seq : test-flag ( Int Int -- Bool ) band 0 i.<> ; : my-actual-test ( -- ) "My test" test.init 5 4 test-flag test.assert test.finish ; ``` Result: ``` Compilation error: In 'main': test-flag: stack underflow - requires 2 value(s), only 0 provided ``` The runner is generating a synthetic `main` that calls `test-flag` as if it were a `( -- )` test entry point. Renaming `test-flag` to `flag-set?` makes the file compile. ## Why this is more than a documentation issue The `test-` prefix is one of the most natural English prefixes for predicates, validators, and harness helpers. A language imposing a convention that this prefix is "magic" forces users to invent unnatural vocabulary (`flag-set?` instead of `test-flag`, `is-mode-active` instead of `test-mode-active?`). It also breaks silently in a way that's hard to diagnose without prior knowledge — the error talks about `main` and stack underflow, not about test discovery. The closer analogue in other ecosystems isn't `test_*` (Rust's `#[test]` is explicit; Python's pytest discovers by `test_` but only inside files that match a pattern, not in arbitrary user code). The current Seq behavior is closer to "any function named `test_*` in any file is a test," which Rust deliberately moved away from. ## Possible directions (non-prescriptive) 1. **Explicit registration** via an attribute or marker word at definition time, e.g. a `@test` annotation or a `test-export` form. Test discovery only finds words explicitly marked. Most invasive, most precise. 2. **Discovery by signature, not name.** A test entry point is any `( -- )` word in a file invoked with `seqc test`. No name-based magic; users with `test-` predicates just keep them. The `test.init`/`test.finish` framing already gives the runner enough handle to count results. 3. **Restrict discovery to files matching a pattern** (e.g., `test-*.seq` only). Inside a test file, `test-` words are tests. Outside, they're plain words. This is pytest's model and the convention `seqlings` already uses (it copies exercise files to `test-<name>.seq` before running). 4. **Reserve a less-natural prefix** for the magic, e.g. `seqtest-` or `t/` namespace. Frees `test-` for user code. Option 3 is probably least disruptive — most existing tests already live in `test-*.seq` files, so the discovery rule could be tightened to "only words named `test-*` in files named `test-*.seq` are entry points" without breaking anything. ## Surfacing context This came up while building the seqlings chapter 18 (bitwise ops). The natural name for a "is this bit set?" predicate was `test-flag`, since it parallels Unix `test`/`stat` etc. Had to rename it to `flag-set?` to dodge the test runner — a reasonable rename in this case, but the next user who hits this on a non-renamable word (or a public API) will be more annoyed. ## What does NOT need to change - The `test.init` / `test.finish` / `test.assert*` words are fine — those are the framework, not the discovery mechanism. - Test files that genuinely live in `test-*.seq` and contain only test entry points keep working under any of the proposed designs.