Consider character-level reader with state type #62

New issue

Open

opened 2026-02-07 17:12:20 +00:00 by navicore · 0 comments

navicore commented

2026-02-07 17:12:20 +00:00

(Migrated from github.com)

Summary

Evaluate whether adopting a character-level reader architecture with an explicit state type would improve maintainability over the current tokenizer approach.

Motivation

seq-lisp-2's reader uses a Reader { input, pos } state type with:

Explicit position tracking at all times
Character classification functions (is-whitespace, is-delimiter, etc.)
Proper escape handling in strings
Clean state-machine architecture

The current seq-lisp reader works well but uses a separate tokenizer → parser pipeline. The seq-lisp-2 approach is arguably more maintainable and makes position tracking (needed for LSP features) more natural.

Scope

Evaluate whether the current reader's architecture is limiting LSP feature development
If so, consider refactoring to a character-level reader with position state
Maintain all current reader features (strings, numbers, floats, comments, quote syntax)

Notes

Lower priority — the current reader works fine and this would be a significant rewrite for primarily architectural benefit. Most valuable if LSP features increasingly need precise source positions.

Inspired by seq-lisp-2/src/reader.seq.

## Summary Evaluate whether adopting a character-level reader architecture with an explicit state type would improve maintainability over the current tokenizer approach. ## Motivation seq-lisp-2's reader uses a `Reader { input, pos }` state type with: - Explicit position tracking at all times - Character classification functions (`is-whitespace`, `is-delimiter`, etc.) - Proper escape handling in strings - Clean state-machine architecture The current seq-lisp reader works well but uses a separate tokenizer → parser pipeline. The seq-lisp-2 approach is arguably more maintainable and makes position tracking (needed for LSP features) more natural. ## Scope - Evaluate whether the current reader's architecture is limiting LSP feature development - If so, consider refactoring to a character-level reader with position state - Maintain all current reader features (strings, numbers, floats, comments, quote syntax) ## Notes Lower priority — the current reader works fine and this would be a significant rewrite for primarily architectural benefit. Most valuable if LSP features increasingly need precise source positions. Inspired by `seq-lisp-2/src/reader.seq`.