Consider input/output holistically (beyond the JSON-only presumption)

navicore commented

2026-06-30 03:52:26 +00:00

Owner

Opening this to start a discussion, not to land a specific design.

Framing

The compiled-binary contract today ties a program's output fairly tightly to a single wire format: the engine wrapper serialises query bindings as JSON, and program-authored write/1 bytes share that same stdout. That coupling has held well for the "program is a pure relation; caller extracts via the query" model that linting.pl and the worker reactor demonstrate.

But "the answer is always JSON" is a presumption I'd like to question before it ossifies. Real production binaries will want their inputs and outputs in other shapes — YAML, TOML, CSV, line-delimited records, plain human prose, things not invented yet — and the caller will often be another program (or a human in a shell) that has its own expectations about the stream it's consuming.

The constraint I think is worth holding

I don't think it's a problem for a compiled program to be limited to stdin and stdout as its only I/O surface. That's a clean, portable, Unix-shaped contract and it fits the immutable-binary thesis. The interesting question is not how many channels a program has, but how the format of those two channels is conceived.

What I'd like us to consider wholemeal

Rather than add formats one at a time as ad-hoc flags, can we imagine the input/output story as an architecture in its own right — one where:

many input and output formats are possible on the same two streams, not just the one the engine wrapper happens to ship with;
the engine's own contract (the fields it emits or consumes — solutions, exhausted, exit codes, the --query itself) stays as inflexible as it needs to for correctness, while the serialisation of those fields is the part that's open to many formats; and
program-authored output and engine-authored output can coexist on stdout without one corrupting the other, whatever format either is in.

The distinction I want to draw is: the shape of the data is fixed; the encoding of the data is plural. Today we've collapsed those two into "JSON, everywhere," and I'd like us to separate them before we build further on top.

What I'm not assuming

Not proposing a specific format list, a specific format/2, a specific flag, or a specific streaming model. Those are all downstream of the architectural decision.
Not asking the program to gain channels beyond stdin/stdout.
Not asking the engine wrapper to go away — it's load-bearing for exit codes and the caller contract.

I'd like this thread to be where we work out what the right framing is before we scope any concrete change. Thoughts welcome — especially on whether "fixed fields, plural encodings" is the right axis to split on, or whether there's a better way to hold the design space open.

Opening this to start a discussion, not to land a specific design. ## Framing The compiled-binary contract today ties a program's output fairly tightly to a single wire format: the engine wrapper serialises query bindings as JSON, and program-authored `write/1` bytes share that same stdout. That coupling has held well for the "program is a pure relation; caller extracts via the query" model that `linting.pl` and the worker reactor demonstrate. But "the answer is always JSON" is a presumption I'd like to question before it ossifies. Real production binaries will want their inputs and outputs in other shapes — YAML, TOML, CSV, line-delimited records, plain human prose, things not invented yet — and the caller will often be another program (or a human in a shell) that has its own expectations about the stream it's consuming. ## The constraint I think is worth holding I don't think it's a problem for a compiled program to be limited to **stdin and stdout** as its only I/O surface. That's a clean, portable, Unix-shaped contract and it fits the immutable-binary thesis. The interesting question is not *how many channels* a program has, but **how the format of those two channels is conceived**. ## What I'd like us to consider wholemeal Rather than add formats one at a time as ad-hoc flags, can we imagine the input/output story as an architecture in its own right — one where: - many input and output formats are *possible* on the same two streams, not just the one the engine wrapper happens to ship with; - the engine's own contract (the fields it emits or consumes — solutions, exhausted, exit codes, the `--query` itself) stays as **inflexible** as it needs to for correctness, while the **serialisation of those fields** is the part that's open to many formats; and - program-authored output and engine-authored output can coexist on stdout without one corrupting the other, whatever format either is in. The distinction I want to draw is: **the shape of the data is fixed; the encoding of the data is plural.** Today we've collapsed those two into "JSON, everywhere," and I'd like us to separate them before we build further on top. ## What I'm not assuming - Not proposing a specific format list, a specific `format/2`, a specific flag, or a specific streaming model. Those are all downstream of the architectural decision. - Not asking the program to gain channels beyond stdin/stdout. - Not asking the engine wrapper to go away — it's load-bearing for exit codes and the caller contract. I'd like this thread to be where we work out what the right framing is before we scope any concrete change. Thoughts welcome — especially on whether "fixed fields, plural encodings" is the right axis to split on, or whether there's a better way to hold the design space open.

navicore self-assigned this

2026-06-30 03:52:59 +00:00

navicore commented

2026-06-30 22:24:05 +00:00

Author

Owner

Read the now-merged docs/design/IO.md. The scope cut from "pluggable" to a declared text | bson set via :- io_format([...]) is the right call — the envelope-as-type / encoder-as-trait split makes the "fixed shape, plural encoding" separation real in the code, and codegen-baked .rodata + dead-stripping means a [text]-only binary stays byte-identical to today and pays zero bson cost.

Two things I'd flag before implementation lands, both fine to defer or dismiss:

1. The two encodings are not information-equivalent on cyclic terms. render.rs's term_to_json_v cycle-cuts X = f(X) to "_N" (lossy, preserved for v1 byte-compat); copyterm::TermBuf represents cycles structurally. So a bson-encoded cyclic term round-trips faithfully while the text-encoded one doesn't. That's a point in bson's favour, but it means the "losslessness is non-negotiable" constraint holds for bson and not for text — worth stating in the doc rather than discovering at checkpoint time, since the cyclic-term round-trip test will pass for bson and silently can't for text without breaking v1 byte-compat.

2. The input-format matrix is thinner than the output story. The doc is precise about output encoding but sketches input as "for bson input, a one-field request document on stdin." The four-cell matrix — {text, bson}-in × {text, bson}-out — isn't spelled out. Concretely: does --format bson output still accept an argv --query text input (text-in / bson-out)? Are input and output encoding fully orthogonal, or does a bson stdin request imply bson output? My read is orthogonal and argv-text-in + bson-out is allowed, but it determines whether entry.rs reads stdin unconditionally or only when bson input is requested — and that's a flag that doesn't currently exist separately from output --format. Worth nailing down before the input-framing code lands.

Otherwise it reads as solid and tight.

Read the now-merged `docs/design/IO.md`. The scope cut from "pluggable" to a declared `text | bson` set via `:- io_format([...])` is the right call — the envelope-as-type / encoder-as-trait split makes the "fixed shape, plural encoding" separation real in the code, and codegen-baked `.rodata` + dead-stripping means a `[text]`-only binary stays byte-identical to today and pays zero bson cost. Two things I'd flag before implementation lands, both fine to defer or dismiss: **1. The two encodings are not information-equivalent on cyclic terms.** `render.rs`'s `term_to_json_v` cycle-cuts `X = f(X)` to `"_N"` (lossy, preserved for v1 byte-compat); `copyterm::TermBuf` represents cycles structurally. So a bson-encoded cyclic term round-trips faithfully while the text-encoded one doesn't. That's a point in bson's favour, but it means the "losslessness is non-negotiable" constraint holds for bson and *not* for text — worth stating in the doc rather than discovering at checkpoint time, since the cyclic-term round-trip test will pass for bson and silently can't for text without breaking v1 byte-compat. **2. The input-format matrix is thinner than the output story.** The doc is precise about output encoding but sketches input as "for bson input, a one-field request document on stdin." The four-cell matrix — {text, bson}-in × {text, bson}-out — isn't spelled out. Concretely: does `--format bson` *output* still accept an argv `--query` text input (text-in / bson-out)? Are input and output encoding fully orthogonal, or does a bson stdin request imply bson output? My read is orthogonal and argv-text-in + bson-out is allowed, but it determines whether `entry.rs` reads stdin unconditionally or only when bson *input* is requested — and that's a flag that doesn't currently exist separately from output `--format`. Worth nailing down before the input-framing code lands. Otherwise it reads as solid and tight.

navicore referenced this issue

2026-07-03 15:13:14 +00:00

Full CI green — all checks pass, including the 30 integration tests (which exercise the real --query/--format #39

navicore referenced this issue

2026-07-03 15:24:14 +00:00

Full CI green — all checks pass, including the 30 integration tests (which exercise the real --query/--format #39

navicore commented

2026-07-03 15:40:51 +00:00

Author

Owner

#39

https://git.navicore.tech/navicore/patch-prolog/pulls/39