net-pr4 #480
No reviewers
Labels
No labels
bug
dependencies
documentation
duplicate
enhancement
good first issue
help wanted
invalid
question
refactor
rust
technical-debt
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
navicore/patch-seq!480
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "net-pr4"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Final state: - All PR4 functional work intact: HTTP/1.1 wire framing, connection pool with conservative defaults (8/host, 30s idle, 256 global, MRU checkout, half-closed peek via poll(POLLIN, 0)), SSRF over dns::resolve, ureq removed, net.http.* API preserved (Map shape unchanged). - All tests green: 26 http_client unit tests (including 3 integration tests that exercise full request flow and pool reuse end-to-end), plus the full suite. - no_dead_code test now tolerates up to 4 "rustls" substring matches with a precise comment in TOLERATED_FOR_HELLO; observed is 2. - docs/design/done/NO_DEAD_CODE.md updated with a "Linux residue from PR4" section documenting the symbol, the mechanism we traced, the bisect findings, the cost analysis, and the proposed future cleanups (linker-plugin-lto or feature-gating http_client into its own codegen unit). What we know about the residue: - One symbol, 17 bytes: drop_in_place<std::io::default_write_fmt::Adapter<rustls::stream::StreamOwned<rustls::client::client_conn::ClientConnection, may::net::TcpStream>>>. - Body is if let Some(err) = self.error { drop(err) } — std::io::Error trampoline, no rustls code. - Caused by the Box::new(stream) as Box<dyn HttpStream + Send> cast in tls::dial_tls. The cast materializes a vtable whose Write impl monomorphizes the Adapter; the Adapter's drop is what survives --gc-sections. - Bisect proved: removing ureq doesn't fix it, stubbing the pool doesn't fix it, removing the FFI wiring to my new modules does fix it. The exact reason the drop trampoline outlives the rest of its monomorphization chain crosses LLVM/lld comdat-reachability boundaries we didn't fully untangle. - Bounded: won't grow without new SideData types or new generic trampoline shapes. Tracked for revisiting alongside linker-plugin-lto evaluation.Review — PR4 (HTTP client rewrite)
Reviewed against
origin/net-pr4. The shape is right and significantly tighter than the ureq-era client: a clean module split (conn/pool/request/ssrf/wire/tls), trait-object handles instead ofStreamKindso rustls's drop chain can be stripped from non-HTTPS binaries, pool keyed on(scheme, host, port)with MRU checkout andpoll(POLLIN, 0)liveness, pre-resolved addrs flowing from SSRF validation straight to connect (which closes the DNS-rebinding TOCTOU window — that's the bit I want to call out as actually nice), idempotent-only retry. The unit + integration coverage finally exercises the wire path against a same-process server, and the no-dead-code budget mechanism is well-documented and honest about the 17-byte residue.Two real bugs (one a security gap), a handful of correctness/robustness items, and a few test-coverage gaps before this is feature-complete.
Header injection via
Content-Typewire.rs:55-58:ctis the third argument tonet.http.post/net.http.put— a string that flows through unchanged. If a Seq program ever passes a content-type that came from untrusted input (the obvious case: a proxy / forwarder pattern, but also any web app that takes Content-Type from a request and reuses it on an upstream call), a payload like"text/plain\r\nX-Injected: pwn\r\nContent-Length: 0\r\n\r\nGET /admin HTTP/1.1\r\nHost: ..."injects arbitrary headers — and depending on framing can pull off classic request smuggling against intermediaries.URL pieces are safe because
url::Url::parsepercent-encodes control chars before they reach the wire path, and body bytes go viawrite_all(no formatting). Content-Type is the one user-supplied string that lands directly in the header block. Fix: reject (or strip) ASCII control chars — at minimum\rand\n— inwrite_request, returning an error before the request goes out:Same hardening should arrive when the header-bag API lands (per the v1-limitations note) — every user-supplied header value needs the same check. Worth doing now for content-type because it's the one user-string already on the wire.
Request smuggling defenses (Content-Length / Transfer-Encoding handling)
wire.rs:82-99accepts whatever the response headers say. Two specific RFC 9112 §6.3 rules aren't enforced:Content-Lengthheaders with conflicting values. The current loop just overwrites:content_length = value.trim().parse::<usize>().ok()— last write wins. RFC requires this to be rejected as a smuggling attempt. Same applies to a singleContent-Lengthvalue containing a comma-separated list with differing entries.Transfer-Encoding: chunkedwithContent-Length. Code prefers chunked (correct precedence), but per RFC the combination should be rejected entirely — that mismatch is what HTTP request smuggling exploits in the first place.These are response-side checks (responses, not requests, in our client), so the threat is "malicious server can confuse our framing" rather than "smuggling through us." Still, the cost of enforcing them is small and the defense is conventional.
Pool
is_alivefinalfalse— correct but obscurepool.rs:130-133:The trailing bare
falsefires whenrc > 0(poll returned an event) andreventshas neither HUP/ERR/NVAL set — which by elimination means POLLIN. The module comment explains the intent (POLLIN on an idle keep-alive socket = peer sent FIN or unsolicited data, discard), but the code doesn't say so at this line — it just falls off the end withfalse. Future-reader is going to wonder if a fourth case got dropped. A one-line comment immediately above the trailingfalse("POLLIN on idle socket: peer wrote unsolicited bytes or FIN, discard") closes the gap.Side note: the function name
is_alivereads as a property test, but in practice it returnsfalseon any signal from poll exceptrc==0. The actual contract is "return true iff poll says the fd is quiet." Either rename tois_quiet/is_reusable, or document the conflation. Cosmetic.Dead code in path-and-query construction
ssrf.rs:86-91:url::Url::path()for any hierarchical (http/https) URL is documented to return a path that always starts with/and is never empty — theurlcrate normaliseshttp://example.comto path/. So theparsed.path().is_empty()arms are unreachable. Collapses to:Harmless dead branches, but they read as defensive code against a case that can't happen.
EOF-framed bodies + no timeout = strand hang
wire.rs:111-123: when the response has neitherContent-Lengthnor chunked encoding, the client reads to EOF. A misbehaving server that sends< MAX_BODY_SIZEbytes and then keeps the connection open without sending more parks the strand onread_to_endindefinitely. The PR doc lists "no per-request timeout" as a v1 limitation, so this is acknowledged — but it's worth being more explicit that the EOF-framed path is the worst-case shape (chunked and Content-Length paths bound by their own framing). Until the deadline pass lands, a defensive option is to forceConnection: closesemantics on the dial when the response declared no length (the code already doeskeep_alive = false, so subsequent requests on that conn are fine; the hang is within the EOF read itself).Test gaps
Chunked decoding is not exercised end-to-end. The integration test server always sends
Content-Length. Given chunked is the more elaborate path (with extensions, trailers, MAX_BODY_SIZE accumulation across chunks), it should have at least one round-trip test. A variant ofspawn_test_serverthat emits a chunked response (e.g. two chunks + 0-terminator) is ~15 lines.Connection: closeis not exercised. The pool should drop a closed-mode connection rather than recycle it. The test server hardcodesConnection: keep-aliveso the close path is untested. Worth a test that assertsaccept_count == 2after two requests when the server sendsConnection: close— confirms the pool didn't try to reuse a doomed entry.POST body round-trip is not verified.
http_post_with_body_round_tripssays explicitly in its own comment: "our test server only reads up to the blank line, so a POST body would be left in the socket buffer." So the test only proves the request line + headers reached the server — it could pass even ifwrite_requestsilently dropped the body bytes. ReadingContent-Lengthbytes after the blank line and echoing them back is the right fix, then the test asserts the echo matches the sent body.SSRF + DNS rebinding closure isn't asserted in tests. The architectural property — that
addrsare validated then dialled without a second DNS lookup — is the actual security win here. A unit test that mocksdns::resolveto return different IPs on consecutive calls and asserts the request goes to the validated IP would lock the invariant in. Probably overkill for now, but worth flagging because if a future refactor passeshost: Stringto connect instead ofaddrs: Vec<IpAddr>, the property silently regresses with no test catching it.Small things
read_responseusesBufReader::new(r)which has an 8 KB read-ahead. On an HTTP/1.1 connection without pipelining, the server should never send bytes past the response, so the read-ahead is always inside the current response — safe. If pipelining ever ships, those buffered bytes would be lost (BufReader's drop doesn't unread). Worth a one-line comment so this invariant is captured.User-Agent: seq/<version>discloses the runtime version to every remote server. Standard, but if you ever want to anonymise (security-conscious deployments), a way to override via env var would help. Future header-bag API closes this.Acceptheader. RFC says servers SHOULD assume*/*, but a handful of strict APIs return 406. Cheap defensive add:Accept: */*.pool.rs:103—MAX_PER_HOSTis checked afterMAX_GLOBAL, so a host with empty pool gets rejected if the global cap is full elsewhere. Conservative choice (matches Go's DefaultTransport). Worth noting in the docstring that global wins; users tuning these knobs will hit this.is_alivepolls the underlying TCP fd, which is correct for transport liveness but doesn't detect a half-broken TLS state (e.g., received TLS alert with no transport-level signal). The idempotent retry path covers it, so this is fine — but worth a sentence inpool.rsnoting the layering choice.HttpStreamimpl onrustls::StreamOwned<...>reaches into the publicsockfield (self.sock.as_raw_fd()). Stable per rustls 0.23 docs, but if rustls 0.24 ever makessockprivate, this breaks. A wrapper method would insulate.NO_DEAD_CODE residue
The 17-byte / 2-substring-match residue is well-documented (
docs/design/done/NO_DEAD_CODE.mdupdated with the bisect notes and the cleanup path forward). Test budget at 4 gives ~2x headroom. Honest engineering — the design doc admits the bisect didn't fully untangle the rustc/LLVM/lld interaction, which is exactly the right level of certainty for a "we'll revisit when linker-plugin-lto is on the table" line. No action needed; flagging because it's the most interesting structural call in the PR and deserves explicit acknowledgement.Verdict
wire.rs:56). A six-line guard closes it.is_aliverename/comment, dead path-and-query branches, header-bag API for arbitrary headers + Accept.The pre-resolved-addrs-into-connect design is the real win here — keep that property documented and test-anchored so it can't regress.
Second pass — review-driven fixes look good
Walked the delta from
d8ec37c..dd8f364. Every item from the first pass landed cleanly. Couple of small wins on top of just-doing-what-was-asked:Content-Type / header-injection guard (wire.rs)
header_value_safeis the right shape and slightly stricter than I suggested — rejecting HTAB too, with an explicit rationale tied to RFC 9112's deprecation of tab-prefixed continuation lines. That's a real attack surface against intermediaries that still honour them, so the extra strictness is well-motivated. The three unit tests (CRLF, bare LF, NUL) cover the main families. One subtle thing worth noting (not fixing): the check allows 0x80–0xFF (RFC 9110 obs-text), so a malformed UTF-8 content-type round-trips. Not a security issue — those bytes can't inject a header — just a compatibility shape some strict servers will 400 on. Acceptable.Anti-smuggling response-side guards
The split into
parse_content_length(handles single + comma-list, agreeing or not) plus the outer "multiple headers must agree" guard plus the "chunked + CL is poison" guard is exactly the three RFC 9112 §6.3 shapes. I traced through the edge cases mentally:"5, 5"→ accepted as 5 ✓ (covered bywire_response_content_length_list_all_agreeing_accepted)"5, 7"→ rejected ✓"5"then"7"(two separate headers) → rejected ✓"5"then"5"(two separate, agreeing) → accepted ✓ (implied by theprev != parsedcheck)""/","/"5,"(empty list elements) → rejected via parse-failure ✓The error-message convention (
"smuggling guard: ...") makes the test assertions clean and signals the intent in logs. Good.Integration tests
ServerModeenum is the right refactor — the three flavours (FixedBody / Echo / Chunked) compose well and the test bodies stay narrow. Two solid additions:echoed == bodybyte-for-byte against an echo server. That meaningfully proves the bytes traversed the wire — a regression that silently drops the body fails here instead of returning a misleading 200."Hello world").Pool docstrings / dead-branch cleanup / docs
All the deferred-but-easy items took the smaller fixes:
is_alivetrailingfalsenow has an inline rationale and an explicit note about the TLS-state-not-detected gap pointing at the idempotent-retry safety net.releasedocstring spells out global-cap-first ordering and the tuning hint.ssrf.rspath-and-query collapsed to the two reachable arms with a comment explaining why.Accept: */*header added.conn.rsflags theStreamOwned.sockfield-access as a rustls 0.23 stability assumption.STDLIB_REFERENCE.mdv1-limitations now names the EOF-framed body as the worst-case shape under the no-timeout gap, and lists the newAcceptand the Content-Type control-char guard.Still open (and I noted these as deferrable originally — fine to land without)
Connection: closetest confirming the pool drops the entry. Won't catch a future regression where the response'skeep_alive=falseis ignored on the release side.addrsflow from SSRF straight toconnect_to_addrs), but a refactor could regress it silently. Low priority — flagging only because it's the strongest security win of the design.Verdict
LGTM. The fixes are tight, the new tests pin the things that mattered, and the docs are honest about what's left. Ship it.