feat(runner): probe a target through the ip of a JSON input line

[mazzz3r]

Summary

Probing a vhost on a given server, or an origin behind a CDN, needs the hostname in the Host header and SNI while the connection goes to another ip, and the same host may need probing on several ips. httpx had no input for that: the vhost,ip form builds the URL from the ip, so it sends no SNI and reports the ip as url and host, and -pa, which probes the ips DNS returns, didn't reach them correctly.

An input line can now be a JSON object with the target and the ip, like curl --resolve. The connection goes to the ip, the hostname stays in the Host header and SNI, and host_ip reports the ip. -pa now pins its ips the same way. Run in a container without network, with app.test resolving to 127.0.0.1 and 127.0.0.2, a server on each whose title is its address, and redir.test redirecting to other.test (127.0.0.3), with -title -ip:

Run	dev	this branch
-l with two lines pinning http://app.test:18080 to .1 and .2	no output	[srv=127.0.0.1 …] [127.0.0.1], [srv=127.0.0.2 …] [127.0.0.2]
-pa on http://app.test:18080	[srv=127.0.0.2 …] [127.0.0.1], [srv=127.0.0.2 …] [127.0.0.2]	[srv=127.0.0.1 …] [127.0.0.1], [srv=127.0.0.2 …] [127.0.0.2]
-pa -fr on http://redir.test:18201	[302,200] [srv=127.0.0.1 host=other.test:18202]	[302,200] [srv=127.0.0.3 host=other.test:18202]
-stream with a malformed line	no output for that line	[ERR] Skipping input: invalid target line …, the other lines probed

Depends on projectdiscovery/fastdialer#580: WithResolveOverride, and the dial cache fix without which the second ip of a host is served from the first one's cached dialer. go.mod points fastdialer at the branch of that PR until a fastdialer release includes it; that replace line is the only change to go.mod.

Changes

• -pa passed each ip in the fastdialer.IP context value, which pins every dial made for the request, so a redirect to another host followed with -fr went to the same ip. The ip now pins only the probed host, with fastdialer.WithResolveOverride. IPv6 ips were passed in brackets, which fastdialer doesn't accept as an ip, so those probes failed; they are passed as is now. Favicon and ztls requests dropped the ip and reached the host through DNS; they keep it now. -maxhr counts errors per ip, and the ip is checked against the network policy as well as the host.
• runner/targetline.go parses input lines that start with {. Only target and ip are accepted, and data after the object is rejected. The ip is validated and canonicalized (::ffff:192.0.2.10 becomes 192.0.2.10; zones and unspecified addresses are rejected), and the target must have a hostname: IP, CIDR, ASN, wildcard and vhost,ip targets are rejected. Lines are read from -l and stdin; -u splits its value on commas, so it cannot carry them. Equivalent lines are stored once, a line without an ip is the plain target, and -pa doesn't resolve a pinned target.
• An invalid line stops the run before probing starts. In stream mode the scan has already started, so the line is skipped with an error message; the three input loops of streamInput share one sendLine helper.
• -proxy, -unsafe, -pipeline, -http2, -jarm, -screenshot and -vhost-input don't connect through the ip, so they are refused with pinned targets. The README note documents the format and says that a proxy set with HTTP_PROXY/HTTPS_PROXY resolves the host itself as well.

Testing

• TestParseTargetLine covers valid lines and each rejected form. TestRunner_normalizeInput covers deduplication of equivalent lines and the flag conflicts. TestRunner_targetLine_targets checks that -pa isn't applied to a pinned target. TestRunner_streamInput_skipsInvalidLines checks that stream mode skips a malformed line and keeps the others.
• TestTargetLinePinsTheConnection: two local servers on 127.0.0.1 and 127.0.0.2 with the same port, each reached for its own line with Host: pinned.test:port. TestTargetLineKeepsSNI: a TLS server sees pinned.test as SNI. Both skip when a proxy from the environment applies to the target.
• go test -race ./runner/ passes except TestRunner_probeall_targets, which resolves one.one.one.one and fails the same way on dev here; the target line tests pass with and without HTTPS_PROXY set, and with -race in a golang:1.27.1 container without network. go test ./common/httpx/ passes. golangci-lint run ./runner/... ./common/...: 0 issues.
• A pinned line with a conflicting flag:

$ httpx -l targets.jsonl -jarm
[FTL] Could not read input file 'targets.jsonl': target "http://app.test:18080": a target with an ip cannot be probed with -jarm

IPv6 could not be exercised here: the kernel of this environment has no IPv6.

Closes #2667

Checklist

• Pull request is created against the dev branch
• All checks passed (lint, unit/integration/regression tests etc.) with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}