Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions TESTING_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -877,6 +877,7 @@ Candidate-ordered parallel-index recovery tests must prove that the fatal result
Issue #5259 pairs `BatchChildErrorParserTests` with `QueryCommandRunnerBatchIssue5259Tests`: preserve direct/child E028 classification and every measured budget/retry field for status explanations and search arrays in serial/parallel batches, alongside mixed success/text failures, explicit raw-stream compatibility, parent-budget accounting, and the existing cancellation/timeout fixtures. Parser coverage must retain malformed/duplicate/type/identity rejection, unpaired Unicode surrogate escapes in values and property names, UTF-8 byte and depth boundaries, unknown-minimum/reduce-size retry variants, nested envelopes, text bounds, and secret/path/control sanitization without unknown-field reflection.
Issue #5344 pairs `QueryCommandRunnerBatchIssue5344Tests` with `BatchChildPartialResultParserTests`. Share a small capped-find fixture for direct/batch row, cursor-resume, count, envelope and zero-row parity; preserve exit 11, incomplete flags, ordered mixed-batch failure accounting, optional raw streams and parent-budget fallback. Share definition fixtures across zero/one/multiple matches, body and supported output variants, retaining unsupported-option errors. Reject malformed, duplicate, invalid-Unicode, over-depth, oversized, mixed and unrelated partial captures. Run on net8/net9 with existing batch cancellation/timeout, JSON-envelope, find and #5259 E028 regressions. Nested batch integration remains console-sensitive; pure parser tests do not mutate shared state.
The #5344 partial fixture also covers scoped regex origin filters in row/count/envelope modes and serial/parallel `--verbose` / `--profile` diagnostics with zero or nonzero results. Parser controls must retain diagnostic objects without counting them as rows, reject missing rows and mismatched terminal/envelope counts, and accept zero-row controls and count records; explicit bounded-envelope truncation keeps its inner scan metadata.
Issue #5423 pairs `BatchChildErrorParserTests` with the console-sensitive `QueryCommandRunnerBatchHintTests`. Preserve slash-separated public option names, short aliases and surrounding punctuation alongside POSIX/Windows path and secret-value controls. Reject path-prefixed/suffixed, assigned and unknown option-like forms, including private filename text after closing punctuation. Compare malformed-find-cursor hints with standalone output in serial and parallel batches on net8/net9; retain parser bounds and the existing E028/cancellation/timeout coverage.
Argument-validation variants that only differ by invalid scalar input share one database fixture and iterate within a fact when no per-case state or discovery identity is required.
Positional `files` glob coverage shares one indexed-file fixture and iterates `*`, `?`, and recursive `**` patterns in a fact, matching the exact tokens that a quoted shell argument passes to the CLI.
Excerpt focus coverage reuses one indexed fixture for line-only leading-window behavior, the focus-length dependency, and focus-column range validation; zero and non-numeric focus-column values share one indexed Markdown fixture.
Expand Down Expand Up @@ -2430,6 +2431,7 @@ results-only・array・envelope・compact の形状、終端と改行の厳密
Issue #5259 は `BatchChildErrorParserTests` と `QueryCommandRunnerBatchIssue5259Tests` を対にし、status 説明と search 配列の直接実行/子実行における E028 分類および全サイズ・再試行フィールドを逐次/並列 batch で維持します。成功/text 失敗の混在、生 stream の明示指定互換性、親上限の計数、既存の取消/timeout fixture も検証してください。parser は不正 JSON/重複/型/identity の拒否、値と property 名の不対 Unicode surrogate escape、UTF-8 byte/深さの境界、最小値不明/サイズ縮小の再試行、nested envelope、文字数上限、および未知フィールドを反映しない機密情報/path/制御文字の除去を検証します。
Issue #5344 は `QueryCommandRunnerBatchIssue5344Tests` と `BatchChildPartialResultParserTests` を組み合わせます。小さな走査上限付きfind fixtureで、直接実行とbatchの結果行・cursor再開・count・envelope・0件結果を比較し、終了コード11、不完全性フラグ、混在batchの入力順と失敗件数、生ストリームの任意指定、親上限によるフォールバックを維持します。definitionは同じfixtureで0件・1件・複数件、本文と対応出力形式、未対応オプションのエラーを検証します。不正JSON、重複、不正Unicode、深さ・サイズ上限超過、混在出力、無関係な部分出力を拒否してください。既存のbatch取消・timeout、JSON envelope、find、#5259のE028回帰とともにnet8/net9で実行します。入れ子のbatch統合テストはconsole-sensitiveとし、純粋なparserテストは共有状態を変更しません。
#5344の部分結果fixtureは、スコープ付き正規表現のoriginフィルターを結果行・count・envelopeで検証し、逐次/並列の `--verbose` / `--profile` 診断も結果が0件・1件以上の両方で確認します。parserは診断オブジェクトを結果行の件数に数えず保持し、行の欠落、終端・envelope件数の不一致を拒否し、0件の制御レコードとcountレコードを受け付け、明示的な上限付きenvelopeの切り詰めでは内側の走査情報を保持してください。
Issue #5423 は `BatchChildErrorParserTests` とconsole-sensitiveな `QueryCommandRunnerBatchHintTests` を対にします。公開オプション名・短縮名のスラッシュ連結表記と周囲の句読点を保持し、POSIX/Windowsパスと秘密値の除去も検証します。閉じ括弧などの後ろに続く私的なファイル名を含め、パスの接頭辞・接尾辞、代入、未知のオプション風表記は例外対象から除外してください。net8/net9の逐次・並列batchで不正なfind cursorのヒントを単独実行と比較し、parserの上限と既存のE028・取消・timeoutの検証を維持します。
invalid scalar input だけが異なる argument-validation variant は、case ごとの state や discovery identity が不要なら1つの database fixture を共有し、fact 内で反復してください。
`files` の positional glob coverage は1つの indexed-file fixture を共有し、quote された shell 引数が CLI に渡す token と同じ `*`、`?`、recursive `**` pattern を fact 内で反復してください。
excerpt の focus coverage は、line-only 時の先頭側 window、focus-length の依存関係、focus-column の範囲検証を1つの indexed fixture で共有してください。focus-column の zero / non-numeric value も1つの indexed Markdown fixture を再利用してください。
Expand Down
9 changes: 8 additions & 1 deletion USER_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2232,7 +2232,11 @@ fields by default, including `E028_RESPONSE_BUDGET_TOO_SMALL`, `requested_bytes`
`effective_bytes`, `minimum_required_bytes`, its known/uncertain flags and reasons,
and `retry` (`action`, `option`, `recommended_bytes`, `maximum_effective_bytes`,
`command`). Parsing is limited to 64 KiB of UTF-8 and depth 16; strings are sanitized
and limited to 1,024 characters. Unknown fields are omitted. Text, malformed,
and limited to 1,024 characters. Standalone slash-separated public CLI option
names such as `--limit/--max-json-bytes` retain their spelling and punctuation in
recovery hints. Actual filesystem paths and secret values remain redacted; unknown
option-like names and lists with path prefixes or suffixes retain normal sanitization.
Unknown fields are omitted. Text, malformed,
over-limit, or invalid error objects retain the safe exit-code-based fallback.
Failed stdout/stderr are available only with `--include-raw-streams`, under the
bounded `raw_streams` object; these diagnostic streams retain their original content.
Expand Down Expand Up @@ -6403,6 +6407,9 @@ countなら `result`)の走査状態と確定性を確認し、同じクエリ
`requested_bytes`、`effective_bytes`、`minimum_required_bytes` と既知・不確実性のフラグ/理由、
`retry`(`action`、`option`、`recommended_bytes`、`maximum_effective_bytes`、`command`)です。
解析は UTF-8 で 64 KiB、深さ 16 までとし、文字列は機密情報を除去して 1,024 文字以内に制限します。
復旧ヒントでは、`--limit/--max-json-bytes` のように独立して記載された公開CLIオプション名の
スラッシュ連結表記と周囲の句読点を保持します。実際のファイルシステムパスや秘密値は引き続き
伏せられ、未知のオプション風の名前やパスの接頭辞・接尾辞が付く一覧には通常の秘匿処理を適用します。
未知のフィールドは省略し、text、不正 JSON、上限超過、不正なエラー object は終了コード由来の安全な
汎用エラーを維持します。失敗時の stdout / stderr は `--include-raw-streams` を指定した場合だけ、
上限付きの `raw_streams` object に元の診断内容のまま追加されます。
Expand Down
16 changes: 16 additions & 0 deletions changelog.d/unreleased/5423.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
---
category: fixed
issues:
- 5423
affected:
- src/CodeIndex/Cli/BatchChildErrorParser.cs
- src/CodeIndex/Diagnostics/DiagnosticSanitizer.cs
---

## English

- **Batch recovery hints preserve slash-separated CLI flags (#5423)** — Public option lists such as `--limit/--max-json-bytes` retain their spelling and punctuation while actual filesystem paths and secret values remain redacted in structured child errors.

## 日本語

- **batchの復旧ヒントでスラッシュ連結されたCLIフラグを保持 (#5423)** — `--limit/--max-json-bytes` のような公開オプション一覧の綴りと句読点を保持し、構造化された子エラー内の実パスや秘密値は引き続き伏せます。
8 changes: 7 additions & 1 deletion src/CodeIndex/Cli/BatchChildErrorParser.cs
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,11 @@ internal static class BatchChildErrorParser
internal const int MaxUtf8Bytes = 64 * 1024;
internal const int MaxDepth = 16;
internal const int MaxTextChars = 1024;
private static readonly IReadOnlySet<string> PublicOptionNames = CliFlagSchema.All
.SelectMany(flag => flag.ShortName is { } shortName ? new[] { flag.Name, shortName } : new[] { flag.Name })
// Help is handled by ArgHelper before the per-command flag schema.
.Concat(["--help", "-h"])
.ToHashSet(StringComparer.Ordinal);

internal static JsonObject? Parse(string stdout, string command, int exitCode)
{
Expand Down Expand Up @@ -171,5 +176,6 @@ private static bool CopyBytes(JsonProperty property, JsonObject target)
private static string Sanitize(string value)
=> DiagnosticSanitizer.ForMessage(
new string(value.Select(ch => char.IsControl(ch) ? ' ' : ch).ToArray()),
MaxTextChars - 3);
MaxTextChars - 3,
PublicOptionNames);
}
47 changes: 44 additions & 3 deletions src/CodeIndex/Diagnostics/DiagnosticSanitizer.cs
Original file line number Diff line number Diff line change
Expand Up @@ -65,8 +65,8 @@ private static string ForPathValue(string? value, bool redactPaths)
public static string ForMessage(string? message)
=> ForMessage(message, MaxDiagnosticFieldLength);

public static string ForMessage(string? message, int maxLength)
=> ForMessage(message, RedactAbsolutePaths, maxLength);
public static string ForMessage(string? message, int maxLength, IReadOnlySet<string>? publicOptionNames = null)
=> ForMessage(message, value => RedactAbsolutePaths(value, publicOptionNames), maxLength);

internal static string ForMessage(string? message, Func<string, string> redactPaths)
=> ForMessage(message, redactPaths, MaxDiagnosticFieldLength);
Expand Down Expand Up @@ -198,11 +198,19 @@ private static string CollapseWhitespace(string value)
return collapsed.ToString();
}

private static string RedactAbsolutePaths(string value)
private static string RedactAbsolutePaths(string value, IReadOnlySet<string>? publicOptionNames)
{
var redacted = new System.Text.StringBuilder(value.Length);
for (int index = 0; index < value.Length;)
{
if (publicOptionNames is not null
&& TryGetPublicOptionListEnd(value, index, publicOptionNames, out var optionEnd))
{
redacted.Append(value, index, optionEnd - index);
index = optionEnd;
continue;
}

if (!TryGetAbsolutePathEnd(value, index, out var end))
{
redacted.Append(value[index]);
Expand All @@ -217,6 +225,39 @@ private static string RedactAbsolutePaths(string value)
return redacted.ToString();
}

private static bool TryGetPublicOptionListEnd(string value, int start, IReadOnlySet<string> optionNames, out int end)
{
end = start;
if (value[start] != '-'
|| (start > 0 && !char.IsWhiteSpace(value[start - 1]) && value[start - 1] is not ('\'' or '"' or '`' or '(' or '[' or '{')))
return false;

// Only complete, catalogued option lists qualify. Assignments, path prefixes,
// unknown names and path/filename suffixes must keep the normal redaction.
var count = 0;
while (end < value.Length)
{
var optionStart = end;
while (end < value.Length && (char.IsAsciiLetterOrDigit(value[end]) || value[end] == '-'))
end++;
if (!optionNames.Contains(value[optionStart..end]))
return false;
count++;
if (end >= value.Length || value[end] != '/')
break;
end++;
if (end >= value.Length)
return false;
}

// Closing punctuation must terminate the token, not introduce a private
// filename suffix such as --limit/--max-json-bytes]customer-project.cs.
var boundary = end;
while (boundary < value.Length && value[boundary] is '\'' or '"' or '`' or ')' or ']' or '}' or ',' or ';' or ':' or '.' or '!' or '?')
boundary++;
return count >= 2 && (boundary == value.Length || char.IsWhiteSpace(value[boundary]));
}

private static bool TryGetAbsolutePathEnd(string value, int start, out int end)
{
end = start;
Expand Down
40 changes: 40 additions & 0 deletions tests/CodeIndex.Tests/BatchChildErrorParserTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,46 @@ public void Parse_PreservesAllowlistedFieldsAndSanitizesUntrustedText()
Assert.Equal(4096, parsed["retry"]!["maximum_effective_bytes"]!.GetValue<long>());
}

[Fact]
public void Parse_PreservesPublicOptionListsWithoutExemptingPaths_Issue5423()
{
var source = JsonNode.Parse(ErrorJson)!.AsObject();
foreach (var options in new[]
{
"--limit/--max-json-bytes", "--json/--compact/--pretty", "-h/--help",
"`--limit/--max-json-bytes`", "(--limit/--max-json-bytes)",
"[--limit/--max-json-bytes]", "'--limit/--max-json-bytes'",
"\"--limit/--max-json-bytes\"", "--limit/--max-json-bytes.",
"{--limit/--max-json-bytes}", "(`--limit/--max-json-bytes`).",
})
{
source["hint"] = $"Use {options} with /private/secret/file.cs and C:\\private\\secret.cs; --token hidden-value";
var parsed = Assert.IsType<JsonObject>(BatchChildErrorParser.Parse(source.ToJsonString(), "search", 1));
Assert.Equal($"Use {options} with <path> and <path>; --token <redacted>", parsed["hint"]!.GetValue<string>());
}

foreach (var path in new[]
{
"/private/--limit/--max-json-bytes", "/--limit/--max-json-bytes", "C:\\--limit\\--max-json-bytes",
"//server/--limit/--max-json-bytes", "./--limit/--max-json-bytes", "../--limit/--max-json-bytes",
"--limit/--max-json-bytes/private", "--limit/--max-json-bytes.private", "--limit/--private",
"--private/--max-json-bytes", "--path=--limit/--max-json-bytes", "prefix--limit/--max-json-bytes",
"'--limit/--max-json-bytes]customer-project.cs'",
"'--limit/--max-json-bytes}customer-project.cs'",
"'--limit/--max-json-bytes`customer-project.cs'",
"--limit/--max-json-bytes]}.customer-project.cs",
})
{
source["hint"] = $"Retry {path} later";
var parsed = Assert.IsType<JsonObject>(BatchChildErrorParser.Parse(source.ToJsonString(), "search", 1));
var hint = parsed["hint"]!.GetValue<string>();
Assert.Contains("<path>", hint);
Assert.DoesNotContain("/--max-json-bytes", hint);
Assert.DoesNotContain("/--private", hint);
Assert.DoesNotContain("customer-project", hint);
}
}

[Fact]
public void Parse_RejectsMalformedMismatchedAndOverBudgetOutput()
{
Expand Down
48 changes: 48 additions & 0 deletions tests/CodeIndex.Tests/QueryCommandRunnerBatchHintTests.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
using System.Text.Json;
using System.Text.Json.Nodes;
using CodeIndex.Cli;
using static CodeIndex.Tests.QueryCommandTestSupport;

namespace CodeIndex.Tests;

[Collection("Console sensitive")]
public class QueryCommandRunnerBatchHintTests
{
[Fact]
public void RunBatch_CursorHintMatchesStandaloneInBothModes_Issue5423()
{
using var project = TestProjectHelper.CreateTempProjectScope("cdidx_batch_hint_5423");
var db = TestProjectHelper.CreateProjectDb(project.Root);
TestProjectHelper.InsertIndexedFile(db, "rows.txt", "text", "Needle");
string[] child = ["find", "Needle", "--json", "--fields", "path", "--path", "rows.txt", "--cursor", "bad"];
var (directExit, directOutput, directError) = CaptureConsole(
() => ProgramRunner.Run([.. child, "--db", db], JsonOptions, "test"));
Assert.Equal(CommandExitCodes.UsageError, directExit);
Assert.Empty(directError);
var direct = JsonNode.Parse(directOutput)!;
var expectedHint = direct["hint"]!.GetValue<string>();
Assert.Contains("--limit/--max-json-bytes", expectedHint, StringComparison.Ordinal);

foreach (var parallelism in new[] { "1", "2" })
{
var input = JsonSerializer.Serialize(child) + "\n" + JsonSerializer.Serialize(child) + "\n";
var (exit, stdout, stderr) = CaptureConsoleWithInput(input, () => QueryCommandRunner.RunBatch(
["--db", db, "--json-summary", "--parallel", parallelism], JsonOptions));
Assert.Equal(directExit, exit);
Assert.Empty(stderr);
var records = stdout.Split('\n', StringSplitOptions.RemoveEmptyEntries).Select(line => JsonNode.Parse(line)!).ToArray();
Assert.Equal(3, records.Length);
for (var index = 0; index < 2; index++)
{
var record = records[index];
Assert.Equal("batch_result", record["record"]!.GetValue<string>());
Assert.Equal(index + 1, record["line"]!.GetValue<int>());
Assert.Equal(directExit, record["exit_code"]!.GetValue<int>());
Assert.Equal("cursor_malformed", record["error"]!["category"]!.GetValue<string>());
Assert.Equal(direct["error_code"]!.GetValue<string>(), record["error"]!["error_code"]!.GetValue<string>());
Assert.Equal(expectedHint, record["error"]!["hint"]!.GetValue<string>());
}
Assert.Equal(2, records[^1]["command_failures"]!.GetValue<int>());
}
}
}
Loading