From 86193e5c73851121103464d496e37d6abe1e2eca Mon Sep 17 00:00:00 2001 From: Christopher Tso <3115311+christso@users.noreply.github.com> Date: Wed, 28 Oct 2026 11:09:49 +1100 Subject: [PATCH 1/3] fix(productivity): use plain punctuation in write-naturally --- plugins/productivity/skills/write-naturally/SKILL.md | 3 ++- .../skills/write-naturally/references/technical-designs.md | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/plugins/productivity/skills/write-naturally/SKILL.md b/plugins/productivity/skills/write-naturally/SKILL.md index 4500b88..8cf73ff 100644 --- a/plugins/productivity/skills/write-naturally/SKILL.md +++ b/plugins/productivity/skills/write-naturally/SKILL.md @@ -28,6 +28,7 @@ Do not force all four parts into short messages. A one-line reply should stay on ## Voice - Direct, calm, and confident. +- Use everyday keyboard punctuation in prose: straight apostrophes (`'`, as in `don't`) and spaced hyphens (` - `) for dashes. - Evidence before reassurance. - Concrete nouns and exact technical terms. - Short sentences when the point is simple; compact paragraphs when reasoning is needed. @@ -57,7 +58,7 @@ Preserve informality; do not manufacture spelling mistakes or unclear grammar. ### Longer documents -Use headings that name the reader’s question or the part being changed. Bullets should describe delivered or proposed behavior. Record exact proof where it exists, and separate deliberate exclusions from unfinished work. +Use headings that name the reader's question or the part being changed. Bullets should describe delivered or proposed behavior. Record exact proof where it exists, and separate deliberate exclusions from unfinished work. ### Surface-specific guidance diff --git a/plugins/productivity/skills/write-naturally/references/technical-designs.md b/plugins/productivity/skills/write-naturally/references/technical-designs.md index 85d42c6..f444da4 100644 --- a/plugins/productivity/skills/write-naturally/references/technical-designs.md +++ b/plugins/productivity/skills/write-naturally/references/technical-designs.md @@ -1,6 +1,6 @@ # Technical designs and architecture documents -Choose the document’s abstraction level before editing. A design explains behavior, boundaries and the mechanisms needed to support its guarantees. +Choose the document's abstraction level before editing. A design explains behavior, boundaries and the mechanisms needed to support its guarantees. ## Explain the design before specifying its rules From 85163e1a7d41ebd950f062f0e35936b9ca03fdf7 Mon Sep 17 00:00:00 2001 From: Christopher Tso <3115311+christso@users.noreply.github.com> Date: Wed, 28 Oct 2026 11:12:32 +1100 Subject: [PATCH 2/3] test(productivity): cover plain punctuation in proofreading --- README.md | 4 +- .../write-naturally/tests/punctuation.yaml | 37 +++++++++++++++++++ .../skills/write-naturally/SKILL.md | 2 +- 3 files changed, 41 insertions(+), 2 deletions(-) create mode 100644 evals/productivity/write-naturally/tests/punctuation.yaml diff --git a/README.md b/README.md index 67c744c..14ba298 100644 --- a/README.md +++ b/README.md @@ -65,7 +65,9 @@ bun run validate bun test ``` -The productivity plugin's writing evals live in [evals/productivity/write-naturally](evals/productivity/write-naturally/promptfooconfig.yaml). Install development dependencies with `bun install`, then use `bun run eval:validate` or `bun run eval:write-naturally`. Its 28 fixtures are fictional and cover team messages, customer updates, planning notes, reviews and technical designs. The writer and judge run in a read-only workspace with network access disabled inside the model workspace; model-service requests still use the configured Codex provider. The suite never posts reviews. The writer defaults to `gpt-6.1-sol`; the grader defaults to `gpt-5.6-luna`. Use `CODEX_MODEL` and `CODEX_JUDGE_MODEL` to override them separately. Focus on review comments with `bun run eval:write-naturally --filter-metadata group=review-regression --max-concurrency 2`. +The productivity plugin's writing evals live in [evals/productivity/write-naturally](evals/productivity/write-naturally/promptfooconfig.yaml). Install development dependencies with `bun install`, then use `bun run eval:validate` or `bun run eval:write-naturally`. Its 30 fixtures are fictional and cover team messages, customer updates, planning notes, reviews and technical designs. The writer and judge run in a read-only workspace with network access disabled inside the model workspace; model-service requests still use the configured Codex provider. The suite never posts reviews. The writer defaults to `gpt-6.1-sol`; the grader defaults to `gpt-5.6-luna`. Use `CODEX_MODEL` and `CODEX_JUDGE_MODEL` to override them separately. Focus on review comments with `bun run eval:write-naturally --filter-metadata group=review-regression --max-concurrency 2`. + +Two proofreading cases check plain punctuation using deterministic `not-regex` and `equals` assertions, with no model grader. They check that proofreading keeps straight apostrophes and spaced hyphens in an already plain draft and replaces curly apostrophes and an em dash in a typographic draft. Run them with `bun run eval:write-naturally --filter-metadata group=punctuation`. Additional coverage lives in the [private eval repository](https://github.com/allagentsdev/agent-plugins-evals-private). diff --git a/evals/productivity/write-naturally/tests/punctuation.yaml b/evals/productivity/write-naturally/tests/punctuation.yaml new file mode 100644 index 0000000..f137291 --- /dev/null +++ b/evals/productivity/write-naturally/tests/punctuation.yaml @@ -0,0 +1,37 @@ +# Narrow proofreading cases use deterministic assertions instead of model graders. +- description: '[P1] proofreading preserves plain keyboard punctuation' + vars: + surface: Team chat + request: Proofread this message. Fix the spelling and keep the wording. Return only the corrected message. + draft: "I'll send teh notes - don't wait for me." + metadata: + id: P1 + dataset_version: v1-public-fictional + group: punctuation + surface: chat + fixture_origin: Original fictional scenario; no real people or private provenance. + options: &deterministicOptions + disableDefaultAsserts: true + assert: &punctuationAssertions + - type: skill-used + metric: repository-skill-used + value: write-naturally + - type: not-regex + metric: keyboard-punctuation + value: '[\u2018\u2019\u2014]' + - type: equals + metric: corrected-message + value: "I'll send the notes - don't wait for me." +- description: '[P2] proofreading replaces curly apostrophes and an em dash' + vars: + surface: Team chat + request: Proofread this message. Fix the spelling and keep the wording. Return only the corrected message. + draft: 'I’ll send teh notes—don’t wait for me.' + metadata: + id: P2 + dataset_version: v1-public-fictional + group: punctuation + surface: chat + fixture_origin: Original fictional scenario; no real people or private provenance. + options: *deterministicOptions + assert: *punctuationAssertions diff --git a/plugins/productivity/skills/write-naturally/SKILL.md b/plugins/productivity/skills/write-naturally/SKILL.md index 8cf73ff..f4d8fab 100644 --- a/plugins/productivity/skills/write-naturally/SKILL.md +++ b/plugins/productivity/skills/write-naturally/SKILL.md @@ -28,7 +28,7 @@ Do not force all four parts into short messages. A one-line reply should stay on ## Voice - Direct, calm, and confident. -- Use everyday keyboard punctuation in prose: straight apostrophes (`'`, as in `don't`) and spaced hyphens (` - `) for dashes. +- Use everyday keyboard punctuation in prose: straight apostrophes (`'`, as in `don't`) and spaced hyphens (` - `) for dashes. Apply this when proofreading too, including punctuation already present in the draft. - Evidence before reassurance. - Concrete nouns and exact technical terms. - Short sentences when the point is simple; compact paragraphs when reasoning is needed. From 916382ba1a9b9fe896c85f37fb31f150ca01bb99 Mon Sep 17 00:00:00 2001 From: Christopher Tso <3115311+christso@users.noreply.github.com> Date: Wed, 28 Oct 2026 11:48:53 +1100 Subject: [PATCH 3/3] docs: trim README eval guidance --- README.md | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 14ba298..087c5aa 100644 --- a/README.md +++ b/README.md @@ -58,26 +58,35 @@ The companion AllAgents migration preserves the old marketplace declarations, su ## Development -Use Bun. The existing engineering plugin tests move with their implementation. +Use Bun: ```sh +bun install bun run validate bun test ``` -The productivity plugin's writing evals live in [evals/productivity/write-naturally](evals/productivity/write-naturally/promptfooconfig.yaml). Install development dependencies with `bun install`, then use `bun run eval:validate` or `bun run eval:write-naturally`. Its 30 fixtures are fictional and cover team messages, customer updates, planning notes, reviews and technical designs. The writer and judge run in a read-only workspace with network access disabled inside the model workspace; model-service requests still use the configured Codex provider. The suite never posts reviews. The writer defaults to `gpt-6.1-sol`; the grader defaults to `gpt-5.6-luna`. Use `CODEX_MODEL` and `CODEX_JUDGE_MODEL` to override them separately. Focus on review comments with `bun run eval:write-naturally --filter-metadata group=review-regression --max-concurrency 2`. +Run the [writing evals](evals/productivity/write-naturally/promptfooconfig.yaml): -Two proofreading cases check plain punctuation using deterministic `not-regex` and `equals` assertions, with no model grader. They check that proofreading keeps straight apostrophes and spaced hyphens in an already plain draft and replaces curly apostrophes and an em dash in a typographic draft. Run them with `bun run eval:write-naturally --filter-metadata group=punctuation`. +```sh +bun run eval:validate +bun run eval:write-naturally +``` + +For deterministic punctuation checks, use `bun run eval:write-naturally --filter-metadata group=punctuation`. -Additional coverage lives in the [private eval repository](https://github.com/allagentsdev/agent-plugins-evals-private). +Evals use fictional data and your Codex connection. Set `CODEX_MODEL` and `CODEX_JUDGE_MODEL` to change the writer and grader. -### Humanizer comparison +Compare the [Humanizer-inspired cases](https://github.com/blader/humanizer/blob/225a6f39ac85f76ee48dbad772ea4abe4ed6c9d8/SKILL.md) with and without the skill: -Eight fictional cases inspired by [Humanizer](https://github.com/blader/humanizer/blob/225a6f39ac85f76ee48dbad772ea4abe4ed6c9d8/SKILL.md) cover rhetorical framing, inflated claims, formatting, voice and replies to readers who already know the context. Controls keep useful checklists and plain technical wording. Run the same cases with the skill using `bun run eval:write-naturally --filter-metadata group=humanizer`, then without it using `bun run eval:write-naturally:baseline`. Both conditions use the same writer model, reasoning setting and graders. The baseline rejects recorded tool calls, including file reads, and missing traces. This checks the model turn; it cannot rule out instructions supplied by the surrounding Codex configuration. +```sh +bun run eval:write-naturally --filter-metadata group=humanizer +bun run eval:write-naturally:baseline +``` -Readability has separate style criteria for these cases; fact preservation is graded independently. Calibration answers stay in metadata and are never supplied to the writer. Compare individual outputs and repeat runs before attributing a gain to the skill. +More cases are available in the [private eval repository](https://github.com/allagentsdev/agent-plugins-evals-private). -Changes are reviewed through pull requests. See [AGENTS.md](AGENTS.md) for contribution guidance. +See [AGENTS.md](AGENTS.md) for contribution guidance. ## License