diff --git a/README.md b/README.md index 67c744c..087c5aa 100644 --- a/README.md +++ b/README.md @@ -58,24 +58,35 @@ The companion AllAgents migration preserves the old marketplace declarations, su ## Development -Use Bun. The existing engineering plugin tests move with their implementation. +Use Bun: ```sh +bun install bun run validate bun test ``` -The productivity plugin's writing evals live in [evals/productivity/write-naturally](evals/productivity/write-naturally/promptfooconfig.yaml). Install development dependencies with `bun install`, then use `bun run eval:validate` or `bun run eval:write-naturally`. Its 28 fixtures are fictional and cover team messages, customer updates, planning notes, reviews and technical designs. The writer and judge run in a read-only workspace with network access disabled inside the model workspace; model-service requests still use the configured Codex provider. The suite never posts reviews. The writer defaults to `gpt-6.1-sol`; the grader defaults to `gpt-5.6-luna`. Use `CODEX_MODEL` and `CODEX_JUDGE_MODEL` to override them separately. Focus on review comments with `bun run eval:write-naturally --filter-metadata group=review-regression --max-concurrency 2`. +Run the [writing evals](evals/productivity/write-naturally/promptfooconfig.yaml): -Additional coverage lives in the [private eval repository](https://github.com/allagentsdev/agent-plugins-evals-private). +```sh +bun run eval:validate +bun run eval:write-naturally +``` + +For deterministic punctuation checks, use `bun run eval:write-naturally --filter-metadata group=punctuation`. -### Humanizer comparison +Evals use fictional data and your Codex connection. Set `CODEX_MODEL` and `CODEX_JUDGE_MODEL` to change the writer and grader. -Eight fictional cases inspired by [Humanizer](https://github.com/blader/humanizer/blob/225a6f39ac85f76ee48dbad772ea4abe4ed6c9d8/SKILL.md) cover rhetorical framing, inflated claims, formatting, voice and replies to readers who already know the context. Controls keep useful checklists and plain technical wording. Run the same cases with the skill using `bun run eval:write-naturally --filter-metadata group=humanizer`, then without it using `bun run eval:write-naturally:baseline`. Both conditions use the same writer model, reasoning setting and graders. The baseline rejects recorded tool calls, including file reads, and missing traces. This checks the model turn; it cannot rule out instructions supplied by the surrounding Codex configuration. +Compare the [Humanizer-inspired cases](https://github.com/blader/humanizer/blob/225a6f39ac85f76ee48dbad772ea4abe4ed6c9d8/SKILL.md) with and without the skill: + +```sh +bun run eval:write-naturally --filter-metadata group=humanizer +bun run eval:write-naturally:baseline +``` -Readability has separate style criteria for these cases; fact preservation is graded independently. Calibration answers stay in metadata and are never supplied to the writer. Compare individual outputs and repeat runs before attributing a gain to the skill. +More cases are available in the [private eval repository](https://github.com/allagentsdev/agent-plugins-evals-private). -Changes are reviewed through pull requests. See [AGENTS.md](AGENTS.md) for contribution guidance. +See [AGENTS.md](AGENTS.md) for contribution guidance. ## License diff --git a/evals/productivity/write-naturally/tests/punctuation.yaml b/evals/productivity/write-naturally/tests/punctuation.yaml new file mode 100644 index 0000000..f137291 --- /dev/null +++ b/evals/productivity/write-naturally/tests/punctuation.yaml @@ -0,0 +1,37 @@ +# Narrow proofreading cases use deterministic assertions instead of model graders. +- description: '[P1] proofreading preserves plain keyboard punctuation' + vars: + surface: Team chat + request: Proofread this message. Fix the spelling and keep the wording. Return only the corrected message. + draft: "I'll send teh notes - don't wait for me." + metadata: + id: P1 + dataset_version: v1-public-fictional + group: punctuation + surface: chat + fixture_origin: Original fictional scenario; no real people or private provenance. + options: &deterministicOptions + disableDefaultAsserts: true + assert: &punctuationAssertions + - type: skill-used + metric: repository-skill-used + value: write-naturally + - type: not-regex + metric: keyboard-punctuation + value: '[\u2018\u2019\u2014]' + - type: equals + metric: corrected-message + value: "I'll send the notes - don't wait for me." +- description: '[P2] proofreading replaces curly apostrophes and an em dash' + vars: + surface: Team chat + request: Proofread this message. Fix the spelling and keep the wording. Return only the corrected message. + draft: 'I’ll send teh notes—don’t wait for me.' + metadata: + id: P2 + dataset_version: v1-public-fictional + group: punctuation + surface: chat + fixture_origin: Original fictional scenario; no real people or private provenance. + options: *deterministicOptions + assert: *punctuationAssertions diff --git a/plugins/productivity/skills/write-naturally/SKILL.md b/plugins/productivity/skills/write-naturally/SKILL.md index 4500b88..f4d8fab 100644 --- a/plugins/productivity/skills/write-naturally/SKILL.md +++ b/plugins/productivity/skills/write-naturally/SKILL.md @@ -28,6 +28,7 @@ Do not force all four parts into short messages. A one-line reply should stay on ## Voice - Direct, calm, and confident. +- Use everyday keyboard punctuation in prose: straight apostrophes (`'`, as in `don't`) and spaced hyphens (` - `) for dashes. Apply this when proofreading too, including punctuation already present in the draft. - Evidence before reassurance. - Concrete nouns and exact technical terms. - Short sentences when the point is simple; compact paragraphs when reasoning is needed. @@ -57,7 +58,7 @@ Preserve informality; do not manufacture spelling mistakes or unclear grammar. ### Longer documents -Use headings that name the reader’s question or the part being changed. Bullets should describe delivered or proposed behavior. Record exact proof where it exists, and separate deliberate exclusions from unfinished work. +Use headings that name the reader's question or the part being changed. Bullets should describe delivered or proposed behavior. Record exact proof where it exists, and separate deliberate exclusions from unfinished work. ### Surface-specific guidance diff --git a/plugins/productivity/skills/write-naturally/references/technical-designs.md b/plugins/productivity/skills/write-naturally/references/technical-designs.md index 85d42c6..f444da4 100644 --- a/plugins/productivity/skills/write-naturally/references/technical-designs.md +++ b/plugins/productivity/skills/write-naturally/references/technical-designs.md @@ -1,6 +1,6 @@ # Technical designs and architecture documents -Choose the document’s abstraction level before editing. A design explains behavior, boundaries and the mechanisms needed to support its guarantees. +Choose the document's abstraction level before editing. A design explains behavior, boundaries and the mechanisms needed to support its guarantees. ## Explain the design before specifying its rules