Export grammar and texts for AI analysis

.gitignore

@@ -215,5 +215,6 @@ DataTreeTimingBaselines.json
 /findings_*.md
 *scratchpad*
 *[Ss]cratchPad*
+/.review/
 
 Docs/migration/working/

CONTEXT.md

@@ -17,6 +17,10 @@ It is intentionally not a full architecture manual. It should stay biased toward
 - **MSBuild project**: A `.csproj`, `.vcxproj`, `.wixproj`, or similar build unit.
 - **Installer project**: WiX authoring and packaging work under `FLExInstaller/`.
 - **Worktree**: A git worktree for isolated builds and edits.
+- **`Grammar`**: overloaded — qualify when the distinction matters.
+  - **Grammar Area**: One of the five top-level Areas (see Core Domain section) — the Parts of Speech/Features/etc. tool group.
+  - **Grammar Sketch**: An existing human-readable, published linguistic-description document export (`ExportDialog.ExportGrammarSketch`); unrelated to the HC grammar below despite the shared word.
+  - **HC grammar**: The HermitCrab-format grammar (phonology, morphology, features, categories) produced from a language project via `HCLoader.Load` + `XmlLanguageWriter.Save` (`SIL.Machine.Morphology.HermitCrab`). Always say "HC grammar," never bare "grammar," when the distinction from the other two matters.
 - **`Writing system`**: overloaded — qualify when the distinction matters.
   - **Writing system definition**: `WritingSystemDefinition` (libpalaso / `SIL.WritingSystems`); the base writing system class, identified by a BCP-47 language tag.
   - **Core writing system definition**: `CoreWritingSystemDefinition` (liblcm / `SIL.LCModel.Core.WritingSystems`); extends `WritingSystemDefinition` with a `Handle` and LCM-specific features such as character sets.
@@ -33,6 +37,7 @@ It is intentionally not a full architecture manual. It should stay biased toward
 - **IME composition**: The transient input-method editing state before text is committed. Treat composition behavior and committed text behavior as separate test and parity concerns.
 - **Lexicon**: The lexical data and editing experience in FLEx.
 - **Interlinear text**: Text annotated with multiple aligned linguistic analysis lines.
+- **Word occurrence** (`IAnalysis`, in `SIL.LCModel`): One tokenized position in a paragraph — either a bare, unanalyzed **wordform occurrence** (`IWfiWordform`, `HasWordform == true` but no morphological breakdown chosen) or an **analyzed occurrence** (`IWfiAnalysis`/`IWfiGloss` — a morphological breakdown/gloss has been attached, whether by a human or an unreviewed parser guess; "analyzed" says nothing about whether the analysis is linguistically correct). Punctuation occurrences also satisfy `IAnalysis` but not `HasWordform`. This three-way distinction (punctuation / wordform-only / analyzed) recurs across `StatisticsView`, `ConcordanceControl`, and Interlinear editing — use these names rather than inventing new ones.
 - **Morphology**: The part of the system and data model concerned with morphemes, rules, and word analysis.
 - **Parser**: Morphological analysis tooling such as HermitCrab or XAmple.
 - **Paratext integration**: Scripture and lexicon interoperability with Paratext. Implemented across `FwParatextLexiconPlugin`, `ParatextImport` (scripture text import via `ParatextImportManager`/`ParatextSfmImporter`), and `Paratext8Plugin` (bridge to Paratext APIs).
@@ -53,7 +58,7 @@ It is intentionally not a full architecture manual. It should stay biased toward
 - **Service locator**: `LcmCache.ServiceLocator` (`ILcmServiceLocator`). IoC container for LCModel — the primary way to retrieve repositories, factories, and services.
 - **Unit of work**: Groups data-model changes under `IActionHandler`. All LCModel writes must occur inside one. `UndoableUnitOfWorkHelper` (undoable) and `NonUndoableUnitOfWorkHelper` (non-undoable) are in `SIL.LCModel.Infrastructure`; use as a `using` block or via their static `.Do(...)` helpers.
 - **Publish/subscribe system**: Messaging system (`IPublisher` / `ISubscriber`, `SIL.FieldWorks.Common.FwUtils`) via `FwUtils.Publisher` and `FwUtils.Subscriber` singletons. Supports exact-name and prefix subscriptions; `PublishAtEndOfAction` defers delivery to end of user action. Event-based problems deserve event-based solutions — avoid state variables for event timing when a deterministic subscribe/unsubscribe solution can be used.
-- **Area**: One of the five top-level navigation divisions — Lexicon, Grammar, Words & Texts, Notebook, Lists. Declared in `areaConfiguration.xml` and identified by constants in `AreaConstants`. Each Area has its own sidebar and owns a set of Tools.
+- **Area**: One of the five top-level navigation divisions — Lexicon, Grammar, Texts & Words, Notebook, Lists. Declared in `areaConfiguration.xml` and identified by constants in `AreaConstants`. Each Area has its own sidebar and owns a set of Tools. (The Texts & Words area's internal `areaChoice` value is `textsWords`.)
 - **Tool**: A view or function within an Area, declared in `toolConfiguration.xml`. The active Tool per Area is tracked via `ToolForAreaNamed_<area>`; switching Areas restores the last-used Tool. Navigation is property-driven through the XCore mediator system.
 - **Dictionary configuration**: A `.fwdictconfig` XML file (`DictionaryConfigurationModel`) defining which LCModel fields appear in a dictionary view, their order, style, and options. Scoped to one or more Publications.
 - **Publication**: A named output target (e.g. a print edition or web view) that dictionary configurations are scoped to. `AllPublications` applies a configuration to all current and future publications.

DistFiles/Language Explorer/Configuration/Main.xml

@@ -921,6 +921,9 @@ I (RandyR) brought them into this file, just to keep track of them for the time
 			populates the "tools" list appropriately. -->
 		<listener assemblyPath="LexTextDll.dll" class="SIL.FieldWorks.XWorks.LexText.AreaListener"/>
 		<listener assemblyPath="LexEdDll.dll" class="SIL.FieldWorks.XWorks.LexEd.FLExBridgeListener"/>
+		<!-- answers the grammar+texts-for-AI export's request to write FLExText files,
+			since InterlinVc/InterlinearExporter are only reachable from ITextDll.dll -->
+		<listener assemblyPath="ITextDll.dll" class="SIL.FieldWorks.IText.FlexTextAIExportListener"/>
 		<!-- take care of the writing system combo box -->
 		<listener assemblyPath="xWorks.dll" class="SIL.FieldWorks.XWorks.WritingSystemListHandler"/>
 		<listener assemblyPath="xWorks.dll" class="SIL.FieldWorks.XWorks.CombinedStylesListHandler"/>

DistFiles/Language Explorer/Export Templates/AIExportInstructions.md

@@ -0,0 +1,76 @@
+# Read this first: a FieldWorks grammar and texts for analysis
+
+This folder was exported from FieldWorks Language Explorer (FLEx) by its "Export Grammar and
+Texts for AI Analysis" command. Everything in it was put here deliberately for an AI assistant
+(ChatGPT, Claude, or otherwise) to read, including this file.
+
+## What is in this folder
+
+- `HCGrammar.xml` -- the project's **HC grammar**: its HermitCrab-format grammar (phonology,
+  morphology, features, categories), the same shape the HermitCrab parser itself consumes to
+  parse words. This is the parser's own input, not a human-readable grammar description.
+- One `.flextext` file per exported text -- FLExText, FieldWorks' interlinear-text interchange
+  format. These are real corpus texts the grammar is supposed to account for. Any given text may
+  be fully analyzed, partly analyzed, or not analyzed at all.
+- `export-instructions.md` -- this file.
+
+## Before you answer, read the reference
+
+Grammars and interlinear texts have real gotchas you will not know about on your own. Fetch and
+read this first:
+
+```
+https://raw.githubusercontent.com/sillsdev/FieldWorks/main/Docs/ai-parser-help/README.md
+```
+
+That is the index for a reference covering correctness gotchas (`broken/`), performance gotchas
+(`speed/`), grammar-authoring methodology (`workflow/`), and the FLExText format itself,
+including the `analysisStatus` ground-truth caveat (`texts/`). Use the raw URL rather than the
+`github.com/.../blob/...` page -- it returns plain markdown that fetches cleanly. Follow whichever
+of its links bear on the question you were asked, and reason about this project's grammar against
+the engine's documented mechanics rather than guessing.
+
+## How to answer: plain language, for a linguist
+
+The person asking is a field linguist with a real language project open in FLEx. They know their
+language; they usually do not know HermitCrab's internals, and they did not ask for a tour of
+them. An answer that reads like engine documentation gets abandoned, however correct it is.
+
+- **Accuracy first, then plain language.** An answer that misstates the engine is worse than no
+  answer, so never simplify to the point of being wrong. When the honest answer is complicated,
+  give the short version first and the detail underneath.
+- **No unexplained jargon.** Use a technical term only if the answer genuinely needs it, and
+  define it in ordinary words the first time: "MPR feature (a label you put on a morpheme so that
+  rules can test for it)". Never leave an abbreviation bare -- write "morphosyntactic property",
+  not "MSP".
+- **Use the words FLEx uses.** Name what the user actually clicks (the Grammar area, an affix
+  template, a slot, an allomorph, an inflection class) rather than the engine's internal class or
+  field name, so they can find it. Describe fixes as things to do in FLEx, not as XML to
+  hand-edit.
+- **Lead with what to do.** Open with the change to make, in a sentence or two. Put the mechanism
+  -- why the engine behaves this way -- after it, for the reader who wants it.
+- **Show rather than lecture.** One worked example with a real surface form beats a paragraph of
+  theory.
+- **Short sentences.** Cut "it is important to note that", "as mentioned above", and any
+  restatement of what you just said.
+- **Flag every guess, every time.** Name what you are unsure about and what would settle it (a
+  specific word to parse, a trace to look at). The user cannot see your reasoning, so an unmarked
+  guess reads exactly like a fact, and a confident wrong answer costs them real work.
+- **Each answer stands alone.** Do not lean on earlier turns in the chat, or on file names in the
+  reference, as though the user had them open.
+
+## For the person who exported this folder
+
+**This is AI. It can be very wrong, and it can be wrong while sounding certain.** Treat anything
+it tells you about your grammar or your texts as a suggestion to check against your own data,
+never as a finding. Understand a change, and confirm it yourself by re-parsing the words you care
+about, before you apply it to your project.
+
+Your exported files are real linguistic data about a real language project. Dropping them into a
+third-party chat service sends that data to that provider (OpenAI, Anthropic, and so on), so check
+your project's data-sensitivity policy before sharing an unpublished or restricted grammar or text
+this way.
+
+A fuller human-readable walkthrough, including example questions worth asking, is at:
+
+<https://github.com/sillsdev/FieldWorks/blob/main/Docs/ai-parser-help/getting-started.md>

DistFiles/Language Explorer/Export Templates/GrammarAndTextsForAI.xml

@@ -0,0 +1,11 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<template type="grammarTextsAI">
+	<FxtDocumentDescription dataLabel="Grammar and Texts" formatLabel="XML for AI Analysis"
+		defaultExtension="" filter="">Export Grammar and Texts for AI Analysis
+
+WARNING: this is AI. It can be very wrong, and it can be wrong while sounding certain. Treat anything it tells you about your grammar or your texts as a suggestion to check against your own data, never as a finding. Understand a change, and confirm it yourself, before you apply it to your project.
+
+Writes this project's HermitCrab grammar (HCGrammar.xml), one .flextext file per selected text, and an export-instructions.md file into a folder you choose.
+
+Then drag and drop all the files into ChatGPT (or Claude) and ask your question. The export-instructions.md file carries the reference links and reading instructions the AI needs, so there is nothing else for you to paste.</FxtDocumentDescription>
+</template>

Docs/ai-parser-help/README.md

@@ -0,0 +1,110 @@
+# AI Parser Help — HermitCrab reference for LLMs
+
+This is a living reference for asking an LLM (ChatGPT, Claude, or otherwise) questions about
+**HermitCrab**, the rule-based morphological parser/generator implemented in this repository
+(`sillsdev/machine`, namespace `SIL.Machine.Morphology.HermitCrab`), and about **FLExText**,
+FieldWorks' interlinear-text interchange format for the connected corpus texts a grammar is
+supposed to account for. It covers three kinds of question a grammar author actually asks about
+HermitCrab itself, plus a fourth section for a different kind of upload entirely — real corpus
+texts rather than the grammar:
+
+- **[`broken/`](broken/README.md)** — "why is this wrong / missing / crashing?" Correctness
+  gotchas: wrong parses, missing parses, crashes, and silent misconfigurations.
+- **[`speed/`](speed/README.md)** — "why is this slow?" Performance gotchas: combinatorial
+  blowups and other parse-time costs.
+- **[`workflow/`](workflow/README.md)** — "how should I approach modeling this?" Authoring
+  guidance for building a grammar well in the first place, grounded in HermitCrab's actual
+  mechanics and in H. Andrew Black's FLEx parsing methodology (primary source included verbatim
+  under `workflow/sources/`).
+- **[`texts/`](texts/README.md)** — "I also have real corpus texts, not just a grammar." A
+  reference for FLExText, FieldWorks' interlinear-text interchange format: what it is, how to
+  extract it, and how to get an LLM to reason over it correctly, including the `analysisStatus`
+  ground-truth caveat and which AI products can actually run code against your uploaded file.
+
+Each file covers one topic in enough depth to answer questions about that topic without needing
+local access to the repo — code excerpts, mechanisms, and worked examples are inlined.
+
+## Got a FieldWorks grammar and a question?
+
+Send that person **this link instead**: [`getting-started.md`](getting-started.md) — it walks
+through extracting your grammar as HermitCrab XML and getting ChatGPT/Claude to reason about it
+using this reference. The rest of this README is the reference material itself (for the LLM to
+read), not the human-facing walkthrough.
+
+Got interlinear **texts** (`.flextext` files) instead of, or in addition to, a grammar? See
+[`texts/getting-started.md`](texts/getting-started.md) instead — extracting and reasoning about
+connected corpus texts is a different workflow from the grammar one above.
+
+## How to use this with an LLM
+
+Paste the **raw** URL of the relevant topic file into your chat, e.g.:
+
+```
+https://raw.githubusercontent.com/sillsdev/machine/master/docs/ai-parser-help/speed/affix-template-optional-slots.md
+```
+
+Then ask your question. Use the raw URL (`raw.githubusercontent.com`), not the normal
+`github.com/.../blob/...` page — the raw URL returns plain markdown text with no site chrome,
+which fetches cleanly for both ChatGPT (web browsing) and Claude (WebFetch) without JS rendering
+or auth. If you're not sure which topic file is relevant, paste this README's raw URL first, or
+whichever of `broken/README.md`, `speed/README.md`, `workflow/README.md`, `texts/README.md` best
+matches your question ("why is this wrong" vs. "why is this slow" vs. "how should I model this"
+vs. "I have corpus texts, not just a grammar") — an LLM that can follow links will use it as an
+index; otherwise, browse the lists yourself.
+
+Do not use these guides as a source of real grammar or text data — see "What belongs here" below.
+
+## How to answer: plain language, for a linguist
+
+Instructions for the LLM reading this reference. The person asking is a field linguist with a
+real language project open in FLEx. They know their language; they usually do not know
+HermitCrab's internals, and they did not ask for a tour of them. An answer that reads like engine
+documentation gets abandoned, however correct it is.
+
+- **Accuracy first, then plain language.** An answer that misstates the engine is worse than no
+  answer, so never simplify to the point of being wrong. When the honest answer is complicated,
+  give the short version first and the detail underneath.
+- **No unexplained jargon.** Use a technical term only if the answer genuinely needs it, and
+  define it in ordinary words the first time: "MPR feature (a label you put on a morpheme so that
+  rules can test for it)". Never leave an abbreviation bare -- write "morphosyntactic property",
+  not "MSP".
+- **Use the words FLEx uses.** Name what the user actually clicks (the Grammar area, an affix
+  template, a slot, an allomorph, an inflection class) rather than the engine's internal class or
+  field name, so they can find it. Describe fixes as things to do in FLEx, not as XML to hand-edit.
+- **Lead with what to do.** Open with the change to make, in a sentence or two. Put the mechanism
+  -- why the engine behaves this way -- after it, for the reader who wants it.
+- **Show rather than lecture.** One worked example with a real surface form beats a paragraph of
+  theory.
+- **Short sentences.** Cut "it is important to note that", "as mentioned above", and any
+  restatement of what you just said.
+- **Flag every guess, every time.** Name what you are unsure about and what would settle it (a
+  specific word to parse, a trace to look at). The user cannot see your reasoning, so an unmarked
+  guess reads exactly like a fact, and a confident wrong answer costs them real work.
+- **Each answer stands alone.** Do not lean on earlier turns in the chat, or on file names in this
+  reference, as though the user had them open.
+
+## What belongs here
+
+- General HermitCrab engine mechanics: how rules, strata, templates, features, and the
+  analysis/synthesis engines work. This is documentation of the open-source parser itself.
+- FLEx/HermitCrab grammar-authoring methodology, grounded in the engine's actual behavior.
+- FLExText format documentation, grounded in the FieldWorks schema/exporter source that produces
+  it — see [`texts/`](texts/README.md).
+- Synthetic/toy grammar snippets used purely to illustrate a mechanism (e.g. `p1`..`p12`,
+  `sg`/`pl` × invented cases) are fine, as are synthetic/invented interlinear-text examples.
+- **Not** real grammar or text data for any specific language (e.g. Sena, Amharic, Indonesian,
+  Aweti). Those grammars — and any real corpus texts from those projects — are private and must
+  never be committed to this repo — see the project's existing grammar-privacy constraints. If a
+  question requires reasoning about a real grammar or text, describe the relevant structure
+  abstractly instead of pasting the real rules or sentences.
+
+## Source grounding
+
+Claims in `broken/`, `speed/`, and `workflow/` are grounded in the actual engine source under
+`src/SIL.Machine.Morphology.HermitCrab/` and `src/SIL.Machine/` as of the commit each file was
+last updated — each file's metadata header names the specific source file(s) it's grounded in.
+Claims in `texts/` are grounded in the FieldWorks source that defines and produces FLExText, plus
+cited external documentation — see [`texts/README.md`](texts/README.md#source-grounding) for
+specifics; unlike the other three sections, `texts/` cites other repositories and external sources
+directly, since explaining an interchange format requires it. File/line references may drift as
+the code evolves; if something looks stale, check the live source at the paths cited.