Update dateModified when a lexicon is changed: two candidate designs

[imnasnainaec]

Problem

date_created and date_modified are inert data fields. The reader parses them, the writer serializes whatever the model holds, and nothing in between ever sets them — there is no datetime.now(), utcnow(), or date.today() anywhere in src/sil_lift/.

The consequence is silent and easy to hit. Edit a gloss and call save(), and the entry goes out with new content under its original dateModified:

lex = sil_lift.load("dictionary.lift")
sense.definition["en"] = "the color of a thing"
lex.save()          # entry content changed; dateModified still says 2011-08-04

That matters because downstream tools key merge and update decisions on it. The Combine's LIFT import maps entry.DateCreated/entry.DateModified straight onto its Word.Created/Word.Modified (Backend/Services/LiftService.cs), and FLEx/Chorus use dateModified when reconciling changes. A lexicon edited by sil-lift currently looks unmodified to all of them.

The two guides that demonstrate editing — docs/en/guides/bulk-edit-glosses.md (load, edit, validate, save) and docs/en/guides/build-export.md (build a lexicon from scratch) — never mention timestamps, so they teach the omission.

What the granularity actually needs to be

Nine model types can carry these attributes: the eight fully-extensible elements (entry, sense, note, example, relation, etymology, pronunciation, variant) plus field. Nothing on the header side has them, nor the <lift> root.

In practice only one of the nine is ever populated. Across the seven FieldWorks exports (FLEx 8.3.12 through 9.0.7) in The Combine's Backend.Tests/Assets:

Element	Present	dateModified	dateCreated
entry	35,318	35,318	35,318
sense	40,497	0	0
note	11,051	0	0
field	6,543	0	0
relation	4,883	0	0
example	4,781	0	0
etymology	1,309	0	0
variant	671	0	0
pronunciation	19	0	0

Every entry carries both stamps; not one of 69,754 sub-entry opportunities carries either. The repo corpus agrees — all 3,546 of its dateModified attributes are on <entry>. The Combine's own data model has no per-sense timestamp at all, and its parser callbacks note that sub-entry dates are read but unused.

All 70,636 date literals in those exports are the same 20-character shape, YYYY-MM-DDTHH:MM:SSZ — no bare dates, numeric offsets, or fractional seconds.

So entry-level stamping is the whole practical requirement, and that is convenient: entry_digest already hashes an entry's entire serialized subtree, so an edit at any depth — a gloss four levels down inside a subsense — registers as a change to its entry.

Shared groundwork

Both routes below need the same three pieces.

1. One internal mutation primitive. A single helper on _ExtensibleNoFields in src/sil_lift/_model.py, inherited by all nine date-bearing types, so the policy lives in exactly one place:

def _stamp(self, when: datetime) -> None:
    self.date_modified = when
    if self.date_created is None:
        self.date_created = when

Whether this is also public API is a separate decision — see Open questions.

2. Parse-time date_modified on the entry record. _EntryRecord in src/sil_lift/_writer.py currently holds a strong entry reference and the parse-time digest. Both routes need to distinguish "content changed and the date was left alone" from "the caller already set the date deliberately", which the digest alone cannot express:

@dataclass(slots=True)
class _EntryRecord:
    entry: Entry
    digest: bytes
    date_modified: datetime | date | None   # new: value at parse time

The predicate both routes use:

def _is_stale(entry: Entry, record: _EntryRecord) -> bool:
    """Content changed, but the caller did not touch dateModified."""
    return entry_digest(entry) != record.digest and entry.date_modified == record.date_modified

3. Stamping must update the entry's baseline. Whichever route does the stamping has to write the new digest and date back into that entry's record. Otherwise a second round of edits on the same in-memory lexicon is silently missed:

1. Load — the baseline is (D0, T0).
2. Edit, stamp, save — the entry is now (D1, T1), but the baseline still reads (D0, T0).
3. Edit again, stamp — the digest differs from D0, but entry.date_modified is now T1 while the baseline says T0, so _is_stale returns False and the second edit ships without a bump.

This is a correctness requirement for both routes, not a property of either. It is also the one piece that cannot be exercised in isolation, since it only becomes observable once something stamps — so it needs tests driving two full edit-and-stamp cycles over a single loaded lexicon.

4. Clock and format. Generated stamps should be seconds-precision UTC, which _fmt_date already renders as ...Z (it rewrites +00:00). Default datetime.now(timezone.utc).replace(microsecond=0), with an explicit when= parameter on every entry point so byte-exact tests are writable. See #9 for honoring SOURCE_DATE_EPOCH as the default clock source.

The change-detection half of this groundwork is separable and useful on its own; see #11.

---

Route 1: stamp changed entries during save()

save() grows two keyword-only parameters and stamps by default:

def save(
    self,
    path: str | os.PathLike[str] | None = None,
    *,
    stamp: bool = True,
    when: datetime | None = None,
) -> None:

Behavior, immediately before serialization:

• For each entry with a parse-time record, stamp it if _is_stale(entry, record).
• For each entry with no record — created in memory, or a from-scratch lexicon with no _source — stamp it and fill date_created if blank.
• stamp=False restores today's behavior exactly.

Stamping updates the baseline per shared groundwork above; note that render_document currently reads lexicon._source and never writes it, so this makes a structure that is write-once at parse time mutable.

Two behaviors fall out correctly and are worth locking in with tests. sort() does not trigger stamping, because reordering does not change any entry's canonical bytes — preserving the guarantee already documented on Lexicon.sort. And an entry the caller stamped by hand is left alone, because _is_stale sees the date differ from its parse-time value.

Pros

• Correct by default. The failure mode disappears for everyone who never reads the docs, which is the population currently getting it wrong.
• No new concepts in the API surface — one keyword on a method every caller already invokes.
• Granularity matches the fidelity contract exactly: the unit that gets stamped is the unit that gets re-serialized.

Cons

• save() mutates the caller's in-memory model, which is surprising for a write method, and more so for save(path) used as "export a copy".
• It creates a discrepancy with iter_problems(), whose docstring promises it validates "what save() would write". After this change that is no longer literally true — validation would see pre-stamp state. Either the docstring narrows to describe content as it stands, or validation stamps too, which means mutating during a read-only-looking call. This needs resolving either way.
• Anyone wanting byte-identical output must now know to pass stamp=False.

---

Route 2: an explicit method, with a validation backstop

Three pieces that together make the omission hard to make silently.

A bulk method on Lexicon, mirroring the semantics The Combine settled on in WordService.UpdateTimes (always bump on edit; fill-only-when-blank on bulk import):

def stamp_modified(
    self,
    when: datetime | None = None,
    *,
    only_missing: bool = False,
) -> list[Entry]:
    """Stamp dateModified on every entry whose content changed since load.

    Returns the entries stamped. With only_missing, fills blank stamps
    without bumping ones that are already set.
    """

Usage stays a single line regardless of how deep or how many the edits were:

lex = sil_lift.load(path)
...                        # edits at any depth, any number of nodes
lex.stamp_modified()       # one call
lex.save()

A validation warning so forgetting the call is loud rather than silent. Problem already carries level/code/entry_id/line, iter_lexicon_problems already receives the whole Lexicon (so _source is in scope) and already imports from _writer (where entry_digest lives), and the CLI already counts, prints, and — under --strict — fails on warnings. A new stale-timestamp code needs no new plumbing:

warning [stale-timestamp] dictionary.lift:4213 (entry a_cd045907-…): entry content
changed but dateModified was not updated; call Lexicon.stamp_modified()

Both existing guides already run validation before saving, so the message appears exactly where someone is already reading output, and it names the method that fixes it.

A knob on the calls everyone already makes, for callers who would rather not think about it again — sil_lift.load(path, stamp=True) recording the policy on the lexicon for later saves, and/or save(stamp=True) as a one-off.

Pros

• No hidden mutation. Stamping runs where the caller put it, so validated bytes and written bytes stay identical and the iter_problems() contract is untouched.
• Consistent with the library's existing posture that nothing happens implicitly — validation is already explicit on load and save.
• Repeated calls without intervening edits stamp nothing, so an accidental double invocation is harmless.
• The warning is independently valuable: it also catches the case where someone edits a lexicon with a different tool in the same pipeline.
• Returning the stamped entries makes scripts self-documenting and assertions easy.

Cons

• Still opt-in. The warning narrows the gap but does not close it — a caller who ignores warnings, or who never validates, gets today's behavior.
• Three overlapping entry points for one concept (stamp_modified, the load knob, the save knob) is a lot of surface pre-1.0.
• stamp_modified() costs a full canonical serialization pass to compute digests, and iter_problems() pays that cost again — two or three passes over a large lexicon on the validate-then-save path unless digests are cached.
• Adding a warning code changes what --strict fails on. Harmless now, since nothing has been released, but it is the kind of change the CLI's documented-interface promise covers.

---

Trade-off summary

	Route 1	Route 2
Correct without reading docs	yes	no (warned, not enforced)
save() mutates the model	yes	no
iter_problems() contract	needs revisiting	unchanged
Extra serialization passes	none	one to two
Byte-identical output	stamp=False	default

Nothing has shipped yet — CHANGELOG.md has only ## [Unreleased] — so a default-behavior change is cheaper now than it will ever be again.

Open questions

• Is the per-node stamp helper public? Entry granularity covers every observed real-world case, and a public per-node method invites for e in lex.entries: e.stamp(), which stamps unchanged entries, changes every digest, and re-serializes the whole file — destroying the byte-identity guarantee wholesale. Keeping it internal avoids that; making it public closes a read/write asymmetry, since the library can preserve a sense-level dateModified it parsed but offers no way to set one.
• Naming. "Touched"/"untouched" already means "differs from its parse-time byte snapshot" throughout _writer.py, _scan.py, and docs/en/fidelity.md. A method named touch() would overload that in the one place the distinction matters most.
• Tombstones. Should setting date_deleted count as a modification and bump dateModified? FLEx merge semantics suggest yes. None of the sampled exports contain a tombstone, so there is no observed precedent to copy.
• Blank dateCreated back-fill. The Combine sets Created from Modified when Created is blank but Modified is not. Worth mirroring, or is that data repair beyond this scope?
• From-scratch lexicons. With no _source, every entry is new. Stamping all of them is the obvious reading, but it is a distinct code path from the loaded case and should be stated explicitly. Given 100% dateCreated coverage in real exports, a from-scratch export that omits them is visibly unlike anything FLEx produces — docs/en/guides/build-export.md needs a line either way.
• Interaction with canonicalize. A full re-serialization pass is not a content edit; it should presumably not stamp anything.
• What does "changed" mean across saves? The baseline update in shared groundwork covers entries that were stamped. An unstamped save — stamp=False, or no stamping call — writes new bytes while the baseline still reflects parse time, so a later edit compares against load rather than against the last write. Whether "changed" should mean "since load" or "since the last save" is a decision both routes owe an answer to, and it also affects Add Lexicon.changed_entries() to report entries edited since load #11.

Alternatives considered

• Dirty tracking via __setattr__ and observable containers. Correct at arbitrary granularity and free of serialization cost — this is broadly what the C# ecosystem does. Rejected: it needs __setattr__ on every slotted node, list.append and Multitext.__setitem__ bypass it unless every container is wrapped too, and the reader sets every field on every node during parse, so loading a 35,000-entry lexicon would pay tracking overhead or need a suppression mode. Large blast radius for a granularity nothing consumes.
• An environment variable toggling stamping. Rejected: it makes byte-identical output depend on something outside the code, so the same script produces different files on different machines, and it would break byte-exact corpus tests whenever it happened to be set in a shell or CI environment. The one defensible environment variable here is the inverse — pinning the clock for determinism, tracked in Honor SOURCE_DATE_EPOCH as the clock source for generated timestamps #9.
• A module-level global policy setter. Rejected: process-global mutable state that changes what gets written to disk, so library code and application code contend over it, and save() becomes unreadable at the call site.
• A public per-node stamp method as the primary API. Rejected as the primary surface for the footgun described above; viable only as a deliberate override alongside one of the routes here.