Skip to content

docs(catalog): add the Web Page file type #669

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aicayzer merged 5 commits into from
Oct 27, 2026
Merged

docs(catalog): add the Web Page file type #669

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9d166dc
docs(catalog): add the Web Page file type and say how HTML files are …
aicayzer Oct 27, 2026
a13819b
docs(folders): name the audio and video subtypes beside the web page
aicayzer Oct 27, 2026
da69288
docs(files): say what a web page limit keeps
aicayzer Oct 27, 2026
24847b1
docs(files): sync the web page bounds rule and count attempts in all
aicayzer Oct 27, 2026
51efb1a
chore(sync): take the contract index from marfa main
aicayzer Oct 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion content/docs/(overview)/files.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Marfa answers `201` with the blob's hash, media type and size:
{ "hash": "sha256:d71b77fe25721711f07e19c5f39e768e1356349c08b58260a42ed97eb996a01b", "mime_type": "image/jpeg", "size_bytes": 2481934 }
```

A file item is an item whose type inherits from `core.file`, such as `core.file.image`. Its `blob_ref` is that hash:
A file item is an item whose type inherits from `core.file`, such as `core.file.image` for a photo or `core.file.webpage` for an HTML page. Its `blob_ref` is that hash:

```http title="Request"
POST /items
Expand Down
8 changes: 5 additions & 3 deletions content/docs/build/handle-files.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ An upload has **no size cap and no total time limit**. Marfa ends one with `408

## Point an item at the bytes

Create a file item whose **`blob_ref`** is the hash. `core.file` takes any file; use `core.file.image`, `core.file.audio` or `core.file.video` where they fit, or a type of your own that inherits from `core.file`.
Create a file item whose **`blob_ref`** is the hash. `core.file` takes any file; use `core.file.image`, `core.file.audio`, `core.file.video` or `core.file.webpage` (a Web Page, an HTML page held as a file) where they fit, or a type of your own that inherits from `core.file`. `marfa items add` and a [folder](/run/folder-files) choose the subtype from the file's media type: `text/html` and `application/xhtml+xml` give a Web Page, and a type you name yourself always wins.

```bash tab="CLI" tab-group="client"
marfa items create --type core.file --prop title=reading-list.txt \
Expand Down Expand Up @@ -94,7 +94,7 @@ _ = try await copy.items.create(Draft(
properties: ["title": "reading-list.txt", "blob_ref": .string(hash), "mime_type": "text/plain"]))
```

Marfa sets the item's `size_bytes` from the stored bytes. Soon after, a housekeeping job reads the bytes and adds what it finds, such as a document's `extracted_text` or an image's `width` and `height`, which moves the item's version.
Marfa sets the item's `size_bytes` from the stored bytes. Soon after, a housekeeping job reads the bytes and adds what it finds, such as a document's or a Web Page's `extracted_text` or an image's `width` and `height`, which moves the item's version.

`marfa items add reading-list.txt` uploads the bytes and creates the item in one command. To make a file belong to another item, such as a photo attached to a note, link the two with an `attached-to` edge from the file item: `marfa items attach <id> photo.jpg` does all three steps. The file is then an **attachment**. An attachment the item's body shows, with `![](path)` or `![[file]]`, is also an **embed**.

Expand All @@ -119,7 +119,9 @@ Authorization: Bearer <key>

The job runs every 30 seconds by default, so a new file is found by its `title`, `description` and tags first and by its text a little later. Search matches only the text the job kept, which [`MARFA_ENRICHMENT_MAX_TEXT_CHARS`](/run/settings) limits. A file in the bin is never found. A device that holds the file finds it by its text once it has fetched the item with its `extracted_text`, which it does when it next catches up.

Enrichment is on unless the [settings](/run/settings) switch it off, and an image has text only while OCR is on.
The job reads the text of a plain text or Markdown file, a Web Page, a PDF, a Word, Excel or PowerPoint file and their OpenDocument equivalents, and, while OCR is on, an image. For a Web Page (`text/html` or `application/xhtml+xml`, with or without a charset) the text is what a reader sees: markup, scripts, styles and comments are left out, character references such as `&amp;` are decoded, and each paragraph, heading, list item and table row starts a new line. A page that is mostly script, or malformed, yields the text Marfa could recover. Marfa reads at most the first 8 MiB of a page and the first 512 levels of nested elements, and keeps `MARFA_ENRICHMENT_MAX_TEXT_CHARS` of text. A page that passes one of these keeps the text found before it, and the file is stored as usual. A page that is not read within the time allowed per file (`MARFA_ENRICHMENT_ITEM_TIMEOUT_MS`) gets no text, and the job tries it up to `MARFA_ENRICHMENT_MAX_ATTEMPTS` times in all.

Enrichment is on unless the [settings](/run/settings) switch it off.

## Download the bytes

Expand Down
2 changes: 1 addition & 1 deletion content/docs/run/folder-files.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ The lines between the `---` fences are the frontmatter. The text after them is t

A new Markdown file needs no frontmatter. It takes its type and tier from the folder's `defaults` or its query, and its title from `defaults` or, without one, its file name. With `query.tier: all`, a new file without an explicit tier uses `defaults.tier` or, without one, `library`. Once the instance creates the item, Marfa writes the frontmatter and `marfa_id` into the file.

A `.txt` file is all body. Any other file the query admits, such as a photo, becomes a file item with its bytes uploaded as a blob, as [Handle files](/build/handle-files) describes. A file item stays a file item at a `.md`, `.markdown` or `.txt` name, such as after a pull places it there or you rename it: its bytes go to the item as they are, and Marfa doesn't read them as a note.
A `.txt` file is all body. Any other file the query admits, such as a photo, becomes a file item with its bytes uploaded as a blob, as [Handle files](/build/handle-files) describes. The file's extension gives its media type, and the media type gives the type: a photo becomes a `core.file.image`, and an `.html`, `.htm` or `.xhtml` file becomes a `core.file.webpage`, so a page an agent writes into the folder is kept as a page. Audio and video files become `core.file.audio` and `core.file.video`, and any other file becomes a `core.file`. The query must admit the type, and `core.file` in `types` admits every type under it. A file item stays a file item at a `.md`, `.markdown` or `.txt` name, such as after a pull places it there or you rename it: its bytes go to the item as they are, and Marfa doesn't read them as a note.

## Files attached to a note

Expand Down
66 changes: 66 additions & 0 deletions src/generated/catalog/builtin-catalog.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -748,6 +748,72 @@
"default": "last_writer_wins"
}
},
{
"id": "core.file.webpage",
"parent": "core.file",
"label": "Web Page",
"description": "An HTML page held as a file, such as a report an agent wrote. Inherits all core.file fields.",
"version": 0,
"fields": {
"blob_ref": {
"type": "string",
"description": "Reference to the binary content (sha256:<hex>)",
"required": true
},
"mime_type": {
"type": "string",
"description": "MIME type",
"required": true
},
"title": {
"type": "string",
"description": "Filename or title"
},
"description": {
"type": "string",
"description": "What the file contains"
},
"url": {
"type": "url",
"description": "Web address"
},
"source_url": {
"type": "url",
"description": "Where the file was sourced from"
},
"author": {
"type": "string",
"description": "Who created the file"
},
"language": {
"type": "string",
"description": "BCP 47 language code",
"format": "bcp47"
},
"notes": {
"type": "string",
"description": "Personal annotations"
},
"extracted_text": {
"type": "string",
"description": "Machine-extracted text content of the referenced blob (server enrichment: document text or image OCR)"
},
"executable": {
"type": "boolean",
"description": "Whether the file may be run as a program; absent means it may not"
},
"size_bytes": {
"type": "integer",
"description": "Size in bytes of the referenced blob (set by the server from the bytes the instance holds; absent where whoever named blob_ref never proved it holds them)"
}
},
"display_hints": {
"title_field": "title"
},
"merge_policy": {
"default": "last_writer_wins"
}
},
{
"id": "core.highlight",
"label": "Highlight",
Expand Down
4 changes: 4 additions & 0 deletions src/generated/catalog/core.file.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,3 +59,7 @@ Video files. Inherits all core.file fields.
| `latitude` | `number` | No | Subject latitude |
| `longitude` | `number` | No | Subject longitude |
| `altitude` | `number` | No | Altitude in meters |

### core.file.webpage [#core-file-webpage]

An HTML page held as a file, such as a report an agent wrote. Inherits all core.file fields.
2 changes: 1 addition & 1 deletion src/generated/catalog/types.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
| [`core.collection`](/catalog/collection) | Items of any type that you gather together. Members join it through the in-collection edge, and it can hold any item but another container. | |
| [`core.entity`](/catalog/entity) | Any named entity: a person, a company, a band, a team, a charity, a brand, a school or a place. `core.entity.person` and `core.entity.place` narrow it. | [`person`](/catalog/entity#core-entity-person), [`place`](/catalog/entity#core-entity-place) |
| [`core.event`](/catalog/event) | Something that happens at a time. | |
| [`core.file`](/catalog/file) | A file or binary reference: the generic fallback for any file that has no more specific type. | [`audio`](/catalog/file#core-file-audio), [`image`](/catalog/file#core-file-image), [`video`](/catalog/file#core-file-video) |
| [`core.file`](/catalog/file) | A file or binary reference: the generic fallback for any file that has no more specific type. | [`audio`](/catalog/file#core-file-audio), [`image`](/catalog/file#core-file-image), [`video`](/catalog/file#core-file-video), [`webpage`](/catalog/file#core-file-webpage) |
| [`core.highlight`](/catalog/highlight) | A user's engagement with content: the highlighted passage plus optional annotation. The canonical relationship (what was highlighted) is carried by a references edge; the moment of highlighting is the item's own `occurred_at`. | |
| [`core.media`](/catalog/media) | Content produced by someone that the user engages with: a film, podcast, book, article, song, show. | [`album`](/catalog/media#core-media-album), [`article`](/catalog/media#core-media-article), [`book`](/catalog/media#core-media-book), [`episode`](/catalog/media#core-media-episode), [`film`](/catalog/media#core-media-film), [`series`](/catalog/media#core-media-series), [`song`](/catalog/media#core-media-song) |
| [`core.message`](/catalog/message) | A light, cross-platform message. Threading is edge-based via in-thread; reply trees via parent-of. Tool-specific richness lives in app namespaces. | |
Expand Down
14 changes: 12 additions & 2 deletions src/generated/contract-index.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -936,7 +936,7 @@
"device/add-file-given": "abeb28065d9d200c",
"device/add-file-tags-wait": "2a0b39209bdc48c6",
"device/file-mime-default": "eba34739fe90b95b",
"device/file-type-default": "0ae572c77a991ce3",
"device/file-type-default": "b2a85304720ea1a3",
"device/file-title-default": "709608204155b1e6",
"device/catalog-replaced-catch-up": "0377b6e5cf959ede",
"device/catalog-replaced-follow": "90f5be33d116a6a7",
Expand Down Expand Up @@ -1219,6 +1219,8 @@
"device/index-order": "5af6c4e3a20e0e3a",
"device/index-unsearchable": "c11eb9679e644c7b",
"device/index-undeclared": "ff822a32be8b729d",
"device/index-file-text": "5f46a42c3f372806",
"device/index-file-text-pending": "c7cb3ea65ca731ab",
"device/index-type-change": "7c558f8f055d0670",
"device/rank-bm25": "a9f48ebef4474fb2",
"device/rank-ties": "357751477aeffd7d",
Expand Down Expand Up @@ -2536,6 +2538,7 @@
"folders/replaced-binding-unpinned": "636e63266359c693",
"folders/embedded-archive-kept": "a3913cc22995ef23",
"folders/binary-created": "2084a92fa1220915",
"folders/binary-webpage": "2a53e4720ed619ca",
"folders/binary-edited": "84021a258e7007bc",
"folders/binary-title-preserved": "eebe4a7ca0612bac",
"folders/blob-file-written": "ff3a47e87da653b2",
Expand Down Expand Up @@ -2951,6 +2954,9 @@
"folders/icloud-ubiquitous-warns": "914ac23dd1b6b948",
"folders/cloud-storage-warns": "00f82c6052ba2428",
"folders/network-storage-warns": "1b61ba5c8a586f51",
"folders/git-tracked-warns": "1df485efa330dd6d",
"folders/git-tracked-continues": "008c13195b8459fe",
"folders/git-untracked-silent": "f34f611550ec6778",
"folders/storage-warning-continues": "6e67f6931e68a418",
"folders/storage-warning-once": "ab1c58dbf82528ca",
"folders/storage-warning-elsewhere": "65aa359fe450bfd7",
Expand Down Expand Up @@ -2990,6 +2996,8 @@
"housekeeping/enrichment-interrupted-counted": "dbbd374997f04da0",
"housekeeping/enrichment-image-by-type": "d08e8d5febaf1a0b",
"housekeeping/enrichment-text-by-type": "89238c57a14f1789",
"housekeeping/enrichment-html-text": "e2948bba2b97c678",
"housekeeping/enrichment-html-bounds": "2068af4533d0d9d5",
"housekeeping/enrichment-needs-file-type": "1dbd4b24aff932b0",
"housekeeping/image-size-from-header": "35d819009cae64fb",
"housekeeping/image-malformed-skipped": "966ba2409ead4fe2",
Expand Down Expand Up @@ -4838,6 +4846,8 @@
"search-and-filters/index-fields": "fc3c5ea99c49a336",
"search-and-filters/index-order": "074148e8ff0798af",
"search-and-filters/index-undeclared": "fe5718b8a6fb8c23",
"search-and-filters/search-file-text": "f75fd5f4370c7062",
"search-and-filters/search-file-text-pending": "9e9cd55973be9607",
"search-and-filters/index-not-string": "33d7f1a786109e49",
"search-and-filters/index-unregistered": "2dd0a003c851c5ca",
"search-and-filters/index-type-change": "af5efe6ffcde8a13",
Expand Down Expand Up @@ -5294,7 +5304,7 @@
"types/source-filter": "4cb80605ef5eb54c",
"types/source-filter-missing-source": "ffaa4dc35611173f",
"types/source-filter-by-id": "6617d24512e658f0",
"types/shipped": "d329bfef050dc7d6",
"types/shipped": "1f92bfb8fd3017a3",
"types/shipped-system": "aa1b148e069da6e4",
"types/message": "b7749ce97bb970c1",
"types/collection": "9b963e89e951391b",
Expand Down
Loading