diff --git a/content/docs/(overview)/files.mdx b/content/docs/(overview)/files.mdx index 9e7ae852..d506fefc 100644 --- a/content/docs/(overview)/files.mdx +++ b/content/docs/(overview)/files.mdx @@ -24,7 +24,7 @@ Marfa answers `201` with the blob's hash, media type and size: { "hash": "sha256:d71b77fe25721711f07e19c5f39e768e1356349c08b58260a42ed97eb996a01b", "mime_type": "image/jpeg", "size_bytes": 2481934 } ``` -A file item is an item whose type inherits from `core.file`, such as `core.file.image`. Its `blob_ref` is that hash: +A file item is an item whose type inherits from `core.file`, such as `core.file.image` for a photo or `core.file.webpage` for an HTML page. Its `blob_ref` is that hash: ```http title="Request" POST /items diff --git a/content/docs/build/handle-files.mdx b/content/docs/build/handle-files.mdx index 503ebfb0..fd250c09 100644 --- a/content/docs/build/handle-files.mdx +++ b/content/docs/build/handle-files.mdx @@ -56,7 +56,7 @@ An upload has **no size cap and no total time limit**. Marfa ends one with `408 ## Point an item at the bytes -Create a file item whose **`blob_ref`** is the hash. `core.file` takes any file; use `core.file.image`, `core.file.audio` or `core.file.video` where they fit, or a type of your own that inherits from `core.file`. +Create a file item whose **`blob_ref`** is the hash. `core.file` takes any file; use `core.file.image`, `core.file.audio`, `core.file.video` or `core.file.webpage` (a Web Page, an HTML page held as a file) where they fit, or a type of your own that inherits from `core.file`. `marfa items add` and a [folder](/run/folder-files) choose the subtype from the file's media type: `text/html` and `application/xhtml+xml` give a Web Page, and a type you name yourself always wins. ```bash tab="CLI" tab-group="client" marfa items create --type core.file --prop title=reading-list.txt \ @@ -94,7 +94,7 @@ _ = try await copy.items.create(Draft( properties: ["title": "reading-list.txt", "blob_ref": .string(hash), "mime_type": "text/plain"])) ``` -Marfa sets the item's `size_bytes` from the stored bytes. Soon after, a housekeeping job reads the bytes and adds what it finds, such as a document's `extracted_text` or an image's `width` and `height`, which moves the item's version. +Marfa sets the item's `size_bytes` from the stored bytes. Soon after, a housekeeping job reads the bytes and adds what it finds, such as a document's or a Web Page's `extracted_text` or an image's `width` and `height`, which moves the item's version. `marfa items add reading-list.txt` uploads the bytes and creates the item in one command. To make a file belong to another item, such as a photo attached to a note, link the two with an `attached-to` edge from the file item: `marfa items attach photo.jpg` does all three steps. The file is then an **attachment**. An attachment the item's body shows, with `![](path)` or `![[file]]`, is also an **embed**. @@ -119,7 +119,9 @@ Authorization: Bearer The job runs every 30 seconds by default, so a new file is found by its `title`, `description` and tags first and by its text a little later. Search matches only the text the job kept, which [`MARFA_ENRICHMENT_MAX_TEXT_CHARS`](/run/settings) limits. A file in the bin is never found. A device that holds the file finds it by its text once it has fetched the item with its `extracted_text`, which it does when it next catches up. -Enrichment is on unless the [settings](/run/settings) switch it off, and an image has text only while OCR is on. +The job reads the text of a plain text or Markdown file, a Web Page, a PDF, a Word, Excel or PowerPoint file and their OpenDocument equivalents, and, while OCR is on, an image. For a Web Page (`text/html` or `application/xhtml+xml`, with or without a charset) the text is what a reader sees: markup, scripts, styles and comments are left out, character references such as `&` are decoded, and each paragraph, heading, list item and table row starts a new line. A page that is mostly script, or malformed, yields the text Marfa could recover. Marfa reads at most the first 8 MiB of a page and the first 512 levels of nested elements, and keeps `MARFA_ENRICHMENT_MAX_TEXT_CHARS` of text. A page that passes one of these keeps the text found before it, and the file is stored as usual. A page that is not read within the time allowed per file (`MARFA_ENRICHMENT_ITEM_TIMEOUT_MS`) gets no text, and the job tries it up to `MARFA_ENRICHMENT_MAX_ATTEMPTS` times in all. + +Enrichment is on unless the [settings](/run/settings) switch it off. ## Download the bytes diff --git a/content/docs/run/folder-files.mdx b/content/docs/run/folder-files.mdx index 8eff5d16..101242ce 100644 --- a/content/docs/run/folder-files.mdx +++ b/content/docs/run/folder-files.mdx @@ -30,7 +30,7 @@ The lines between the `---` fences are the frontmatter. The text after them is t A new Markdown file needs no frontmatter. It takes its type and tier from the folder's `defaults` or its query, and its title from `defaults` or, without one, its file name. With `query.tier: all`, a new file without an explicit tier uses `defaults.tier` or, without one, `library`. Once the instance creates the item, Marfa writes the frontmatter and `marfa_id` into the file. -A `.txt` file is all body. Any other file the query admits, such as a photo, becomes a file item with its bytes uploaded as a blob, as [Handle files](/build/handle-files) describes. A file item stays a file item at a `.md`, `.markdown` or `.txt` name, such as after a pull places it there or you rename it: its bytes go to the item as they are, and Marfa doesn't read them as a note. +A `.txt` file is all body. Any other file the query admits, such as a photo, becomes a file item with its bytes uploaded as a blob, as [Handle files](/build/handle-files) describes. The file's extension gives its media type, and the media type gives the type: a photo becomes a `core.file.image`, and an `.html`, `.htm` or `.xhtml` file becomes a `core.file.webpage`, so a page an agent writes into the folder is kept as a page. Audio and video files become `core.file.audio` and `core.file.video`, and any other file becomes a `core.file`. The query must admit the type, and `core.file` in `types` admits every type under it. A file item stays a file item at a `.md`, `.markdown` or `.txt` name, such as after a pull places it there or you rename it: its bytes go to the item as they are, and Marfa doesn't read them as a note. ## Files attached to a note diff --git a/src/generated/catalog/builtin-catalog.json b/src/generated/catalog/builtin-catalog.json index 6f120498..4077cdbc 100644 --- a/src/generated/catalog/builtin-catalog.json +++ b/src/generated/catalog/builtin-catalog.json @@ -748,6 +748,72 @@ "default": "last_writer_wins" } }, + { + "id": "core.file.webpage", + "parent": "core.file", + "label": "Web Page", + "description": "An HTML page held as a file, such as a report an agent wrote. Inherits all core.file fields.", + "version": 0, + "fields": { + "blob_ref": { + "type": "string", + "description": "Reference to the binary content (sha256:)", + "required": true + }, + "mime_type": { + "type": "string", + "description": "MIME type", + "required": true + }, + "title": { + "type": "string", + "description": "Filename or title" + }, + "description": { + "type": "string", + "description": "What the file contains" + }, + "url": { + "type": "url", + "description": "Web address" + }, + "source_url": { + "type": "url", + "description": "Where the file was sourced from" + }, + "author": { + "type": "string", + "description": "Who created the file" + }, + "language": { + "type": "string", + "description": "BCP 47 language code", + "format": "bcp47" + }, + "notes": { + "type": "string", + "description": "Personal annotations" + }, + "extracted_text": { + "type": "string", + "description": "Machine-extracted text content of the referenced blob (server enrichment: document text or image OCR)" + }, + "executable": { + "type": "boolean", + "description": "Whether the file may be run as a program; absent means it may not" + }, + "size_bytes": { + "type": "integer", + "description": "Size in bytes of the referenced blob (set by the server from the bytes the instance holds; absent where whoever named blob_ref never proved it holds them)" + } + }, + "display_hints": { + "title_field": "title" + }, + "merge_policy": { + "default": "last_writer_wins" + } + }, { "id": "core.highlight", "label": "Highlight", diff --git a/src/generated/catalog/core.file.md b/src/generated/catalog/core.file.md index a3f6d626..209cf262 100644 --- a/src/generated/catalog/core.file.md +++ b/src/generated/catalog/core.file.md @@ -59,3 +59,7 @@ Video files. Inherits all core.file fields. | `latitude` | `number` | No | Subject latitude | | `longitude` | `number` | No | Subject longitude | | `altitude` | `number` | No | Altitude in meters | + +### core.file.webpage [#core-file-webpage] + +An HTML page held as a file, such as a report an agent wrote. Inherits all core.file fields. diff --git a/src/generated/catalog/types.md b/src/generated/catalog/types.md index f5e6cfe4..fc368517 100644 --- a/src/generated/catalog/types.md +++ b/src/generated/catalog/types.md @@ -6,7 +6,7 @@ | [`core.collection`](/catalog/collection) | Items of any type that you gather together. Members join it through the in-collection edge, and it can hold any item but another container. | | | [`core.entity`](/catalog/entity) | Any named entity: a person, a company, a band, a team, a charity, a brand, a school or a place. `core.entity.person` and `core.entity.place` narrow it. | [`person`](/catalog/entity#core-entity-person), [`place`](/catalog/entity#core-entity-place) | | [`core.event`](/catalog/event) | Something that happens at a time. | | -| [`core.file`](/catalog/file) | A file or binary reference: the generic fallback for any file that has no more specific type. | [`audio`](/catalog/file#core-file-audio), [`image`](/catalog/file#core-file-image), [`video`](/catalog/file#core-file-video) | +| [`core.file`](/catalog/file) | A file or binary reference: the generic fallback for any file that has no more specific type. | [`audio`](/catalog/file#core-file-audio), [`image`](/catalog/file#core-file-image), [`video`](/catalog/file#core-file-video), [`webpage`](/catalog/file#core-file-webpage) | | [`core.highlight`](/catalog/highlight) | A user's engagement with content: the highlighted passage plus optional annotation. The canonical relationship (what was highlighted) is carried by a references edge; the moment of highlighting is the item's own `occurred_at`. | | | [`core.media`](/catalog/media) | Content produced by someone that the user engages with: a film, podcast, book, article, song, show. | [`album`](/catalog/media#core-media-album), [`article`](/catalog/media#core-media-article), [`book`](/catalog/media#core-media-book), [`episode`](/catalog/media#core-media-episode), [`film`](/catalog/media#core-media-film), [`series`](/catalog/media#core-media-series), [`song`](/catalog/media#core-media-song) | | [`core.message`](/catalog/message) | A light, cross-platform message. Threading is edge-based via in-thread; reply trees via parent-of. Tool-specific richness lives in app namespaces. | | diff --git a/src/generated/contract-index.json b/src/generated/contract-index.json index 9507eb66..5a9aea2d 100644 --- a/src/generated/contract-index.json +++ b/src/generated/contract-index.json @@ -936,7 +936,7 @@ "device/add-file-given": "abeb28065d9d200c", "device/add-file-tags-wait": "2a0b39209bdc48c6", "device/file-mime-default": "eba34739fe90b95b", - "device/file-type-default": "0ae572c77a991ce3", + "device/file-type-default": "b2a85304720ea1a3", "device/file-title-default": "709608204155b1e6", "device/catalog-replaced-catch-up": "0377b6e5cf959ede", "device/catalog-replaced-follow": "90f5be33d116a6a7", @@ -1219,6 +1219,8 @@ "device/index-order": "5af6c4e3a20e0e3a", "device/index-unsearchable": "c11eb9679e644c7b", "device/index-undeclared": "ff822a32be8b729d", + "device/index-file-text": "5f46a42c3f372806", + "device/index-file-text-pending": "c7cb3ea65ca731ab", "device/index-type-change": "7c558f8f055d0670", "device/rank-bm25": "a9f48ebef4474fb2", "device/rank-ties": "357751477aeffd7d", @@ -2536,6 +2538,7 @@ "folders/replaced-binding-unpinned": "636e63266359c693", "folders/embedded-archive-kept": "a3913cc22995ef23", "folders/binary-created": "2084a92fa1220915", + "folders/binary-webpage": "2a53e4720ed619ca", "folders/binary-edited": "84021a258e7007bc", "folders/binary-title-preserved": "eebe4a7ca0612bac", "folders/blob-file-written": "ff3a47e87da653b2", @@ -2951,6 +2954,9 @@ "folders/icloud-ubiquitous-warns": "914ac23dd1b6b948", "folders/cloud-storage-warns": "00f82c6052ba2428", "folders/network-storage-warns": "1b61ba5c8a586f51", + "folders/git-tracked-warns": "1df485efa330dd6d", + "folders/git-tracked-continues": "008c13195b8459fe", + "folders/git-untracked-silent": "f34f611550ec6778", "folders/storage-warning-continues": "6e67f6931e68a418", "folders/storage-warning-once": "ab1c58dbf82528ca", "folders/storage-warning-elsewhere": "65aa359fe450bfd7", @@ -2990,6 +2996,8 @@ "housekeeping/enrichment-interrupted-counted": "dbbd374997f04da0", "housekeeping/enrichment-image-by-type": "d08e8d5febaf1a0b", "housekeeping/enrichment-text-by-type": "89238c57a14f1789", + "housekeeping/enrichment-html-text": "e2948bba2b97c678", + "housekeeping/enrichment-html-bounds": "2068af4533d0d9d5", "housekeeping/enrichment-needs-file-type": "1dbd4b24aff932b0", "housekeeping/image-size-from-header": "35d819009cae64fb", "housekeeping/image-malformed-skipped": "966ba2409ead4fe2", @@ -4838,6 +4846,8 @@ "search-and-filters/index-fields": "fc3c5ea99c49a336", "search-and-filters/index-order": "074148e8ff0798af", "search-and-filters/index-undeclared": "fe5718b8a6fb8c23", + "search-and-filters/search-file-text": "f75fd5f4370c7062", + "search-and-filters/search-file-text-pending": "9e9cd55973be9607", "search-and-filters/index-not-string": "33d7f1a786109e49", "search-and-filters/index-unregistered": "2dd0a003c851c5ca", "search-and-filters/index-type-change": "af5efe6ffcde8a13", @@ -5294,7 +5304,7 @@ "types/source-filter": "4cb80605ef5eb54c", "types/source-filter-missing-source": "ffaa4dc35611173f", "types/source-filter-by-id": "6617d24512e658f0", - "types/shipped": "d329bfef050dc7d6", + "types/shipped": "1f92bfb8fd3017a3", "types/shipped-system": "aa1b148e069da6e4", "types/message": "b7749ce97bb970c1", "types/collection": "9b963e89e951391b",