Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 37 additions & 23 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,12 @@ model. The reference RepViT-M0.9 artifact has 4.76 million parameters and is
19.45 MiB in FP32.

> [!IMPORTANT]
> This repository contains research and training code. It does not distribute a
> pretrained checkpoint. The CPC and GAICD archives reviewed during development
> do not state explicit image or annotation licenses; confirm training and model
> redistribution rights before publishing a derived checkpoint.
> Pretrained ONNX and PyTorch checkpoints are on
> [Hugging Face](https://huggingface.co/appwrite/focalnet) and the
> [2026-09-14-rc1 GitHub release](https://github.com/appwrite/focalnet/releases/tag/2026-09-14-rc1).
> The CPC and GAICD archives reviewed during development do not state explicit
> image or annotation licenses; the published weights do not grant rights to
> those datasets.

## How it works

Expand Down Expand Up @@ -93,9 +95,15 @@ ONNX Runtime, so a runtime-only environment can use `pip install .`.

## Inference

Predict an importance map and an aspect-aware crop:
Download the published checkpoints, then predict an importance map and an
aspect-aware crop:

```sh
mkdir -p artifacts
curl -fsSL \
"https://huggingface.co/appwrite/focalnet/resolve/main/focalnet.onnx" \
-o artifacts/focalnet.onnx
# same file: https://github.com/appwrite/focalnet/releases/download/2026-09-14-rc1/focalnet.onnx
uv run focalnet predict artifacts/focalnet.onnx photo.jpg \
--aspect-ratio 16:9 \
--heatmap artifacts/photo-importance.npy
Expand All @@ -120,26 +128,27 @@ the same image; it is not a calibrated confidence value.
### Browser demo

A static crop lab under `web/` runs the same format-v2 ONNX in the browser with
ONNX Runtime Web. Photos stay on-device. This repository does not publish a
checkpoint; the 19.45 MiB weights stay gitignored. Export a model, then point
the lab at that file. Bun is the package manager, bundler, test runner, and
local static server:
ONNX Runtime Web. Photos stay on-device. The 19.45 MiB weights stay gitignored;
`bun run fetch-model` downloads `focalnet-human.onnx` from
[Hugging Face](https://huggingface.co/appwrite/focalnet) and verifies the
published SHA-256. Bun is the package manager, bundler, test runner, and local
static server:

```sh
cd web
bun install
FOCALNET_ONNX=../artifacts/focalnet-human.onnx bun run fetch-model
bun run fetch-model
bun test
bun run dev
```

`FOCALNET_ALLOW_DUMMY=1 bun run fetch-model` writes a tiny placeholder for UI
layout only. Set `FOCALNET_ONNX_SHA256` to verify a known file, or
`FOCALNET_ONNX_URL` to fetch a checkpoint you already host. Source order is
existing `web/public/focalnet-human.onnx`, then `FOCALNET_ONNX`, then
`FOCALNET_ONNX_URL`; dummy mode still runs if a configured URL fails. The demo
compares the human-ranked crop with the importance-retention crop from the same
map.
Override the source with `FOCALNET_ONNX` (local file) or `FOCALNET_ONNX_URL`.
`FOCALNET_ONNX_SHA256` defaults to the published ranking-model hash, including
when the variable is set but empty. Source order is existing
`web/public/focalnet-human.onnx`, then `FOCALNET_ONNX`, then the Hub URL.
`FOCALNET_ALLOW_DUMMY=1` writes a tiny placeholder for UI layout only if a
configured source is missing or fails. The demo compares the human-ranked crop
with the importance-retention crop from the same map.

## Training

Expand Down Expand Up @@ -300,8 +309,9 @@ crop optimization, and ONNX parity. They use synthetic inputs and do not require
teacher models or pretrained weights.

`modal_app.py` and the scripts in `scripts/` provide reference workflows for
remote data preparation and GPU training. Review their resource names, storage
paths, and expected dataset hashes before running them in another environment.
remote data preparation, GPU training, and Hugging Face publishing. Review
their resource names, storage paths, and expected dataset hashes before running
them in another environment.

## Limitations

Expand All @@ -311,12 +321,16 @@ paths, and expected dataset hashes before running them in another environment.
- The GAICD test split was inspected during several development iterations, so
it should be treated as an engineering benchmark rather than an untouched
final test.
- Dataset rights must be resolved before distributing a trained checkpoint.
- CPC and GAICD archives do not state explicit licenses; published checkpoints
do not grant rights to those datasets.
- Model latency depends on the ONNX Runtime build, processor, thread count, and
surrounding service workload.

## License

FocalNet's source code is available under the [MIT License](LICENSE). This
license does not grant rights to third-party datasets, teacher model weights, or
derived model artifacts; review their respective terms before redistribution.
FocalNet's source code and the published checkpoints on
[Hugging Face](https://huggingface.co/appwrite/focalnet) and the
[GitHub release](https://github.com/appwrite/focalnet/releases/tag/2026-09-14-rc1)
are available under the [MIT License](LICENSE). This license does not grant
rights to third-party datasets or teacher model weights; review their
respective terms before redistribution.
Binary file added docs/hub-images/comparison-faces.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/hub-images/comparison-retriever.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
151 changes: 151 additions & 0 deletions docs/huggingface-model-card.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
---
license: mit
library_name: onnxruntime
pipeline_tag: image-to-image
tags:
- onnx
- computer-vision
- image-cropping
- saliency
base_model: timm/repvit_m0_9
---

# FocalNet

Appwrite FocalNet is a compact vision model for content-aware image cropping. It
predicts a 64×64 importance map, then ranks crop candidates with a small
composition head trained on human preferences. Training code lives in
[appwrite/focalnet](https://github.com/appwrite/focalnet).

The reference RepViT-M0.9 ranker has 4.76 million parameters. The published
`focalnet-human.onnx` artifact is 19.45 MiB in FP32.

## Examples

These photographs come from [Autogravity](https://github.com/appwrite/autogravity).
A centered 1:1 crop of the park scene keeps grass. FocalNet keeps the dog.

![Center 1:1 crop of empty grass versus FocalNet 1:1 crop of the golden retriever](images/comparison-retriever.png)

On a two-person portrait, FocalNet holds the nearer subject instead of splitting
the frame.

![Center 1:1 crop versus FocalNet 1:1 crop of a two-person portrait](images/comparison-faces.png)

## Files

| File | Role | SHA-256 |
| --- | --- | --- |
| `focalnet-human.onnx` | Importance map + crop ranking (format v2) | `59164c601c98cea3f62b25166710831dac63e1a872fc64767c65316ad5385439` |
| `focalnet.onnx` | Importance map only (format v1) | `87becceb269a2973c359df789783be49a9f840f47170a015d3776d7c4145a2ce` |
| `focalnet-human.pt` | PyTorch ranking checkpoint | `82c4d695310c220e5f3bcfbf5e64d315a5e826cfd77e945e61e032460014480a` |
| `focalnet.pt` | PyTorch importance checkpoint | `d1942f0652f8ea85f75ffc0cb1bf40d7e70b38e7ad102ab0e6df5c2e07ce52cf` |

JSON sidecars next to the ONNX files record the preprocessing contract, parameter
counts, and export verification.

Use `focalnet-human.onnx` for inference. The PyTorch files are for continued
training, not for the ONNX Runtime path.

## Usage

Download the ranking model and crop a photo:

```sh
mkdir -p artifacts
curl -fsSL \
"https://huggingface.co/appwrite/focalnet/resolve/main/focalnet-human.onnx" \
-o artifacts/focalnet-human.onnx
uv run focalnet predict-human artifacts/focalnet-human.onnx photo.jpg \
--aspect-ratio 16:9
```

From Python:

```python
from pathlib import Path
from urllib.request import urlretrieve

from focalnet.human_runtime import HumanCropPredictor

url = "https://huggingface.co/appwrite/focalnet/resolve/main/focalnet-human.onnx"
path = Path("artifacts/focalnet-human.onnx")
path.parent.mkdir(parents=True, exist_ok=True)
urlretrieve(url, path)
crop = HumanCropPredictor(path).predict("photo.jpg", aspect_ratio=16 / 9)
```

Importance-only inference:

```sh
mkdir -p artifacts
curl -fsSL \
"https://huggingface.co/appwrite/focalnet/resolve/main/focalnet.onnx" \
-o artifacts/focalnet.onnx
uv run focalnet predict artifacts/focalnet.onnx photo.jpg --aspect-ratio 16:9
```

`predict-human` generates up to 125 crops across five positions and five zoom
levels, scores a padded maximum of 128 candidates in one ONNX call, and selects
the highest preference score among crops whose retained importance is within
0.05 of the best candidate. The returned human score is a relative logit.
Compare it only among crops for the same image.

## ONNX contract

| Field | Contract |
| --- | --- |
| Image input | `image`, float32 `[1,3,256,256]`, RGB NCHW |
| Normalization | `(pixel/255 - [0.485,0.456,0.406]) / [0.229,0.224,0.225]` |
| Resize | Apply EXIF orientation, then aspect-fit with bilinear interpolation |
| Alpha and padding | Composite alpha on black; normalized padding values are zero |
| Importance output | `importance`, float32 `[1,1,64,64]`, sigmoid applied |
| Ranking inputs | `boxes` `[1,128,4]` and letterbox `content` `[1,4]` |
| Ranking output | `crop_scores`, float32 `[1,128]` |

## Results

The importance model was trained on 500,000 teacher-labeled Open Images V7
samples (RepViT-M0.9 encoder, 48-channel decoder, 10 epochs, best epoch 7).
Teacher agreement on the group-disjoint 10,000-image validation split:

| Metric | Result |
| --- | ---: |
| Map MAE | 0.09975 |
| Normalized centroid error | 0.06430 |
| 1:1 teacher importance retained | 91.09% |
| 16:9 teacher importance retained | 90.86% |
| 4:5 teacher importance retained | 85.56% |

The ranking head (7,069 parameters) was pretrained on CPC and fine-tuned on
GAICD. The importance network stays frozen. On GAICD's 500-image test split:

| Metric | GAICD only | CPC → GAICD |
| --- | ---: | ---: |
| Spearman correlation | 0.75696 | **0.76450** |
| Pearson correlation | 0.78211 | **0.79192** |
| Top-5 accuracy | 42.48% | **43.70%** |
| Top-10 accuracy | 64.45% | **64.54%** |
| Pairwise accuracy | 86.00% | **86.44%** |

The GAICD test split was inspected during several development iterations. Treat
these figures as engineering benchmarks, not an untouched final test. INT8
post-training quantization did not pass the crop-quality gate; FP32 is the
published artifact.

## Training data

- Open Images V7 (teacher-labeled importance maps from U²-Net + YuNet)
- [Comparative Photo Composition (CPC)](https://www3.cs.stonybrook.edu/~cvl/projects/wei2018goods/VPN_CVPR2018s.html)
- [GAICD](https://github.com/HuiZeng/Grid-Anchor-based-Image-Cropping-Pytorch)

The reviewed CPC and GAICD archives do not state explicit image or annotation
licenses. This checkpoint is published under MIT for the model files; that
license does not grant rights to the third-party datasets or teacher weights
used during training.

## License

Model files are available under the [MIT License](https://github.com/appwrite/focalnet/blob/main/LICENSE),
the same license as the training code. Review dataset and teacher-model terms
before redistribution of derived artifacts.
6 changes: 5 additions & 1 deletion docs/results-500k.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,4 +78,8 @@ focalnet.onnx 87becceb269a2973c359df789783be49a9f840f47170a015d3776d7c4145a2ce
best.pt d1942f0652f8ea85f75ffc0cb1bf40d7e70b38e7ad102ab0e6df5c2e07ce52cf
```

The files identified by these hashes are not distributed in this repository.
These files are published at
[huggingface.co/appwrite/focalnet](https://huggingface.co/appwrite/focalnet)
and the
[2026-09-14-rc1 GitHub release](https://github.com/appwrite/focalnet/releases/tag/2026-09-14-rc1)
as `focalnet.onnx` and `focalnet.pt`. They stay gitignored in this repository.
12 changes: 8 additions & 4 deletions docs/results-cpc-gaic-v2.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,11 @@ CPCDataset.tar.gz dfa4ec73c9d9b4b525a8f79aee5670fac4797bad2eb1bd0e1f26f051ac
GAIC.zip b895a3f9e03c8f70c37194370441dc2f17bb60e0ab437447ee240b003cd6550b
```

The files identified by these hashes are not distributed in this repository.
Neither reviewed dataset archive states an explicit license for its images or
annotations. Confirm training and model redistribution rights before publishing
a derived checkpoint.
The ranking checkpoint identified by these hashes is published at
[huggingface.co/appwrite/focalnet](https://huggingface.co/appwrite/focalnet)
and the
[2026-09-14-rc1 GitHub release](https://github.com/appwrite/focalnet/releases/tag/2026-09-14-rc1)
as `focalnet-human.onnx` and `focalnet-human.pt`. Dataset archives stay off the
Hub and the release. Neither reviewed CPC nor GAICD archive states an explicit
license for its images or annotations; the published weights do not grant rights
to those datasets.
Loading
Loading