CELL-FM is a virtual microscopy model. From a protein's amino-acid sequence and a cellular context, it generates the fluorescence image that protein would give; from an image, it generates sequences consistent with it.
Code for Virtual experiments bridge sequence and microscopy with generative models (Dihan Zheng, Kibeom Hong, Bo Huang).
| Paper | bioRxiv, doi:10.64898/2026.09.13.751243 |
| Weights | huggingface.co/BoHuangLab/CELL-FM |
Predictive models usually compress an experiment into task-specific labels, such as a localization class, before learning. CELL-FM instead learns to generate the experimental readout itself: the image of a protein in a cell. Phenotypes are then measured on generated images with the same analysis used on real ones — a nucleus-to-cytoplasm ratio, a condensate classifier, an image embedding. One model serves many downstream questions without retraining, and its results stay directly comparable to the real assay.
CELL-FM models images as a continuous distribution and protein sequences as a discrete one, with a shared encoder and a separate decoder for each output.
Tokens. The protein channel and the cell-context channels (the nucleus, plus microtubules and ER for HPA) are encoded by a VAE and cut into patch tokens. The sequence is embedded with the ESM-C 600M protein language model.
Unified encoder. A single transformer attends over image and sequence tokens together and fuses them into one shared representation.
Three heads read from that representation:
| Head | Objective | Role |
|---|---|---|
| Image generator (MMDiT) | Flow matching in VAE latent space | sequence + cell → protein image |
| Sequence generator | Discrete diffusion (order-agnostic autoregressive) | image + cell → sequence |
| Image reconstruction | Denoising masked autoencoder | auxiliary, stage 1 only |
Training. Stage 1 optimizes all three losses with half of the image patches masked, to learn a robust shared representation. Stage 2 turns masking off and fine-tunes the two generative losses for fidelity.
Sampling. An image is generated by integrating the learned flow from Gaussian noise and decoding the latent with the VAE. A sequence starts fully masked and is filled in one position at a time, in random order.
Expression level. For CondenSeq, protein expression level is an extra continuous conditioning token, so the same sequence can be imaged at any concentration. This is what makes in silico titration possible.
| Dataset | What it images | Scale | Used for |
|---|---|---|---|
| Human Protein Atlas | Immunofluorescence in fixed cells; nucleus, microtubule and ER context | 12,894 proteins, 1.1 M single-cell images | Pretraining, localization, signal design |
| OpenCell | Endogenously tagged proteins in live HEK293T; nucleus context | 1,311 proteins, ~96,000 single-cell images | Fine-tuning from HPA for virtual staining |
| CondenSeq | Intrinsically disordered peptides in U2OS nuclei, pooled screen; nucleus context | 14,598 peptides of 66 aa, ~8 M single-cell images | Condensate formation across expression levels |
Each application is a Colab notebook in notebooks/. Choose a GPU runtime (a Colab
T4 works); the notebook installs its own dependencies and downloads the weights on first run.
| Notebook | What it does | |
|---|---|---|
| Condensate titration | Images a 66-aa intrinsically disordered peptide along a ladder of expression levels, calls each image condensed or diffuse with a ViT classifier, and turns the calls into a titration curve summarised by AUC (propensity to condense) and AAC (re-entrant dissolution at high concentration). |  |
| NLS screening | Slides a window along a protein sequence, virtually stains a cell with each fragment, and scores the nucleus-to-cytoplasm signal ratio per window to locate nuclear localization signals. | |
| Localization signal generation | Runs the model from image to sequence: shown a cell whose protein sits in the nucleus or in the cytoplasm, it designs NLS or NES peptides, then compares their residue composition with the human proteome. | |
| OpenCell virtual staining | Paints any OpenCell protein onto the same nucleus from its sequence, then embeds the generated cells with a ViT to map where proteins localize. | |
Notes on the notebooks' dependency pins are in notebooks/README.md.
| Path | Contents |
|---|---|
cell_fm/ |
Models (CELL-FM, VAE, ViT classifiers), datasets, training pipeline and tasks |
scripts/ |
Launch scripts for training and evaluation |
notebooks/ |
Application notebooks |