LightOnOCR-2-1B fails to load in Docker deployment: missing LightOnOcrForConditionalGeneration in bundled transformers

[tatankam]

Description

I'm running SIE using the official Docker deployment with the image image: ghcr.io/superlinked/sie-server:latest-cuda12-default

When trying to use the model lightonai/LightOnOCR-2-1B, model loading fails with the following error:

ModelLoadFailedError: Model 'lightonai/LightOnOCR-2-1B' failed to load (DEPENDENCY, attempts=1):
ImportError: cannot import name 'LightOnOcrForConditionalGeneration' from 'transformers'

Investigation

Inside the running container:

import transformers
print(transformers.__version__)

returns:

4.57.6

and:

from transformers import LightOnOcrForConditionalGeneration

fails with:

ImportError: cannot import name 'LightOnOcrForConditionalGeneration'

The bundled Transformers package used by SIE is:

/app/bundle-libs/transformers

I also verified that there is no lighton_ocr model implementation present under:

/app/bundle-libs/transformers/models

Expected behavior

Since lightonai/LightOnOCR-2-1B appears in the available model catalog, I would expect it to either:

1. Load successfully, or
2. Be excluded/marked as unsupported when the required backend is not available.

Question

Is lightonai/LightOnOCR-2-1B currently supported by the Docker release of SIE?

If not, is support planned in a future image/version, or is there a recommended workaround?

Thanks a lot
Mario

[jturner421]

I am seeing the same thing. I attempted to use this server tag: ghcr.io/superlinked/sie-server:v0.6.19-cuda12-transformers5. The Lighton model loads and works, but it very slow.

[svonava]

Interesting, checking!

[svonava]

@jturner421 what hardware are you running on and what perf are you seeing? we will do some more optimization here

[svonava]

Update: we have merged a broader set of generative OCR performance improvements, and they will be included in the next release.

On our canonical NVIDIA L4 benchmark (500 requests at concurrency 4), LightOnOCR-2-1B improved from 125.7s to 20.8s p50 latency and from 450.7s to 44.4s p99 latency, while page throughput increased from 0.024 to 0.164 pages/s. That is approximately 6.0× lower median latency, 10.1× lower p99 latency, and 6.8× higher page throughput.

The same optimized serving path now also covers PaddleOCR-VL-1.5 and GLM-OCR. Their new canonical L4 results are:

• PaddleOCR-VL-1.5: 12.4s p50, 26.0s p99, 0.275 pages/s
• GLM-OCR: 16.8s p50, 67.2s p99, 0.199 pages/s

We also reran the full 1,403-page quality benchmark for all three models; the quality gates passed with results comparable to the previous serving path.

[jturner421]

@svonava Sorry it took so long to get back to you. I am running this on hardware that you have not benchmarked. I am using an Ubuntu 24.04 LTS server with an Nvidia RTX 5090. Although this is technically a consumer level card, it does have 32GB of VRAM. Using this setup, I am able to serve inference, OCR, and embeddings. I've attached a screenshot of Nvidia-smi for reference.

I don't expect you to be able to check against this. But I do think that the updates that you've made that will be in the next release will be helpful. Have those already been checked into source? I ask because in order to run the project locally, I had to make some small changes to run the gateway locally to serve inference since I don't have Kubernetes set up. The project as it currently stands does not allow inference to be served unless it's on a local Mac or using Kubernetes.

[svonava]

Very cool - cc @huronat any idea when the OCR speedup gets released?

@jturner421 I'm jealous of the 5090, sitting here with 4090 locally still :-) Very curious what numbers you'll be getting once the next release hits.

[huronat]

@jturner421 hey, thanks for flagging, the OCR speedup optimized image has been rolled out with the v0.6.20 release and is already available as ghcr.io/superlinked/sie-server:v0.6.20-cuda12-sglang.
By the way, we're also carving the latest public release for v0.6.21, as well, soon. We'd love to hear your numbers on an RTX 5090/SM120 path.

[jturner421]

@huronat @svonava Are your OCR tests, metrics, and test documents publically available? I did not see them in the repo. I'd be happy to run a set of benchmarks against my config for you.

[huronat]

@jturner421 hey, thanks for running benchmarks. Currently we don't publish it onto the public OSS repo, but we do have measurement benchmarks on the SIE webpage, please find e.g.: https://superlinked.com/models/lightonai-lightonocr-2-1b