Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,22 @@ build/
*.egg-info/

.env
.cache/
.pytest_cache/
.ruff_cache/
.idea/
.vscode/
*.log
tmp/
.DS_Store

weights/
wheels_*/
apps/gradio_cache/
apps/sessions/
thirdparty/infinigen/

Dockerfile
.dockerignore

outputs/
outputs/
60 changes: 60 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
name: CI

on:
pull_request:
push:
branches: [master]

jobs:
release-smoke:
runs-on: ubuntu-22.04
steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.10"

- name: Install validation tools
run: |
python -m pip install --upgrade pip
python -m pip install build ruff

- name: Lint Python
run: ruff check embodied_gen apps tests

- name: Validate shell scripts
run: bash -n install.sh install/*.sh

- name: Build wheel
run: python -m build --wheel --outdir dist

- name: Validate wheel contents
run: |
python - <<'PY'
from pathlib import Path
from zipfile import ZipFile

wheels = list(Path("dist").glob("*.whl"))
assert len(wheels) == 1, wheels

with ZipFile(wheels[0]) as archive:
names = archive.namelist()

required = {
"embodied_gen/__init__.py",
"embodied_gen/scripts/imageto3d.py",
"embodied_gen/scripts/textto3d.py",
"embodied_gen/scripts/gen_texture.py",
"embodied_gen/utils/gpt_config.yaml",
}
missing = sorted(required.difference(names))
assert not missing, f"Missing wheel files: {missing}"
assert any(name.endswith("/LICENSE") for name in names)
assert not any(
"__pycache__" in name or name.endswith((".pyc", ".pyo"))
for name in names
)
PY
2 changes: 1 addition & 1 deletion .github/workflows/deploy_docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ jobs:
run: echo "PYTHONPATH=$(pwd)" >> $GITHUB_ENV

- name: Build docs (validation)
run: mkdocs build
run: mkdocs build --strict

deploy:
needs: build
Expand Down
File renamed without changes.
5 changes: 4 additions & 1 deletion MANIFEST.in
Original file line number Diff line number Diff line change
@@ -1 +1,4 @@
graft embodied_gen
graft embodied_gen
global-exclude __pycache__
global-exclude *.py[cod]
global-exclude .DS_Store
63 changes: 41 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,17 +6,18 @@
[![📄 arXiv](https://img.shields.io/badge/📄-arXiv_v1-b31b1b)](https://arxiv.org/abs/2506.10600)
[![📄 arXiv](https://img.shields.io/badge/📄-arXiv_v2-b31b1b)](https://arxiv.org/abs/2607.07459)
[![🎥 Video](https://img.shields.io/badge/🎥-Video-red)](https://youtu.be/MIkJJSVM8L4)
[![中文介绍](https://img.shields.io/badge/中文介绍-07C160?logo=wechat&logoColor=white)](https://mp.weixin.qq.com/s/RbMPX3vTTSwrW0MjwjeXuA)

[![🤗 Dataset](https://img.shields.io/badge/🤗-Dataset-blue)](https://huggingface.co/datasets/HorizonRobotics/EmbodiedGenData)
<!-- [![中文介绍](https://img.shields.io/badge/中文介绍-07C160?logo=wechat&logoColor=white)](https://mp.weixin.qq.com/s/HH1cPBhK2xcDbyCK4BBTbw) -->
[![🤗 Hugging Face](https://img.shields.io/badge/🤗-EmbodiedGen_Asset_Gallery-blue)](https://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Gallery-Explorer)
[![🤗 Hugging Face](https://img.shields.io/badge/🤗-Image_to_3D_Demo-blue)](https://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Image-to-3D)
[![🤗 Hugging Face](https://img.shields.io/badge/🤗-Text_to_3D_Demo-blue)](https://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Text-to-3D)
[![🤗 Hugging Face](https://img.shields.io/badge/🤗-Texture_Gen_Demo-blue)](https://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Texture-Gen)

> **From intent to executable 3D worlds.**
> ***EmbodiedGen*** compiles language, images, and edit commands into **simulation-ready 3D worlds** — physically plausible assets, large-scale scenes, and task-driven interactive environments, deployable across every major robotics simulator.
> ***EmbodiedGen*** compiles language, images, and edit commands into **policy-ready 3D worlds** — physically plausible assets, large-scale scenes, and task-driven interactive environments, deployable across major simulators.

<img src="docs/landing/assets/img/overview.jpg" alt="EmbodiedGen V2 Overview" width="700"/>
<img src="docs/landing/assets/img/overview.jpg" alt="EmbodiedGen V2 Overview" width="800"/>

---

Expand All @@ -27,7 +28,7 @@
- 📦 **One world, every simulator** — a standardized layout loads with consistent geometry, collision, and physics across **SAPIEN, Isaac Sim, Isaac Gym, MuJoCo, Genesis, and PyBullet**.
- 🧩 **Pluggable 3D backends** — switch between **SAM3D**, **TRELLIS**, and the **Hunyuan3D Pro** cloud API with a single flag.
- 🧥 **Beyond rigid bodies** — text-conditioned garments deploy as deformable meshes in Genesis.
- 🦾 **Part-level affordance** — functional part segmentation, per-part semantics, and simulation-validated 6-DoF grasp poses for any generated asset.
- 🕹️ **Part-level affordance** — functional part segmentation, per-part semantics, and simulation-validated 6-DoF grasp poses for any generated asset.
- 🤖 **Closed-loop robot learning** — policies trained purely in EmbodiedGen-generated worlds transfer to real robots (task success **9.7 → 79.8%** in sim, **21.7 → 75.0%** on real robots, from [sim2real RL paper](https://arxiv.org/abs/2603.18532)).

## 📋 Table of Contents
Expand All @@ -42,6 +43,7 @@
- [🤖 Train — Robot Learning](#robot-learning)
- [⚙️ Articulated Object Generation](#articulated-object-generation)
- [🧩 3D Scene Completion](#3d-scene-completion)
- [🔄 Sim-to-Real Reinforcement Learning](#sim-to-real-reinforcement-learning)

---

Expand All @@ -50,26 +52,34 @@
```sh
git clone https://github.com/HorizonRobotics/EmbodiedGen.git
cd EmbodiedGen
git checkout v2.0.1
git checkout v2.1.0
conda create -n embodiedgen python=3.10.13 -y
conda activate embodiedgen
# Manually install one CUDA toolkit when needed. cu126 remains the default.
# bash install.sh cu126 # RTX 40-series.
# bash install.sh cu128 # RTX 50-series / Blackwell.
bash install.sh basic # around 10 mins
conda deactivate && conda activate embodiedgen
bash install.sh basic # around 10 mins
```

Set up the GPT agent (required by most pipelines): update the API key in `embodied_gen/utils/gpt_config.yaml`. Then generate your first sim-ready asset:
Set up one GPT agent backend (required by most pipelines):

- **Azure OpenAI / OpenRouter:** select the backend and update its API key in
`embodied_gen/utils/gpt_config.yaml`.
- **Codex CLI:** run `codex login`, then set `agent_type: codex` in
`embodied_gen/utils/gpt_config.yaml`; no project API key is required.

See the [GPT Agent Setup guide](docs/documentation/gpt_agent.md) for complete
configuration, environment variables, and deployment recommendations.

Then generate your first sim-ready asset:

```sh
img3d-cli --image_path apps/assets/example_image/sample_00.jpg \
--n_retry 2 --output_root outputs/imageto3d
# → outputs/imageto3d/sample_00/result: URDF + mesh(.obj/.glb) + 3DGS(.ply) + video
```

A pre-built Docker image is also available on [Docker Hub](https://hub.docker.com/repository/docker/wangxinjie/embodiedgen).

➡️ Full guide: [Installation & Setup](https://horizonrobotics.github.io/EmbodiedGen/docs/install.html) · [Docker](docker/README.md)

---
Expand Down Expand Up @@ -133,7 +143,7 @@ room-cli -m embodied_gen.scripts.room_gen.gen_room \
# or specify the profile explicitly: --room-type Kitchen --seed 42 --complexity simple

# 3DGS background scene from text (requires `bash install.sh scene3d`)
scene3d-cli --prompts "Art studio with easel and canvas" --output_dir outputs/bg_scenes/ --seed 0
CUDA_VISIBLE_DEVICES=0 scene3d-cli --prompts "Art studio with easel and canvas" --output_dir outputs/bg_scenes/ --seed 0
```

➡️ Full guides: [Room Generation](https://horizonrobotics.github.io/EmbodiedGen/docs/tutorials/room_gen.html) · [3D Scene Generation](https://horizonrobotics.github.io/EmbodiedGen/docs/tutorials/scene_gen.html)
Expand Down Expand Up @@ -252,6 +262,14 @@ See our paper **3D-Fixer** published in CVPR 2026:

---

<h2 id="sim-to-real-reinforcement-learning">🔄 Sim-to-Real Reinforcement Learning</h2>

See our paper **Scaling Sim-to-Real Reinforcement Learning for Robot VLAs with Generative 3D Worlds** [[link]](https://arxiv.org/abs/2603.18532).

<img src="docs/documentation/assets/sim2real_work.jpg" alt="Sim-to-real reinforcement learning with EmbodiedGen-generated worlds" width="600">

---

## For Developer

```sh
Expand All @@ -263,17 +281,6 @@ python -m pytest # Pass all unit-test are required.

If you use EmbodiedGen in your research or projects, please cite:

```bibtex
@misc{wang2026embodiedgenv2agenticsimulationready,
title={EmbodiedGen V2: An Agentic, Simulation-Ready 3D World Engine for Embodied AI},
author={Xinjie Wang and Liu Liu and Taojun Ding and Andrew Choi and Chaodong Huang and Mengao Zhao and Ziang Li and Jackson Jiang and Chunlei Yu and Shengxiang Liu and Wei Xu and Zhizhong Su},
year={2026},
eprint={2607.07459},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2607.07459},
}
```
```bibtex
@misc{wang2025embodiedgengenerative3dworld,
title = {EmbodiedGen: Towards a Generative 3D World Engine for Embodied Intelligence},
Expand All @@ -286,6 +293,18 @@ If you use EmbodiedGen in your research or projects, please cite:
}
```

```bibtex
@misc{wang2026embodiedgenv2agenticsimulationready,
title={EmbodiedGen V2: An Agentic, Simulation-Ready 3D World Engine for Embodied AI},
author={Xinjie Wang and Liu Liu and Taojun Ding and Andrew Choi and Chaodong Huang and Mengao Zhao and Ziang Li and Jackson Jiang and Chunlei Yu and Shengxiang Liu and Wei Xu and Zhizhong Su},
year={2026},
eprint={2607.07459},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2607.07459},
}
```

---

## 🙌 Acknowledgement
Expand All @@ -297,4 +316,4 @@ EmbodiedGen builds upon the following amazing projects and models:

## ⚖️ License

This project is licensed under the [Apache License 2.0](docs/LICENSE). See the `LICENSE` file for details.
This project is licensed under the [Apache License 2.0](LICENSE). See the `LICENSE` file for details.
2 changes: 1 addition & 1 deletion apps/app_style.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
# permissions and limitations under the License.

from gradio.themes import Soft
from gradio.themes.utils.colors import gray, neutral, slate, stone, teal, zinc
from gradio.themes.utils.colors import gray, stone

lighting_css = """
<style>
Expand Down
2 changes: 0 additions & 2 deletions apps/visualize_asset.py
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,6 @@ def fake_gpu_init():
"dataset/**/*.mp4",
],
local_dir=HF_LOCAL_DIR,
local_dir_use_symlinks=False,
)
DATA_ROOT = os.path.join(HF_LOCAL_DIR, "dataset")
else:
Expand All @@ -115,7 +114,6 @@ def ensure_hf_files(rel_patterns: list[str]) -> None:
repo_type="dataset",
allow_patterns=[f"dataset/{p}" for p in rel_patterns],
local_dir=HF_LOCAL_DIR,
local_dir_use_symlinks=False,
)


Expand Down
40 changes: 40 additions & 0 deletions docker/5090/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
FROM pytorch/pytorch:2.8.0-cuda12.8-cudnn9-devel

LABEL maintainer="Horizon Robotics" \
description="EmbodiedGen for RTX 5090 (sm_120, CUDA 12.8)"

ENV DEBIAN_FRONTEND=noninteractive \
LANG=C.UTF-8 \
LC_ALL=C.UTF-8

RUN apt-get update && apt-get install -y --no-install-recommends \
git build-essential ninja-build pkg-config \
libgl1-mesa-glx libglib2.0-0 libsm6 libxext6 libxrender1 \
libegl1 libgles2 libgbm1 \
&& rm -rf /var/lib/apt/lists/*

ENV CUDA_HOME=/usr/local/cuda \
CUDA_PATH=/usr/local/cuda \
PATH=/usr/local/cuda/bin:$PATH \
LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH \
EMBODIEDGEN_TORCH_INDEX_URL=https://download.pytorch.org/whl/cu128 \
PIP_NO_CACHE_DIR=1 \
TORCH_CUDA_ARCH_LIST="12.0" \
FORCE_CUDA=1 \
MAX_JOBS=8

ENV HF_HOME=/hf_cache \
TORCH_HOME=/hf_cache/torch_hub \
OPENCV_IO_ENABLE_OPENEXR=1

WORKDIR /EmbodiedGen
COPY . /EmbodiedGen/

RUN conda create -n embodiedgen python=3.10.13 -y && \
conda run -n embodiedgen bash install/install_basic.sh && \
conda run -n embodiedgen python -m pip install numpy==1.26.4

RUN /opt/conda/bin/conda init bash && \
echo "conda activate embodiedgen" >> /root/.bashrc

CMD ["bash", "--login"]
61 changes: 61 additions & 0 deletions docker/5090/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# RTX 5090 Docker Image

This image builds the EmbodiedGen `basic` stage for RTX 5090 / Blackwell
(`sm_120`) with PyTorch 2.8 and CUDA 12.8. Model weights are downloaded at
runtime and are not included in the image.

## Prerequisites

- Linux with an RTX 5090 and a CUDA 12.8-compatible NVIDIA driver.
- Docker with NVIDIA Container Toolkit.
- Access to PyPI and GitHub while building, and Hugging Face at runtime.

## Build

`.git` is excluded from the build context, so initialize the basic-stage
submodules on the host first:

```bash
git clone <repository-url> EmbodiedGen
cd EmbodiedGen
git submodule update --init --recursive --progress \
thirdparty/TRELLIS thirdparty/sam3d
docker build -t embodiedgen:cu128-sm120 -f docker/5090/Dockerfile .
```

Lower `MAX_JOBS` in the Dockerfile if CUDA extension builds exhaust memory.

## Verify

```bash
docker run --rm embodiedgen:cu128-sm120 \
conda run -n embodiedgen python -c "import embodied_gen, gsplat, kaolin, pytorch3d; print('imports OK')"
docker run --rm embodiedgen:cu128-sm120 \
conda run -n embodiedgen img3d-cli --help
docker run --rm --gpus all embodiedgen:cu128-sm120 \
conda run -n embodiedgen python -c "import torch; x=torch.randn(1024,1024,device='cuda:0'); print('matmul', (x@x).sum().item())"
```

The last command must print a numeric result. A `no kernel image` error usually
means a cu126 PyTorch wheel replaced the cu128 wheel.

## Run

```bash
docker run --gpus all --shm-size=64g -it --rm \
-v embodiedgen_hf_cache:/hf_cache \
-v "$PWD":/workspace \
embodiedgen:cu128-sm120 bash
```

Inside the container:

```bash
conda activate embodiedgen
img3d-cli --image_path <image> --output_root /workspace/outputs
```

## Notes

- `EMBODIEDGEN_TORCH_INDEX_URL` keeps newer installers on the cu128 index.
- For RTX 4090, A100, or H100, use `docker/Dockerfile` instead.
10 changes: 5 additions & 5 deletions docker/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
FROM pytorch/pytorch:2.4.0-cuda11.8-cudnn9-devel
FROM pytorch/pytorch:2.8.0-cuda12.6-cudnn9-devel
LABEL maintainer="Horizon Robotics <xinjie.wang@horizon.auto>" \
version="0.1.2" \
version="2.1.0" \
description="Docker image for EmbodiedGen"

RUN apt-get update && apt-get install -y --no-install-recommends \
Expand All @@ -16,9 +16,9 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
&& rm -rf /var/lib/apt/lists/*

# TORCH_CUDA_ARCH_LIST="8.0;8.9;9.0" -> A100/4090/H100
ENV CUDA_HOME=/usr/local/cuda-11.8 \
PATH=/usr/local/cuda-11.8/bin:$PATH \
LD_LIBRARY_PATH=/usr/local/cuda-11.8/lib64:$LD_LIBRARY_PATH \
ENV CUDA_HOME=/usr/local/cuda \
PATH=/usr/local/cuda/bin:$PATH \
LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH \
DEBIAN_FRONTEND=noninteractive \
TORCH_CUDA_ARCH_LIST="8.0;8.9;9.0" \
TCNN_CUDA_ARCHITECTURES=80,89,90
Expand Down
Loading
Loading