DepthVision

Native monocular depth for Adobe After Effects and Premiere Pro.

English · Türkçe

DepthVision turns an ordinary frame into a relative depth map inside the Adobe render pipeline. It is a native C++ effect, not a panel or a script: frames remain in the host, inference runs locally through ONNX Runtime, and the result is returned in the same effect stack as any other video filter.

The project is deliberately conservative about what it claims. DepthVision estimates relative inverse depth from one image. It is useful for mattes, depth-guided grading, relighting studies, normal and edge passes, and compositing. It does not recover calibrated metres, camera intrinsics, or occluded geometry.

Release status: the current Windows build is a beta. The core, model, DirectML and CPU paths, SDK build, PiPL resource, package integrity, and Premiere plug-in discovery have been validated locally. A complete multi-version host and GPU certification matrix is still in progress. The beta installer is not code-signed.

Download and installation

Download the latest Windows x64 setup from GitHub Releases. Each installer is published with a separate SHA-256 file.

1. Close After Effects, Premiere Pro, and Adobe Media Encoder.
2. Verify the downloaded installer against its .sha256 file.
3. Run DepthVision-Setup-<version>-Windows-x64.exe as an administrator.
4. Restart the Adobe host and locate DepthVision → DepthVision in the Effects panel.

The installer writes the complete runtime to:

C:\Program Files\Adobe\Common\Plug-ins\7.0\MediaCore\DepthVision

Do not copy DepthVision.aex by itself. The effect also needs the model, ONNX Runtime and DirectML files distributed beside it. A manual installation must preserve the complete directory shown in Runtime layout.

What it provides

Area	Implementation
Hosts	After Effects SmartFX and Premiere Pro software effect paths
Host formats	AE ARGB 8-bit, 16-bit and 32-bit float; Premiere BGRA 8-bit and 32-bit float
Inference	Depth Anything V2 Small, ONNX Runtime 1.22.1
Acceleration	DirectML on compatible Windows GPUs; explicit CPU mode; automatic CPU fallback
Resolution presets	Draft 252 px, Preview 392 px, Production 518 px, Ultra 770 px on the long edge
Range handling	Robust percentile clipping or fixed raw model-space endpoints
Refinement	Edge-aware 3 × 3 filtering, detail recovery, contrast and gamma shaping
Temporal processing	Optional confidence-weighted history with seek and scene-cut reset
Outputs	Depth, depth × source, near matte, far matte, normals, depth edges, false color, focus slice
Alpha	Straight-color inference with optional source-alpha preservation and correct host premultiplication

Output modes

DepthVision keeps its working depth buffer as float until the final host write. With normalized depth (d\in[0,1]), near surfaces are white by default and far surfaces are black.

• Depth — grayscale normalized depth.
• Depth over Source — source RGB multiplied by depth, useful as an immediate spatial reference.
• Near Matte / Far Matte — complementary smooth threshold mattes around Matte Center.
• Normals — tangent-style normals derived from central differences of the depth field.
• Depth Edges — the magnitude of the depth gradient, scaled by Edge Gain.
• False Color — a compact blue-to-red diagnostic mapping of the normalized depth.
• Focus Slice — a soft band-pass matte around Matte Center; useful for selecting a depth interval rather than one side of a threshold.

These passes are derived from the estimated depth; they are not separate neural-network outputs.

Controls

Control	Meaning
Output	Selects one of the eight generated passes.
Quality	Chooses the neural-network working resolution. It does not resize the final frame.
Compute	Automatic prefers DirectML and falls back to CPU; GPU fails clearly when DirectML cannot start; CPU never creates a DirectML session.
Range Mode	Automatic Percentile normalizes each frame robustly; Manual keeps fixed raw inverse-depth endpoints across a shot.
Low / High Clip Percentile	Removes the low and high tails of the raw inverse-depth distribution. Defaults are 2% and 98%.
Manual Near / Far	Raw model-space endpoints. Manual Near must be greater than Manual Far.
Invert	Reverses near and far after normalization.
Gamma	Shapes mid-depth values after refinement.
Contrast	Expands or compresses separation around 0.5.
Detail	Adds back high-frequency depth detail after edge-aware smoothing.
Edge Preservation	Reduces smoothing across source-luminance changes.
Stability	Amount of temporal history used in stable regions. Zero is frame-independent.
Scene Cut Sensitivity	Thumbnail luminance-difference threshold that discards temporal history.
Matte Center / Softness	Threshold location and transition width for matte and focus-slice outputs.
Normal Strength	Scales the depth gradient before normal-vector normalization.
Edge Gain	Scales the gradient magnitude in the depth-edge pass.
Preserve Source Alpha	Reuses the input alpha instead of returning an opaque generated pass.

Processing pipeline

Adobe effect world
  → row-stride-aware ARGB/BGRA decode
  → unpremultiply and sanitize non-finite float pixels
  → aspect-preserving, patch-aligned resize
  → ImageNet channel normalization
  → Depth Anything V2 inference
  → crop/resize to the source frame
  → robust or manual depth normalization
  → edge-aware refinement and tone shaping
  → optional temporal stabilization
  → selected derived output
  → source-alpha restore, premultiply, host-format encode

Pixel decode and alpha

Adobe worlds may have padding between scanlines, so every row is addressed with the host-provided rowbytes rather than assuming a tightly packed buffer. RGB is converted from premultiplied to straight form before inference:

[ c = \begin{cases} \operatorname{clamp}(c_p / \alpha, 0, 1), & \alpha > 10^{-6} \ 0, & \text{otherwise} \end{cases} ]

NaN and infinity in float footage are replaced with zero before preprocessing. HDR RGB is clipped only for the network input; the depth pipeline itself remains float. The effect does not silently perform color-profile conversion—the network receives the numeric RGB values supplied by the host after unpremultiplication.

Working resolution and patch alignment

For dynamic-shape models, the long edge (L) is selected by the quality preset. The short edge keeps the source aspect ratio and is rounded to the nearest multiple of the model patch size, 14:

[ S = \max\left(14,;14\cdot\operatorname{round}\left(\frac{L,r}{14}\right)\right) ]

where (r) is the short-to-long source ratio. Fixed-shape backends use a centred letterbox. After inference, only the valid content rectangle is sampled back to the original frame size.

Network input

The resized straight RGB image is converted from interleaved RGBA to planar NCHW. Each channel uses the ImageNet normalization expected by the exported model:

[ x'_c = \frac{x_c - \mu_c}{\sigma_c} ]

with (\mu=(0.485,0.456,0.406)) and (\sigma=(0.229,0.224,0.225)).

The model returns a single relative inverse-depth field (z). Larger raw values generally indicate nearer content; the absolute scale is not stable between unrelated shots or model variants.

Robust range normalization

Automatic mode computes low and high quantiles of finite predictions. With low percentile (p_l) and high percentile (p_h):

[ z_f = Q_{p_l}(z), \qquad z_n = Q_{p_h}(z) ]

[ d = \operatorname{clamp}\left(\frac{z-z_f}{z_n-z_f},0,1\right) ]

This rejects isolated extremes before mapping near to 1 and far to 0. Manual mode substitutes user-supplied (z_n) and (z_f), which is preferable when a shot must keep the same remap across frames. Invert applies (d\leftarrow1-d) after normalization.

Edge-aware refinement

Depth upsampling can soften object boundaries. DepthVision first computes a small joint, luminance-guided filter. For neighbour (q) around pixel (p):

[ w_{pq}=w_s(p,q)\exp\left(-\frac{(Y_q-Y_p)^2}{2\sigma_Y^2}\right) ]

[ b_p=\frac{\sum_q w_{pq}d_q}{\sum_q w_{pq}} ]

Edge Preservation controls (\sigma_Y); a smaller value gives less weight across strong source edges. Detail recovery is an unsharp operation:

[ d'_p=\operatorname{clamp}\left(d_p+1.5k(d_p-b_p),0,1\right) ]

where (k) is the normalized Detail control. Contrast and gamma are then applied:

[ d''=\operatorname{clamp}((d'-0.5)C+0.5,0,1)^{1/\gamma} ]

This is a restrained spatial refinement, not a second segmentation network. Source edges can therefore guide the result, but they cannot invent geometry missing from the depth prediction.

Temporal stabilization

Temporal processing is optional and deliberately sequential. DepthVision resets history on a seek, non-consecutive frame number, size change, explicit discontinuity, or scene cut. Scene cuts use the mean absolute luminance difference of 64 × 36 thumbnails.

For valid consecutive frames, the per-pixel history confidence is:

[ q_p=\exp(-18|d_{t,p}-d_{t-1,p}|) ]

[ h_p=\operatorname{clamp}(s,0,0.95)q_p ]

[ \hat d_{t,p}=(1-h_p)d_{t,p}+h_p\hat d_{t-1,p} ]

Stable regions retain more history; large changes trust the current frame. This is not optical-flow warping and should not be described as a video-depth model. Keep Stability at zero for arbitrary frame-order rendering or unrestricted timeline scrubbing.

Derived passes

Normals use central depth differences and a user scale (s_n):

[ \mathbf n=\operatorname{normalize}(-s_n\partial_xd,-s_n\partial_yd,1) ]

Depth edges use:

[ e=\operatorname{clamp}\left(g\sqrt{(\partial_xd)^2+(\partial_yd)^2},0,1\right) ]

The focus slice is a soft band around centre (m). If (r) is Matte Softness, the matte is 1 near the centre and falls to 0 between (r/2) and (r) using cubic smoothstep.

Runtime and host integration

The model session is created lazily on the first render and reused for that effect instance. Changing the compute device rebuilds the session. DirectML sessions use sequential execution, disabled memory patterns, and a guarded Run call. If Automatic cannot create a DirectML session, DepthVision creates a clean CPU session instead; explicit GPU returns an actionable error.

On Windows, onnxruntime.dll is loaded by absolute path from the DepthVision package. This avoids accidentally binding to an ABI-incompatible ONNX Runtime installed by another plug-in or found in a global search path.

The PiPL advertises pixel independence, deep-color awareness, SmartFX support, and float-color awareness. Multi-Frame Rendering is intentionally not advertised in this beta because sequential temporal history and out-of-order evaluation conflict. Each effect instance also serializes initialize/render transitions so an animated compute-device change cannot race an in-flight inference.

Runtime layout

DepthVision/
├── DepthVision.aex
├── DirectML.dll
├── onnxruntime.dll
├── onnxruntime_providers_shared.dll
├── models/
│   ├── depth_anything_v2_small.onnx
│   └── depthvision-model.json
├── LICENSES/
├── LICENSE
├── THIRD_PARTY_NOTICES.md
└── SHA256SUMS.txt

The model file is about 99 MB and is intentionally excluded from Git history. Fetch and package scripts verify its pinned SHA-256 before accepting it.

Building from source

Requirements

• Windows 10/11 x64
• Visual Studio 2022 Build Tools with Desktop C++
• CMake 3.25 or newer
• Official After Effects C++ SDK 25.6 (61) or a compatible newer SDK
• PowerShell 7 recommended
• Inno Setup 6.7 for the .exe installer

Adobe's SDK is not redistributed in this repository. Obtain it from Adobe and keep it outside the source tree.

Fetch verified dependencies

.\tools\fetch_model.ps1
.\tools\fetch_onnxruntime.ps1

The model acquisition is pinned to revision 64fe43eba7f8a384b02fe3fadaa26cbba35548d8 and SHA-256:

afb6a5c28f3b6bf1618c6e43f02073ef9dfdc70e937502d51603e57b0a1df10c

Configure, build and test

AE_SDK_ROOT may be the extracted SDK itself or the outer directory containing Adobe's versioned *.AfterEffectsSDK folder.

cmake --preset windows-release `
  -DAE_SDK_ROOT="C:\SDKs\AfterEffectsSDK_25.6"

cmake --build --preset windows-release --parallel
ctest --preset windows-release

The release plug-in is written to:

build/windows-release/adobe/Release/DepthVision.aex

Assemble the runtime and installer

.\tools\package_windows.ps1 `
  -PluginPath .\build\windows-release\adobe\Release\DepthVision.aex `
  -OnnxRuntimeRoot .\.deps\onnxruntime-directml-1.22.1

.\tools\build_installer.ps1 -Version 1.0.0-beta.1

The installer build writes both the setup executable and its SHA-256 file to dist/.

Validation

The automated suite covers range statistics, letterbox round-trips, invalid settings, all eight output modes, temporal first/sequential/seek behaviour, real model inference, explicit CPU selection and DirectML selection. Release checks also inspect the x64 PE, PiPL resource, Windows version resource, exported entry points, package hashes and runtime dependencies.

The detailed record and remaining host gates live in docs/VALIDATION.md and docs/QA.md. A passing unit test is not a substitute for host certification; After Effects color depths, Premiere export, Media Encoder, multiple GPU vendors, cancellation, memory pressure, and long-running timelines still need matrix coverage before a stable release.

Known limits

• Relative depth is not metric distance.
• Single-frame inference can flicker or change ordering on ambiguous, reflective, transparent, very dark, or textureless content.
• Temporal stabilization is history blending, not motion compensation.
• Input RGB follows the host's numeric working values; DepthVision does not perform independent ICC/OCIO conversion.
• DirectML performance depends on the GPU, driver, frame size and quality preset.
• The beta installer and plug-in are unsigned; verify published checksums.
• macOS packaging and Metal/Core ML execution are not implemented.

Project structure

adobe/      After Effects/Premiere adapter, PiPL and Windows resources
assets/     Brand assets used by project documentation and installer
docs/       Architecture, controls, build/release notes, validation and QA
include/    Public C++ core interfaces
installer/  Reproducible Inno Setup definition
models/     Model manifest and license; fetched ONNX file is ignored
src/        Core processing and ONNX Runtime backend
tests/      Deterministic core tests and optional real-model smoke test
tools/      Fetch, validation, packaging and installer scripts

Licensing and attribution

DepthVision's source is released under the MIT License. The packaged model is Depth Anything V2 Small under Apache-2.0. ONNX Runtime and DirectML retain their respective licenses and third-party notices. See THIRD_PARTY_NOTICES.md and the package's LICENSES directory.

The logo and banner in this repository are original DepthVision brand artwork. They are not screenshots and are not presented as model output. This README intentionally contains no simulated product UI; future product images should be captured from the tagged plug-in build they document.