cd ~/
3D ReconstructionMeshingSLAMSpatial ComputingMagic Leap

Real-Time 3D Reconstruction: Meshing at Interactive Rates

Advances in spatial mapping for V2 - dense reconstruction, mesh quality, and the compute challenge of real-time 3D.

Evyatar Bluzer
3 min read

Meshing is next on my V2 list. V1's spatial mapping was good enough for plane detection, and for V2 we want realistic occlusion and physics, which takes dense, accurate meshes.

V1 Limitations

Current meshing runs at roughly 5cm voxel resolution, regenerates the full mesh every 2-3 seconds, does minimal hole filling, and produces blocky, noisy surfaces. For V2 we need 1-2cm resolution, incremental sub-second updates, intelligent hole completion, and surfaces that come out smooth and watertight, which is half of what the 640x480 depth upgrade is buying.

Reconstruction Pipeline

Depth Filtering

Raw depth has holes and noise, so we filter before integration: bilateral filtering for edge-preserving smoothing, temporal averaging to accumulate confidence, statistical outlier rejection.

TSDF Integration

The core representation is a Truncated Signed Distance Function, an implicit surface. A voxel grid stores distance to the nearest surface, new depth frames update those distances via a running average, and memory stays manageable because only near-surface voxels get stored. V1 used a dense voxel array at 5cm resolution over a limited volume; V2 moves to sparse voxel structures (octree or hash) at 1cm resolution, room-scale.

Mesh Extraction

Marching cubes pulls the triangle mesh out of the TSDF. It's a classic algorithm, well-understood and parallelizable per voxel, and mesh complexity grows with surface area rather than volume.

Mesh Simplification

Raw marching cubes output carries far too many triangles. We decimate vertices with quadric error metrics, preserve sharp edges and features, and hold to a triangle budget the renderer can live with.

GPU Acceleration

Reconstruction is embarrassingly parallel. Each depth pixel updates independent voxels, each voxel processes independently, and marching cubes runs per voxel. V1 did all of this on the CPU, slowly and at a power cost; V2 moves it to a GPU compute pipeline. Expected improvement: 10x throughput at similar power. In November I argued for a dedicated NPU so the GPU could stay reserved for graphics. The NPU half stands. The reserved half lasted five months; meshing is moving onto GPU compute.

Reconstruction pipelineSix boxes in two rows joined by labeled arrows, read left to right then back along the bottom row: depth frames, depth filtering, TSDF integration, mesh extraction, mesh simplification, collision mesh; a dashed highlighted boundary encloses TSDF integration and mesh extraction, the stages moving to GPU compute. V2: GPU compute, 10x throughput Depth frames 640x480 depth in V2 Depth filtering bilateral, temporal averaging, outlier rejection TSDF integration voxel grid, running average V1 dense 5 cm, V2 sparse 1 cm Mesh extraction marching cubes, per voxel Mesh simplification quadric error decimation, triangle budget Collision mesh physics and occlusion raw depth filtered depth implicit surface triangle mesh decimated mesh
The V2 reconstruction pipeline from depth frames to a collision mesh; the two per-voxel stages inside the dashed boundary move from CPU to GPU compute in V2, for an expected 10x throughput at similar power.

Learned Completion

Depth sensors don't see everything. Occlusions, range limits, and specular surfaces all leave gaps, so the question becomes whether we can complete what's missing. Geometric priors help (planes extend, rooms have floors and ceilings), a neural network can predict unobserved geometry outright, and semantic reasoning fills in the rest - if it's a chair, it probably has four legs.

We're prototyping the learned version: train on complete 3D models, feed in a partial observation, get a completed mesh out. Early results are promising for common objects. Generalization to arbitrary scenes is harder.

Memory Management

Room-scale at 1cm resolution means billions of voxels if you store them densely, which we can't. So we use hierarchical structures that only subdivide where needed, LRU caching that keeps recent observations and pages out old areas, and level-of-detail that holds high resolution near the user and goes coarse far away.

The memory budget is 500MB for reconstruction. The target is a kitchen-sized space at 1cm with dynamic updates.

Comments