HeliSDG

Problem

Detecting, and even more so tracking aircraft from a helicopter-mounted camera is data-starved. Footage of helicopters flying in close proximity is rare, expensive to requisition via charter, and comes with a slurry of safety concerns and strict flight regulations.

Furthermore, from a machine learning standpoint such data is almost never labelled at the pixel level, if at all, and sports poor domain randomization be it airframe, weather condition, or viewing geometry.

Thus synthetic data generation of short, temporally coherent video snippets is useful.

Idea

In a given snippet,

• an ego helicopter carries the camera along flight path in a photoreal scene.
• one or more target helicopters fly through the field of view with believable motion.
• every frame ships with segmentation masks and bounding boxes.
• the dataset is balanced across a scenario taxonomy.

On that last point, random sampling wastes expensive renders on unusable frames (occluded targets, terrain collisions, impossible trajectories). I elected to construct a valid flight corridor first and plan trajectories within it.

Architecture

flowchart LR
    CFG[scenario families<br/>+ terrain pool]

    subgraph py["Python sampler"]
        GEN[domain randomisation]
    end

    subgraph cpp["C++ planner"]
        direction TD
        EGO[ego A*] --> SM[smoothing] --> CAM[camera corridor] --> TGT[target planner]
    end

    subgraph render["Isaac Sim + Cesium"]
        REN[photoreal render]
    end

    VIZ[three.js visualizer]
    OUT[frames + masks + bboxes]

    CFG --> GEN
    GEN -- scenario.yaml --> EGO
    TGT -. infeasible → resample .-> GEN
    TGT -- plan.json --> VIZ
    TGT -- plan.json --> REN
    REN --> OUT

C++17 owns all planning logic, Python only interfaces with Isaac Sim/Cesium.

scenario.yaml is the numeric planning contract C++ reads while manifest.json is a sidecar carrying model, lighting, weather, etc. such that the planner is scenario-agnostic.

C++ planner

Terrain

There is no way natively to export the terrain that Cesium streams into Isaac Sim. But trajectory planning must know the terrain to avoid collisions. Does that mean the planner must run in the sim, thereby burning GPU time on non-rendering tasks?

No. Recall we are working with real-world terrain, so we may use a DEM of the same area we intend to sim. Notably, Cesium streams a DSM mesh, but for a helicopter $AGL >> \varepsilon$. Indeed, benchmarking Cesium’s Google Photoreal 3D Tiles against Copernicus GLO-30 over nine 2km × 2km patches gives a vertical RMSE of 0.7–5 m on open/flat terrain and 10–26m on steep terrain.

Terrain is represented as a HeightFieldPatch in local ENU with WGS84 ellipsoidal heights, constructed by sampling a DEM.

Ego path

26-neighbour voxel A* over a clearance-carved volume. The obvious approach is to voxelise the entire space over terrain into an occupancy grid. At 5m over a 2km x 2km with just a 500m AGL ceiling is 16M voxels, per scenario.

Given terrain is stored as a height field, whether an arbitrary point is “flyable” is a continuous query rather than a discrete lookup. Thus a voxel is navigable iff the bilinear interpolated height at its centre meets minimum clearance.

bool is_navigable(const Vec3& pos) const noexcept {
    if (!in_bounds(pos.x(), pos.y())) return false;
    return kernels::clearance_at(patch, pos) >= min_clearance && pos.z() <= ceiling;
}

This way the search touches only what it expands into. This also decouples search resolution from height field resolution, so the grid is 5 m regardless of the source DEM resolution.

A* returns a staircase, the smoother turns it into an aircraft path. Pose is derived along that path under bank-rate and vertical-speed limits.

Targets

The planner defines target position as (range, az, el) off the camera boresight at the first and last frame rather than as absolute world coordinates. This

1. guarantees it is inside the frustum at both ends by construction.
2. enables expressing flight profiles as anchor pairs (ex. crossing is left edge → right edge).

Vec3 camera_relative_point(const Pose& cam, double range, double az, double el) {
    const Vec3 dir(cos(el) * cos(az), cos(el) * sin(az), sin(el));
    return cam.position + range * (cam.orientation * dir);
}

Between the anchors the target flies a curve. The planner was a time-layered DAG where voxels are frame indexed, edges forward in time only, Dijkstra in topological order inside the moving frustum. Correct and fast, but it only ever produced straight-ish paths. Reject-sampling to seed to chord and bend it proved simpler and more effective (curvature as a geometric knob).

Rendering

• Cesium refines tiles against Kit viewports, and Replicator render product on headless isn’t one. Binding the capture camera to a real viewport enforced screen-space error.
• The bottleneck is tile streaming, not the GPU. No render knob moved wall-clock. Sorting the render queue by terrain patch, so each region cold-streams once, did.

Visualizer

A three.js app wrapping the planner to scrub frames over the heightfield, drag ego endpoints and target anchors, probe candidates per family, save as a scenario YAML.

C++17 · Eigen · Python · NumPy · pyproj · rasterio · Isaac Sim 5.1 · Omniverse Replicator · Cesium for Omniverse · Google Photoreal 3D Tiles · Copernicus GLO-30 · USGS 3DEP · three.js