Synthetic Data Generation with Omniverse Replicator for Real-Time Segmentation

As a robotics software company, we have worked with many vision models. SAM3 is one of the most impressive: give it a one-word prompt and it segments the object, without any training.

For practical robotics applications, though, two other questions matter: can the model run in real time on the robot's own hardware, and how much GPU memory does it need?

Smaller models like RF-DETR are much faster, but they need labeled training data, which is often unavailable for niche industrial parts. Synthetic data generation solves this: you render the part in a simulator, and every image comes with an exact label.

In this article, we use Isaac Sim and Omniverse Replicator to generate 2,000 labeled images of steel flanges, train RF-DETR entirely on them, and compare it with SAM3 on real photos.

‍

How Synthetic Data Works

Instead of photographing real parts and labeling them, we render them in a simulator. The simulator knows exactly which pixel belongs to which object, so every image comes with an exact mask, and no manual labeling or annotation tool is needed.

Omniverse Replicator does this inside Isaac Sim. It renders photorealistic images, saves labels such as segmentation masks with each one, and lets you randomize the scene from a Python script.

The key to good synthetic data is closing the sim-to-real gap, the difference between rendered and real images. Isaac Sim's RTX renderer brings the appearance of rendered images close to real photos, and domain randomization adds variety by changing the lighting, materials, positions and camera in every frame. With enough variety, a real photo looks to the model like one more variation.

‍

‍

Building the Flange Dataset

We picked steel flanges, a common industrial part with very little public segmentation data. The scene is simple: a conveyor belt, a few flanges from CAD models, and a camera looking down at them.

‍

‍

Replicator saves each rendered image together with its label. Here, the label is an instance mask that outlines every flange separately and marks which pixels belong to it. Both come from the same render, so the mask matches the image pixel for pixel.

‍

‍

Before every frame, we randomized:

• the number and position of the flanges
• the flange finish (painted, bare, rusted)
• the belt material
• clutter around the flanges
• the lighting
• the camera angle

‍

This gave us 2,000 labeled images in about 19 minutes on a single RTX 5060 Ti.

‍

Training RF-DETR on Synthetic Data Only

We converted the masks to the standard COCO format and fine-tuned RF-DETR Seg, a real-time segmentation model released under the Apache 2.0 license. We used the Small size as a balance between speed and mask quality.

Starting from its pre-trained weights, we trained it on the 2,000 synthetic images with one class, “flange”. Training took about 35 minutes on the same GPU. No real image was collected or labeled at any point.

‍

Results: SAM3 vs RF-DETR on Real Photos

We tested both models on 50 real flange photos (58 flanges in total) from a public dataset on Roboflow Universe. Neither model had seen them before. SAM3 got a one-word prompt, “flange”. We ran RF-DETR in full precision (FP32) and half precision (FP16) . Full precision stores each model weight with 32 bits. Half precision uses 16 bits, which lets the GPU process images faster, usually with little or no loss in accuracy.

‍

‍

IoU (intersection over union) measures how closely a predicted mask overlaps the real outline of the part: 1.0 is a perfect match and 0 is no overlap.

mAP (mean average precision) scores segmentation accuracy from 0 to 1. It checks how many masks are correct and how many flanges were found at ten IoU cutoffs, from 0.5 (a loose fit) to 0.95 (an almost exact outline), and averages the results.

The results:

• SAM3: mask mAP 0.856, time per image 202 ms (5 FPS), peak GPU memory 3.94 GB
• RF-DETR, FP32: mask mAP 0.864, time per image 11.1 ms (90 FPS), peak GPU memory 0.22 GB
• RF-DETR, FP16: mask mAP 0.863, time per image 5.0 ms (198 FPS), peak GPU memory 0.12 GB

‍

Result summary:

• Accuracy is the same. Both models found every flange at a 0.5 cutoff, with no false masks, and scored a perfect 1.0 there. Their overall mAP is below 1.0 because mAP also checks stricter cutoffs, up to 0.95, where a mask must match the flange's outline almost exactly. RF-DETR scored slightly higher.
• RF-DETR is 40 times faster. 5.0 ms per image in FP16, against about 200 ms for SAM3.
• RF-DETR uses about 33 times less GPU memory. Small enough to share a GPU with the rest of the robot's software.

‍

When Synthetic Data Makes Sense

SAM3 is a good fit when the task keeps changing and there is no time to train anything. On a moving line with a known set of parts, a small model trained on synthetic data gives the same accuracy at a fraction of the time and memory.

With synthetic data, the time goes into building the simulation scene once, not into taking and labeling photos. When a new part is added to the line, you add its CAD model to the same scene and generate a fresh dataset. There is no new photo shoot and no manual labeling.

‍

At Black Coffee Robotics, we build Isaac Sim simulations and perception pipelines for robotic arm and other applications. If you need a vision model for a line where no labeled data exists yet, get in touch.