World model dataset: what real-world scenes need to include

A world model dataset is the collection of images, video, camera poses and 3D geometry that a model uses to learn how real places look and change as an agent moves through them. This guide is for research leads at AI labs and operations leads at robotics companies who are sourcing real-world data for physical AI.

You will find what such a dataset should contain, how real captures compare with synthetic scenes, a six-step pipeline from capture to training, and the questions to ask before you license data.

Key takeaways:

• Metric camera poses matter as much as raw image count.
• Real scenes add clutter, light and wear that simulation rarely reproduces.
• Pair video with 3D reconstructions so one scene serves training and simulation.
• Check privacy masking, licensing and diversity before you look at volume.
• Start with a small pilot subset, then scale.

What is a world model dataset?

A world model dataset is a large set of real or simulated scenes in which every image or video frame is tied to where the camera was and what the space looks like in 3D. Models trained on it learn to predict the next view of a place, which robots use for planning, navigation and testing policies.

A classic image dataset answers 'what is in this picture'. A world model dataset answers 'what will I see if I move one meter to the left'. That requires sequences and consistent geometry behind them.

Teams use this data for three families of models:

• Video world models. Predict future frames from past frames and actions.
• 3D-consistent scene models. Learn geometry and appearance that stay stable across viewpoints.
• Action-conditioned models. Predict how the scene responds when a robot moves or acts.

The same scenes also feed vision-language navigation (VLN) models, which follow spoken or written directions through a building, and vision-language-action (VLA) models, which turn instructions into motor commands.

What should a world model dataset for robotics include?

A world model dataset for robotics should include multi-view images or video, metric camera poses, depth, a 3D reconstruction per scene, semantic labels and privacy masking. It also needs diversity across countries, building types and lighting, plus clear licensing terms, so your team can train, evaluate and publish without legal surprises later.

Here is what each layer contributes:

• Multi-view imagery with metadata. EXIF data records device, lens and capture time.
• Metric camera poses. Position and orientation in real meters, often from ARKit or ARCore tracking.
• Depth maps. Distance per pixel, useful for supervision and collision checks.
• Structure-from-motion reconstructions. COLMAP is an open-source tool that recovers camera poses and sparse 3D points from overlapping photos.
• Gaussian splatting. A 3D scene representation built from millions of small colored ellipsoids that renders photorealistic new viewpoints.
• Semantic annotations. Labels for floors, doors, stairs, shelves and other classes.
• Privacy masking. Faces and license plates blurred before data leaves the pipeline.

Walked paths at human height matter for humanoids and other robots that share hallways with people.

A scene dataset shows places, not contact. Grasping or pushing also needs manipulation data with robot actions and forces.

Is real-world data better than synthetic data for world models?

Real-world data is not better in every case, but it covers what synthetic scenes often miss: cluttered layouts, reflections, changing light, people and wear. Most teams combine both. Synthetic data gives volume and perfect labels, while real captures anchor the model in physical reality and narrow the sim-to-real gap on real robots.

Synthetic data does several things well:

• Perfect ground truth for depth, segmentation and object pose.
• Cheap variation of lighting, textures and object placement.
• Safe rehearsal of rare events.

Real data does other things well:

• Real materials, glare, signage and visual noise.
• Layouts that architects and tenants actually built, not idealized ones.
• Sensor artifacts your robot cameras will meet in deployment.

The hybrid route is real-to-sim: reconstruct a real site as a sim-ready digital twin, export it to NVIDIA Isaac Sim (OpenUSD scene files) or MuJoCo, and randomize conditions on top of real geometry. Policies trained there go back to the real site for testing. Our article on humanoid robot training data from real sites covers that loop for humanoids.

How to build a world model data pipeline in 6 steps

1. Define the target behavior. Write down what the model must predict or do, and where. This sets scene mix and capture height.
2. Capture. Smartphones scale across many contributors and record device tracking poses. 360-degree cameras cover a full field of view in one walk and suit large indoor sites.
3. Pre-process. Filter blurred frames, align timestamps, check pose quality and apply privacy masking before anything else.
4. Reconstruct. Build a dense point cloud, a COLMAP reconstruction, depth maps and a Gaussian splatting model per scene.
5. Annotate and export. Add semantic labels and export sim-ready assets in the formats your simulator expects.
6. Train, test and replicate. Hold out whole scenes or cities for evaluation, train, then compare results in simulation and on the real site.

What to measure along the way:

• Pose error and reprojection error per scene.
• Novel-view quality on frames held out from training.
• Coverage: countries, building types, lighting conditions, floor counts.
• Policy success rate in simulation versus on the real site.

Honest limits: a capture is a snapshot, and furniture moves after it. Consumer devices vary in optics and sensors. Data volumes are large, so plan storage and transfer early.

Checklist before you license a world model dataset

• Are camera poses metric, and how were they computed?
• Which reconstructions ship per scene: point cloud, COLMAP, splats, depth?
• How many countries, cities and building types are covered?
• How is privacy masking done, and is it audited?
• Can you get a pilot subset before a full license?
• Do terms allow commercial training, model release and publication?
• Is custom capture of your own sites available?
• How often does the dataset grow, and how are updates delivered?

How OVER does it

OVER (Over the Reality), a spatial computing company based in Udine, Italy, offers the OverMaps dataset by Over Holding Srl in two variants, OverMaps-Smartphone and OverMaps-360Cam. It totals 171,635 scenes, 86M+ high-resolution images and about 672 TB, with multi-view images with EXIF, metric camera poses from ARKit and ARCore, 3D Gaussian Splatting per scene, COLMAP reconstructions, depth maps, semantic annotations and privacy masking. A curated subset, OverMaps_1k (1,000 scenes, about 3.9 TB), is on Hugging Face, and the dataset grows continuously through Map2Earn contributors.

OverMaps-360Cam covers 2,712 scenes in 29 countries and 125 cities: 500+ hours of dual-fisheye 360-degree video captured with an Insta360 X5, about 60 TB and 2,200 km of walked paths. The same capture method mapped Città Fiera, 150,000 m² in Udine, in about 90 minutes with one operator during opening hours. The OVER platform counts 270,000+ locations mapped, and its visual positioning system API relocalizes a camera at centimeter-level accuracy in under 100 ms.

For teams that need their own sites, OVER runs 3D spatial data for robotics end to end: capture, pre-processing, digital twin reconstruction, annotation, sim-ready export for NVIDIA Isaac Sim (OpenUSD) and MuJoCo, simulation, VLN and VLA model training, policy training for humanoids, quadruped robot dogs and other mobile robots, testing and real-to-sim-to-real replica. Licensing terms are shared on request.

Frequently asked questions

How much does a world model dataset cost?

Cost depends on factors, not a list: number of scenes, data volume, exclusivity, commercial training rights, annotation depth, delivery format and whether you need custom capture of your own sites. Share your target behavior and scene mix to get a quote.

Can I test a world model dataset before licensing it?

Ask for a pilot subset. A small curated sample is enough to check pose quality, formats and fit with your training code before a full license.

Do world model datasets include robot actions?

Scene datasets usually contain camera motion, not robot actions or contact forces. For manipulation you need separate robot or teleoperation data. Scene data is strongest for navigation, visual prediction and simulation environments.

If you are scoping real-world data for a world model or a robot policy, talk to our team about scenes, formats and licensing.