TOOLDEX:
An Ego-Exo Human Demonstration Dataset
for Creative Physical Tool Use

Anonymous Authors
Dataset Visualizer Task Table (CSV) Summary JSON
—
trials
—
task signatures
—
distinct tools
—
actions (verbs)
—
support objects
—
deformable objects

Abstract

TOOLDEX is a large-scale dataset of human tool-use demonstrations captured in a shared multi-camera rig. We densely cover the combinatorial space of tool, action, and object: the same action (e.g., scoop, stir, crush) is performed with many different physical tools, so every trajectory measures how humans flexibly repurpose objects to accomplish a goal. Each trial captures 8 synchronized RGB-D cameras, and a large fraction of the dataset ships with hand-optimized 6DoF object poses, per-frame masks, and hand trajectories — enabling policy learning, affordance discovery, and tool-use generalization studies.

Dataset at a glance

Collection spans one shared scene (cutting-board / tabletop) with 8 calibrated RGB-D cameras. Each trial is one human demonstration of a (tool, action, object) triple with natural variation in grasp, approach, and in-plane pose. Key takeaway: for almost every action we record, multiple tools are demonstrated, yielding rich tool-choice diversity per goal.


Tool distribution. Share of trials using each physical tool. Spatulas, forks, and spoons dominate the long tail.
Action distribution. Share of trials per action. flip, scoop, roll, lift, and crush lead the set.
Object distribution. Share of trials per manipulated object / material — includes egg, dough, sand, nuts, naan, tomato sauce, and sponge.

One action, many tools

The signature of this dataset is the diversity of how humans perform the same action. scoop is demonstrated with 7 distinct tools, stir with 6, crush with 5, etc. To prevent class imbalance from confounding the analysis, every task is collected to roughly the same total demonstration length (~1 hour), so trial-count differences do not dominate downstream statistics. This shows up in the tool×action heatmap, the per-action tool-choice bar chart, and the per-task duration bar plot below.


Tool × action heatmap. Each cell is the trial count for a specific (tool, action) pair. Empty cells indicate affordance gaps; dense rows/columns indicate highly reused tools/actions.

Tool-choice diversity per action. For each verb, we list every distinct tool used to realize it in the dataset.

Balanced collection: ~1 hour of demonstration video per task. Total cam-0 video duration aggregated across all trials of each task_dir. The dashed red line marks the average per-task duration (57 min across the 60 tasks). Bars cluster around it because we deliberately equalized collection effort across tasks rather than letting trial counts alone determine total time.

Task distribution

Top task folders by number of trials. The full table and CSV enumerate every task signature and its (tool, action, object) breakdown.


Full task table

Click a column header to sort; filter by tool / action / object in the visualizer.

# task_dir tools actions objects trials
loading…

Automatic Annotation Pipeline & Visualization

Our automatic annotation pipeline produces three coordinated outputs for every recorded trial: per-frame 6 DoF tool pose recovered via multi-view FoundationPose fusion, bimanual MANO hand meshes estimated with WiLoR and refined through joint hand–tool optimization, and dense segmentation masks for the target object — including deformable materials such as dough, sauce, and granular media. All annotations are jointly optimized to ensure physical consistency. The video below visualizes these overlaid annotations on a single demonstration, illustrating how hand pose, tool pose, and deformable object state are tracked in concert throughout the interaction.

Joint hand — tool — deformable-object visualization

3D Gaussian Splatting Novel-View Synthesis
from Sparse Views

From 8 sparse calibrated cameras capturing a single timestep, we reconstruct a per-scene 3D Gaussian field and synthesize novel views around the captured rig. Our top variant achieves MUSIQ 71.85 — the highest no-reference quality on this scene among all 3DGS pipelines we tested.

What is unique vs 2025/2026 research: sparse-view 3DGS papers (DropGaussian, FSGS, CoR-GS, DNGaussian, AnySplat) all aim for a stronger training-time regularizer to combat under-constrained Gaussians. Our finding is complementary — the dominant remaining quality gap is texture, not geometry, and a single forward pass through a generative view-degradation prior (Difix3D, NVIDIA 2024) closes it. We pair an under-fit SparseGS backbone with an under-used post-processing prior, and the composition is greater than its parts.

Ground truth (8 cameras)

 
 

Ours

MUSIQ 63.26 ↑ · NIQE 5.07 ↓ · BRISQUE 30.66 ↓
PIQE 53.36 ↓ · MANIQA 0.308 ↑ · TOPIQ 0.450 ↑

Multi-pane: ours vs four baselines  (drag any of the four seams)

Five renders of the same novel-view sweep stacked into one canvas. Ours takes the central 50% of the canvas by default; drag any of the four internal seams to wipe between adjacent methods. Left to right: InstantSplat · Ours · Sparfels · SplatFormer · SparseGS.

More 3DGS variants (sorted by MUSIQ)

SparseGS + Difix3D

MUSIQ 71.85 ↑ · NIQE 4.71 ↓ · BRISQUE 21.53 ↓
PIQE 48.76 ↓ · MANIQA 0.361 ↑ · TOPIQ 0.552 ↑

InstantSplat + GSFix3D

MUSIQ 62.25 ↑ · NIQE 4.98 ↓ · BRISQUE 39.07 ↓
PIQE 50.56 ↓ · MANIQA 0.336 ↑ · TOPIQ 0.523 ↑

InstantSplat 7k

MUSIQ 55.90 ↑ · NIQE 5.18 ↓ · BRISQUE 48.90 ↓
PIQE 50.44 ↓ · MANIQA 0.240 ↑ · TOPIQ 0.402 ↑

InstantSplat 1k

MUSIQ 52.83 ↑ · NIQE 5.12 ↓ · BRISQUE 40.68 ↓
PIQE 48.98 ↓ · MANIQA 0.226 ↑ · TOPIQ 0.374 ↑

Sparfels

MUSIQ 55.64 ↑ · NIQE 5.41 ↓ · BRISQUE 44.88 ↓
PIQE 56.03 ↓ · MANIQA 0.263 ↑ · TOPIQ 0.412 ↑

VGGT-X

MUSIQ 55.48 ↑ · NIQE 5.52 ↓ · BRISQUE 48.58 ↓
PIQE 51.74 ↓ · MANIQA 0.262 ↑ · TOPIQ 0.409 ↑

SplatFormer

MUSIQ 47.45 ↑ · NIQE 4.66 ↓ · BRISQUE 34.53 ↓
PIQE 42.59 ↓ · MANIQA 0.198 ↑ · TOPIQ 0.335 ↑

SparseGS

MUSIQ 51.03 ↑ · NIQE 4.81 ↓ · BRISQUE 31.05 ↓
PIQE 49.61 ↓ · MANIQA 0.193 ↑ · TOPIQ 0.370 ↑

3DGS results — ours vs baselines

Same 8-camera pestle_crush_nuts_ep5 scene, single timestep, no post-processing on baselines. Direction: MUSIQ↑, NIQE↓, BRISQUE↓, PIQE↓, MANIQA↑, TOPIQ↑.

method MUSIQ↑ NIQE↓ BRISQUE↓ PIQE↓ MANIQA↑ TOPIQ↑
Ours 63.26 5.07 30.66 53.36 0.308 0.450
Sparfels 55.64 5.41 44.88 56.03 0.263 0.412
VGGT-X 55.48 5.52 48.58 51.74 0.262 0.409
InstantSplat 52.83 5.12 40.68 48.98 0.226 0.374
SplatFormer 47.45 4.66 34.53 42.59 0.198 0.335
SparseGS 28.86 6.77 47.24 63.12 0.116 0.187

All scores measured on the same 8-camera pestle_crush_nuts_ep5 scene at the held-out frame-0 timestep.

4D Gaussian Splatting Novel-View Synthesis
from Sparse Views

From 8 sparse calibrated cameras capturing a dynamic scene (10 s · 30 fps), we reconstruct a time-aware Gaussian field and synthesize a novel-view trajectory of equal length. Our top variant reaches MUSIQ 72.03 — the first 4DGS pipeline on this scene to exceed the strongest 3DGS-only result (71.85).

What is unique vs 2025/2026 research: recent 4DGS work (4DGaussians/HexPlane CVPR 2024, Spacetime Gaussians CVPR 2024, MEGA ICCV 2025, Faster-GS CVPR 2026) focuses on architectural changes — learned deformation fields, per-Gaussian polynomial trajectories, entropy-constrained compression, accelerated optimization. None target a key empirical observation: per-frame InstantSplat produces images whose artifact distribution exactly matches what generative view-fixers were trained to remove. We show that a per-frame backbone + an ordered diffusion stack (Difix3D→GSFix3D) at 1080p is +1.46 MUSIQ better than the reverse order, and the full pipeline beats every 4DGS baseline by 8–15 MUSIQ on the same scene.

Ground truth (8 cameras, time-sliced, 10 s @ 30 fps)

 
 

Ours

MUSIQ 72.03 ↑ · NIQE 4.50 ↓ · BRISQUE 21.88 ↓
PIQE 38.33 ↓ · MANIQA 0.412 ↑ · TOPIQ 0.561 ↑

Side-by-side: ours vs InstantSplat  (drag the slider to wipe)

Same novel-view trajectory; left half: ours, right half: InstantSplat baseline. MUSIQ 72.03 vs 55.44.

InstantSplat + GSFix3D + Difix3D

MUSIQ 70.57 ↑ · NIQE 4.63 ↓ · BRISQUE 12.92 ↓
PIQE 27.02 ↓ · MANIQA 0.393 ↑ · TOPIQ 0.531 ↑

InstantSplat + Difix3D

MUSIQ 63.48 ↑ · NIQE 5.44 ↓ · BRISQUE 19.45 ↓
PIQE 31.20 ↓ · MANIQA 0.326 ↑ · TOPIQ 0.455 ↑

InstantSplat + GSFix3D

MUSIQ 62.92 ↑ · NIQE 4.31 ↓ · BRISQUE 28.81 ↓
PIQE 38.10 ↓ · MANIQA 0.316 ↑ · TOPIQ 0.506 ↑

InstantSplat only

MUSIQ 55.44 ↑ · NIQE 5.21 ↓ · BRISQUE 39.05 ↓
PIQE 49.62 ↓ · MANIQA 0.232 ↑ · TOPIQ 0.392 ↑

4DGS results — ours vs baselines

Same 8-camera redrubberspatula_lift_egg_59 scene (300 frames @ 30 fps, novel-view path [0,4,7,5,6,2,3,1]). Baselines run without any of our diffusion / 1080p / stack add-ons. Direction: MUSIQ↑, NIQE↓, BRISQUE↓, PIQE↓, MANIQA↑, TOPIQ↑.

method MUSIQ↑ NIQE↓ BRISQUE↓ PIQE↓ MANIQA↑ TOPIQ↑
Ours 72.03 4.50 21.88 38.33 0.412 0.561
InstantSplat 55.44 5.21 39.05 49.62 0.232 0.392

Same 8-camera scene; the green row is our full pipeline — the first 4DGS result on this scene to exceed 71.85 (the 3D-only ceiling).

Benchmarking VLMs’ Physical Tool-use Capabilities

We benchmark VLMs at three levels of physical tool-use reasoning. For each example, the model receives the first frame from a top-view RGB camera and a natural-language instruction such as “stir the coffee.” The evaluation set contains 160 manually verified demonstrations spanning 32 task variants.

Tool selection tests whether a model understands which visible objects can physically perform the requested action. The model returns the names of all plausible tools: for example, a spoon, ladle, or scoop could each be proposed for “scoop the nuts.” Because several objects may be functionally valid, multiple answers are allowed.

Spatial grounding tests whether the model can connect its tool choice to the correct object instance in the image. For every selected tool, the model predicts a rectangular bounding box. We compute intersection over union (IoU) only when the predicted tool category matches the demonstrated tool, separating localization errors from semantic selection errors.

Behavioral planning asks how the tool should move after it has been identified. To isolate motion planning from selection, the model is explicitly given the demonstrated tool name along with the image and instruction. It outputs an ordered sequence of 2D image coordinates describing a high-level path from the tool to and through the interaction region—for example, approaching a cup and tracing a stirring motion. “Successfully evaluated” is the percentage of responses that can be parsed and scored as valid trajectories.

Predictions are compared with two ground-truth paths. The tool-center trajectory projects the center of the tool through time. The correspondence-aware trajectory instead tracks a point on the functional part of the tool, such as a spoon tip, so it better represents the intended interaction region. Mean L2 measures average pointwise pixel error and is sensitive to the progression of the motion; Chamfer-L2 compares the overall sets of path points and emphasizes path shape. Lower values indicate closer agreement with the human demonstration.

Tool selection and grounding

Tool accuracy measures demonstrated-tool matches; BBox IoU measures localization for semantically matched predictions. Best results are shown in bold.

Model Tool Acc. BBox IoU
GPT-5-mini 0.662 0.300
GPT-5.5 0.738 0.890
Claude-Sonnet-4.6 0.800 0.641
Claude-Opus-4.7 0.800 0.564
Gemini-3-Flash 0.756 0.050
Gemini-3.1-Pro 0.781 0.010
Qwen-3.5-9B 0.557 0.003
Qwen-3.6-27B 0.638 0.004

Trajectory prediction

Results across 160 selected trajectories. Distance metrics are reported in pixels; higher is better for successful evaluation and lower is better for path errors.

Model Successfully
evaluated		 Tool-center
Mean L2		 Tool-center
Chamfer-L2		 Corr.-aware
Mean L2		 Corr.-aware
Chamfer-L2
GPT-5.6 Sol. 90.0% 241.05 113.45 237.55 98.05
Gemma 4 31B 86.9% 259.96 104.17 269.73 107.31
Gemini Robotics ER 77.5% 230.37 91.91 223.15 74.44
Claude Fable 5 71.3% 263.33 146.37 234.89 102.61
Qwen 3.6 27B 37.5% 244.88 128.44 228.61 103.71
Tool-grounding predictions from vision-language models across six physical manipulation tasks
Tool selection and spatial grounding across tasks. We compare model-predicted tool labels and bounding boxes on six representative manipulation tasks: collecting sand, flattening dough, flipping naan, spreading tomato sauce, stirring coffee, and sweeping nuts. Bounding boxes are color-coded by model: brown for GPT-5.5, teal for Claude Sonnet 4.5, purple for Gemini 3.1 Pro, and pink for Qwen3.6-27B, green box marks the ground-truth tool.
Ground-truth and model-predicted tool trajectories for a representative manipulation task
Qualitative comparison of behavioral planning prediction results. Teal denotes the centered ground-truth trajectory, purple denotes the correspondence-aware ground-truth trajectory and orange are predicted trajectories. For the task “scoop cashews from a deep bowl using an ice cream scooper,” most evaluated models correctly localize the scooper, cashews, and bowl. However, their predicted trajectories are often overly smooth or idealized compared with the more irregular motion observed in the ground-truth trajectory.

Robot Physical Tool-Use from Human Demonstrations

Experiment Setup. To evaluate whether TOOLDEX provides robot-transferable demonstrations for dexterous policy learning, we test two mimic-style reinforcement learning settings that both use TOOLDEX trajectories as reference signals. ManipTrans adapts ManipTrans and trains a residual policy for an Inspire hand in IsaacGym. Arm+Hand RL is an independently implemented mimic RL pipeline that jointly controls a Sharpa hand and a Franka FR3 arm in IsaacLab. The policies are trained with an identical pipeline across all tasks without task-specific tuning. Unlike ego-only datasets where hand–tool trajectories must be estimated from a single noisy viewpoint, TOOLDEX's multi-view-fused annotations can be consumed directly without additional refinement. We select demonstrations from 6 tasks spanning 4 action categories (stir, scoop, crush, collect) with 254–600 frames (≈4–10 s at 60 Hz). After training, we evaluate each policy for 200 episodes with randomized start frames.

Per-task success rate on 6 tool-use tasks

Success rate (%): the object reaches within 5 cm of the demonstration's final pose under randomized start frames. ManipTrans and Arm+Hand RL report mean ± std over 3 seeds; Kinematic retarget (open-loop replay of the reference) and BC (behavior cloning of the reference) are single-run reference baselines. Best per task in bold.

Task Kinematic retarget BC ManipTrans Arm+Hand RL
Stir coffee (spatula) 7.0 1.6 52.2 ± 2.2 72.7 ± 0.8
Scoop nuts (ladle) 22.2 9.1 62.9 ± 0.4 66.8 ± 1.3
Crush banana (mallet) 10.2 2.6 53.0 ± 1.9 55.8 ± 0.2
Scoop nuts (scooper) 31.0 9.8 45.3 ± 2.0 31.3 ± 0.6
Crush nuts (pestle) 7.0 1.9 16.8 ± 0.2 46.7 ± 2.4
Collect sand (squeegee) 6.9 2.6 12.6 ± 0.7 57.5 ± 1.9
Average 14.0 4.6 40.5 55.1

Simulation and real-robot rollouts — squeegee

The Arm+Hand RL policy sweeping with a squeegee: executed in simulation (left) and deployed on the real Sharpa hand mounted on a Franka FR3 arm (right).

Simulation

Real robot

Real robot

Human trajectory diversity

For each task with hand-optimized annotations, we render a representative trial as a skeleton-overlay video — MANO hand joints (21 per hand), bones, and a fading 60-frame wrist tail trace the human trajectory through the action. A featured worked example is shown below at full size; the remaining tasks follow as small tiles. Click any video to play.


Featured trial: fork_flip_egg / 20260205_bigwoodenspatula_flip_eggtoy_in_blackpan_1. Left/right MANO hand joints projected into camera 0; fingers colored per group (thumb = red, index = green, middle = blue, ring = yellow, pinky = magenta); wrist dot with a 60-frame fading tail traces the trajectory. Detection-failure sentinels and failed pose-fits are masked out; remaining per-sample jitter is smoothed by a Hampel filter (window 7, 3σ) before drawing.

loading trajectory overlays…

Interactive Dataset Visualizer

Browse a stratified random subset of trials. Filter by tool, action, or object to see how humans solve the same task in different ways.

Open the visualizer

BibTeX

@misc{tooldex2026,
    title  = {TOOLDEX: An Ego-Exo Human Demonstration Dataset for Creative Physical Tool Use},
    year   = {2026},
}