top of page

When Multimodality Hurts: The Effect of Language Ambiguity on Tabletop Manipulation Policies

Ethan Tian, Ivy He
21/11/26

We study how language ambiguity, scene complexity, input modality, instruction type, and object-set size affect target-slot prediction in simulated tabletop manipulation. A modality-configurable behavioral cloning classifier receives wrist-camera images, table-camera images, instruction text, or their combinations. Accuracy measures agreement with the recorded target slot, not successful grasp execution or physical manipulation. When averaged within each object-set size, full multimodal input is not the best configuration. Scene complexity has a significant but non-monotonic effect, while the planned tests do not support a consistent ambiguity effect or ambiguity–complexity interaction. Accuracy decreases from 61.11% with 3 objects to 49.77% with 5 and 2.78% with 9, but sparse per-slot coverage confounds the 9-object result. Failure analysis identifies similar-object confusions, spatial-relation errors, and instruction-valid alternatives penalized by exact-target scoring.

bottom of page