Conversational Rule Creation in XR: User’s Strategies in VR and AR Automation

Carcangiu, Alessandro;Manca, Marco;Mereu, Jacopo;Santoro, Carmen;Simeoli, Ludovica;Spano, Lucio Davide

2025-01-01

---

Abstract

Rule-based approaches allow users to customize XR environments. However, the current menu-based interfaces still create barriers for end-user developers. Chatbots based on Large Language Models (LLMs) have the potential to reduce the threshold needed for rule creation, but how users articulate their intentions through conversation remains under-explored. This work investigates how users express event-condition-action automation rules in Virtual Reality (VR) and Augmented Reality (AR) environments. Through two user studies, we show that the dialogues share consistent strategies across the interaction setting (keywords, difficulties in expressing conditions, task success), even if we registered different adaptations for each setting (verbal structure, event vs action first rules). Our findings are relevant for the design and implementation of chatbot-based support for expressing automations in an XR setting.