Code in Space: Redefining Tech Creation with AI and XR - The JetBrains Blog

JetBrains Research

Research is crucial for progress and innovation, which is why at JetBrains we are passionate about both scientific and market research

About JetBrains Research

Research<br>Code in Space: Redefining Tech Creation with AI and XR

Ilya Zakharov<br>Ekaterina Koshchenko<br>Agnia Sergeyuk

The main message

Advances in AI are already single-handedly changing how we interact with technologies. This moment might become the first interaction revolution in roughly 60 years , since the mouse-and-windowing paradigm crystallized in the 1970s and 1980s. For us at JetBrains, the most interesting part of this interaction is related to code. LLMs have introduced the conversational format to software engineering. Something that might appear no less consequential than AI several years from now is extended reality (XR) hardware, and how it is (slowly) becoming good enough to be taken seriously as a working medium. XR is relevant for AI not merely as a more immersive output device; it is, even more importantly, an unprecedentedly rich input surface: using gaze, hands, voice, head pose, body posture, spatial context, and even physiological signals. Altogether, with all these new types of data, it enables a more efficient, focused, and personalized multimodal human-AI experience. Crucially, all these inputs are available without a dedicated lab.

When the input richness of XR is paired with modern AI, the result is a qualitatively new interaction substrate, one we’re just starting to figure out how to use. Our Human-AI eXperience (HAX) team at JetBrains Research is now studying what this means specifically for tech creators. By tech creators, we mean software engineers, of course, but also hardware engineers, UI/UX designers, and data scientists, as the question is broader than any one role.

What we asked

To anchor our exploration of what XR can bring to HAX empirically, we conducted semi-structured expert interviews. Thirteen senior researchers and practitioners from leading academic institutions and industry labs (including groups at Cambridge, Aarhus, Stuttgart, and Meta) participated. We analyzed these interviews using the classic Braun and Clarke’s (2006) six-phase approach. The interviews continued until we reached thematic saturation: when three consecutive interviews introduced no new parent themes.

We were interested in understanding:

1.What are the needs and recurring pain points of tech creators that multimodal AI-XR tools could plausibly address?

2.Which multimodal interaction techniques are most promising for addressing those needs, and what might the resulting tooling look like?

3.What are the main constraints – technical, cognitive, organizational – for building such tools today?

What we found, in brief

We’ve identified more than 150 topics of interest from the answers to our interview questions. All these topics were then clustered (first independently by two of the authors and then updated until a consensus was reached) into five overarching themes.

AI-XR consolidated codebook — tabbed

The initial codes and thematic clusters of AI-XR for multimodal Human-AI Experience

156 consolidated codes across 5 themes

Input & Interaction 34<br>AI Context Layer 33<br>Adoption Barriers 38<br>Creation & Work 31<br>Ethics & Control 20

Theme 1<br>The input problem and new interaction paradigms

How do humans communicate intent to AI-XR systems?

New interaction paradigms for spatial computing<br>Future of interactions<br>Natural interaction<br>Multimodal context and input<br>No preferences in multimodality<br>New inputs and UI placement<br>Matching input with the task<br>Explicit and implicit interactions<br>AI-based hand and gesture tracking<br>AI-based voice control<br>AI-based body language<br>AI-based haptics and tactile feedback<br>Gaze control<br>No controllers for XR<br>Embodied interaction<br>Personalization of inputs<br>Spatial reasoning<br>Wearables for XR<br>AI-XR communication and telepresence<br>Switching perspectives for communication<br>Human communication-inspired interaction<br>Better UI<br>Visual overlaying<br>Mobile/on-the-go interaction<br>Flexible interaction patterns<br>Feeling depth in XR<br>Audio is underused<br>Vocalization will not be enough<br>Lack of freedom for interaction<br>Gorilla arm / interaction fatigue<br>Midas touch problem<br>"No interface" / zero-UI paradigm<br>Constraints can aid efficiency<br>2D metaphors are poor fits for 3D

Theme 2<br>AI as the contextual layer

Can AI make XR environments understand and adapt to you?

AI as a context-aware co-creator and mediator<br>Agentic AI<br>AI for understanding context/user<br>Inherent personalization of AI<br>Personalized XR through AI<br>AI lacking personal/user context<br>AI for content creation<br>Communicating/decoding intent<br>AI for disambiguation<br>Immediate interactive context/content<br>Interactive environment<br>Directing attention<br>Attention as a scarce resource<br>Digital twinning<br>Semantic graphs from visual scenes<br>Environmental sensing and physical-world understanding<br>Multimodal...