The Future of XR and Metaverse Measurement
The global $28bn Extended Reality (XR) industry of Augmented Reality (AR), Virtual Reality (VR) and Metaverse technologies is forecast to grow to $250bn by 2028 with leading brands and tech companies developing business solutions to compete in this space. It’s a computing platform shift which necessitates questions about the role of XR in people’s lives and how businesses demonstrate value. Specifically, how does the industry measure, analyse and optimise human behaviour in a post-screen world?
Source: Statista XR/AR/VR/MR market size 2021-2028 report 2022
Research into the state of the art of XR data analytics
Gorilla in the room, a specialist XR research and development company, conducted a state of the art research study with 65 companies actively creating XR solutions to discover how AR and VR is currently measured, how companies are demonstrating business results and, most importantly, what type of data metrics the industry needs to grow. The focus of this research is to understand the future of data analytics to adequately measure user behaviour in XR and Metaverse experiences.
Research between September 2021 and June 2022 was conducted in three stages; qualitative interviews with XR experts, desktop research to establish the context of current market data and quantitative surveys to ensure the broadest view possible. Companies covered all sectors and included brands (Jaguar Land Rover), agencies (Ogilvy), platforms (Unity), manufacturers (HTC), developers (nDreams) and media owners (Yahoo). Contributors offered their time freely and, where they granted permission, are referenced at the end of the report.
Special thanks to Jeremy Dalton, Head of Metaverse Technologies at PwC UK and author of Reality Check, for his support.
83% think data analytics are necessary to grow their business
There is clear appetite for better metrics as the industry matures however a knowledge gap exists in the understanding of user behaviour and the business impact of Extended Reality. The common view is that data analytics have not kept pace with XR and Metaverse developments as many data solutions are based on legacy metrics designed for 2D computer and mobile screen interactions which don’t capture the unique nature of 3D experiences.
“It’s vital for our industry to be able to report on the success of campaigns and attribute that value back to the brand.”
“We struggle most of the time to find ROI metrics linked to increased sales”.
Immersion is a key selling point for the industry however few credible metrics exist to truly measure immersion levels as it’s difficult to quantify. Taking this as a starting point, the research explored the metrics being used and how companies pitch AR / VR to attract new clients and grow existing relationships. 3D experiences with Six Degrees of Freedom (6DOF) create a sense of presence which exceeds other forms of digital experiences so there is appetite for XR data analytics to go beyond screen-based web metrics.
“We want to quantify immersion but don’t know the data fields we can use.”
“We need to have more of the data that shows people that XR is more effective than other approaches.”
A definitive gap in the need to provide clients with metrics which prove the value of XR exists, so an audit of current practices was conducted.
Current measurements are broad and based on legacy metrics
Contributors to the study reported on the analytics currently being used – or tools they are aware of – and current practices are broad with many measurement techniques driven by legacy measures. Interestingly, many approaches reflect those used in the consumer research sector. Some metrics are adopted from either consumer research or academic psychological practices, which are frequently misused or misinterpreted. It’s therefore beneficial, along with reporting the metrics being used, to critique the applications of measures and their scalability as a standardised method.
Time
The time spent in an XR experience is the most common single metric. It was reported as indicating both enjoyment and willingness to stay in the experience, or as a measure of experience complexity. For Gaming, the length of time spent and break points were important however, for Education, time could be an indicator of learning speed/task difficulty or confidence in completing a task. This means it does need to be used in addition to other measures to be interpreted meaningfully. Time using an experience was also reported by some as being an indicator of how ergonomically an experience was designed. In other words, time could indicate if the UX was performing as expected.
Dwell Time on specific elements of the XR experience is used and is considered a measure of interest in specific content such as an area of a VR scene or a specific element of an AR experience. Please note: whilst eye-tracking technology uses dwell time metrics, the metrics reported in this study were based on user location and orientation as opposed to eye fixation (as used in eye-tracking).
Google Analytics (or similar)
The most common data capture method is Google Analytics (or similar) which is used by 26% of respondents in this research study to track visits / user numbers, user ID, location, dwell time, clickthrough and social media activity. Time is also tracked by these types of analytics tools however isn’t grouped under Google Analytics as there are some specific solutions, most notably in VR. Overall, this legacy measurement approach doesn’t deliver metrics unique to XR and doesn’t demonstrate the value of 3D experiences above and beyond traditional 2D digital media.
“Industry processes and KPIs just go down old paths of metrics and there’s lack of maturity in use of data collection.”
Bespoke feedback
Specific feedback points coded into the experience (i.e. certain actions are tracked) are used as a test to see when actions were completed and if this action, for example, had been done within a certain time frame. It also indicates which features of an experience were being used, showing where users spent their time and the order of actions. These are unique to each XR project and did not generalise across other XR experiences.
“If an experience has specific triggers we need to know how long it takes the user to notice them and then interact with them.”
Customer survey and polling data (after the event) can be useful to measure overall experiences however they rely too heavily on self-report which is proven – by Behavioural Science – to not capture real behaviour. Self-report can also be unreliable if the influences over behaviour are those where users are unreliable witnesses to their own actions.
“With survey data you can’t get all the answers, as there are some questions that can’t be answered.”
Specific questions coded into the experience to collect feedback on a response at a particular time, or in response to a specific prompt, are unique questions that corresponded to specific actions within that XR experience.
Pre/post measures on XR experience test an ability, skill or knowledge prior to using the XR experience and, afterwards, demonstrating any change in that measure. These tests are tailored towards particular goals, such as learning a specific task or skill, however this kind of feedback isn’t diagnostic so doesn’t measure how and why the XR experience changed behaviour. The main interest was in demonstrating that performance had changed.
Amongst survey questions there are some generalisable questions that are used infrequently. However, these mostly focussed on the specifics of what the task had been designed to do. Hence it was mostly bespoke to feedback action measures and questions related to outcomes that were specific to that experience.
The main aim for companies was demonstrating the value of their experience by focussing on the objectives that the XR experience was designed to address. These are limited, as standardised metrics, so these bespoke metrics can’t be scaled to generate a business case for XR. Targeted data has a role in measuring performance, but there is little diagnostic data showing why the nature of an AR or VR experience was superior at delivering results.
Heatmaps
Tracking 6DOF movement in 3D space is being used by some companies however it’s relatively uncommon even though spatial data was reported as being informative about user behaviour. Data visualisations using heatmaps to represent the magnitude of movement provides clear interpretation of complex data with use cases relevant to all XR sectors. Heatmaps are primarily being used on both AR and VR projects to measure user attention towards specific content to optimise UX and prove effectiveness. Whilst heatmaps are commonly used to measure attention on websites, the complexity of 6DOF data means it’s currently a relatively underdeveloped measurement technique in the Extended Reality industry.
Biometrics
Heart Rate, Galvanic Skin Response, Respiratory Rate and Pupil Dilation are methods used by a small group of companies however usage reflects the kinds of claims used in other sectors. Biometric measures are used commonly within psychological research however the best and most comprehensive definition of what these metrics measure is usually termed as arousal which can be both positive (joy) and negative (fear). The pattern of certain kinds of feedback has been used to measure factors such as attention but specialised equipment is commonly needed for any accurate assessment. The data is commonly ‘noisy’ and hence large data samples are needed for any clear interpretation. In the consumer research sector, these measures have been claimed as measuring a wide number of cognitive and emotional processes but little empirical evidence exists that they measure what some claim they do. Additionally, biometrics is not readily scalable so the least useful of the measures reviewed.
Eye tracking
Eye-tracking is used by a specialist group of companies to provide a strong indicator of visual attention and it’s a successful technique across multiple sectors. Eye-tracking indicates where an experience is being visually attended, the order in which things are viewed and the dwell time on specific elements in the XR experience. Accurate eye-tracking requires specialist equipment for HMDs (Head Mounted Display devices) so it is not readily scalable. Some solutions based on front facing cameras do exist however there are no precise claims for the accuracy beyond those quoted by the manufacturers of the technology, hence any ability to verify efficacy is difficult.
Facial coding
Although this wasn’t mentioned by respondents, facial coding (where facial muscle movements are tracked using AI) is also used by the industry. These data points are similar to front facing cameras used in (non XR) advertising testing. These detect facial responses and infer a reaction however the Scientific American Magazine (2022) points out that these systems only detect muscular facial movements and not the cause. Crucially, there is no linear relationship between facial movement and emotional condition so the accuracy of interpreting internal mental states is questionable. These systems may produce some useful results however the general view is that there is not a 1:1 correlation between facial muscle movements and specific emotions, therefore these systems may only be partially useful.
Voice capture
Additionally, some use cases exist where people describe their experiences however the qualitative feedback needs to be analysed and interpreted. Some attempts at sentiment analysis have been used but these only provide a broad measure of positive or negative responses.
XR community want to better understand user experiences
Our panel of experts and companies taking part in the research cited the measures they need to improve their understanding of user behaviour. As an open question, people weren’t limited to any specific response – as those creating XR experiences are aware of what they’d like to measure – but aren’t necessarily aware of how these measures might be produced. For this reason, we report findings in two categories. Firstly, the metrics which are clearly defined and readily implementable. Secondly, requests for metrics which aren’t as easy to define and where respondents had varying ideas about the practical applications. The second category is therefore open ended and inconclusive about how these measures can be obtained.
Clearly defined metrics with a clear pathway to measurement
Heatmaps and Attention
Attention measures were the most requested metric. This includes heatmaps and data visualisations which indicate attention over time, where user attention was directed and when people looked at specific elements. There were a number of reasons given for this. In Educational and Training, the ability to track if attention was being drawn to the correct place, for example, in a simulation, where attention needs to be directed towards an event that was critical for learning to take place. For others, attention was important as it indicated which parts of an experience needed the most consideration in terms of aesthetics and detail, and which parts are less important as they are commonly overlooked by users. The implication is that attention would correlate with awareness of elements within an experience as well as being a strong indicator of attention.
“You can get heatmaps for website use, but it would be helpful to have the same capability within an XR experience.”
“Where they look at the most, gaze tracking, is important, especially in marketing, and you want to understand attention and feed back to what they experience.”
Telemetry and Navigation
Although there is overlap with requests for attention data, there is a specific need for movement of the device (handset or headset) to track user behaviour in 3D space. Understanding yaw and rotation, the direction or orientation, is considered important to measure a user’s primary visual field. Additionally, metrics on navigation around an object or location as well as direction of travel is seen as valuable. The key objective is to understand whether people can intuitively navigate the experience. NB: although not stated these are taken as an additional implicit request for attention. Telemetry is also identified as a way to understand reactions, interactions and how users move around/through XR experiences. Hand tracking and head position are also mentioned as indicators of correct performance of user tasks.
Tracking responses
Bespoke feedback to assess specific actions within experiences, with reactions measured via an integrated questionnaire, is deemed important and supports current qualitative feedback approaches. As mentioned earlier, whilst this measurement technique is valuable on custom projects, it doesn’t deliver standardised metrics for the industry.
Time, View Count and User ID
Whilst standardised metrics available in Google Analytics (or similar) are commonly used, they rank low in terms of priority metrics which companies need to advance their understanding of user behaviour. This is symptomatic of the data analytics available to AR/VR creators and highlights the importance of progressing industry measurement approaches beyond legacy tools. Please note there is some overlap between tracking responses using bespoke feedback (above) and these analytics tools.
Not clearly defined metrics with unclear measurement method
Engagement
The definition for engagement differed from enjoyment through to interest and emotional reaction. It demonstrates the breadth of XR practitioner backgrounds and the need for the industry to agree a definition, particularly with regards to legacy definitions in digital marketing. Engagement metrics are clearly important and therefore requires further exploration. Engagement has legacy issues with multiple definitions however this study provides some ideas. Engagement, for example, can be measured by the amount of movement in an XR experience based on the volume or distance travelled in a VR environment and a high amount of movement around an AR experience. More work needs to be done to assess how people engage with XR experiences however this study indicates that relatively high movement levels can be associated with high engagement.
Assessing Behaviour
There’s clear desire to assess overall patterns of behaviour in XR experiences however the exact behaviours are not defined and are likely to be different by sector and for each experience. It is, from a cognitive science perspective, possible to assess behaviour with telemetry and attention measures however this requires further development.
Immersion
Interestingly, immersion rarely appears in open responses in the quantitative survey, however was highlighted in expert interviews as an important metric for the immersive tech industry to quantify. This, we believe, is due to the definition and measurement technique remaining a significant challenge. The unique nature of AR, VR and Metaverse experiences requires these phenomenological experiences to be fully understood and is therefore a big question for the industry to discuss.
Ease of use
This measure was prioritised by a small number of XR creators and, whilst it’s not defined, could potentially be solved by other metrics such as attention and telemetry.
XR metrics most likely to drive future business value
What is clear from this study is that – whilst the XR industry have lots of ideas about the metrics which will advance AR, VR and Metaverse measurement – the definitions and methodologies aren’t always consistent. Based on people’s feedback, as well as a review of academic papers and industry case studies, the metrics which are perceived by companies as important to demonstrate future business value were created as a set of closed multiple choice questions. These tailored questions aimed to surface the metrics which are the most important candidates for future development.
Overall, the need for metrics to demonstrate the utility of XR was clear. All respondents in this study stated that their business would benefit from these kinds of analytics with the average respondent selecting at least three of the six options.
Attention
Measuring attention to specific content elements and assessing the content areas which are viewed the most (i.e. what people look at) is important to the XR community. This consolidates the fact that attention measures are also the highest priority in terms of perceived value. This is seen as the best way to assess the greatest level of interest given to content in the experience to optimise content development as it infers which elements are the most engaging. Attention is also highly valuable as a metric to demonstrate business value e.g. brand recognition in the experience.
Navigation
Analysing navigation around the experience, the user journey and the sequence of user events (i.e. how do people move in 3D environments) is also a highly rated metric. This supports telemetry as a measurement technique with a clear route for future XR data analytics. Measuring user position and pathways is essential to understanding how people travel through VR environments and navigate AR objects or portals. Heatmaps are considered the most valuable type of data visualisation to interpret and understand user journeys.
Immersion
Measuring immersion levels remains important but a real challenge for the XR community as this needs to measure an internal mental state. This state of mind isn’t something individuals have the ability to accurately report. Immersion and sense of presence are very similar with presence both a subjective–phenomenal interpretation (i.e. people ‘feel’ like they are in the virtual environment in contrast to the real world) however an objective–functional interpretation of presence is considered the ability to interact. Presence can be applied to VR through telepresence (i.e. people feeling they are transported into a virtual place) or the sensation of presence when AR content is overlayed onto the real world.
In expert interviews, we discussed meaningful ways to measure immersion and many liked the idea of assessing movement in 3D space as a useful measurement. Commonly, the concept is that relatively higher levels of movement compared to a baseline measure would indicate higher levels of immersion.
Surveys
Questionnaires (bespoke feedback) in AR and VR experiences are considered the best way to address specific answers to questions about custom brand or user objectives.
Standardised metrics are required for XR industry to grow
This research study demonstrates a consistent pattern. Companies are creating a broad range of creative solutions for clients and end users however, beyond human interpretation, there is little consensus on the definition of a good AR or VR experience. This is particularly pertinent to the definition and measurement of immersion. Critically, there isn’t a consistent data analytics approach or agreement on metrics which can be applied across companies, sectors and different types of XR experiences.
This is an essential challenge for the industry to compete with traditional forms of media which have established measurement standards to demonstrate ROI. There is therefore a significant gap in the ability of XR creators to produce business results and benchmark one experience against the other. For the XR industry to flourish, standard measures across different experiences are essential for companies to demonstrate the effectiveness of XR as a solution for multiple business challenges.
Attention is highlighted as a strong candidate for the industry to pursue, especially as this metric is commonly used in other industries, most notably Media and Advertising. It’s a strong measure of interest and evidence of cognitive processing with the Attention Economy associated with more positive results in the Media and Advertising industry. There are clear benefits to the XR community in understanding attention better, specifically time spent viewing content, what gets attention first and the priority and order of viewed content. Attention is seen as an important metric across multiple XR sectors, as well as AR and VR formats, as it measures interest in the content, helps optimise user experiences and can be attributed to brand or company objectives. Importantly, this metric can be scaled through current headset and handset technologies.
Navigation is also important across both AR and VR formats to track position, pathway and user journeys in 3D environments. This represents a significant divergence from legacy measures designed for 2D screens as ease of navigation and ergonomics are entirely different behaviours in 6DOF experiences. The relationship between headset or handset devices and the AR or VR content can be tracked using device sensors so this metric can also be scaled across current platforms. Importantly, the relationship between people in shared Metaverse experiences, and their interconnected behaviours, necessitates accurate measurements of their movements in 3D space.
The start of a discussions on the future of XR data analytics
The aim of this study was to assess what XR data analytics are needed – in the context of current methodologies – to highlight the metrics which the industry can communicate the benefits of Extended Reality to existing clients as well as those who are not yet engaged. This is especially pertinent to the growth of the Metaverse – where a persistent simulated world has collective experiences with shared goals – as measuring human behaviour will only accelerate the measurement challenge.
The XR industry is both creative and resourceful in trying to advance the understanding of AR, VR and Metaverse technologies in a post-screen world. The industry is approaching maturity so communicating the benefits of XR and demonstrating business value requires standardised methods of measurement. We hope this report is a starting point for a discussion around what are – and what aren’t – the best methods to demonstrate the value which we know Extended Reality and Metaverse can deliver.
Please contact hello@gorillaitr.com for more information.
Thanks to the XR community who took part in this research
Selection of companies who granted permission to be referenced in this report.
Recent Comments