CORTEXR, the XR data analytics company, has published results of a global study into Extended Reality and Metaverse measurement. Research shows the industry is maturing quickly but the computing platform shift necessitates questions about the role of XR in people’s lives and how businesses demonstrate value. Specifically, how does the industry standardise Extended Reality measurement with data analytics to optimise human behaviour in a post-screen world?
83% think XR data analytics are necessary to grow their business
There is clear appetite for standardised XR data analytics as the industry matures however a measurement gap exists between the understanding of user behaviour and the business impact of Extended Reality. Most people know that data analytics have not kept pace with creative and technical developments as they’re mainly based on legacy metrics designed for computer mouse and mobile touchscreens, which don’t capture the unique nature of 3D experiences.
“It’s vital for our industry to be able to report on the success of campaigns and attribute that value back to the brand.”
“We struggle most of the time to find ROI metrics linked to increased sales.”
“We want to quantify immersion but don’t know the data fields we can use.”
“We need to have more of the data that shows people that XR is more effective than other approaches.”
Current XR measurements are broad and based on legacy metrics
This XR and Metaverse research uncovers data analytics currently being used – or tools people are aware of – and current practices are broad with many measurement techniques driven by legacy measures. Interestingly, many approaches reflect those used in the consumer research sector, with metrics adopted from either consumer research or academic psychological practices, which are frequently misused or misinterpreted.
Time
Time spent in an XR experience is the most common single metric, with people saying it indicated both enjoyment and willingness to stay in the experience, or as a measure of experience complexity. For Gaming, the length of time spent and break points were important however, for Education, time could be an indicator of learning speed/task difficulty or confidence in completing a task. Time therefore needs to be combined with other measures to be interpreted meaningfully.
Dwell Time on specific elements of an XR experience is considered a measure of interest in specific content such as an area of a VR scene or a specific element of an AR experience. It’s important not to confuse this with eye-tracking technology, which also uses Dwell Time metrics, which is based on eye fixation measures. More on eye-tracking later.
Google Analytics (or similar)
The most common data capture method is Google Analytics (or similar) which is used by 26% of respondents in this research study to track visits / user numbers, user ID, location, dwell time, clickthrough and social media activity. Time is also tracked by Google Analytics however it’s separated out as there are some specific solutions, most notably in VR. This legacy approach doesn’t deliver metrics unique to XR and Metaverse measurement and doesn’t demonstrate the value of 3D experiences above and beyond traditional 2D digital media.
“Industry processes and KPIs just go down old paths of metrics and there’s lack of maturity in use of data collection.”
Bespoke feedback
Specific feedback points coded into the experience (i.e. certain actions were tracked) are used as a test to see when actions were completed and if this action, for example, had been done within a certain time frame. It also indicates which features of an experience were being used, showing where users spent their time and the order of actions. These are unique to each XR project so can’t be standardised across other XR experiences.
“If an experience has specific triggers we need to know how long it takes the user to notice them and then interact with them.”
Customer survey and polling data (after the event) can be useful to measure overall experiences however they rely too heavily on self-report which is proven – by Behavioural Science – to not capture real behaviour. Self-report can also be unreliable if the influences over behaviour are those where users are unreliable witnesses to their own actions.
“With survey data you can’t get all the answers, as there are some questions that can’t be answered.”
Specific questions coded into the experience to collect feedback on a response at a particular time, or in response to a specific prompt, are unique questions that corresponded to specific actions within that XR experience. Pre/post XR measurement of experiences to test an ability, skill or knowledge prior to using the XR experience and, afterwards, demonstrating any change in that measure. These tests are tailored towards particular goals, such as learning a specific task or skill, however this kind of feedback isn’t diagnostic so doesn’t measure how and why the XR experience changed behaviour. The main interest is in demonstrating that performance had changed.
Amongst survey questions there are some generalisable questions that are used infrequently. However, these mostly focus on the specifics of what the task had been designed to do. Hence it’s mostly bespoke to feedback action measures and questions related to outcomes that were specific to that experience. The main aim for companies is demonstrating the value of their experience by focussing on the objectives that the XR experience was designed to address. These are limited, as standardised metrics, so these bespoke metrics can’t be scaled to generate a business case for XR. Targeted data showing performance has a role, but there is little diagnostic data showing why the nature of an XR experience was superior at delivering results.
Heatmaps
Tracking 6DOF movement in 3D space – the solution which CORTEXR delivers – is still relatively uncommon even though spatial data is reported as being informative about user behaviour. Data visualisations using heatmaps to represent the magnitude of movements provides clear interpretation of complex data with use cases relevant to all XR sectors. Heatmaps are primarily being used on both AR and VR projects to measure user attention towards specific content in order to optimise UX and prove effectiveness. Whilst heatmaps are commonly used to measure attention
on websites, applying complex AR and VR data analytics is an underdeveloped XR and Metaverse measurement approach in the industry.
Biometrics
Heart Rate, Galvanic Skin Response, Respiratory Rate and Pupil Dilation are methods used by a small group of companies however the usage reflects the kinds of claims used in other sectors. Biometric measures are used commonly within psychological research however the best and most comprehensive definition of what these metrics measure is usually termed as arousal which can be both positive (joy) and negative (fear). The pattern of certain kinds of feedback has been used to measure factors such as attention but generally specialised equipment is commonly needed for any accurate assessment. The data is commonly ‘noisy’ and hence large data samples are needed for any clear interpretation. In the consumer research sector, these measures have been claimed as measuring a wide number of cognitive and emotional processes but little empirical evidence exists that they measure what some claim they do!
Eye tracking
Eye-tracking is used by small number of companies to provide a strong indicator of visual attention and it’s a successful technique across multiple sectors. Eye-tracking indicates where an experience is being visually attended, the order in which things are viewed and the dwell time on specific elements in the XR experience. Accurate eye-tracking requires specialist equipment for HMDs so it is not readily scalable. Some solutions based on front facing cameras do also exist however there are no precise claims for the accuracy beyond those quoted by the manufacturers of the technology, hence any ability to verify efficacy is difficult.
Facial coding
Although this wasn’t mentioned in the study, facial coding (where facial muscle movements are tracked using AI) is used by some companies. These data points are similar to front facing cameras used in (non XR) advertising testing. These detect facial responses and infer a reaction however the Scientific American Magazine (2022) points out that these systems only detect muscular facial movements and not the cause. Crucially, there is no linear relationship between facial movement and emotional condition so the accuracy of interpreting internal mental states is questionable. These systems may produce some useful results however the general view is that there is not a 1:1 correlation between facial muscle movements and specific emotions.
XR community want to better understand user experiences
The panel of experts and companies taking part in this XR and Metaverse research highlighted measurement methods they need to improve understanding of user behaviour. Firstly, the metrics which are clearly defined and readily implementable. Secondly, requests for metrics which aren’t as easy to define and where people had varying ideas about the practical applications. The second category is therefore inconclusive in terms of practical use of these measures.
Clearly defined metrics with a clear pathway to measurement
Heatmaps and Attention
Attention measures were the most requested metric. This includes heatmaps and data visualisations which indicate attention over time, where user attention was directed and when people looked at specific elements. There were a number of reasons given for this. In Educational and Training, the ability to track if attention was being drawn to the correct place or where attention needs to be directed towards an event that was critical for learning to take place. In other sectors, attention is important to indicate which parts of an experience needed the most consideration in terms of aesthetics and detail, and which parts are less important as they are commonly overlooked by users. Attention therefore correlates with awareness of elements within an experience as well as being a strong indicator of user attention.
“You can get heatmaps for website use, but it would be helpful to have the same capability within an XR experience.”
“Where they look at the most, gaze tracking, is important, especially in marketing, and you want to understand attention and feed back to what they experience.”
Telemetry and Navigation
Measuring movement of the device (handset or headset) to track user behaviour in 3D space is important. Understanding yaw and rotation, the direction or orientation, is considered essential to measure a user’s primary visual field. Additionally, metrics on navigation around an object or location as well as direction of travel is seen as valuable. The key objective is to understand whether people can intuitively navigate the experience. Telemetry can also understand reactions, interactions and how users move around/through XR experiences.
Tracking responses
Bespoke feedback to assess specific actions within experiences, with reactions measured via an integrated questionnaire, supports current qualitative feedback approaches. As mentioned earlier, whilst this measurement technique is valuable on custom projects, it doesn’t deliver standardised metrics for the industry.
Time, View Count and User ID
Whilst standardised metrics available in Google Analytics are commonly used, they rank low in terms of priority metrics which companies need to advance their understanding of user behaviour. This is symptomatic of the data analytics available to AR/VR creators and highlights the importance of progressive data analytics like CORTEXR which goes beyond legacy tools.
Not clearly defined metrics with unclear measurement method
Engagement
The definition of engagement, depending on who you speak to, differs from enjoyment through to interest and emotional reaction. This demonstrates the breadth of XR practitioner backgrounds and the need for the industry to agree a definition which doesn’t go down the same path as legacy definitions in digital marketing. Engagement, for example, could be measured by the amount of movement in an XR experience based on the volume or distance travelled in a VR environment and magnitude of movement around an AR experience. We have live tests to assess how people engage with XR experiences to see if high movement levels can be associated with high engagement.
Assessing Behaviour
There’s clear desire to assess overall patterns of behaviour in XR experiences however the exact behaviours are not defined and are likely to be different by sector and for each experience. It is, from a cognitive science perspective, possible to assess behaviour with navigation and attention measures.
Immersion
Interestingly, immersion was rarely mentioned in this study, but we think it’s an essential benchmark metric for the immersive tech industry to quantify. The unique nature of AR, VR and Metaverse experiences requires these phenomenological experiences to be fully understood which is why CORTEXR launched our Immersion Index after 3 years of R&D and live tests with beta customers.
Ease of use
This measure was prioritised by a small number of XR creators, probably because it’s a subjective definition, however the suite of metrics provided by CORTEXR allow you to understand ease of use and optimise experiences.
XR metrics most likely to drive future business value
What is clear from this XR and Metaverse research is that – whilst the industry have lots of ideas about the metrics which will advance AR, VR and Metaverse measurement – the definitions and methodologies aren’t always consistent. A set of closed multiple choice questions to surface the metrics which are the most important to the XR community did however highlight attention and navigation as standardised metrics.
Attention
Measuring attention to specific content elements and assessing the content areas which are viewed the most (i.e. what people look at) is important to the XR community. This consolidates the fact that attention measures are also the highest priority in terms of perceived value. This is seen as the best way to assess the greatest level of interest given to content in the experience to optimise content development as it shows which elements are the most engaging. Attention is also highly valuable as a metric to demonstrate business value e.g. brand activity in the experience.
Navigation
Analysing navigation around the experience, the user journey and the sequence of user events (i.e. how do people move in 3D environments) is also a highly rated metric. This supports telemetry as a measurement technique with a clear benefit to the XR industry. Measuring user position and pathways is essential to understanding how people travel through VR environments and navigate AR objects or portals. Heatmaps are considered the most valuable type of data visualisation to interpret and understand user journeys.
Immersion
Understanding immersion involves measuring an internal mental state. Immersion and sense of presence are very similar with presence a subjective–phenomenal interpretation (i.e. people ‘feel’ like they are in the virtual environment in contrast to the real world) and an objective–functional interpretation of presence is the ability to interact. Presence in VR is telepresence (i.e. people feeling they are transported into a virtual place) and the sense of presence in AR is effect of content overlayed onto the real world. Our Immersion Index algorithms compare movement within a particular project to baseline databases to score the overall level of immersion.
Surveys
Questionnaires (bespoke feedback) in AR and VR experiences are considered the best way to address specific answers to questions about custom brand or user objectives.
Standardised metrics are required for XR industry to grow
This global XR and Metaverse research demonstrates a consistent pattern. Companies are creating a broad range of creative solutions for clients and end users however, beyond human interpretation, there is little consensus on the definition of a good AR or VR experience. Critically, there isn’t a consistent data analytics approach or agreement on metrics which can be applied across companies, sectors and different types of XR experiences.
This is an essential challenge for the industry to compete with traditional forms of ‘media’ which have established measurement standards to demonstrate ROI. There is therefore a significant gap in the ability of XR creators to produce business results and benchmark one experience against the other. For the XR industry to flourish, standard measures across different experiences are essential for companies to demonstrate the effectiveness of XR as a solution for multiple business challenges.
Attention is highlighted as a strong candidate for the industry, especially as this metric is commonly used in other industries, most notably Media and Advertising. It’s a strong measure of interest and evidence of cognitive processing with the ‘Attention Economy’ associated with more positive results in the Media and Advertising industry. There are clear benefits to the XR community in understanding attention better, specifically time spent viewing content, what gets attention first and the priority and order of viewed content. Attention is seen as an important metric across multiple XR sectors, as well as AR and VR formats, as it measures interest in the content, helps optimise user experiences and can be attributed to brand or company objectives. Importantly, this metric is scaled through current technologies using CORTEXR.
Navigation is also important across both AR and VR formats to track position, pathway and user journeys in 3D environments. This represents a significant divergence from legacy measures designed for 2D screens as ease of navigation and ergonomics are entirely different behaviours in 6DOF experiences. The relationship between headset or handset devices and the AR or VR content can be tracked using device sensors so this metric is also scalable across current platforms with a CORTEXR plug-in.
Extended Reality measurement study spotlights the metrics that matter
A state-of-the-art research study with 65 companies creating XR solutions summarises how AR and VR measurement has been approached, how companies demonstrate business results and what type of data metrics the industry needs to grow. The future of XR data analytics is the core mission of CORTEXR, it’s a long overdue debate within the XR community and the results of this XR and Metaverse research highlight the solution to measuring user behaviour in Extended Reality.
The research was conducted in three stages; qualitative interviews with XR experts, desktop research to establish the context of current market data and quantitative surveys to ensure a broad perspective. The XR community taking part included brands (e.g. Jaguar Land Rover), agencies (e.g. Ogilvy), platforms (e.g. Unity), manufacturers (e.g. HTC), developers (e.g. nDreams) and media owners (e.g. Yahoo) so it’s a credible snapshot from the industry.
The start of a discussions on the future of XR data analytics
The aim of this XR and Metaverse research was to assess what data analytics are needed – in the context of current methodologies – to highlight the metrics which the industry think are the best way to communicate the benefits of Extended Reality to existing clients as well as those who are not yet engaged. This is especially relevant to developments in the Metaverse – where a persistent simulated world has collective experiences with shared goals – as measuring human behaviour in these experiences is essential to grow the industry.
The XR industry is both creative and resourceful in trying to advance the understanding of AR, VR and Metaverse technologies in a post-screen world. The industry is approaching maturity so communicating the benefits of XR and demonstrating business value requires standardised methods of measurement. Visit CORTEXR to find out more and get in touch at contact@cortexr.com to join our community of XR companies advancing the metrics of human behaviour in XR.
