Multimodal feature fusion in VR based on EEG and eye-tracking data for the prediction of customers’ preferences

This project focuses on revolutionizing the e-commerce industry by leveraging multimodal feature fusion within a Virtual Reality (VR) environment.

Overview

This project focuses on revolutionizing the e-commerce industry by leveraging multimodal feature fusion within a Virtual Reality (VR) environment. By integrating and analyzing EEG (Electroencephalography) and eye-tracking data, the system can predict customers’ preferences with a high level of precision, allowing for the creation of personalized and immersive shopping experiences in online retail spaces.

The novel approach of this project combines neuroscientific data from EEG with visual attention metrics from eye-tracking, providing deeper insights into how customers engage with products in VR. By understanding and predicting customer preferences, the project aims to reshape the future of online retail by offering tailored experiences that cater to individual tastes.

Technology and Features

  • EEG Data Acquisition: EEG data is collected using an advanced setup that records brainwave activity while users interact with the VR shopping environment. Differential entropy (DE) and connectivity features are then extracted from the raw EEG signals, providing insights into the user’s cognitive state and emotional responses to various stimuli within the virtual environment.
  • Eye-tracking Data: Eye-tracking data is captured using an HTC Vive Pro headset, which tracks pupil diameter, position sensor data (left and right eye), gaze origin, and gaze direction. This data provides valuable information about what products or areas of the virtual store are capturing the user’s attention and how long they spend looking at particular items.
  • Multimodal Feature Fusion: The core innovation of this project lies in the fusion of EEG and eye-tracking data. By combining these two data streams, the system can achieve a more detailed and nuanced understanding of customer preferences, offering a multimodal approach to predicting user behavior. This allows for the creation of a highly personalized shopping experience, where product recommendations and store layouts can be tailored to the individual’s preferences in real-time.

Key Benefits

  • Enhanced Precision in Predicting Preferences: By integrating both EEG and eye-tracking data, the system can predict customer preferences with greater accuracy compared to traditional single-modality approaches. This leads to a more targeted shopping experience, where users are presented with products that align with their subconscious preferences and interests.
  • Immersive Retail Experience: The use of VR technology allows for an immersive and interactive shopping experience, where customers can virtually browse products, inspect them up-close, and receive real-time recommendations based on their cognitive and visual responses.
  • Real-time Data Processing: The system processes EEG and eye-tracking data in real-time, allowing for immediate adjustments to the virtual shopping environment based on the user’s preferences. This ensures a dynamic experience where user interactions and preferences are continuously analyzed and used to refine recommendations.
  • Applications in E-Commerce: This technology has the potential to transform online shopping by making it more engaging and personalized. Retailers can use the insights gained from EEG and eye-tracking data to create virtual stores that adapt to each customer’s preferences, improving customer satisfaction and conversion rates.

Experimental Setup

The experimental setup involves the use of an HTC Vive Pro headset for eye-tracking data acquisition and EEG sensors for brainwave activity monitoring. The system is designed to seamlessly integrate both data streams for multimodal analysis. This configuration allows researchers to collect and analyze both cognitive and visual engagement metrics simultaneously, providing a comprehensive understanding of user behavior in the VR environment.

This project demonstrates the powerful potential of combining EEG and eye-tracking data in Virtual Reality to predict customer preferences and reshape the e-commerce landscape. By offering highly personalized and immersive shopping experiences, this technology is poised to revolutionize the way customers shop online.