© Inserm / Hirsch, Philippe
[break]A brain-computer interface enables users to send commands to a computer purely by using their mental activity. BCIs use electroencephalographic (EEG) devices based on electrodes that are placed on the surface of the head, and that record electrical signals corresponding to exchanges of electric current between the neurones. The electrical activity produced by the brain is then analysed in real time and translated into a command sent to the computer or any other automated system thus making it possible to move a cursor to the right or left, simply by mentally imaging the movements a hand would make.
Initiated in 2005 by INRIA and INSERM, the first OpenViBE (2005-2009) project, which was also supported by the ANR, was completed in 2009 with the creation of open-source software that made it possible to easily design, develop and test brain-computer interfaces (http://openvibe.inria.fr). Today, this is recognised software that is used throughout the world. The OpenViBE2 software is a sequel to this initial project and is aimed at exploring these technologies more thoroughly by exploring them in a particularly complex applicational context – that of video games. This is a relatively new field of application for BCIs. But the advent of low-cost EEG helmets has opened the way to such applications.
The purpose of OpenViBE2 is thus to improve the current abilities of BCIs and test their use and their potential in the field of video games.
[break]As a development of the “gesture-controlled” command (joystick, mouse, gamepad), the “mental” command is a new and very promising route in the field of video-gaming. Since the early 2000s, scientists have been tacking the scientific issue of how to incorporate BCI technologies to enable them to interact with virtual environments.
In this context, the aim of the OpenViBE2 project is to identify and use the gamer’s mental state and brain responses to interact with the game and/or adapt the actual content of the video game accordingly. A very original approach suggested in the project was to consider BCIs not as a substitution technique for the classic interfaces (joystick, mouse, gamepad) but rather to consider using the BCIs as a new way of playing that complements the traditional techniques. In this way, the gamer can continue to use a joystick while at the same time or at certain key moments in the game, he/she can also mobilise his/her brain activity for playing.
[break]Whether in respect of diseases and neurological syndromes, the world of entertainment or daily life, OpenViBE2 opens the way to innovative technologies and major interests for human beings. During the course of the OpenViBE2 project, major scientific advances have been obtained in these three fields:
In Neuroscience: identification of new mental activities linked to the attention processes.
In the first leg of the OpenViBE project, the INSERM team known as “Mental Dynamics and Cognition (Dycog)” at the Lyon Neuroscience Research Centre (INSERM/CNRS) participated in the development of software capable of “writing through thought”, thus facilitating communication by people suffering from a motor handicap. OpenViBE2 offers new advances in the field of neurosciences that makes it possible to use interfaces brain-computer for therapeutic purposes in order to improve certain neurological disorders such as attention deficit disorders. Thanks to the device, researchers were able to analyse:
– attention paid to the outside world by measuring in real time and selectively the level of engagement of the cerebral network responsible for seeking specific information in a visual scene.
– attention paid by the user to an internal representation, c’est-à-dire i.e. the level of engagement of the cerebral network responsible for maintaining mental representation.
– The level of distraction of a person by determining the real time in which a person is distracted and for how long.
The principle of virtual reality, used mainly in the serious game ADHD developed thanks to OpenViBE2, enabled researchers to obtain results with respect to attention deficit disorder. The virtual environment is similar to the real environment and the user has to concentrate in order to perform a task that he/she knows from the real world. This virtual training employs the process known as NeuroFeedback in which the user is required to self-regulate his/her mental activity.
In processing signals: the CEA and GIPSA-Lab researchers have found new techniques to better filter and interpret electrical signals from the brain.
These make it possible in particular to eliminate noise interference (especially when it is linked to muscle activities such as facial muscles, blinking or jaw-clenching), and extracting the relevant brain activity as accurately and in as focused a manner as possible. These techniques are very useful in a gaming context, in which gamers may be very mobile.
In virtual reality and man-machine interaction: OpenViBE2 has enable the INRIA researchers to invent new concepts to interact with video games in the most original and effective way and suggest new world firsts:
– The brain-computer “multi-player” interface: the INRIA researchers designed the very first collaborative or competitive gaming application in which the brain activities of two gamers were analysed simultaneously. Both players could play on the same side or in opposition to each other, in a simplified video football game.
– The natural incorporation of BCI into the virtual world: researchers studied how to best incorporate the stimulation necessary for certain brain-computer interfaces based on mental responses, by incorporating visual stimuli into the virtual world. For example, some BCIs “visual flashes” that are recognisable in the user’s brain. These can be incorporated into a video game, for example in the form of butterflies that beat their wings at different frequencies.
– Automatic adaptation of the virtual world to the gamer’s mental state: Finally, researchers have offered radically new approaches in which elements of the virtual environment have been automatically modified depending on the player’s mental state. In a virtual maze game, guides are thus activated automatically if the user presents with too great a mental burden. This work has benefited internationally through the award of several science prizes (Best Paper Award, Eurohaptics 2012, BCI Award 2012 nominee).
[break]The OpenViBE2 project has made it possible to develop numerous proofs of the academic concept and commercial video game prototypes all of them activated by the brain.
Video game manufacturers have worked directly with the INRIA university laboratories, as well as with INSERM, CEA and the GIPSA-Lab, to develop video games based on brain activity. This collaboration has made it possible to better define scientific research initiatives throughout the project and bring all the research together into solutions that are best suited to the technological constraints of the field of application.
During the course of the project, the partners introduced a huge campaign of experiments involving “brain-computer interfaces and video games” run by CHART at the Cité des Sciences in the course of which nearly 400 testers tested the prototypes. The results of these experiments made it possible to highlight the attractions of these technologies to a wider public, and provide the consortium of partners with a large amount of user feedback. A standardisation project was also run by UBISOFT to define the “ideal” helmet best suited to the video-games market.
– MindShooter (INRIA): In this game, the gamer controls a spaceship shown at the bottom of the screen and he/she has to destroy the enemy ships at the top of the screen. Three commands are available: move right, move left, and fire. To activate them, the gamer must concentrate on the part of the ship that corresponds to the action he/she wishes to trigger.
– Brain Invaders (GIPSA-LAb): This game is inspired by the Japanese “Space Invaders” game. Once again, the aim is to destroy the spaceships on the screen, thanks to a mental response that is produced as soon as a rare but expected event occurs.
– Brain Arena (INRIA): This simplified “multi-player” football game enables two users to play together or one against the other using their mental powers.
Three commercial prototypes of video games based on mental activity have also been developed by the consortium.
– Cocoto Brain (Kylotonn Games): A “Casual game” based on mental activity, Entertainment for the Nintendo Wii console. The gamer needs to protect a fairy by preventing all her enemies from getting near her. The gamer has to concentrate on targets positioned over the enemies in order to neutralise them.
– BCI Training Center (Black Sheep Studio), a game of the “mental training” type using the EEG developed by Black Sheep Studio. The game enables the gamer to practice brain-training gaming activities (visual searching, finding a word in a grid) associated with a real-time adaptation of the game, depending on the player’s mental state measured by the EEG device (relaxation and concentration).
– A serious game for the treatment of attention deficit disorder (CLARTE): a prototype is designed for children destined for children suffering from attention deficit hyperactivity disorder (ADHD) has been developed by CLARTE. The virtual environment of this platform is a classroom in which the child suffering from ADHD is invited, for example, to watch a video on a screen in the class, of which the perceived quality is directly linked to his/her mental activity. Thus, if he/she wants to continue to watch the video under optimal conditions, he/she must continue to maintain his/her activity under the conditions indicated by the therapist. Several mini-games are also offered to improve his/her attentional ability using the EEG.
In conclusion, the OpenViBE2 project made it possible to acquire scientific expertise and technological know-how that is unique in the world, concerning the use of this very promising technology in a developing market. Important scientific advances have been obtained in many different fields, such as neuroscience, electrical signal processing from the brain, man-machine interfaces and virtual reality. The prototypes made it possible to illustrate numerous scientific results for the project in actual commercial settings.
The results of the project were capitalised in the form of know-how, methods and software such as the free OpenViBE software that ought to make it possible to have better future control over the opening up of these BCI technologies to the public at large. Discussions are currently taking place concerning the marketing of prototypes in the near future with help of the French start-up MENSIA TECHNOLOGIES.
More than 50 scientific papers have been presented at conferences or published in journals of reference.
The cerebellum is essential for sensorimotor control but also contributes to higher cognitive functions including social behaviors. An international research consortium uncovered how dopamine in the cerebellum modulates social....