Brain2Qwerty: What it takes to turn brain activity into text 

For people who have lost the ability to speak or move, restoring communication is a major goal of brain-computer interface research. Jean-Rémi King and his colleagues are exploring whether progress towards that goal can be made using brain recordings taken without surgery. 

In our latest Spotlight, King, a CNRS researcher who leads Meta’s Brain & AI team, joins MEGIN’s Nicole Brough to discuss Brain2Qwerty. The project combines magnetoencephalography (MEG) with AI to decode text from brain activity recorded while people type. The ambition is substantial, but the experiment begins with something familiar: a keyboard. 

Typing gives researchers a way to connect brain activity with a specific action. As King explains, “we know precisely the moment at which they press each button.” Those repeated, timed events provide examples from which an AI system can learn. 

Participants physically type during the experiments. Developing a communication tool for people unable to do so remains a future research goal. Choosing how to record that activity was itself part of the investigation. 

“Well, we actually tried both EEG and MEG. We weren’t sure which one would be better.” 

Both methods offered the fast temporal resolution needed to study individual keystrokes. What interested the team was how their relative performance changed as the AI developed. King describes MEG’s advantage over EEG becoming more pronounced as the researchers introduced deeper, more complex models. 

The finding points to the importance of studying the recording technology and the decoding method together. In this experiment, the choice of brain signal mattered increasingly as the models became more sophisticated. 

That work also depended on experimental expertise. King describes collaborating with Svetlana Pinet at the Basque Center on Cognition, Brain and Language, whose research investigates the neural basis of typing. The original Brain2Qwerty study used a 306-channel MEGIN system, as documented in its published methods. 

Some of the challenges were unexpectedly practical. A keyboard designed to be MRI compatible turned out to introduce artifacts into the MEG recordings. The team had to modify its springs to reduce the interference. Before the algorithms could interpret the signals, the experimental setup had to support usable recordings. 

The first version established that decoding keystrokes from MEG activity was possible. King describes the results as better than anticipated, while still modest from a practical perspective. 

For Brain2Qwerty v2, the team increased recording time from approximately one hour to ten hours per participant. It also addressed a limitation of the original approach: the need to supply the timing of individual typing events. As King explains, “we designed the AI system such that it did not require knowing when each word or each key press was started.” 

Meta’s account of Brain2Qwerty v2 describes this progression towards decoding sentences directly from continuous brain recordings. It is an important step in investigating an interface that could operate without externally supplied keystroke timings. 

More data brought better decoding, but King is clear about the demands of further improvement: “but it takes more and more hours to actually improve your performances.” 

There is another challenge beyond recording for longer. A model trained on one person currently offers only modest benefits when applied to another. As King puts it, “the algorithms are not really transferring between individuals.” Developing approaches that work more effectively across people is therefore a priority.

Alongside the longer-term communication ambition, he sees a continuing opportunity to understand how the brain works. Better decoding models could help researchers examine how language becomes a sequence of actions. “So even though the real-time decoding performances may be challenging, I think the progress from the theoretical and fundamental point of view is likely to continue.” 

For more on that intersection of neuroscience and AI, explore King’s research and the TITAN laboratory, where he is a principal investigator. 

Watch the full MEGIN Spotlights interview with Jean-Rémi King for the story behind Brain2Qwerty, the lessons from version two and the questions guiding the next stage of research.