Brain2Qwerty — Typing, decoded from the brain (video hero, zoomed start)

Loading the scanner…

Brain2Qwertyv2

A non-invasive brain-computer interface for online sentence generation.

Scroll to read more

The Challenge

Every year, thousands of people lose the ability to speak after a stroke, an accident, or a brain disorder. While communication can be restored with brain implants on the motor cortex, such neuroprostheses require open-brain surgery.

Today, we introduce Brain2Qwerty v2 : a model to decode natural sentences from non-invasive magnetoencephalography (MEG) recordings.

--><br>--><br>Brain2Qwerty in one picture: brain activity &rarr; an AI decoder &rarr; the sentence being typed. --><br>-->

Architecture

For this, we built off the Brain2Qwerty v1 architecture released last year and recently accepted at Nature Neuroscience. Brain2Qwerty v1 consisted of predicting keystrokes from MEG brain activity patterns recorded at BCBL, but could not work in real time because it needed the timing of every keypress.

Brain2Qwerty v2 overcomes this limit and generates the sentences directly from a continuous recording of brain activity.

This new model now combines three hierarchical modules to jointly improve the decoding of letters, words, and sentences.

The Conformer detects characters, the Aligner groups them into word embeddings, and the language model reconstructs the final sentence . -->

Performance

Trained on 10&times; more data per participant, Brain2Qwerty v2 can decode complete and meaningful sentences solely from MEG signals of healthy volunteers, and reaches up to 78% word accuracy for the best participant.

32%

average character error rate with MEG (v1)

19%

best participant — whole sentences decoded verbatim

2×

better than scalp EEG (67% error)

-->

Typed sentences

~2,200<br>v1

~22,000<br>v2

Word accuracy

48%40%<br>v1

78%61%<br>v2

mean<br>best participant

Explore results

Explore examples of the sentences decoded by Brain2Qwerty v2 from the brain recordings of three subjects.

Remaining challenges

Two major challenges remain before this method can be hoped to transfer to the clinic. First, decoding performance is not yet good enough for everyday use: this method still makes too many word-level or character-level errors to be practical. Still, our results follow a scaling law : the more data used for training, the better the decoder, without -- at least for now -- a detectable performance plateau. We are hopeful that larger datasets will thus further improve decoding and reduce the remaining gap with invasive neuroprostheses.

Second, the MEG device used in the presented study consists of a large scanner -- i.e. a setup inaccessible to most patients. However, we are optimistic about this challenge: MEG sensors continue to improve, and some are now wearable, and could thus be more practical in the clinics.

Open Science

As always, we're thrilled to share our publications, code and data:

Brain2Qwerty v1 Nature Neuroscience

Brain2Qwerty v2 Meta preprint

Full Code for v1 and v2 GitHub

v1 Dataset collected by BCBL HuggingFace

v2 Dataset collected by BCBL embargoed until journal publication.