Although Elon Musk’s Neuralink may be the most famous Brain-Computer Interface (BCI) currently being studied, those trials have involved only animal subjects.
However, since 2009 a team of researchers from Stanford University’s Howard Hughes Medical Institute and Brown University under the BrainGate2 pilot clinical trial have been making steady progress on BCIs tested by human trial subjects.
Now, that team has announced their latest groundbreaking results, a BCI that allows people who have lost the ability to type or often even talk the opportunity to create usable text by envisioning handwriting the individual letters in their mind. And according to the study published in the May issue of the journal Nature, the 65-year-old test subject did so at a rate similar to a non-disabled person his age using a smartphone texting app.
“Imagine if you could only move your eyes up and down but couldn’t move anything else,” said team member and Stanford neuroscientist Frank Willett in an e-mail to The Debrief. “A device like this could enable you to type your thoughts at speeds that are comparable to that of normal handwriting or typing on a smartphone.”
Background: What is “Mindwriting?”
Led by study coauthor Krishna Shenoy, a Howard Hughes Medical Institute Investigator at Stanford University, and jointly supervised by Stanford neurosurgeon Jaimie Henderson, the team of researchers already had a string of previous successes under their belts.
In 2017, they published the results of a BCI device that allowed study participants to use their thoughts to control a cursor on a screen, a method that returned a usable 40 characters per minute average. At the time, it was the fastest typing accomplished by a BCI implanted subject.
In 2019, the team then announced they had decoded the brain’s neural activity associated with speech, all with the hopes of being able to reproduce that activity.
Unfortunately, although relatively fast, the 2017 thought-to-text method had some drawbacks, including an overall similarity in the signals produced by the brain when trying to move the cursor around. It was this realization by Shenoy and Henderson that brought Willett to his idea of trying handwriting for their project.
“When we first started exploring the concept of a handwriting BCI, we had no idea whether attempted handwriting movements would still evoke rich and structured patterns of neural activity many years after paralysis,” Willett told The Debrief. “It was exciting to see that when we asked our participant to try to handwrite different letters, even though his hand had been paralyzed for more than ten years, we could still see clear patterns in the neural activity – enough to even reconstruct the motions of his imaginary pen and visualize the letters he was trying to write.”
Analysis: How “Mindwriting” Works and Why It Allows People To Write Using Only Their Thoughts
According to the research, the primary study participant, a man referred to only as T5, lost practically all movement below his neck because of a spinal cord injury in 2007.
In 2016, Henderson placed two brain-computer-interface chips on the left side of T5’s brain, and each chip has 100 electrodes that pick up signals from neurons firing in the part of the motor cortex that governs hand movement.
“This particular participant has engaged in a lot of different studies in addition to this one,” Willett told The Debrief, “including moving a robotic arm and controlling a tablet computer with a brain-computer interface.”
Once connected to their interface, researchers instructed T5 to envision writing individual letters. This imagined activity trained their AI to interpret the brain’s signals during this visualization allowing the team to turn those signals into actual text. After some trial and error, they found their new system’s performance was twice as fast and significantly more accurate than their already record-breaking BCI text creation method.
“We found that asking the participant to handwrite different letters evoked very distinct patterns of neural activity in his brain, which we think enabled us to achieve high accuracies at speeds greater than what was previously possible,” Willett told The Debrief. “Compare this to point-and-click typing that was the previous state-of-the-art: moving in straight lines towards different keys evokes very similar patterns of neural activity because the motions involved are simply straight-line movements with a different angle or distance.”
Willett also said that the experience was even unique for the experienced BCI test subject, who similarly reported some unexpected results.
“Although T5’s hand was almost completely still during attempted handwriting (and he did not hold a pen), T5 reported feeling as though an imaginary pen in his hand was physically moving and tracing out the letter shapes as he attempted to handwrite.”
This subjective feeling of touch and motion, Willett noted, appeared to obey some type of physical constraints, as T5 also reported being able to “write” more quickly if he attempted to write smaller letters.
“Often, when this illusory feeling of motion was less clear, he would think that his brain activity was less clear too and that we wouldn’t be able to decode what he was trying to write – but we always could!”
As far as the actual speed achieved by T5?
“The most relevant speed is 90 characters per minute,” Willett said in his e-mail, “which is the speed the participant plateaued at after 5 days of using the system to copy on-screen sentences as quickly as he could. It represents the average speed taken over everything that was written on that last day of copy-typing.”
When asked about how accurate this ‘mindwriting’ (as it is called in the Stanford University press release) was, Willett noted it was actually surprisingly accurate.
“After an autocorrect was applied, [like in a smartphone texting app] the results were very accurate, over 99% accuracy, meaning that less than 1 in 100 characters were wrong. This translates into a ‘word error rate’ of 3.4% – meaning that only 3.4% of the words were incorrect.”
Willett also pointed out that this level of accuracy is comparable to state-of-the-art speech recognition systems that translate speech into text, which he indicated typically have a word error rate of 4-5%.
Outlook: Brain-Computer Interfaces Are Not ‘Science Fiction’ anymore
In his e-mail to The Debrief, Willett made sure to point out that this is only a first step investigational device and that any practical application is still a long way away.
“To translate the technology into a real product, it needs to be streamlined – the user should be able to use the BCI without needing to take too much time to train it. In addition, it should be smart enough to automatically track how neural activity changes over time so that the user does not have to pause to retrain the system each day. This work is a proof-of-concept demonstration, not a product that someone can buy today.”
He did, however, note that once a system like this was available, there are already a large number of people who could immediately benefit.
“In the near term, people who have ‘locked-in’ syndrome, or paralysis of nearly all voluntary muscles, stand to benefit most,” he said.
In a final note on how significant this type of method could be over previous methods available to people in these extremely limiting physical conditions, Willett highlighted the famous case of a disabled Frenchman who was able to write a book without speech or hand movement.
“Take the famous case of Jean-Dominique Bauby, who was locked-in and wrote “The Diving Bell and the Butterfly” to chronicle his experience,” explained Willett. “To write this book, a visitor had to list all letters in the French alphabet one-by-one, and he blinked his left eye to select the letter he wanted. It could take minutes to write a single word. With the handwriting BCI [we have created], he might have been able to write 18 words per minute.”
Follow and connect with author Christopher Plain on Twitter: @plain_fiction
Don’t forget to follow us on Twitter, Facebook, and Instagram, to weigh in and share your thoughts. You can also get all the latest news and exciting feature content from The Debrief on Flipboard, and Pinterest. And subscribe to The Debrief YouTube Channel to check out all of The Debrief’s exciting original shows: The Official Debrief Podcast with Michael Mataluni– DEBRIEFED: Digging Deeper with Cristina Gomez –Rebelliously Curious with Chrissy Newton