Paralysed man writes again

A new system pioneered in America has helped a paralysed patient to "write" again for the first time in a decade. He's able to do it at the rate of 90 characters a minute, roughly the time it takes you or me to tap out text messages, so it's pretty slick. The system works by eavesdropping on the neurological "chatter" that goes on in the brain's motor centre, which fires up when a person makes movements. The neurological signals produced here have specific patterns of activity that correspond to the pattern or shape of a movement a person intends to make. So when you write a letter or number, this involves a specific sequence of movements, so it's represented by a specific sequence of nerve impulses. And even though a person is paralysed and cannot transmit those signals to their muscles, the signals are still there, so if you pick them up and teach a computer to recognise them, you can get that person writing again - albeit on a screen - which is what Frank Willett and his team at Stanford University has managed to achieve…

Frank - This is really about trying to restore communication to people with severe paralysis, or people who are locked in. So imagine if you can't move any part of your body, how do you communicate your thoughts? And this is about making a device to let people do that.

Chris - We have made some forays in that direction though, haven't we? People have recorded brain activity and used a computer to decode that and turn it into movement.

Frank - Yep. One of the biggest ways previously, was to enable control of a computer cursor. So someone could use a brain computer interface to move a cursor around the screen and click on individual keys on an onscreen keyboard and type things out that way.

Chris - So what have you done that's different?

Frank - So what's different here is handwriting. So to use the brain computer interface, the person tries to make their hand write each individual letter that they want to type. And it turns out that this method lets us go twice as fast as previous work.

Chris - How did you do it?

Frank - It starts with recording the signals. We have two tiny sensors about the size of a baby aspirin that get placed on the outer layer of the brain, and a brain area that deals with motor control of the hand. And these sensors pick up electrical impulses from individual neurons. And then we translate these impulses into text. So when he tries to write each letter, that evokes a specific pattern of impulses across the different neurons, and we detect that and figure out what he's trying to write and type it on the screen.

Chris - How do you figure out what letter they are trying to write? Is that, you basically say, right, I want you to imagine writing a letter A or letter B, and you do this enough times, the computer therefore can learn when the person tries to do an a, this is the pattern of activity that that bit of the brain generates?

Frank - Exactly. It's a pattern recognition problem. And that was one of the main challenges in this work was trying to make an algorithm that was accurate enough to, you know, reliably tell from these electrical impulses, exactly what letter you're trying to write, because we don't have the luxury of recording from every neuron and motor areas of the brain. We can only record from a handful. These neurons are variable and we call them noisy. Like they don't always have a clear signal. So we have to see through that noise and be able to reliably pick out what the letters are.

Chris - I've got a copy of your paper in front of me. And you've presented in that paper, a facsimile of what the "writing" in inverted commas that the person produces looks like. It's stunning.

Frank - Thanks. Yeah. We were excited and surprised that, you know, even after 10 years of paralysis, so he hasn't written or moved his hand after 10 years, still the brain activity in the motor areas of the brain still has this very fine structure where we could even figure out, you know, where his imaginary pen is trying to move and what these letters look like.

Chris - How accurate is your system though? Because obviously if you want to extrapolate this, so a person can write notes and write things down for people quickly in this way, if they're trying to write the word ship, they don't want to make a spelling error there, do they? So how accurate is it?

Frank - It's quite accurate. And that's one of the exciting things about it, is that we think it might actually be usable in the real world. The raw accuracy before you apply any kind of auto correction type of system is about 95%. So one out of every 20 letters is wrong, but if you use modern auto correction techniques, like on your smartphone, we found that when we applied that the accuracy was above 99%.

Chris - And if the person starts to actually write whole sentences, because when you just formulate one letter in your mind's eye, that's just one sequence of movements. But often when you write a series of letters, you'd think about joining them up and then joining words into sentences. Does that not blur the movement signals and does that not make the machine make more mistakes?

Frank - Yeah, well, that's definitely a part of a challenging problem, is these transitions between the letters. We actually asked the participant to write as if you were writing on a palm pilot, actually. So to write in print, not in cursive and write each letter on top of the previous one.

Chris - And how did the patient actually respond to this?

Frank - Well, I think compared to a lot of things, it was pretty easy to use. So one exciting part of this was that even on the first day, when we asked him to try to write letters, we got beautiful neural activity back, that was highly interpretable. So we didn't have to like, train up to use this over a long period of time.

Chris - I presume your patient's right-handed?

Frank - Yes.

Chris - Because 90% of the population are. So I had a 90% chance of saying that correctly, but have you tried swapping around to the other hand, can you, for the first time in your life, write neatly with the wrong hand as it were with your system potentially?

Frank - Well, yeah, it's interesting because the left side of the brain actually controls the right side of the body and vice versa. So these implants were placed into the left side of his brain and we have him use his right hand to use the system. But actually we found that contrary to what you might think, this brain area does encode both of the hands. So we can actually ask him to do things with his left hand and pick up activity as well, but it's not quite as strong.

Chris - And if he gesticulates in his mind's eye as he's writing, does that totally confuse your system?

Frank - Yeah. Yeah. We haven't made the system robust, to be able to use any hand or anything. We've only focused on the right hand so that probably wouldn't go well.