Researchers are using AI to finetune a robotic prosthesis to improve manual dexterity by finding right balance between human and machine control.

Whether you’re reaching for a mug, a pencil, or someone’s hand, you don’t need to consciously instruct each of your fingers on where they need to go to get a proper grip.

The loss of that intrinsic ability is one of the many challenges people with prosthetic arms and hands face. Even with the most advanced robotic prostheses, these everyday activities come with an added cognitive burden as users purposefully open and close their fingers around a target.

Researchers at the University of Utah are now using artificial intelligence to solve this problem. By integrating proximity and pressure sensors into a commercial bionic hand and then training an artificial neural network on grasping postures, the researchers developed an autonomous approach that is more like the natural, intuitive way we grip objects. When working in tandem with the artificial intelligence, study participants demonstrated greater grip security, greater grip precision, and less mental effort.

Critically, the participants were able to perform numerous everyday tasks, such as picking up small objects and raising a cup, using different gripping styles, all without extensive training or practice.

The study was led by engineering professor Jacob A. George and Marshall Trout, a postdoctoral researcher in the Utah NeuroRobotics Lab, and appears in the journal Nature Communications.

“As lifelike as bionic arms are becoming, controlling them is still not easy or intuitive,” Trout says. “Nearly half of all users will abandon their prosthesis, often citing their poor controls and cognitive burden.”

One problem is that most commercial bionic arms and hands have no way of replicating the sense of touch that normally gives us intuitive, reflexive ways of grasping objects. Dexterity is not simply a matter of sensory feedback, however. We also have subconscious models in our brains that simulate and anticipate hand-object interactions; a “smart” hand would also need to learn these automatic responses over time.

The Utah researchers addressed the first problem by outfitting an artificial hand, manufactured by TASKA Prosthetics, with custom fingertips. In addition to detecting pressure, these fingertips were equipped with optical proximity sensors designed to replicate the finest sense of touch. The fingers could detect an effectively weightless cotton ball being dropped on them, for example.

For the second problem, they trained an artificial neural network model on the proximity data so that the fingers would naturally move to the exact distance necessary to form a perfect grasp of the object. Because each finger has its own sensor and can “see” in front of it, each digit works in parallel to form a perfect, stable grasp across any object.

But one problem still remained. What if the user didn’t intend to grasp the object in that exact manner? What if, for example, they wanted to open their hand to drop the object? To address this final piece of the puzzle, the researchers created a bioinspired approach that involves sharing control between the user and the AI agent. The success of the approach relied on finding the right balance between human and machine control.

“What we don’t want is the user fighting the machine for control. In contrast, here the machine improved the precision of the user while also making the tasks easier,” Trout says. “In essence, the machine augmented their natural control so that they could complete tasks without having to think about them.”

The researchers also conducted studies with four participants whose amputations fell between the elbow and wrist. In addition to improved performance on standardized tasks, they also attempted multiple everyday activities that required fine motor control. Simple tasks, like drinking from a plastic cup, can be incredibly difficult for an amputee; squeeze too soft and you’ll drop it, but squeeze too hard and you’ll break it.

“By adding some artificial intelligence, we were able to offload this aspect of grasping to the prosthesis itself,” George says. “The end result is more intuitive and more dexterous control, which allows simple tasks to be simple again.”

“The study team is also exploring implanted neural interfaces that allow individuals to control prostheses with their mind and even get a sense of touch coming back from this,” George says. “Next steps, the team plans to blend these technologies, so that their enhanced sensors can improve tactile function and the intelligent prosthesis can blend seamlessly with thought-based control.”

Additional coauthors are from the University of Utah and the University of Colorado, Boulder

Funding came from the National Institutes of Health and the National Science Foundation.

Source: University of Utah

Related

Algorithm for robotic prosthetics could ease pain for amputees

Watch: Prosthetic robot hand ‘knows’ what it’s touching

AI-trained exoskeleton saves user’s energy

Engineers have developed a pioneering prosthetic hand that can grip plush toys, water bottles, and other everyday objects like a human.

The hand carefully conforms and adjusts its grasp to avoid damaging or mishandling whatever it holds.

The system’s hybrid design is a first for robotic hands, which have typically been too rigid or too soft to replicate a human’s touch when handling objects of varying textures and materials.

The innovation offers a promising solution for people with hand loss and could improve how robotic arms interact with their environment.

Details about the device appear in Science Advances.

“The goal from the beginning has been to create a prosthetic hand that we model based on the human hand’s physical and sensing capabilities—a more natural prosthetic that functions and feels like a lost limb,” says Sriramana Sankar, a Johns Hopkins University PhD student in biomedical engineering who led the work.

“We want to give people with upper-limb loss the ability to safely and freely interact with their environment, to feel and hold their loved ones without concern of hurting them.”

The device, developed by the same Neuroengineering and Biomedical Instrumentations Lab that in 2018 created the world’s first electronic “skin” with a humanlike sense of pain, features a multifinger system with rubberlike polymers and a rigid 3D-printed internal skeleton. Its three layers of tactile sensors, inspired by the layers of human skin, allow it to grasp and distinguish objects of various shapes and surface textures, rather than just detect touch. Each of its soft air-filled finger joints can be controlled with the forearm’s muscles, and machine learning algorithms focus the signals from the artificial touch receptors to create a realistic sense of touch, Sankar says.

“The sensory information from its fingers is translated into the language of nerves to provide naturalistic sensory feedback through electrical nerve stimulation,” Sankar says.

In the lab, the hand identified and manipulated 15 everyday objects, including delicate stuffed toys, dish sponges, and cardboard boxes, as well as pineapples, metal water bottles, and other sturdier items. In the experiments, the device achieved the best performance compared with the alternatives, successfully handling objects with 99.69% accuracy and adjusting its grip as needed to prevent mishaps. The best example was when it nimbly picked up a thin, fragile plastic cup filled with water, using only three fingers without denting it.

“We’re combining the strengths of both rigid and soft robotics to mimic the human hand,” Sankar says. “The human hand isn’t completely rigid or purely soft—it’s a hybrid system, with bones, soft joints, and tissue working together. That’s what we want our prosthetic hand to achieve. This is new territory for robotics and prosthetics, which haven’t fully embraced this hybrid technology before. It’s being able to give a firm handshake or pick up a soft object without fear of crushing it.”

To help amputees regain the ability to feel objects while grasping, prostheses will need three key components: sensors to detect the environment, a system to translate that data into nerve-like signals, and a way to stimulate nerves so the person can feel the sensation, says Nitish Thakor, a Johns Hopkins biomedical engineering professor who directed the work.

The bioinspired technology allows the hand to function this way, using muscle signals from the forearm, like most hand prostheses. These signals bridge the brain and nerves, allowing the hand to flex, release, or react based on its sense of touch. The result is a robotic hand that intuitively “knows” what it’s touching, much like the nervous system does, Thakor says.

“If you’re holding a cup of coffee, how do you know you’re about to drop it? Your palm and fingertips send signals to your brain that the cup is slipping,” Thakor says.

“Our system is neurally inspired—it models the hand’s touch receptors to produce nervelike messages so the prosthetics’ ‘brain,’ or its computer, understands if something is hot or cold, soft or hard, or slipping from the grip.”

While the research is an early breakthrough for hybrid robotic technology that could transform both prosthetics and robotics, more work is needed to refine the system, Thakor says. Future improvements could include stronger grip forces, additional sensors, and industrial-grade materials.

“This hybrid dexterity isn’t just essential for next-generation prostheses,” Thakor says.

“It’s what the robotic hands of the future need because they won’t just be handling large, heavy objects. They’ll need to work with delicate materials such as glass, fabric, or soft toys. That’s why a hybrid robot, designed like the human hand, is so valuable—it combines soft and rigid structures, just like our skin, tissue, and bones.”

Additional authors are from Florida Atlantic University, Johns Hopkins, and the University of Illinois Chicago.

Funding for the research came from the Department of Defense through the Orthotics and Prosthetics Outcomes Research Program and the National Science Foundation.

Source: Johns Hopkins University

Related

Stretchy ‘skin’ could give robots sensitivity of human touch

Robotic prosthetic ankle boosts user stability

Robotic gripper is gentle enough to pick up a drop of water