Researchers from Northwestern University have created an app that transforms your smartphone into a movie-quality full-body motion capture system.
Unlike competitive systems that require numerous stationary cameras in a fixed environment, the new app, called MobilePoser, is fully mobile and allows for real-time motion capture in any environment.
According to the team behind MobilePoser, their technology leverages the various sensors already built into mobile devices, including smartphones, smartwatches, tablets, and wireless earbuds. The app captures and converts this data using a customized, psychics-based algorithm, resulting in full-body motion capture without any added cost or equipment.
Once MobilePoser is commercially available, the team says it could lead to fully immersive virtual reality experiences and hands-free gaming applications. The app could also allow indoor navigation where GPS systems can fail and advanced biometric tracking options for medical applications.
“Running in real-time on mobile devices, MobilePoser achieves state-of-the-art accuracy through advanced machine learning and physics-based optimization, unlocking new possibilities in gaming, fitness, and indoor navigation without needing specialized equipment,” said Karan Ahuja, an Assistant Professor of Computer Science at Northwestern University’s McCormick School of Engineering and the study’s leader. “This technology marks a significant leap toward mobile motion capture, making immersive experiences more accessible and opening doors for innovative applications across various industries.”
Inventing Full-Body Motion Capture with a Smartphone
State-of-the-art motion capture technologies have helped Hollywood bring otherwise fictional characters to life on the screen. Unfortunately, capturing the movements of an actor like Andy Serkis and turning them into Gollum on Lord of the Rings is extremely complicated and expensive.
The main limitations of movie-quality motion capture tech are the expensive cameras and sensor-laden body suits used to capture the full range of motions these actors employ to make them look and move as realistically as possible. Those costs and complications only increase for movies like Avatar or mega-budget superhero franchises that involve numerous actors portraying digital characters.
“This is the gold standard of motion capture, but it costs upward of $100,000 to run that setup,” Ahuja said of the technology employed by movie studios. “We wanted to develop an accessible, democratized version that basically anyone can use with equipment they already have.”
To accomplish their goal, the team evaluated the sensors that already exist in most mobile devices. Called inertial measurement units (IMUs), these include gyroscopes, magnetometers, and accelerometers that allow these devices to determine if they are rotated, count the wearer’s individual steps, or even alert emergency services if they detect a fall. Unfortunately, these devices are not accurate enough for accurate motion capture without some help.
Ahuja’s team tackled this problem by building a customized, multi-stage artificial intelligence algorithm to refine the IMU data. The team then trained their algorithm using an extensive database of “synthesized” IMU measurements generated from high-quality motion capture data.
Next, the team refined their algorithm with a physics-based optimizer, ensuring that the predicted movements match the actual movements of the smartphone carrier’s real-life body movements. This means a joint cannot bend backward, or a head cannot turn all the way around.
After testing and training, the team said their device accurately captured the subject’s motion and body pose to within 8 to 10 centimeters. For comparison, the now-defunct Microsoft Kinect motion capture system, which employs multiple cameras and requires a fixed environment, had a tracking error of 4 to 5 centimeters.
More Devices Increase App’s Accuracy
While the MobilePoser app isn’t as accurate as the expensive motion capture technologies used by large movie studios, its inventors say that wearing and/or carrying multiple devices could reduce its tracking error margins even further. However, they also say it is still robust enough to offer high-quality motion capture from just one device.
“The accuracy is better when a person is wearing more than one device, such as a smartwatch on their wrist plus a smartphone in their pocket,” Ahuja said. “But a key part of the system is that it’s adaptive. Even if you don’t have your watch one day and only have your phone, it can adapt to figure out your full-body pose.”
Along with offering movie-quality motion capture technology to everyday aspiring filmmakers or the potential to revolutionize immersive gaming experiences, the team says their app could also upgrade the technology available to medical professionals who could benefit from more accurate motion capture of their patients.
“Right now, physicians track patient mobility with a step counter,” Ahuja said. “That’s kind of sad, right? Our phones can calculate the temperature in Rome. They know more about the outside world than about our own bodies.”
“We would like phones to become more than just intelligent step counters,” the researcher added. “A phone should be able to detect different activities, determine your poses, and be a more proactive assistant.”
In hopes of encouraging other researchers and engineers to take their app to the next level, the team has made its pre-trained models, data pre-processing scripts, and all model training code available to the public as open source.
After a successful presentation of MobilePoser and its high-quality motion capture smartphone technology on October 15th at the 2024 ACM Symposium on User Interface Software and Technology in Pittsburgh, Ahuja also says the app is expected to be available soon for iPhone, AirPods, and Apple Watch owners.
Christopher Plain is a Science Fiction and Fantasy novelist and Head Science Writer at The Debrief. Follow and connect with him on X, learn about his books at plainfiction.com, or email him directly at christopher@thedebrief.org.