A research team at the Technical University of Munich (TUM) has created an intelligent robot that can locate lost items on demand by combining knowledge gathered from the internet with its own internally generated spatial map.
The TUM research team behind the futuristic-sounding robotic ‘retriever’ is already working to improve the intelligent robot’s design, including adding the ability to search for missing objects in a box, a drawer, or even behind a door.
“We have taught the robot to understand its surroundings,” said Prof. Angela Schoellig, who leads the Robotics Lab at the TUM Chair of Safety, Performance, and Reliability for Learning Systems, which focuses on the development of safe, adaptive robotics and AI.
Intelligent Robot Maps Its Surroundings to Find Misplaced Objects
According to a statement from researchers working in Prof. Schoellig’s TUM Learning Systems and Robotics Lab, the new robot resembles “a broomstick on wheels with a camera mounted at the top.” However, they also note that the robot’s intelligence is far more complex than its comparatively simple appearance suggests.
Instead, the TUM team’s invention is among the first of its kind to integrate knowledge-based image understanding and apply that new information to a specific task. For example, if the intelligent robot is tasked with finding a lost set of keys last seen in the bedroom, it will begin by creating a three-dimensional image of the room.
Although the captured images are two-dimensional, the research team notes that the image pixels contain ‘depth information’ about the objects within the image. Using this spatial data, the TUM team’s robot can create an accurate map to the centimeter. After capturing the initial images and creating a special map, the robot continuously updates it as new information comes in.
“(It) is important for all robots that move in spaces that are constantly changing,” Schoellig explained.
Using the Internet to Understand the Meaning of Objects
Although the robot’s actions are autonomous, the research team does play a critical role in its current operations. The most direct interaction involves retrieving information about visible objects from the internet, including the significance that the object has to humans.
Professor Schoellig said that this added layer of interaction has allowed the team to teach the robot to “understand its surroundings” by putting the items it encounters into a human context.
The intelligent robot was also equipped with a language model similar to other emerging artificial intelligence systems. According to Schoellig, the model they used captures the relationships between objects, and then its human ‘assistants’ convert that information into the robot’s language.
In one example, two-digit numbers are assigned to locations within a three-dimensional map. As the robot moves, these numbers change as the robot recalculates the likelihood that its target object, in this case, missing keys, is in that location. An analysis of the robot’s movements revealed that searching in areas it deemed “probable locations” was 30 percent more efficient than searching randomly throughout the room.
Another quality that improves the robot’s search-and-find capabilities is its ability to remember previous images and compare them with new ones encountered in its surroundings. So, the research team notes, if the missing keys were lost in the bedroom, the robot can recognize the change “with a high degree of certainty (95 percent).” These areas are remembered as “highly probable” search locations.
Improved Capabilities Include Making the Robot Smarter and Adding Arms and Hands
Although the current intelligent robot can find items with a little help from its humans, the TUM team is already working on designs for robots that can navigate any environment without human intervention. They also want to expand the robot’s capabilities to include searching inside a drawer or behind a closed door.
According to the team’s statement, achieving this increased level of autonomy will require upgrades in both hardware and software. On the software side, that means increasing the robot’s ability to ‘draw knowledge’ from the internet and apply that information to its understanding of the surrounding environment.
A hardware upgrade that would make the robot capable of searching within boxes and drawers would inevitably include attachments for manipulating objects. Humans call them arms and hands
“Robotic arms and hands must open a cupboard and determine whether it opens upwards or sideways and how best to grasp the handle,” they explained.
The study “Where Did I Leave My Glasses? Open-Vocabulary Semantic Exploration in Real-World Semi-Static Environments” was published in IEEE Robotics and Automation Letters.
Christopher Plain is a Science Fiction and Fantasy novelist and Head Science Writer at The Debrief. Follow and connect with him on X, learn about his books at plainfiction.com, or email him directly at christopher@thedebrief.org.