Northwestern University engineers have unveiled an extremely energy-efficient 3D camera, called SpiderCam, inspired by how jumping spiders judge depth before making a calculated leap.
The research team behind the novel device said their technology could help engineers build wearable technologies, assistive devices, robots, drones, and other futuristic technologies that benefit from quickly and efficiently gauging their surroundings.
How Jumping Spiders Inspired Energy-Efficient SpiderCam
According to a statement announcing SpiderCam, 3D cameras typically estimate image depth in two ways. The first approach involves captured images from multiple viewpoints and comparing them, and the second involves projecting a light source, often an infrared (IR) laser, and then measuring the reflection.
Although both methods can produce realistic three-dimensional images, they can require significant computing power, additional operational energy, and often expensive hardware such as lasers.
Fortunately, millions of years of evolution have resulted in a creature that possesses a hyper-efficient way of estimating depth: the jumping spider.
“Jumping spiders jump to catch prey, to avoid predators and to get around, and that requires excellent vision,” said Northwestern’s Emma Alexander, the study’s corresponding author and an expert in bio-inspired computer vision. However, the researcher notes that their brains are extremely small, requiring the ability to calculate distances in an energy- and computationally efficient way.
According to Alexander, jumping spider eyes have multiple layers of retinas in each eye, rather than just one as in human eyes. When measuring distance, each retina focuses on an image at a slightly different distance, resulting in some images being blurrier than others.
“They see multiple levels of focus at all times,” Alexander explained. “So, they are always collecting pairs of images.”
After collecting these images in real time, the fly’s poppyseed-sized brain compares their sharpness, yielding a surprisingly accurate depth estimate, as evidenced by the creature’s ability to jump with precision.
Prototype Captures Images at Two Distances and Compares Their Blurriness
Inspired by nature’s hyper-efficient, extremely accurate depth estimation method, Alexander’s team decided to build a 3D camera using the same basic approach.
“We wanted to understand whether we could borrow some of the same principles to create an extremely energy-efficient depth sensor that could be used in resource-constrained situations where users don’t have unlimited access to power,” Alexander explained.
Instead of capturing images at multiple distances, as the jumping spider’s eye does, the Northwestern team’s initial design captures two images of the same scene at different focal lengths. Next, the system uses a custom algorithm to analyze the sharpness of the edges and textures captured in each image. The team said this analysis acts as a “translator between blur and distance” in the fly’s brain, resulting in “depth measurements in real time.”
To best approximate the naturally evolved energy efficiency of a jumping spider’s eye, Alexander’s team built the algorithm directly into a customizable computer chip specifically optimized for energy efficiency called a field-programmable gate array (FPGA).
The final prototype integrating the team’s dual-lens system generates depth maps at 32.5 frames per second. Critically, it generates this level of accurate 3D maps while consuming 624 milliwatts of power. For comparison, the team notes that 624 milliwatts is “less energy than used by a standard nightlight.”
Future Versions Could be Even More Energy Efficient
Although the team said that the current version of the SpiderCam is the “first passive FPGA-based 3D camera system to operate below one watt,” they hope to improve its optics and expand its field of view. Once they have a commercial-grade version, the team hopes to integrate SpiderCam into wearable devices and small robots.
Beyond proposed applications, the Northwestern team said that they “envision” a future, ultra-energy-efficient version of the chip capable of “bringing 3D vision to applications where conventional depth sensors are impractical.”
“I’m very interested in settings where you’re very resource constrained and can’t just plug a camera into a wall,” Alexander explained. “For example, it could be deployed in field settings with limited power.”
“Separately, I also think it’s particularly exciting for applications like augmented reality where you’re interfacing with the physical world and need to know the locations of objects around you,” the researcher added.
The study’s co-first authors, Marcos Ferreira and Tianao Li, presented this work at the Computer Vision Foundation’s Conference on Computer Vision and Pattern Recognition (CVPR) in Denver. Complimentary registration is available for members of the media.
Christopher Plain is a Science Fiction and Fantasy novelist and Head Science Writer at The Debrief. Follow and connect with him on X, learn about his books at plainfiction.com, or email him directly at christopher@thedebrief.org.