The frontier of physical AI is a game of Jenga in a warehouse in San Leandro, California.
That warehouse is occupied by engravera company that creates data tools used to train AI models. Andrew Ceja is a pilot (the company’s term for its robotic trainers) and he’s carefully removing blocks of wood from a wobbly tower while wearing a headset with a camera that tracks what it sees. That alone is pretty common for collecting robot training data, but this headset includes sensors that measure your brain waves as you carefully disassemble the tower of blocks.
Encord is one of a small but growing number of startups betting that the next real limitation for humanoid and warehouse robotics will not be model architecture, but the sheer scarcity of real-world physical training data. Instead of just helping robotics companies manage the data they have, Encord is building a business around manufacturing the data they don’t have.
The brainwave headset that Ceja wears was built by Zander Laboratoriesa German neuroscience startup that is betting that measuring brain activity (to infer mental states like error, intention, and surprise) can create a more useful data set for training models. Encord’s work with Zander is currently a test; Encord says the goal is to build an initial set of brainwave-labeled data, run it through robotic client models, and evaluate whether it actually improves performance before deciding whether to scale it up.
Lucas Gehrke, a Zander neuroscientist supervising the work, says the amount of brain activity used at any time during a given task offers clues to model builders trying to determine when they need to deploy their highest-effort models.
This is the “cutting edge” of the effort to solve the robotic data bottleneck, according to Vineeth Velmurugan, director of robot learning at Encord. Velmurugan, a veteran of OpenAI’s robot lab and Berkshire Grey, the warehouse automation company, joined Encord to form the company’s internal data creation team.
Encord was founded to help companies building computer vision applications annotate data and evaluate models. As his clients (Velmurugan says they work with many leading robotics companies but he’s not authorized to name them) began applying end-to-end learning to robotic manipulation tasks, executives realized they would have to produce training data themselves, rather than simply manage it. “The data simply doesn’t exist,” Velmurugan said.
The bet that generative AI can do for robots what it has done for chatbots continues to hit the same wall. Self-driving car companies collect data from the physical world themselves, but that’s difficult to scale. Training from video can work, but it lacks the fidelity of real-world data. Velmurugan says it will take a data set about five times the size of YouTube’s video corpus to achieve this, a scale that helps explain why data generation itself has become a business and not just a research problem.
Feed your egocentric data needs
Companies building robot brains are now turning to two main sources: “egocentric” videos collected by workers wearing cameras, often augmented with additional camera angles and other metrics, and robot data remotely operated. Encord does both: It pulls egocentric data from various factories around the world and uses its San Leandro facility to experiment with new modalities, such as brain waves, or collect data sets on specific abilities to make adjustments.
When TechCrunch visited, pilots were using leader-follower teams (paired robotic arms, one controlled directly by a human operator and another that mimics their movements) to create data on tasks like pouring coffee from a coffee maker into cups (very sloppy) and stacking poker chips. “All the humanoid companies have asked us for these parts,” says Velmurugan.
The storage shelves held boxes of artificial flowers in vases, books, plastic vegetables, cat litter trays and scoops, bags and bundles of wire, the stuff available for training handlers for housework.
At one of these stations, another pilot, Sofía Infante, maneuvers robotic arms to connect and disconnect Ethernet cables from the back of a server—the kind of work that data center operators would love to automate, if only robots could manipulate them with the required precision. Taking a walk behind the controls, I could see why that’s still out of my reach: Grippers are much less dexterous than human fingers and lack the degrees of freedom we take for granted in our arms.
Another new data modality Encord is developing uses a set of sensors attached to the forearm to detect electrical signals in the muscles. Videos taken of human hands manipulating objects typically don’t capture the entire hand, but Velmurugan hopes to build a 3D representation of where the hand is at any time based on the arm’s sensors, creating a more robust understanding of the models.
Encord’s data sets are annotated with physical descriptions of what each video contains (“right hand tightens bolt”) to help LLM-based models understand what is happening. Velmurugan estimates that this type of dense annotation is worth 100 times more than “ego junk data” for training specific tasks, and it only costs 20 times more to produce, which is a good deal, on paper.
But “20 times more” is still real money, and that’s the problem: mining text from the Internet, the way LLM’s creators built their models from Stack Overflow and the rest of the web, costs cutting-edge labs next to nothing. The generation of physical training data does not, and that is the limit of the comparison of physical AI as LLM. This type of data must be manufactured, not simply collected, and that changes the economics of building these models.
Velmurugan says progress is being made: With Encord’s visibility in programs across the industry, he can see both startups and cutting-edge labs figuring out what works and what doesn’t to improve physical AI models. That point of view (among many robotics companies at once) is also part of Encord’s speech. You can detect which data techniques are gaining traction across the industry before a single customer can.
That will keep the dozen pilots at the Encord facility busy. Both Infante and Ceja are part of a growing workforce developing the building blocks of neural networks; They previously worked at Scale, another AI data annotation company, before joining Encord.
Ceja had worked at a waste management company where his interest in technology led him to be in charge of keeping a robotic trash sorter in good working order. Now, as the Jenga tower collapses, he says he enjoys the challenge of solving robot training tasks: “It’s something new every day!”
When you purchase through links in our articles, we may earn a small commission. This does not affect our editorial independence.
