Elon Musk's new idea: Optimus may learn to fold clothes by watching videos

Tesla (TSLA.O) is drawing on a time-tested blueprint to train its humanoid robot. According to sources familiar with the matter, the electric vehicle manufacturer informed employees at the end of June that the Optimus project will focus more on a "pure vision" approach.

Previously, Tesla used motion capture suits and virtual reality headsets to record data from human operators and remotely control the robot. Now, the company will primarily train the robot by filming videos of workers performing tasks, such as teaching it how to pick up objects or fold T-shirts.

Sources said the company stated that abandoning motion capture suits and remote control will allow the team to scale up data collection more quickly.

This shift marks a significant adjustment in Tesla's robotics strategy, aligning Optimus with CEO Elon Musk's long-held belief that artificial intelligence can master complex tasks using only cameras. Tesla has been using a similar approach to train its autonomous driving software.

This change comes shortly after the resignation of Optimus project lead Milan Kovac. Sources revealed that AI chief Ashok Elluswamy has taken over the project.

Motion capture and remote control are standard practices in the robotics industry. For example, leading robotics company Boston Dynamics has used remote control to train its Atlas robot. During training, workers wear motion capture suits to perform various tasks, and the data is then fed into the robot. Motion capture suits can also be used to remotely control robots.

It is still unclear whether Tesla will prioritize motion capture and remote control again in the future, or continue developing with video data based on previously collected information.

Robert Griffin, Senior Research Scientist at the Institute for Human and Machine Cognition, said that a large amount of remote control data allows robots to learn through physical interaction with the environment. He stated that relying solely on video data makes it difficult for robots to accurately translate actions from video into the real world.

"If you only use video data, there is no direct physical interaction," he said.

Folding T-shirts and Picking Up Objects

Musk first announced in 2021 that Tesla planned to develop a humanoid robot called Optimus. The billionaire said the robot would eventually be able to handle tasks such as factory labor and caregiving work.

Last year, the company recruited "data collection operators." These positions involved performing and recording basic household tasks. According to job postings, operators needed to wear motion capture suits and virtual reality headsets for extended periods.

Until the end of June, the project still included training Optimus through remote control and motion capture suits. Sources said workers spent a lot of time dealing with issues related to the suits and the robot itself, which limited the amount of data the team could collect.

Since the shift in training methods, workers have started using a set of five custom-made Tesla cameras to record their actions. Sources said these cameras are mounted on helmets and heavy backpacks worn by workers, filming in all directions to provide the AI model with precise environmental positioning data.

Christian Hubicki, director of the Robotics Lab at the joint College of Engineering of Florida A&M University and Florida State University, said that these cameras from different angles might allow Tesla to capture finer details, "such as the position of joints and fingers," and better position the robot. He added that these videos could also supplement data previously collected through remote control.

During training, workers receive specific task instructions, especially regarding hand movements, to ensure the actions look as human-like as possible. One employee said they might spend months repeatedly performing the same simple task.

Jonathan Aitken, a robotics expert at the University of Sheffield, said Tesla might need to find a way for Optimus to learn various tasks through some generalizable actions.

"At this scale, they must have a set of universal actions, otherwise training for all tasks would take an extremely long time," Aitken said.

He added that Tesla might adopt a strategy similar to that of Physical Intelligence, a company that teaches robots transferable skills and flexible application by inputting large amounts of demonstration data, rather than rote memorization of single tasks.

"A Very Tesla-Style Approach to Robotics Development"

This new strategy is consistent with how Tesla trains its autonomous driving software. Other autonomous driving companies use sensors such as lidar and millimeter-wave radar to train their software, while Tesla mainly relies on cameras.

The company collects data from millions of Tesla vehicles equipped with 8 to 9 cameras. Musk has said that Tesla launched its assisted driving software in China after training its AI system with publicly available videos of Asian streets.

In the earnings call in January this year, Musk admitted that "the training needs for the Optimus humanoid robot may ultimately be at least 10 times that of cars."

"This is a very Tesla-style approach to robotics development. No other company is attempting this at such a scale," Aitken said. "They need a data volume as massive as that used to train cars."

Alan Fern, an AI and robotics expert at Oregon State University, said that training Optimus is even more challenging for Tesla than developing autonomous vehicles.

"Driving is just one task," he said. Relying mainly on video learning "requires the robot to both understand what is happening in the video and have the skills to complete the task. Some things can be learned by observation, but others require hands-on practice in a simulator or the real world."