Researchers from NYU created an open-source AI model that is capable of teaching a robot common household tasks in as little as 20 minutes.
The team, led by fourth-year Ph.D. student Mahi Shafiullah, named the model Dobb-E as a tip of the hat to the house elf from Harry Potter.
The project targeted one of the hardest aspects of training a robot in real-world scenarios.
Robots are extremely proficient at repeated tasks in labs and factories where the environment can be controlled to eliminate the unexpected.
Yet, using robots as common helpers in home settings has proven difficult because of the sheer number of unexpected problems that they can encounter.
The researchers asked participants in 22 homes in the New York area to record simple tasks on an iPhone mounted on a reacher-grabber arm.
With only 13 hours of video, the team trained a working open-source AI model that was then used to train a commercially available robot from Hello Robot.
The robot used the lidar data from the iPhone recordings to match movement, depth, and rotation via another iPhone attached to the robot.
The results were remarkable, and the team was able to “teach” the robot new tasks in as little as 20 minutes – five to consume the video, and another 15 to fine-tune its movements.
The team was able to train the robot to open curtains, close garbage bags and lift them out of a garbage pail, and even fine-tune tasks like pouring a cup of liquid and turning a stove knob.
The robot was successful over 80% of the time, but more work is needed for obstacles like reflections in glass and mirrors.
The work, while still a small prototype, provides a glimpse into the possibilities of the future of robotics.