Co-painting is by its nature collaborative, and developing data that trains a robot to collaborate is difficult and time-consuming. To get around this complication, CoFRIDA uses self-supervised training data based on FRIDA’s stroke simulator and planner. The researchers created a self-supervised, fine-tuning dataset by having FRIDA simulate paintings that consisted of a sequence of brush strokes, from which some strokes could be removed to produce examples of partial paintings. The team had to determine how to remove elements from drawings in the training data while leaving enough of the image for CoFRIDA to recognize it. For example, researchers took away details like the rim of a wheel or windows in a car but left the outline of the vehicle.
“We tried to simulate different states of the drawing process,” Zhu said. “It’s easy to get to the final sketch, but it’s quite hard to imagine the intermediate stage of this process.”
Using the dataset of partial and complete paintings, the researchers fine-tuned a text-to-image model, InstructPix2Pix(opens in new window), that enabled CoFRIDA to add brush strokes and work with existing content on the canvas. This approach, which relies on data created using CoFRIDA’s brush simulator, means that generating a painting incorporates the robot’s real constraints, such as its limited set of tools.
Outside the lab, researchers hope CoFRIDA can teach people about robotics and expand creativity, encouraging people who may doubt their artistic abilities. CoFRIDA can also help make users’ visions come to life or take the artwork in a whole new direction.
“If you start from a very simple sketch, CoFRIDA takes the artwork in vastly different directions. If you ask for six different drawings, you’ll get six very different options,” Schaldenbrand said. “It’s nice to be able to make decisions at a high level because it makes me feel like an art director. The robot makes these low-level decisions of where to put the marker, but I get to decide what the overall thing will look like. I still feel in control of the creative process, and in a world where artists fear replacement by AI, CoFRIDA as an example of a robot designed to support human creativity is incredibly relevant.”
Researchers hope further work can integrate personalization into CoFRIDA, giving users even more control over the style of the finished product.
The team’s paper, “CoFRIDA: Self-Supervised Fine-Tuning for Human-Robot Co-Painting(opens in new window),” won the Best Paper Award on Human Robot Interaction at the 2024 IEEE International Conference on Robotics and Automation(opens in new window) (ICRA) in Yokohama, Japan. An accompanying CoFRIDA demonstration was a finalist for the Best Demo at the ICRA EXPO.
“Carnegie Mellon University is a private research university in Pittsburgh, Pennsylvania. The institution was originally established in 1900 by Andrew Carnegie as the Carnegie Technical School. In 1912, it became the Carnegie Institute of Technology and began granting four-year degrees.”