That these agents have arrived at a moment plagued by broader AI doomsday warnings about their potential threat to humanity brings this friendly packaging into sharp relief.
“When these companies speak to investors, regulators and researchers, AI is described as extraordinarily powerful technology with serious and unresolved questions around safety, security, and control,” says Dr. Sarah Saska, a sociotechnologist and specialist in tech’s relationship with culture and power. “Yet, increasingly intimate versions of the same technology are presented to consumers as friendly, playful, frictionless, and low-stakes.”
In the same week, OpenAI CEO Sam Altman unveiled Dots, the company pulled its latest AI model and delayed its IPO due to existential safety concerns around the very same AI agent tech. Meanwhile, just weeks before its Muse launch, Meta agreed to pay an $18 billion settlement to 48 US states — the largest-ever corporate settlement in tech sector history — to resolve claims that the algorithmic design of its Facebook and Instagram platforms harm children.
Safety warnings are coming from inside the house at major rivals, too: Anthropic CEO Dario Amodei called on the industry to slow the pace of frontier AI development, a plea publicly backed by Altman, xAI’s Elon Musk, and Google DeepMind’s Demis Hassabis. (Zuckerberg, for his part, has rejected calls for an industry-wide slowdown in the pace of AI development, arguing instead that individual companies should decide for themselves if and when to slow down.)
Users are nevertheless intrigued about these AI agents. Within 10 days of launch, Muse had racked up more than 730,000 US downloads and knocked ChatGPT off the top of Apple’s US App Store rankings. Days later, investor excitement around the agent helped send Meta shares up 11% in a single session, adding some $192 billion to its market value. Yet, within the first two weeks of its release, several reports raised concerns about Muse overstepping users’ intended permissions — including sharing sensitive information and taking actions users said they’d not explicitly approved.
“My concern is that these designs make powerful systems feel harmless and familiar, encouraging people to share more and question less.”
Lidia Velkova, managing director of Clever Together Futureproof
