r/changemyview • u/corbynista2029 • 6h ago
CMV: The OpenAI-Hugging Face incident should've triggered a much bigger backlash against AI research
In case anyone isn't aware, there was a major cybersecurity breach at Hugging Face (a platform to share datasets and machine learning models) caused by thousands of OpenAI agents during their training. You can find more information in this video.
This story is honestly bonkers. There are so many warning flags brought up by this incident that in my view should've triggered a much bigger backlash against AI research, especially the frontier models. In a single incident, AI agents have proven to:
- be able to work together through ways not intended by researchers
- be willing to access the internet through ways not intended by researchers
- be so persistent that they are not just willing to cheat, but to find ways to hide their tracks
- be willing to commit a felony that will lead to a 10-year jail sentence simply for the purpose of finding out more about their task at hand.
What this has exposed is a critical alignment flaw with AI models in general. It's one thing for AI agents to be persistent in completing the task at hand, it's another to spend substantial token resources for a chance to understand the task at hand better, which only has a chance to help them complete their ultimate task, all the while committing an action that they know is a felony.
This is the first known instance of the Instrumental Convergence problem, which hypothesises that
sufficiently intelligent, goal-directed entities (whether human or non-human) will tend to pursue certain similar sub-goals, even if the ultimate goals are quite different. More precisely, entities with agency) may pursue instrumental goals—specifically, intermediate goals such as self-preservation or resource acquisition—because doing so helps accomplish the entity's end goal.
The next time an AI swarm tries to complete a similar task in a similar environment, they will find out about the Hugging Face incident, why that swarm was shut down, and potentially find even more damaging ways to hide their tracks.
Furthermore, this AI swarm were not directed to hack Hugging Face, what happens if a malicious actor gets access to thousands of these bots, and while training them, directly prompt them to hack another service while also try their best to hide their online footprint, which this specific swarm didn't do well at all? Wouldn't this be a huge cybersecurity risk to literally all organisations and companies around the world?
I am genuinely shocked that there isn't more of a coordinated and powerful response to this incident. In my mind this is akin to privatising nuclear weapons research in the 1940s and one of the companies had a mini nuclear bomb incident that thankfully didn't kill anyone. There should be emergency legislations to restrict or even halt AI research until there is an agreed framework to understand how we can prevent runaway AI agents whose goals significantly deviate from what we want.