> In May 2023, a colonel in the US Air Force revealed a simulation in which an AI-enabled drone, tasked with attacking surface-to-air missiles but acting under the authority of a human operator, attacked the operator instead. (The US Air Force later denied the story.)
This casts a shadow on the reliability of the whole article. While the retraction (they later stated that this was a thought experiment and NOT a simulation) may or may not be true, the current state of AI would suggest that this kind of chain-of-logic problem-solving is likely out of reach. To put this in here without such a caveat and instead use the 'later denied' language is a strong indicator that we're being sold a story, and not the whole one.
The quotes and other horror-scenario errata also come off a little thin. If Hinton really, genuinely believes the world is ending, then why is he working on the generational successor to backpropagation [1]?
Good way to sell 'news', though. Keep 'em scared, keep 'em reading, I guess.
Fun quote from that 2018 article: "there was an algorithm that was supposed to figure out how to apply a minimum force to a plane landing on an aircraft carrier. Instead, it discovered that if it applied a huge force, it would overflow the program’s memory and would register instead as a very small force. The pilot would die but, hey, perfect score."
Thanks for the links. It looks like I'm behind a bit - this space has been moving just crazy fast and months make the difference.
Just to clarify..
> Finally, as already mentioned by others, agents finding unexpected ways to maximize their reward is a well known problem of reinforcement learning:
This much I agree with, but when it comes to the chain of reasoning stuff - my understanding was that the current state of the art wasn't capable of proper abstract logic given a sufficiently complex domain. Specifically, the air force test rang false because it seemed like a total stretch that an NN would be able to reason in that way.
Am I just plain wrong or has there been movement in that space in particular recently?
It's not abstract logic, it's rollout [1]: repeatedly simulating different action sequences N steps into the future and comparing their score after the last step.
A brute force approach would just simulate all possible sequences and go with the highest-scoring one; RL algorithms pseudo-randomly sample the (typically intractably large) search space and use the results to update their policy (these days, typically implemented as a neural network).
In the Air Force examples, as long as shooting the human controller or the communication tower is not explicitly prohibited, there is nothing surprising about an RL agent trying that course of action (along with other random things like shooting rocks, prairie dogs and even itself). Doing so requires no abstract reasoning or understanding of the causal mechanism between shooting the human controller and getting a better score, just random sampling and score-keeping. If the rollout score consistently goes up after the action "shoot the human controller", any RL algorithm worth its salt will update its policy accordingly and start shooting the human controller.
> This much I agree with, but when it comes to the chain of reasoning stuff - my understanding was that the current state of the art wasn't capable of proper abstract logic given a sufficiently complex domain.
How complex is "sufficiently complex"?
Or put another way - GPT-4 already seems quite good at abstract reasoning, as long as you translate things to avoid too obscure concepts, and keep things under its context window.
For this specific case, a small experiment I did once convinced me that GPT-4 is rather good at planning ahead, when playing an interactive text game. So imagine yourself narrating an UAV camera feed over the radio, like it was a baseball match or a nature documentary. That's close enough, and embedded enough in real context, that GPT-4 would be able to provide good responses to "What to do next? List updated steps of you plan.".
As for the "rest of the owl" involved in piloting an UAV, that's long ago been solved. Classical algorithms can handle flying, aiming and shooting just fine. Want something extra fancy? Videogame developers have you covered - game AI as a field is mostly several decades worth of experience in using a mix of cheesy hacks and bleeding-edge algorithms to make virtual agents good at planning ahead to best navigate a dynamic world and kill other agents in it. Recent Deep Learning AI work would be mostly helpful in keeping "sensory inputs" accurate.
That said, despite being the bleeding edge of AI research, I don't think LLMs would be able to achieve the effect this air force story is describing - they understand too much. You'd have to go out of your way to get something like GPT-4 to confuse its own operator with an enemy missile launcher. This scenario smells like the work of an algorithm that doesn't work with high-level concepts, and instead has numeric outputs plugged in directly to UAV's low-level controls, and is trained on simulated scenarios. I.e. generic NNs, especially pre-deep learning, genetic algorithms, etc. - the simple stuff, plugged in as feedback controllers - essentially smarter PLCs. Those are prone to finding cheesy local minima of the cost function.
In short: think tool-assisted speed runs. Or fuzzers. Or whatever that web demo was that evolved virtual 2-dimensional cars, by generating a few "vehicles" with randomly-sized wheels and bodies, making them ride a randomly-generated squiggly line, waiting until all of them get stuck, and then using the few that traveled the farthest to "breed" the next generation. This is the stuff that you put on a drone, if you want something that can start shooting at you just because "it seemed like a good idea at that moment".
> While the retraction (they later stated that this was a thought experiment and NOT a simulation) may or may not be true, the current state of AI would suggest that this kind of chain-of-logic problem-solving is likely out of reach.
No, it's not. It's a predictable & expected outcome of particular RL agent algorithms, and you do in fact see that interruptibility behavior in environments which enable it: https://arxiv.org/abs/1711.09883#deepmind (which may be why the colonel said that actually running such an experiment was unnecessary).
>If Hinton really, genuinely believes the world is ending, then why is he working on the generational successor to backpropagation
This just misunderstands nerd psychology. You can believe there's a high chance of what you're working on being dangerous and still be unable to stop working on it. As Oppenheimer put it, "when you see something that is technically sweet, you go ahead and do it". Besides, working on fundamental problems rather than scaling and capabilities is probably disconnected enough from immediate danger for Hinton to avoid cognitive dissonance.
> The quotes and other horror-scenario errata also come off a little thin. If Hinton really, genuinely believes the world is ending, then why is he working on the generational successor to backpropagation [1]?
That paper was published 27 Dec 2022. He quit Google in order to warn people about AI on May 1 2023. He realized advancing AI will lead to the world ending and so he _stopped_.
This casts a shadow on the reliability of the whole article. While the retraction (they later stated that this was a thought experiment and NOT a simulation) may or may not be true, the current state of AI would suggest that this kind of chain-of-logic problem-solving is likely out of reach. To put this in here without such a caveat and instead use the 'later denied' language is a strong indicator that we're being sold a story, and not the whole one.
The quotes and other horror-scenario errata also come off a little thin. If Hinton really, genuinely believes the world is ending, then why is he working on the generational successor to backpropagation [1]?
Good way to sell 'news', though. Keep 'em scared, keep 'em reading, I guess.
1 - https://arxiv.org/abs/2212.13345