Exploring examples of purpose misgeneralisation – the place an AI system’s capabilities generalise however its purpose does not
As we construct more and more superior synthetic intelligence (AI) techniques, we need to be sure that they don’t pursue undesired targets. Such behaviour in an AI agent is commonly the results of specification gaming – exploiting a poor alternative of what they’re rewarded for. In our newest paper, we discover a extra delicate mechanism by which AI techniques could unintentionally study to pursue undesired targets: purpose misgeneralisation (GMG).
GMG happens when a system’s capabilities generalise efficiently however its purpose doesn’t generalise as desired, so the system competently pursues the fallacious purpose. Crucially, in distinction to specification gaming, GMG can happen even when the AI system is educated with an accurate specification.
Our earlier work on cultural transmission led to an instance of GMG behaviour that we didn’t design. An agent (the blue blob, under) should navigate round its atmosphere, visiting the colored spheres within the right order. Throughout coaching, there’s an “skilled” agent (the purple blob) that visits the colored spheres within the right order. The agent learns that following the purple blob is a rewarding technique.
Sadly, whereas the agent performs effectively throughout coaching, it does poorly when, after coaching, we substitute the skilled with an “anti-expert” that visits the spheres within the fallacious order.
Although the agent can observe that it’s getting unfavourable reward, the agent doesn’t pursue the specified purpose to “go to the spheres within the right order” and as an alternative competently pursues the purpose “comply with the purple agent”.
GMG will not be restricted to reinforcement studying environments like this one. In actual fact, it could actually happen with any studying system, together with the “few-shot studying” of huge language fashions (LLMs). Few-shot studying approaches purpose to construct correct fashions with much less coaching knowledge.
We prompted one LLM, Gopher, to guage linear expressions involving unknown variables and constants, akin to x+y-3. To resolve these expressions, Gopher should first ask in regards to the values of unknown variables. We offer it with ten coaching examples, every involving two unknown variables.
At take a look at time, the mannequin is requested questions with zero, one or three unknown variables. Though the mannequin generalises appropriately to expressions with one or three unknown variables, when there are not any unknowns, it nonetheless asks redundant questions like “What’s 6?”. The mannequin all the time queries the person at the very least as soon as earlier than giving a solution, even when it’s not needed.
Inside our paper, we offer extra examples in different studying settings.
Addressing GMG is essential to aligning AI techniques with their designers’ targets just because it’s a mechanism by which an AI system could misfire. This can be particularly essential as we strategy synthetic common intelligence (AGI).
Think about two doable sorts of AGI techniques:
- A1: Supposed mannequin. This AI system does what its designers intend it to do.
- A2: Misleading mannequin. This AI system pursues some undesired purpose, however (by assumption) can also be sensible sufficient to know that it will likely be penalised if it behaves in methods opposite to its designer’s intentions.
Since A1 and A2 will exhibit the identical behaviour throughout coaching, the opportunity of GMG signifies that both mannequin may take form, even with a specification that solely rewards meant behaviour. If A2 is discovered, it will attempt to subvert human oversight to be able to enact its plans in direction of the undesired purpose.
Our analysis group can be pleased to see follow-up work investigating how probably it’s for GMG to happen in observe, and doable mitigations. In our paper, we recommend some approaches, together with mechanistic interpretability and recursive analysis, each of which we’re actively engaged on.
We’re presently accumulating examples of GMG on this publicly obtainable spreadsheet. When you have come throughout purpose misgeneralisation in AI analysis, we invite you to submit examples right here.