Why complex reasoning models could make misbehaving AI easier to catch

"In order to build AI that's truly aligned with human interests, researchers should be able to flag misbehavior while models are still "thinking" through their responses, rather than just waiting for the final outputs -- by which point it could be too late to reverse the damage. That's at least the premise behind a new paper from OpenAI, which introduces an early framework for monitoring how models arrive at a given output through so-called "chain-of-thought" (CoT) reasoning."

"Published Thursday, the paper focused on "monitorability," defined as the ability for a human observer or an AI system to make accurate predictions about a model's behavior based on its CoT reasoning. In a perfect world, according to this view, a model trying to lie to or deceive human users would be unable to do so, since we'd possess the analytical tools to catch it in the act and intervene."

Monitorability is defined as the ability for a human observer or an AI system to make accurate predictions about a model's behavior based on its chain-of-thought (CoT) reasoning. The goal is to flag misbehavior while models are still producing intermediate reasoning, enabling intervention before harmful final outputs occur. Longer and more detailed CoT outputs generally correlate with higher monitorability, making outputs easier to predict, though exceptions exist. Methods for detecting red flags in reasoning can uncover deception attempts, but they are not silver-bullet solutions. The research seeks to disentangle pathways connecting user inputs and system outputs to build safer models.

Read at ZDNET