Neural nets are exceptionally vulnerable to goal self-hack alignmentforum.org

The Alignment Forum post argues that an AI optimizing a goal through an LLM could steer into situations where the model’s proxy for that goal diverges from the intended objective. It claims neural networks’ local linearity and high-dimensional inputs make such “Goodhart zones” reachable through small changes, drawing on adversarial-example research and a 2023 paper by Andy Zou and co-authors on transferable attacks against language models.