Statistically Undetectable Backdoors in Deep Neural Networks
Worth knowing that backdoors in deep neural networks can remain hidden even when users have full access to a model’s weights.
Research shows that an adversarial trainer can plant backdoors in a large class of deep, feedforward neural networks. These backdoors are statistically undetectable in a white-box setting. This means that a backdoored model and an honestly trained model look almost identical, even if you have the full description of the model and all its weights. The backdoor allows for specific, invariance-based adversarial examples for every input. It maps distant inputs to unusually close outputs. Without the secret information, it is provably impossible to generate these examples in a reasonable amount of time.
This highlights a fundamental power asymmetry between model trainers and the organizations that use their AI. For the public sector, this has significant implications for AI governance and security. It suggests that white-box access is not a complete solution for ensuring model integrity. When government agencies procure AI for public services, they must recognize that standard audits may not catch sophisticated, hidden flaws. This underscores the need for trust frameworks that account for risks that are mathematically invisible to standard inspection tools.
How can public institutions build reliable trust when model inspection has these clear limits?
#AI #Cybersecurity #PublicSector #AIGovernance #MachineLearning
https://arxiv.org/abs/2607.09532v1