Interviews with AI red team heads at Microsoft, Google, Nvidia, and Meta on why breaking AI models matters for safety, the challenges of fixing them, and more
Rashi Shrivastava / Forbes : X: @forbes , @rashishrivast18 , @evijitghosh , @forbes , @rashishrivast18 , and @rashishrivast18 . LinkedIn: Rashi Shrivastava X: @forbes : Forbes spoke to the leaders of AI red teams at Microsoft, Google, Nvidia and Meta, who are tasked with looking for vulnerabilities in AI systems so they can be fixed. “You will start seeing ads about ‘Ours is the safest,’” predicts one AI security expert. https://www.forbes.com/... Rashi Shrivastava / @rashishrivast18 : At hacking events, red teamers have been able to make AI models to cough up credit card information and spin political misinformation. Experts say AI red teams will be the moat of the game going forward. https://www.forbes.com/... Avijit Ghosh / @evijitghosh : I had a great time discussing DEFCON and potential pitfalls of Generative AI with Rashi for @Forbes! https://www.forbes.com/... @forbes : AI red teamers are often walking a tightrope, balancing safety and security of AI models while also keeping them relevant and usable. https://www.forbes.com/... Rashi Shrivastava / @rashishrivast18 : Forbes spoke to the leaders of AI red teams at Microsoft, Google, Nvidia and Meta about how breaking AI models has come into vogue and the challenges of fixing them. https://www.forbes.com/... Rashi Shrivastava / @rashishrivast18 : With tech giants like Google, Meta, Nvidia and Microsoft racing to ship new generative AI tools, AI red teams, which hack models to ensure they are safe and secure, play a crucial role, often balancing safety while keeping AI tools useful and relevant. https://www.forbes.com/... LinkedIn: Rashi Shrivastava : With tech giants racing to ship new generative AI tools, AI red teams, which hack models to ensure they are safe and secure, play a crucial role …
Context & Ripple Effects
This coverage places AI red teaming at several major model developers in an arc that began with Facebook’s internal effort to hack its own AI systems to identify blind spots before outside attackers do. The interviews show the practice becoming a shared operational concern across Microsoft, Google, Nvidia, and Meta rather than a one-company security function.
The later release of Microsoft’s PyRIT risk-testing tool illustrates the direction implied here: turning adversarial probing from a specialist exercise into a more repeatable part of generative-AI development and deployment.
First-order effects
- Red-team findings give the named companies a mechanism to surface harmful outputs, data leakage, and other model vulnerabilities for remediation before or alongside broader use.
- Safety and engineering teams must contend with the practical difficulty highlighted in the interviews: discovering a failure mode does not make it straightforward to fix without affecting a model’s usefulness.
Second-order effects
- As vendors increasingly present safety as a differentiator, red-team capability becomes part of how enterprise customers and partners evaluate model providers, not merely an internal security expense.
- Repeatable testing tools can shift adversarial evaluation toward developer workflows, reducing reliance on one-off exercises while making uncovered risks easier to document and compare.
Third-order effects
- If this pattern holds, operational AI assurance will become a durable layer of the model-development stack, with testing evidence and remediation processes carrying more weight alongside model performance.
- The same testing practice can inform more explicit limits on deployment and access for dual-use capabilities, as reflected in Meta’s later definition of AI systems it considers too risky to release.
The trend: AI labs are moving from ad hoc model breakage exercises toward operationalized assurance processes that tie safety testing to release and access decisions.