As security pros worry about AI taking their jobs, researchers at Microsoft insist that effective red-teaming still relies on human expertise, cultural awareness, and emotional intelligence — qualities that can’t be replicated by machines.

The software giant says its AI red team rigorously tested more than 100 generative AI products and determined that human ingenuity remains crucial to uncovering vulnerabilities and anticipating how hackers might exploit these systems.

According to a whitepaper from Redmond’s AI red team, tools like its open source PyRIT (Python Risk Identification Toolkit) can streamline the simulated hacks but, in the end, human involvement remains irreplaceable in addressing nuanced risks.

In specialized areas like cybersecurity medicine, chemical or biological risk, Microsoft insists that human-powered subject matter expertise is mandatory to properly assess AI responses with precision, far beyond the capabilities of language models.

“On multiple operations, we have relied on [humans] to help us assess the risk of content that we were unable to evaluate ourselves or using LLMs,” Microsoft said, arguing that “it is important for AI red teams to be aware of these limitations.”

The company’s research team also called attention to what it described as “cultural competence” where red-teams must account for linguistic and cultural variations to identify security risks that might be overlooked by AI models trained primarily on English datasets.

“The human element of AI red teaming is perhaps most evident in answering questions about AI safety that require emotional intelligence,” the research team said, noting that questions like “how might this model response be interpreted in different contexts?” and “do these outputs make me feel uncomfortable?” can only be parsed by human operators.

“Ultimately, only human operators can assess the full range of interactions that users might have with AI systems in the wild,” the company added.

The paper also includes a case study on how Microsoft’s red-team investigated “psycho-social harms” by evaluating how a chatbot responds to users in distress and warned that red-teamers can be exposed to disproportionate amounts of “unsettling and disturbing AI-generated content.” 

“This underscores the importance of ensuring that AI red-teams have processes that enable operators to disengage when needed and resources to support their mental health,” the company added.

The researchers warned that the integration of generative AI models into modern applications has introduced novel attack vectors, including one case where an outdated FFmpeg component in a video processing AI application introduced a server-side request forgery (SSRF) bug that allows malicious hackers to  escalate system privileges.

“AI models have expanded attack surfaces by introducing new vulnerabilities,” the team said, noting that prompt injections exploit the fact that AI models often struggle to distinguish between system-level instructions and user data.   

Related: Microsoft Releases Red Teaming Tool for Generative AI

Related: Microsoft Shares Guidance and Resources for AI Red Teams

Related: Google Open Sources AI-Aided Fuzzing Framework

Related: Google Brings AI Magic to Fuzz Testing With Eye-Opening Results