AI is not replacing us yet, but those who know how to use AI will replace those who do not know how to use AI.

Artificial intelligence is on everyone’s mind right now, especially the cybersecurity industry. In a constant game of whack-a-mole, both defenders and attackers are harnessing AI to tip the balance of power in their respective favor. Before we can understand how defenders and attackers leverage AI, we need to acknowledge the three most common types of AI models currently in circulation.

How Three AI Models Augment Defenders

• Imagine generating an AI summary, a human-readable report of all security events and alerts with a click of a button, which even a non-technical person can read and understand. That’s a great example of Generative AI in action.
• Imagine a converged cloud-native network and security philosophy like SASE (Secure Access Service Edge) that processes trillions of network flows. Practically speaking, it’s impossible for any individual (or even teams) to sift through such a large volume of network and security data to identify malicious patterns. Unsupervised machine learning can not only find patterns, but it can also correspond it to other security events.
• Imagine analyzing all the security and technical data, finding patterns and predicting attacks before they happen: A dream come true for security researchers thanks to supervised machine learning.

The Misuse of AI Models by Threat Actors

Despite the media hype, the usage of AI by cybercriminals is still at nascent stage. This doesn’t mean that AI is not being exploited for malicious purposes, but it’s also not causing the decline of human civilization like some purport it to be. Cybercriminals use AI for very specific tasks, namely:

• Writing hyper-targeted BEC emails.
• Creating new variants of existing malware to evade detection (a.k.a., polymorphic malware).
• Scanning and analyzing code to identify vulnerabilities in target systems.
• Creating video and voice impersonations (synthetic media, deepfakes) for social engineering attacks.

Threat actors have also begun creating their own private and uncensored models (such as PoisonGPT, WolfGPT, and FraudGPT) because publicly available AI models like ChatGPT receive ethical training and are governed by content moderation rules that apply a degree of censorship. Some of these tools, like WormGPT, have been ridiculed for being cheap connectors to ChatGPT and still prone to the same restrictions.

How Cybercriminals Attack, Abuse, and Weaponize LLMs

There are six main ways threat actors attack and abuse Large Language Models.

1. By targeting the prompt: Let’s say a sandbox uses AI to analyze malware code. Attackers insert a “malicious” prompt in the code that misguides its prediction or detection. This technique is called “prompt injection”. There are also other adversarial prompting techniques such as obfuscation and “Do Anything Now” that attackers can use to deceive, compromise, or weaponize AI with words.
2. By targeting the response: What happens if attackers analyze some piece of code or a dataset that includes private or confidential information such as hard-coded passwords or trade secrets. This insecure output handling can lead to data leakage and other security incidents.
3. By targeting the model: AI systems are susceptible to biases. An AI security system might categorize traffic from Japan as low risk. Attackers might take advantage of this flaw and use a VPN to mask their traffic. Attackers can also exploit AI hallucinations. A threat actor encounters a hallucination of an unpublished package, creates a malicious payload with the same name, publishes it, and fools the system into thinking it is part of the system.
4. By targeting the training data: The media frequently mentions attacks on AI training data however, it is quite difficult to gain access to that. Attackers are more likely to use techniques like “feedback poisoning,” where human feedback is used to manipulate the LLM. For example, when you ask an LLM a question, you receive an answer with a “thumbs up” or “thumbs down” feedback option, which helps the model learn what is relevant or irrelevant. This can be used by attackers to manipulate the LLM.
5. By targeting the infrastructure: AI models are hosted on cloud computing servers and data centers. Attackers can launch a distributed denial of service (DDoS) attack in an attempt to disrupt or disable the technology. Supply chain attacks are also a possibility. Attackers can attack a supply chain partner and use privilege escalation and lateral movement to manipulate or disable the AI model.
6. By targeting humans, APIs, or systems using the results: Malicious actors can infect victim laptops with malware and use GenAI to hijack a company’s financial transactions. LLMs have their own APIs, which third parties can use to build custom instructions and integrations. If these integrations have vulnerabilities, bad actors can exploit these or build malicious plugins to run man-in-the-middle types of attacks.

AI is not replacing us yet, but those who know how to use AI will replace those who do not know how to use AI. With technology evolving so fast and in so many simultaneous directions, it is difficult to stay updated on its evolution and mounting risks. Any rapid development like AI introduces many new tools, tactics, and opportunities for exploitation previously deemed unimaginable. Security teams must invest in advanced and converged security models that provide holistic overview and management of diverse attack surfaces. Utilizing machine learning algorithms can further improve threat detection and response capabilities.