The growing complexity of modern multicloud environments has created a pressing need to converge observability and security analytics.

Security analytics is a discipline within IT security that focuses on proactive threat prevention using data analysis. As attackers become more skilled, threats can become more detrimental to organizations if they go undetected. A proactive approach to application security is essential: by constantly collecting and analyzing data, security analytics enables teams to catch problems before they escalate.

In the past five years, delivering innovation more efficiently has remained at the forefront of customer demand. As environments began to scale with cloud-native and microservices architectures to meet this demand, security approaches didn’t evolve accordingly. The result: Environments are more vulnerable to threats and more difficult to secure. In fact, according to a recent survey, nearly 70% of chief information security officers agreed that vulnerability management has become more difficult as the complexity of their software supply chain and cloud ecosystem has increased.

The ripple effect of increased risk compounds the problem. With more alerts and greater difficulty identifying false positives, security teams experience frustration and burnout. A lack of common tooling, common language, and collaboration further inhibits productivity and prolongs remediation.

At the 2024 Dynatrace Perform conference in Las Vegas, Gerhard Byrne, Dynatrace principal product manager, and Susan St. Clair, principal security solutions engineer, discussed how organizations can take a more proactive approach to threat detection and incident resolution.

During their breakout session, Byrne and St. Clair demonstrated how the Dynatrace platform accelerates remediation by enriching security data with observability context and actionable insights to protect environments against exploitation or lateral movement.

Threat hunting expectations vs. reality

In a perfect world, threat hunting and incident resolution would be a linear, straightforward process. Ideally, after fetching data and filtering, an organization could enrich the findings with observability data to get better insights into the nature of the alert. With these insights, the team could identify the threat and understand the nature of the incident. This allows them to react accordingly and return the system to a secure state.

But with the complexity of modern cloud environments — including the associated software supply chains and siloed toolchains — and the increasing sophistication of today’s attackers, threat hunting is unpredictable. In reality, security teams aren’t aware of all the unknown unknowns in their environments. After fetching, filtering, and enriching information with observability data, security teams might change their hypothesis about what’s occurring. This revision of assumptions might happen multiple times, causing engineers to lose track of previous hypotheses, patterns, and evidence. Keeping threats documented is a challenge: Engineers typically open numerous tabs to maintain context, which is tedious and can create error. Remediating a vulnerability can thus take far longer than anticipated, which can be detrimental when the risk is high.

“As defenders, we need to embrace different paths and possibilities like our adversaries are doing today,” Byrne said. “Just going down a checklist will not help you find new threats.”

Streamline threat hunting and accelerate resolution with Dynatrace Security Investigator

During the conference, Dynatrace announced the new Security Investigator app on the platform. The app enables security teams to investigate threats faster, obtain accurate and observability-enriched results, and maintain context throughout the weaving paths on which security investigations might lead.

Security Investigator demo

St. Clair began her demonstration of the app with the following scenario: She receives a Slack alert that an anomaly was detected and there has been unauthorized access to a Kubernetes cluster monitored via OneAgent.

To begin, St. Clair filtered the alert for a time window of a couple of hours to focus her analysis. Using Dynatrace Query Language in Grail, St. Clair determined what log data was available to her. With each execution, data appears in a query tree. “As I’m building out this investigation, each of these nodes is being created for me automatically,” she said. “As part of that documentation, I can easily go back and forth to see what was executed.”

St. Clair then used the Dynatrace Pattern Language (DPL) to make the data more usable. She used the DPL Architect to create her own patterns in addition to the platform’s out-of-the-box patterns for parsing the data. After running an audit log pattern, she wanted to understand why she received the Slack alert and which object the attacker had accessed so she could then focus on the unauthorized responses. Running this query, suspicious IPs arose, and she saved them as evidence in a new folder she created within the app.

As she continued to execute queries, St. Clair’s hypothesis began to change. She saw substantial traffic in a specific port, which was not necessarily malicious, but it was abnormal enough to warrant additional investigation. Fortunately, the query tree automatically creates new “branches” to support changing hypotheses and help engineers keep track of evidence and patterns. The query tree thus frees security professionals from tedious manual documentation, allowing them to focus entirely on finding the unknown unknown.

Finally, St. Clair found running malware on the system, pivoting the investigation from the initial authentication alert. By the end of an investigation, she had a visual representation of the process from start to finish.

“I can keep track of where I went. [The data is] documented, shareable, collaborative, and available for further investigation,” St. Clair said.

The road ahead: enriched security analytics with Dynatrace

Organizations can benefit most from their security investigations using the abundant data in the Dynatrace platform. The platform’s broad and deep observability identifies where problems initiate as well as their dependencies using PurePath. Teams can use Real User Monitoring and Session Replay to track user-facing activity in real time. This helps them understand how an attacker accessed an application, how they interacted with it, how traces originated, and more. Teams can also create management reports and use Dynatrace Notebooks to easily share their security investigation data with their peers.

“Security is a team sport,” Byrne said. “The next time you’re in a war room, you can be the person who provides insights and conclusions based on the wealth of data that is available at your fingertips with Dynatrace platform.”