Mapping the user experience is one of the most persistent challenges a business can face. Fullstory, a leading behavioral data analytics platform, helps organizations identify pain points and optimize digital experiences by reproducing  user sessions and sharing strong analytics highlighting areas for improvement in the customer's journey. This boosts conversion rates, reduces churn, and enhances customer satisfaction. 

AI has made this even stronger. Fullstory's comprehensive AI-powered autocapture technology, Fullcapture, removes the need for manual instrumentation and uncovers hidden patterns that might otherwise be missed. 

Today, we’ll share how Fullstory leverages Vertex AI serving Gemini 1.5 Pro to strengthen their autocapture technology.

How Vertex AI and AI agents help Fullstory measure the user experience

Think of Fullcapture as a video recorder for your website or app, capturing every interaction in detail. Traditional autocapture methods are more like transcription services, logging only selected highlights and often missing the complete picture. With Fullcapture, no user action goes unrecorded, with minimal impact on device performance. Operating server-side, Fullcapture allows for revisiting any aspect of user behavior as needed. If a new signal is required, it can be easily retrieved from the recorded data without affecting client-side performance.

The table below breaks down how Fullcapture goes beyond traditional autocapture capabilities to give users a deeper understanding of their customer data.

By integrating its Fullcapture capabilities with Google's Vertex AI serving Gemini 1.5 Pro, Fullstory empowers customers to effortlessly analyze this extensive data and focus on what truly matters. Driven by a proactive AI agent, Fullstory enables faster data discovery by highlighting important elements and automatically categorizing user interactions into semantic components, providing even deeper levels of analysis.

AI-powered data discovery 

Data discovery is a 6-step process that involves exploring, classifying, and analyzing data from various sources to uncover patterns and extract actionable insights. This process allows users to visually navigate data relationships and apply advanced analytics to optimize business decisions and performance.

To effectively analyze user behavior, businesses need to identify and label key elements on their websites (e.g., buttons, forms). This process can be tedious and time-consuming. Fullstory’s AI agent, powered by Gemini 1.5 Pro, automates this critical task by scraping data from user interactions and making intelligent decisions at various stages—identifying key elements, determining their significance, and autonomously categorizing them. This multi-stage decision-making process not only streamlines workflows but also ensures businesses can focus on deriving actionable insights rather than manual labeling.

Within Fullstory, "elements" allow users to label UI components based on specific CSS selectors. A CSS selector is a pattern used to target elements in a webpage, such as classes, IDs, or attributes. For instance, a "Checkout Button" element might be created with the selector .checkout-page-container [data-testid="primary-button"]. These labels help categorize UI components and utilize them for product analytics. Broad semantic labeling is crucial for long-term success with Fullstory, and automating this process simplifies workflows for users.

Vertex AI with Gemini 1.5 Pro offers a unique opportunity to add a human touch at scale. It proactively identifies and describes web components, ultimately providing actionable insights that benefit Fullstory customers. Gemini 1.5 Pro is trained on extensive web expertise, including web implementation from CSS and web frameworks like React, along with a vast dataset of website images.

For example, the model can analyze a website screenshot and accurately describe its components, understanding both the overall structure, visible text, and the logical structure of the web page. This understanding can be further enhanced with web implementation details, such as CSS selectors, to gain a deeper understanding of specific components.

Optimizing for accurate element identification

Fullstory employs a meticulous approach to ensure the model provides high-quality element suggestions in four critical ways:

1. Strategic prompt engineering: Complex tasks are broken down into smaller, manageable steps, allowing the model to build a foundational understanding and deliver consistent and accurate results.

2. Pre-filtering with heuristics: Heuristics pre-filter potential elements before requests are sent to Vertex AI with Gemini, optimizing efficiency.

3. Validation with Vertex AI with Gemini: The model's expertise validates potential elements, ensuring that only useful suggestions are presented.

4. Contextualized suggestions: Each suggestion includes a screenshot, CSS selector, and occurrence metrics, providing valuable context for informed decision-making.

This process ensures effective and efficient use of Gemini's AI capabilities, resulting in accurate and valuable element suggestions.

Pinpointing and perfecting: How we identify and label key web elements

Optimizing the digital experience requires identifying and understanding key web elements. These elements need to be defined in a way that remains resilient to website changes. This presents a challenge, given the diverse nature of websites and user behaviors.

In the real world, an element selector can look something like:

While metrics like "most clicked buttons" provide some insight, a more sophisticated approach is needed to uncover elements that drive engagement, signal errors, or reveal hidden opportunities. Effective management of potentially long and brittle element selector definitions is also crucial for maintaining data quality.

The search for meaningful elements

Fullstory captures every user interaction, generating a wealth of data. The platform continuously monitors unrecognized components, prioritizing:

• New feature discovery: Identifying elements on newly launched feature pages.

• Power user behavior: Understanding how experienced users interact with the website.

• Error signals: Detecting elements with CSS that suggests potential errors.

• Content analysis: Analyzing elements containing text that indicates user intent or potential issues.

These searches utilize CSS selectors to precisely target elements for granular analysis and efficient refinement.

Analyzing user behavior in Fullstory often involves crafting complex CSS selectors. With Vertex AI and Gemini 1.5 Pro, this process is simplified.

• Deep indexing: Components of website CSS selectors and associated events are tokenized and indexed, enabling efficient searching through countless variations.

• Semantic relevance: The model understands the meaning behind selectors. For example, when tracking an "Add to Cart" button, the model recognizes that the class .add-to-cart is more relevant than a generic class like .primary-button.

• Powerful search: Combining semantic understanding with advanced search capabilities, the model identifies the best match for selectors.

This results in high-quality selectors without requiring in-depth CSS expertise, allowing users to focus on uncovering valuable insights from their Fullstory data.

Here’s an example of a Fullstory element being optimized:

The Importance of accurate labeling

Once an element is identified through its CSS selector, accurate labeling becomes crucial. This involves:

• Name: A clear and concise name reflecting the element's function.

• Description: A detailed explanation of the element's purpose and behavior.

• Role: Assigning a predefined role from Fullstory's library (e.g., "Add to Cart Button," "Validation Error").

Adding context to the equation

To ensure high-quality and consistent element labeling, Vertex AI with Gemini 1.5 Pro leverages Fullstory's rich data and advanced AI capabilities to provide comprehensive context:

• Visual representation: Screenshots of the element in action, generated from session playbacks.

• Textual analysis: Examining text occurrences associated with the element.

• Location tracking: Identifying the URLs where the element appears.

This approach, combining Fullstory's data capture with Vertex AI and Gemini 1.5 Pro, allows for AI-powered analysis that moves beyond basic metrics to truly understand user behavior. By identifying and labeling key web elements with precision and context, businesses can unlock valuable insights and create exceptional digital experiences.

Delivering real-world results with Vertex AI API for Gemini 1.5 Pro

The collaboration between Fullstory and Google Cloud has yielded tens of thousands of element suggestions generated for customers, with Gemini 1.5 Pro intelligently filtering out a significant portion of irrelevant suggestions. The model has also identified numerous error elements that were previously unrecognized.

Beyond element identification, the mapping between element configuration and screenshots has opened up new opportunities for improving site configuration and enhancing analytics. This ongoing collaboration between Fullstory and Google Cloud continues to drive significant value for customers, empowering them to gain a deeper understanding of user behavior and optimize their digital experiences.

Ready to unlock the power of behavioral data? Visit Fullstory on Google Cloud Marketplace today! With Fullstory, you can gain a deeper understanding of your customers by uncovering hidden insights into their behavior and identifying key opportunities to optimize their digital experience.

Want to learn more about leveraging AI to enhance your Fullstory experience? Explore Fullstory's documentation or try out this collaboration to see how AI can accelerate your journey.