As blockchain and Web3 continue to revolutionize financial services and other industries, tech professionals have had to adapt to changing requirements for developing and managing IT environments. Application development has been especially impacted by the move from Web2 to Web3. Fortunately, NEAR is working to ease the development challenges of an open web.

As an easy-to-use, carbon-neutral blockchain protocol, NEAR was built from the ground up to be secure, scalable, and performant. “NEAR is developer-friendly and provides robust services for Web3 applications to publish decentralized apps with ease,” says Eduardo Ohe, a staff data engineer on the NEAR data platform team. “My team’s goal is to help internal and external teams to query their data in the NEAR blockchain as easily as possible.”

NEAR chose to work with Google Cloud and partner Databricks to improve data querying experiences and dramatically accelerate retrieving analytics from apps in its NEAR blockchain operational system (B.O.S).

Optimizing data querying in blockchain

NEAR previously relied on indexers made by customers or supplied by the NEAR Protocol. The indexers used a complex process to retrieve blockchain data, normalize it, and load it into PostgreSQL.

“Our old process took a lot of time and resources,” says Ohe. “It required constant monitoring of both indexers and databases to make sure data flowed properly. Additionally, the indexers took weeks to build and set up, meaning developers would not be able to query data or run analytics on the apps they built in the NEAR B.O.S.”

Recognizing the need to make a change, NEAR looked to the Databricks Spark Engine as a way to improve Blockchain Data offload and indexing. The team also knew they could implement Databricks Lakehouse on Google Cloud, which is tightly integrated with BigQuery.

“The decision to migrate to Databricks Platform was clear,” explains Ohe. “The Google Cloud and Databricks teams are extremely collaborative and helped us design the best solution for us. The partnership between Databricks and Google Cloud enabled us to create an integrated IT environment built on best practices from both companies. It proved to be a very efficient, straightforward strategy to achieve our goals.”

NEAR also has its public dataset on BigQuery, offering its users more blockchain data at speed. The public dataset along with Databricks on Google Cloud allows developers to access data in near real-time and query applications using simple SQL within minutes of launch. This new setup also eliminated the maintenance demands of legacy indexers.

More sophisticated systems, less maintenance

NEAR uses Google Kubernetes Engine (GKE) to run and manage its compute resources, and it acts as the underlying layer to launch Databricks compute clusters.

“Databricks on GKE is easy to scale and allows our specialists to manage their processes inside a standardized, reliable Kubernetes policy,” says Ohe. “We don’t have issues with clusters going down — GKE brings stability to the Databricks process we run.”

Scalability is critical for NEAR, as new application onboarding within its B.O.S can bring millions of users on the same day to the same chain. Autoscaling in GKE enables NEAR to deal with these spikes smoothly without bogging internal teams down with maintenance.

“The reliability we’ve achieved builds confidence in the data that we provide to users,” says Ohe. “Our internal teams can see these massive spikes in users without worrying people are not getting the insights that they need.”

Since launching the Databricks on Google Cloud solution and NEAR BigQuery Public Dataset, NEAR has been able to provide historic on-chain data queries at scale and speed. Its intuitive systems provide easy-to-use querying capabilities that show results in minutes rather than weeks. The company also sees cost benefits because it eliminates the need to store and process bulk NEAR Protocol data.

These savings have allowed NEAR to provide queries of NEAR data for free in BigQuery up to one terabyte.

Unlocking blockchain for more developers

Internally, reception of the new Databricks on Google Cloud and BigQuery deployment has been positive, which helps NEAR attract and retain talent. Externally, the new setup furthers the company’s goal of making an open web more accessible to more people.

“In the past, developers had to build analytics in their apps and specialized machines,” says Ohe. “All of this complexity and cost could prevent developers from using the NEAR protocol. Now that we’ve removed obstacles with the help of Databricks on Google Cloud, developers can onboard more easily and start experimenting quickly.”

Looking ahead, NEAR expects its platform to be an important resource for developers wanting to find decentralized apps. To help grow its audience and get more developers familiar with the NEAR blockchain, the company offers its solutions on Google Cloud Marketplace, which is all part of its goal to drive Web3 adoption.

“Open web will grow faster in the coming years, and we’re committed to giving developers the tools they need to build and maintain Web3 applications,” says Ohe. “Google Cloud and Databricks are our partners in this journey. We trust in the scalability and reliability of their services and are confident we can create best-in-class solutions for developers.”

To learn more about how partners like Databricks can help augment your IT performance with Google Cloud, check out the Google Cloud Marketplace.