The Evolution of Blockchain Data Indexing Technology: From Raw Nodes to AI-Driven Full-Chain Data Services

1. Introduction

Since the first decentralized applications in 2017, to the flourishing of various financial, gaming, and social applications on blockchain today, have we ever considered the sources of the various data used in the interactions of these applications?

In 2024, AI and Web3 became hot topics. In the field of artificial intelligence, data is like the lifeblood of its growth and evolution. Just as plants need sunlight and water to thrive, AI systems similarly rely on vast amounts of data to continuously "learn" and "think". Without data, no matter how sophisticated the algorithms of AI are, they cannot exert their intended intelligence and effectiveness.

This article will conduct an in-depth analysis of the evolution of blockchain data indexing in the process of industry development from the perspective of blockchain data accessibility, and compare established data indexing protocols with emerging blockchain data service protocols, with a particular focus on the similarities and differences in data services and product architecture features of the new protocols that incorporate AI technology.

2. The Complexity and Simplicity of Data Indexing: From Blockchain Nodes to Full Chain Database

2.1 Data Source: Blockchain Node

Blockchain is described as a decentralized ledger. Blockchain nodes are the foundation of the entire network, responsible for recording, storing, and disseminating all transaction data on the chain. Each node has a complete copy of the blockchain data, ensuring the decentralized nature of the network. However, for ordinary users, building and maintaining a node is not an easy task, requiring specialized technical skills and high costs.

To solve this problem, Remote Procedure Call (RPC) node providers have emerged. These providers are responsible for the costs and management of nodes, providing data through RPC endpoints. Users can access Blockchain data without the need to build their own nodes. Public RPC endpoints are free but have rate limits. Private RPC endpoints offer better performance but are inefficient for complex data queries. The standardized API interface provided by node providers lowers the threshold for users to access on-chain data.

2.2 Data Parsing: From Prototype Data to Usable Data

The data obtained from blockchain nodes is often raw data that has been encrypted and encoded. Although this data retains the integrity and security of the blockchain, its complexity increases the difficulty of data parsing. For ordinary users or developers, directly handling this prototype data requires a substantial amount of technical knowledge and computing resources.

The data parsing process is particularly important in this context. By converting complex prototype data into a more understandable and operable format, users can intuitively understand and utilize this data. The success or failure of data parsing directly determines the efficiency and effectiveness of blockchain data applications, making it a key step in the entire data indexing process.

2.3 The Evolution of Data Indexers

As the volume of Blockchain data increases, the demand for data indexers is growing. Indexers play an important role in organizing on-chain data and sending it to databases for easy querying. The way indexers work is by indexing Blockchain data and making it available at any time through a SQL-like query language. By providing a unified query interface, indexers allow developers to quickly and accurately retrieve the information they need using a standardized query language.

Different types of indexers optimize data retrieval in various ways:

1. Full Node Indexer: Extracts data directly from full Blockchain nodes, ensuring data completeness and accuracy, but requires a large amount of storage and processing power.
2. Lightweight Indexer: Relies on full nodes to fetch specific data as needed, reducing storage requirements but potentially increasing query time.
3. Dedicated Indexer: Optimizes retrieval for specific use cases for certain types of data or specific Blockchains.
4. Aggregated Indexer: Extracts data from multiple Blockchains and sources, including off-chain information, providing a unified query interface.

Currently, mainstream indexer protocols not only support multi-chain indexing but also customize data parsing frameworks for the data needs of different applications.

The emergence of indexers has greatly improved the efficiency of data indexing and querying. Compared to traditional RPC endpoints, indexers can efficiently index large amounts of data and support high-speed queries. These indexers allow users to perform complex queries, easily filter data, and analyze it after extraction. Additionally, some indexers also support aggregation from multiple Blockchain data sources. By running distributed across multiple Nodes, indexers not only provide stronger security and performance but also reduce the risk of interruptions and downtimes that centralized RPC providers may bring.

2.4 Full Blockchain Database: Aligning to Flow Priority

Using index nodes to query data often means that the API becomes the sole gateway to digest on-chain data. However, when projects enter the expansion phase, more flexible data sources are often required. As application demands become more complex, primary data indexers and their standardized indexing formats gradually struggle to meet the increasingly diverse query needs.

In modern data pipeline architecture, the "stream-first" approach has become a solution to overcome the limitations of traditional batch processing, enabling real-time data ingestion, processing, and analysis. The development of blockchain data service providers is also moving towards building blockchain data streams, as traditional indexing service providers have successively launched products for obtaining real-time blockchain data in a data stream manner.

These services aim to address the need for real-time analysis of Blockchain transactions and provide more comprehensive query capabilities. By redefining the challenges of on-chain data from the perspective of modern data pipelines, we can view the management, storage, and delivery of on-chain data in a whole new light.

3. AI + Database? In-depth comparison of data indexing protocols

3.1 The Graph

The Graph network achieves multi-chain data indexing and query services through a decentralized network of nodes, facilitating developers to conveniently index Blockchain data and build decentralized applications. Its main product models are the data query execution market and the data indexing cache market.

Subgraphs are the fundamental data structure in The Graph network, defining how to extract and transform data from the Blockchain into a queryable format. Anyone can create a subgraph, and multiple applications can reuse these subgraphs.

The Graph network consists of four key roles: indexers, curators, delegates, and developers, who together provide data support for web3 applications.

Currently, The Graph has shifted to a fully decentralized subgraph hosting service, where circulating economic incentives among different participants ensure the system operates.

The Graph's products are also rapidly evolving in the AI wave. The tools AutoAgora, Allocation Optimizer, and AgentC developed by Semiotic Labs enhance the performance of the ecosystem in various aspects.

3.2 Chainbase

Chainbase is a full-chain data network that integrates all Blockchain data into one platform. Its unique features include:

• Real-time Data Lake
• Dual-Chain Architecture
• Innovative Data Format Standards
• Encrypted World Model

These features make Chainbase stand out in the Blockchain indexing protocol, with a particular focus on the accessibility of real-time data, innovative data formats, and the creation of smarter models to enhance insights through the combination of on-chain and off-chain data.

Chainbase's AI model Theia is a key highlight that distinguishes it from other data service protocols. Theia is based on the DORA model developed by NVIDIA, combining on-chain and off-chain data along with temporal and spatial activities to learn and analyze encryption patterns, and respond through causal reasoning.

3.3 Space and Time

Space and Time (SxT) aims to create a verifiable computation layer that extends zero-knowledge proofs on a decentralized data warehouse, providing trusted data processing for smart contracts, large language models, and enterprises.

SxT introduces Proof of SQL technology, an innovative zero-knowledge proof technique that ensures SQL queries executed on a decentralized data warehouse are tamper-proof and verifiable. This approach changes the resource consumption of multiple nodes redundantly indexing the same data under the consensus mechanism, enhancing the overall performance of the system.

SxT also collaborates closely with Microsoft's AI Innovation Lab to accelerate the development of generative AI tools, making it easier for users to process blockchain data through natural language.

Conclusion and Outlook

Blockchain data indexing technology has evolved from the initial node data sources, through the development of data parsing and indexing, to the AI-enabled full-chain data services, undergoing a gradual improvement process. The continuous evolution of these technologies has not only improved the efficiency and accuracy of data access but also brought users an unprecedented intelligent experience.

With the continuous development of new technologies such as AI and zero-knowledge proofs, Blockchain data services will become further intelligent and secure. In the future, Blockchain data services will continue to play an important role as infrastructure, providing strong support for industry progress and innovation.