Open source Substrate Indexer in Rust with a decentralized Blockchain Explorer, by Mosaic Chain

Proponent: Peter Molnar, CTO of Mosaic Chain: 14dC5pCtc7mvRZuUZA5d4R1uxd7w2T8Pj8HsuWJ92ESgRDU6
Date: 2024.09.01.
Requested budget: 79,520 USDT

Short Description: As mentioned in our previous Polkadot forum post, where we open-sourced our NFT-based consensus mechanism as promised, we are now presenting the concept of the Rust based Substrate Indexer (Mosaic Indexer) and the decentralized blockchain explorer (Mosaic Explorer).

The Mosaic Chain Team aims to decentralize not only the Mosaic Chain but also its associated infrastructures, such as the blockchain indexer and explorer, to ensure they are not reliant on any single entity. In this proposal, the Mosaic Team highlights the benefits of the Rust based Mosaic Indexer and Mosaic Explorer, and explains its functionality in simple terms to the Polkadot Community. Additionally, we showcase the Mosaic Indexer in action through a DEMO version of the UI of Mosaic Chain Explorer. This demo queries on-chain data from the Mosaic Chain’s local testnet via the Mosaic Indexer and displays key on-chain events.

Link to the proposal

Problem statement

The Polkadot ecosystem is built on principles of decentralization, security, and interoperability. However, many of the current data indexing solutions are centralized or rely on specific entities, which can create single points of failure, limit transparency, and reduce the overall trustlessness of the ecosystem. Existing solutions may also involve significant costs or technical barriers that discourage broader adoption.

A Rust based indexer and community owned, decentralized explorer is crucial for the Polkadot ecosystem to fully realize its potential as a truly decentralized Web3 platform. By decentralizing the on-chain data querying through the decentralized explorer, we could eliminate reliance on a single party, enhancing security, transparency, and trust. Additionally, a decentralized approach aligns with Polkadot's ethos, ensuring that all data queries are handled in a trustless manner directly from the parachain nodes.

What is Blockchain Data Indexing and Querying?

Blockchain indexing is a method of organizing on-chain data so that users can quickly and easily query the information.
Some common indexing methods include:

Indexing transaction-related information such as the sender, recipient, timestamp, and transaction hash.
Indexing addresses (accounts) to track their balances, transaction history, and other related data.
Indexing various crypto tokens and NFTs to track token activities and balances.
Indexing smart contract interactions, events, and parameters.
Indexing blocks by block height, timestamp, and other relevant parameters.

API Integration: The indexed data is then made accessible through powerful APIs and user interfaces, enabling users and applications to easily access the data. These APIs mainly include GraphQL endpoints and RESTful APIs.

Solution: The Mosaic Indexer and the Mosaic Explorer

The unique combination of a Rust-based, easily deployable blockchain indexer and a community-driven, decentrally run explorer infrastructure enables seamless transmission of on-chain data from Substrate chains to blockchain explorer interfaces, while it does not rely on third party services.

Our solution for indexing on-chain data in a decentralized manner consists two pillars:

The Mosaic Indexer is a Rust based indexing processor which can query on-chain data directly from parachains to a local database.
The Mosaic Explorer runs on multiple blockchain nodes' hardware and maintains connection via a p2p connection through a VPN and forwards on-chain information to Explorer UIs in a decentralized manner.

You can access the DEMO versions of Mosaic Indexer and Mosaic Explorer from this link: https://demo-indexer.mosaicchain.io/.

Please note that this is not the final version, as our team is still in the process of making refinements and improvements. The perfect version of the open-source code will be released once the development is fully completed.

Rationale behind the new concept of blockchain indexing

The Rust based Mosaic Indexer we provide to the ecosystem aims to broaden the indexing landscape by offering a free alternative for teams with limited financial resources. The Mosaic Explorer Our goal is not to create a branded indexing protocol but to open-source the code of both the Mosaic Indexer and Mosaic Explorer as the new framework for decentralized blockchain indexing that the community can own and run.

The Mosaic Indexer and the Mosaic Explorer is designed in a Web3-native, Polkadot-like fashion as it is based on Rust and together, they create a significant advantage: ultimate decentralization for indexing blockchains.

Our concept for blockchain indexing is straightforward:

We developed the Mosaic Indexer in Rust, rather than TypeScript, making it easily compatible with any Substrate-based chain.
We created “processors” for the Mosaic Indexer that can query fundamental runtime modules, such as the balances pallet and staking pallet. (We are continuing to develop new processors for FRAME pallets)
We built Mosaic Explorer as a special decentralized explorer which runs on a community owned p2p network as it is deployed on multiple blockchain nodes.

Advantages of these open-source tools:

An indexer that can be hosted by anyone for any Substrate chain.
A more cost-effective, free option compared to subscribing indexer models. Not reliant on a third party.
A free codebase that the entire Polkadot community can further develop over time, in the spirit of open-source culture.

With this semi-retroactive proposal, we are also eagerly looking forward to publishing the open-source code for the Mosaic Indexer and Decentralized Explorer on the Mosaic Chain's GitHub, where we already released our NFT based Aura consensus improvement implementation.

We are confident that Mosaic Indexer and Mosaic Explorer will be a gap filler for the ecosystem and the project teams. Please leave your comment below, we welcome any questions from the community.

Here is a summary of the SubSquare post:

Mosaic Chain Team has proposed a Rust based Substrate Indexer (Mosaic Indexer) and a decentralized blockchain explorer (Mosaic Explorer) to decentralize the Mosaic Chain and its associated infrastructures.
The current data indexing solutions are centralized or rely on specific entities, which can create single points of failure, limit transparency, and reduce the overall trustlessness of the ecosystem.
Blockchain indexing is a method of organizing on-chain data so that users can quickly and easily query the information.
The Mosaic Indexer is a Rust based indexing processor which can query on-chain data directly from parachains to a local database, and the Mosaic Explorer runs on multiple blockchain nodes' hardware and maintains connection via a p2p connection through a VPN and forwards on-chain information to Explorer UIs in a decentralized manner.
The Mosaic Indexer and Mosaic Explorer are designed in a Web3-native, Polkadot-like fashion as it is based on Rust and together, they create a significant advantage: ultimate decentralization for indexing blockchains.