SQD (fka Subsquid) - Public data indexing Infrastructure for Polkadot and Kusama (Q1 2024)

Proponent: Subsquid Lab Official - 13bfKSQXoBn3AMLtZaW6BKv797fqZzsD3PYF6xpJDir3tdoE

Beneficiary: Subsquid Lab Official - 13bfKSQXoBn3AMLtZaW6BKv797fqZzsD3PYF6xpJDir3tdoE

Contact Details: Subsquid Labs GmbH 6300 Zug, Switzerland mf@subsquid.io

Short description: Ongoing development and maintenance costs for public SQD Archives

Archive raw data: Archive Infrastructure Metrics - January, February, March

Requested DOT: 47,597.91 DOT = $202 291.10

Previous proposals: https://polkadot.polkassembly.io/referenda/995

Motivation

A) Archives

Archives are an important piece of Polkadot and Kusama data infrastructure provided by SQD (formerly Subsquid). They provide access to on-chain data for public chains in the Polkadot ecosystem. They are being used as a data source for Squids and for efficient data exploration and ad-hoc queries.

The Archives, as performant data sources of historical on-chain data, are critical to the operation of backend APIs run by major projects in the Polkadot ecosystem (the list is not exhaustive):

• Polkassembly
• Talisman
• RMRK
• SubWallet 
• KodaDot
• ChainSafe
• Multix
• Phala dApp
• Subsocial + Polkaverse
• Stellaswap
• Beamswap
• Polkascan 
• Calamar 
• Moonsama/Exosama
• Reef
• Manta Wallet
• ZooDAO
• Zenlink

Since the first quarter of 2024, SQD has transitioned to a paid model for supporting parachains. However, public chains continue to be available for free use. Throughout the first quarter of 2024, we provided a free public data indexing service for Substrate blockchains, allowing interested parties to access indexed data on Substrate Events, Extrinsics, Storage Items, and EVM logs. Below is a list of the public chains we currently support at no cost:

• polkadot

• kusama

• asset-hub-kusama

• asset-hub-polkadot

• collectives

• collectives-polkadot

• rococo

• statemine

• statemint

• westend-collectives

As of Q1 2024, SQD has sunsetted support for Giant Squid, with LimeChain taking over its further development, detailed in the proposal at https://polkadot.polkassembly.io/treasury/758

B) Contribution

During the first quarter of 2024, we made significant contributions to support ink! v5 and address actual use cases that were highly demanded by the community:

• We updated @subsquid/ink-typegen and @subsquid/ink-abi to support ink! v5.

• We enabled handling of multi-block migrations in subsquid/substrate-processor.

Achievements

• FireSquid Sunset and Full Migration to ArrowSquid: With the launch of ArrowSquid, we have significantly enhanced our processing capabilities by enabling the ingestion of unfinalized blocks directly from network nodes. This advancement allows for bypassing the initial need for archiving, facilitating near real-time responses to on-chain activities with subsecond latency. ArrowSquid’s capability to handle forks and rewrite database contents when orphaned blocks are indexed, along with its support for detailed transaction execution receipts, traces, and state diffs, marks a significant step forward in our technology. Following the complete sunset of FireSquid, all consumers successfully migrated to ArrowSquid by Q1’24

• Decentralizing Archives Tethys Testnet: Following the successful sunset of the Panthalassa testnet, which had initially launched in partnership with CoinList with over 40,000 participants, we have now launched the Tethys testnet. The Panthalassa testnet, renowned for deploying several thousand indexers to index substrate chains including Polkadot and Kusama chains, has paved the way for its successor. The Tethys testnet introduces significant enhancements, including improved query execution, increased network stability, and enhanced communication between nodes.

• The mainnet bootstrap: Following the successful testnets, we have bootstrapped the mainnet. Throughout these test phases, we gathered valuable insights on enhancing quality, query execution speed, and reliability. These advancements have been fully implemented in the mainnet, ensuring a robust and efficient network.

• The biggest Polkadot projects have fully migrated or are in the process of finishing the migration from FireSquid to ArrowSquid, including: Polkassembly, Subsocial, HydraDX, KodaDot, Talisman, Multix Multisig,  RMRK, Apillon, StellaSwap

Statistics and data

Over the course of Q1 2024:

• Served a total of 1,725,006,525,189 (~1.57 terabytes) of data from the substrate based archives in Q1 2024

• Served ~ 80m archive requests from substrate based archives in Q1 2024

• The highest daily total data transfer recorded for Substrate was 313.12 Gb within a single 24-hour period.

Costing

This proposal encompasses the combined costs for running Archives for Polkadot, Kusama, and other public chains, along with the development contributions, amounting to a total of 

47,597.91 DOT — Raw data for Archives can be seen here.

The proposal is submitted towards the Polkadot treasury. Here is a summary of the running costs for Archives, in a tabular format:

Contribution costs: