- Treasury/
- Proposals/
- #937/
SQD (fka Subsquid) - Public data indexing Infrastructure for Polkadot and Kusama (Q1 2024)
Proponent: Subsquid Lab Official - 13bfKSQXoBn3AMLtZaW6BKv797fqZzsD3PYF6xpJDir3tdoE
Beneficiary: Subsquid Lab Official - 13bfKSQXoBn3AMLtZaW6BKv797fqZzsD3PYF6xpJDir3tdoE
Contact Details: Subsquid Labs GmbH 6300 Zug, Switzerland mf@subsquid.io
Short description: Ongoing development and maintenance costs for public SQD Archives
Archive raw data: Archive Infrastructure Metrics - January, February, March
Requested DOT: 47,597.91 DOT = $202 291.10
Previous proposals: https://polkadot.polkassembly.io/referenda/995
Motivation
A) Archives
Archives are an important piece of Polkadot and Kusama data infrastructure provided by SQD (formerly Subsquid). They provide access to on-chain data for public chains in the Polkadot ecosystem. They are being used as a data source for Squids and for efficient data exploration and ad-hoc queries.
The Archives, as performant data sources of historical on-chain data, are critical to the operation of backend APIs run by major projects in the Polkadot ecosystem (the list is not exhaustive):
- Polkassembly
- Talisman
- RMRK
- SubWallet
- KodaDot
- ChainSafe
- Multix
- Phala dApp
- Subsocial + Polkaverse
- Stellaswap
- Beamswap
- Polkascan
- Calamar
- Moonsama/Exosama
- Reef
- Manta Wallet
- ZooDAO
- Zenlink
Since the first quarter of 2024, SQD has transitioned to a paid model for supporting parachains. However, public chains continue to be available for free use. Throughout the first quarter of 2024, we provided a free public data indexing service for Substrate blockchains, allowing interested parties to access indexed data on Substrate Events, Extrinsics, Storage Items, and EVM logs. Below is a list of the public chains we currently support at no cost:
polkadot
kusama
asset-hub-kusama
asset-hub-polkadot
collectives
collectives-polkadot
rococo
statemine
statemint
westend-collectives
As of Q1 2024, SQD has sunsetted support for Giant Squid, with LimeChain taking over its further development, detailed in the proposal at https://polkadot.polkassembly.io/treasury/758
B) Contribution
During the first quarter of 2024, we made significant contributions to support ink! v5 and address actual use cases that were highly demanded by the community:
We updated @subsquid/ink-typegen and @subsquid/ink-abi to support ink! v5.
We enabled handling of multi-block migrations in subsquid/substrate-processor.
Achievements
FireSquid Sunset and Full Migration to ArrowSquid: With the launch of ArrowSquid, we have significantly enhanced our processing capabilities by enabling the ingestion of unfinalized blocks directly from network nodes. This advancement allows for bypassing the initial need for archiving, facilitating near real-time responses to on-chain activities with subsecond latency. ArrowSquid’s capability to handle forks and rewrite database contents when orphaned blocks are indexed, along with its support for detailed transaction execution receipts, traces, and state diffs, marks a significant step forward in our technology. Following the complete sunset of FireSquid, all consumers successfully migrated to ArrowSquid by Q1’24
Decentralizing Archives Tethys Testnet: Following the successful sunset of the Panthalassa testnet, which had initially launched in partnership with CoinList with over 40,000 participants, we have now launched the Tethys testnet. The Panthalassa testnet, renowned for deploying several thousand indexers to index substrate chains including Polkadot and Kusama chains, has paved the way for its successor. The Tethys testnet introduces significant enhancements, including improved query execution, increased network stability, and enhanced communication between nodes.
The mainnet bootstrap: Following the successful testnets, we have bootstrapped the mainnet. Throughout these test phases, we gathered valuable insights on enhancing quality, query execution speed, and reliability. These advancements have been fully implemented in the mainnet, ensuring a robust and efficient network.
The biggest Polkadot projects have fully migrated or are in the process of finishing the migration from FireSquid to ArrowSquid, including: Polkassembly, Subsocial, HydraDX, KodaDot, Talisman, Multix Multisig, RMRK, Apillon, StellaSwap
Statistics and data
Over the course of Q1 2024:
Served a total of 1,725,006,525,189 (~1.57 terabytes) of data from the substrate based archives in Q1 2024
Served ~ 80m archive requests from substrate based archives in Q1 2024
The highest daily total data transfer recorded for Substrate was 313.12 Gb within a single 24-hour period.
Costing
This proposal encompasses the combined costs for running Archives for Polkadot, Kusama, and other public chains, along with the development contributions, amounting to a total of
47,597.91 DOT — Raw data for Archives can be seen here.
The proposal is submitted towards the Polkadot treasury. Here is a summary of the running costs for Archives, in a tabular format:
Contribution costs:
Subsquid Lab Official has requested 47,597.91 DOT ($202,291.10) from the Polkadot treasury to cover ongoing development and maintenance costs for public SQD Archives. These archives provide on-chain data access for public chains in the Polkadot ecosystem and are used by major projects like Polkassembly, Talisman, and RMRK. In Q1 2024, SQD transitioned to a paid model for supporting parachains but continues to offer free access to public chains. They also made significant contributions to support ink! v5 and addressed community demands. Achievements include the full migration to ArrowSquid, decentralizing Archives Tethys Testnet, and the mainnet bootstrap. In Q1 2024, they served 1.57 terabytes of data and 80 million archive requests, with the highest daily data transfer being 313.12 Gb.