substrate-etl is a large scale index of Polkadot + Kusama and their parachains based on Polkaholic.io, a multichain block explorer/indexer. Using this data warehouse, anyone can query decoded blocks, extrinsics, events, calls across both ecosystems.

We onboarded Polkadot Data into Google Public Datasets, announced in September 2023:

The substrate-etl index results in two large datasets (crypto_polkadot and crypto_kusama) covering the 2 relay chains and over 70 parachains. This represents Polkadot/Kusama alongside other Web3 ecosystems and supports complex large scale analyses with SQL using BigQuery's lightning-fast query performance on extremely large datasets. Having substrate-etl enables Dune-like analytics to guide business intelligence, inform engineering decision making, and showcase ecosystem growth.

Power users include Parity Data (led by Karim Jedda and Pavla Mijic), Subwallet/DotInsights, Web3metrics,
Dolpha.com, ChainViz and many data analysts within parachains and outside the ecosystem.

Quick Start

For a full guide, see substrate-etl, but you can get started with the following Analytics Hub links:

Roadmap

In addition to continuing support for substrate-etl/Polkaholic.io for existing and future chains, our plans for 2024 are to:

chart a course within the Decentralized Futures for 2024, supporting Web3 Foundation's Data Analysis Tools for Substrate-based Blockchains (e.g. traces and staking), chart a course for substrate-etl + Polkaholic APIs to support Dune Analytics Style Data Service for Polkadot/Kusama with leading teams in the ecosystem
develop + support a Polkadot Builder Collective with a "builders supporting builders" orientation
build support for CoreJam+CorePlay+DA and extend support for ink!/WASM contracts
extend substrate-etl to support raw + fully-decoded EVM datasets (transactions+logs), enabling Substrate+EVM hybrid chain analytics, as more parachains incorporate hybrid chain designs

Proposal Details

Requested: 99,025 DOT

Proposal Details here

Proposal Costs here

About Colorful Notion

Colorful Notion is led by Sourabh Niyogi and Michael Chung, based in the SF Bay Area. We have been active in the Polkadot ecosystem over the last 2 years and look forward to building with others in CoreJam/Polkadot 2.0’s “map reduce” ubiquitous computing future.

We are grateful for everyone who has helped us and hope we can pay it forward by helping others

Substrate ETL is a big tool for analyzing data from Polkadot and Kusama.
It allows people to look at information like blocks and transactions from these blockchains.
A new dataset was added to Google Cloud in September 2023 to help people use this data.
The dataset includes information about over 70 parachains, which are smaller chains that work with Polkadot and Kusama.
Many analysts and teams use this tool to help make important decisions and understand how things are growing.
Plans for 2024 include adding more features and supporting more data analyses for these blockchains.
Colorful Notion, the team behind this project, has been involved with Polkadot for two years and aims to collaborate with others in the community.

substrate-etl Retroactive funding: Q3+Q4 2022-Q1+Q2 2023

Quick Start

Roadmap

Proposal Details

About Colorful Notion