substrate-etl 2.0: Polkadot + Kusama Data on Dune (12 months, 3/15/24-3/14/25)

[Dune](https://dune.com/browse/dashboards) is the industry leader in highly shareable dashboards widely used by industry leaders, data analysts, marketers, and builders across the broader Web3 ecosystem. For example here is [Brian Armstrong sharing a Dune dashboard about Base](https://twitter.com/brian\_armstrong/status/1690158283632586752), compared to [Moonbeam sharing dolpha.com](https://twitter.com/MoonbeamNetwork/status/1702721492834755061), [Astar sharing subscan](https://twitter.com/WatanabeSota/status/1731863081943277787) or [Polkadot sharing DotLake](https://twitter.com/Polkadot/status/1722980186189070760):

Common to the above is that Web3 ecosystem participants want to share on-chain data, which Dune is optimized for, better than a block explorer ([Subscan - #311](https://polkadot.polkassembly.io/referenda/311)) or expensive dashboards ([Dolpha.com - #341](https://polkadot.polkassembly.io/referenda/341)). Colorful Notion has built [substrate-etl - #248](https://polkadot.polkassembly.io/referenda/248), here proposed to be ingested by Dune as *substrate-etl 2.0* in 2024. ## Proposal - Onboard Polkadot + Kusama Data on Dune (12 months, 3/15/24-3/14/25) Colorful Notion proposes to onboard the Polkadot + Kusama relay chains by mid-March 2024, available for the community for at least _2 years_. This proposal funds the _first_ year after a PoC is completed in early 2024, for 3/15/24 - 3/14/25. Onboarding Polkadot will be Dune’s second non-EVM ecosystem integration, after Solana. Separate from this OpenGov proposal, Colorful Notion has negotiated pricing for all parachains with Dune, beyond the Polkadot and Kusama Relay Chain. This will enable Polkadot + Kusama Substrate data to be represented in Dune for both relay chains (using "polkadot" and "kusama") and all Polkadot parachains in an official integration with hourly indexing. # Demonstration: Polkadot Staking Dashboard In December 2023, Colorful Notion developed a proof of concept with key Dune engineers for the Polkadot Relay Chain with a few representative datasets. Here is one of the first Polkadot dashboards built in Dune: [https://dev.dune.com/mkchungs/polkadot-staking-validator\](https://dev.dune.com/mkchungs/polkadot-staking-validator) Once data is ingested from Colorful Notion’s substrate-etl into Dune, this dashboard can be rapidly constructed within a couple hours from SQL knowledgeable engineers such as Colorful Notion cofounder [@mkchungs](https://polkadot.polkassembly.io/user/mkchungs). Here is one query: Any on-chain dataset from Polkadot or Kusama involves writing a query such as the against some substrate-etl 2.0 dataset. Colorful Notion’s commercial terms have been developed to allow for a certain amount of storage ingesting a certain number of chains so that “Dune wizards” can query these tables, views, and materialized views and produce shareable charts like the above. Everyone with a Dune account can query and build these dashboards in Dune. While Google BigQuery can be connected to open source Apache Superset (and we have, see our W3F grant work on [Deep Account Analytics](https://github.com/w3f/Grants-Program/pull/1286) and inkubator-supported [ChainIDE + Polkaholic.io WASM Contract Explorer Integration](https://github.com/use-inkubator/Grant-Milestone-Delivery/blob/master/deliveries/chainide-polkaholic-integrated-wasm-contract-explorer-integration-milestone\_1.md)), Dune is more usable and more shareable. Whereas much of 2022 involved developing Polkaholic.io and 2023 involved substrate-etl 1.0 bringing data into Google BigQuery, Colorful Notion will have 2024 involve robust ingestion of Polkadot and Kusama data into Dune and ensuring the community can use it to support blockchain analytics and marketing. # Proposal Details [Full Proposal](https://docs.google.com/document/d/1UMpD152JmvHm6O9Q76TlF8GR1O\_FRJydjf7bndClj-k/edit) [Cost Breakdown](https://docs.google.com/spreadsheets/d/1c7-3kdfAjWdtWTkx2anEQgYpToUiWo3vav1WrapFuEI/edit#gid=1479719106) [Slides](https://docs.google.com/presentation/d/1wulreQof9V7QwCTuJZ9e64eDHhzAtqTS/edit#slide=id.g2a79f725924\_0\_8) **Requested Amount:** 325,000 DOT [$1,968,200] # December 2023 Decision For DOT Tokenholders: Dune Now or late 2024? Colorful Notion has been advocating a Polkadot Data Alliance Bounty along with Parity Data throughout much of 2023 (see [this from CN](https://docs.google.com/document/d/1ryC6dxcd9tiQsB7KiCc2BY\_TwBJ5jKloGfCyVAGCkKo/edit#heading=h.gjdgxs) and [this from Parity Data](https://docs.google.com/document/d/1fA5ARHy-frzgZC66rniKZ5o7CSbDvCTkS--kWaMzuMs/edit#heading=h.foftxwdr3st2)). In November, we finally decided to submit #248 and reach out to Dune as an outstanding goal of both teams. From the DOT Tokenholder point of view, once it is accepted that an ecosystem wide Dune integration is desirable and both teams are competent (as we do), we believe the decision is largely a tradeoff between Dune NOW vs Dune LATER: A simplified comparison is as follows: Which boils down to the following choice: * If you believe you want a Dune Integration NOW (early 2024), vote Aye on #366. * If you believe you want a Dune Integration LATER (or don’t want it at all), vote Nay on #366. We believe the answer is to integrate Dune NOW to support marketing and general ecosystem growth. # About Colorful Notion Colorful Notion is led by [Sourabh Niyogi](https://forum.polkadot.network/u/sourabhniyogi) and [Michael Chung](https://forum.polkadot.network/u/mkchungs). We have been active in the Polkadot ecosystem over the last 2 years and look forward to building with others in Polkadot 2.0’s “map reduce” ubiquitous computing future. ## Updates * 12/21/2023 - [AAG #93 - et-l brute?](https://www.youtube.com/watch?v=3jlA51DLz2o) * 12/22/2023 - PoC of Polkadot Relay Chain completed -- see [Polkadot Proof-of-Concept Dashboard](https://dune.com/sourabhniyogi/polkadot-proof-of-concept) * 1/3/2024 - Happy New Year 2024! Basic CN-Dune terms are being developed by Dune General Counsel; Partially redacted summary of Terms used for MSA draft may be posted here, _with permission_. * 1/4/2024 - CN received MSA from Dune. Publicly sharable CN-Dune terms are included below for transparency:  * 1/5/2024 - 3 Order forms developed for MSA. * 1/9/2024 - [Relay Chain/Parachain tables](https://docs.google.com/spreadsheets/d/1eG14AFNpJBx8Ng46XIy19hM1R\_POcT94t5mvfWaQoFE/edit#gid=1994869400) (shared with Dune permission) * 1/11/2024 - MSA finalized * 3/19/2024 - Dune public announcement: https://twitter.com/DuneAnalytics/status/1770087920994603325 # Ongoing Dashboard development in Q2 2024: [Polkadot Wiki PR](https://github.com/w3f/polkadot-wiki/pull/5933) ## Polkadot Relay Chain Polkadot Governance: * General: https://dune.com/substrate/polkadot-gov * Refs: https://dune.com/substrate/polkadot-referendum Polkadot Staking: https://dune.com/substrate/polkadot-staking ## Polkadot Parachains: AssetHub: https://dune.com/substrate/assethub Interlay: https://dune.com/substrate/interlay Astar: https://dune.com/substrate/astar-dapp-staking Bifrost: https://dune.com/substrate/bifrost HydraDX: https://dune.com/substrate/hydradx Centrifuge: https://dune.com/substrate/centrifuge Moonbeam: https://dune.com/substrate/moonbeam-governance Stellaswap: https://dune.com/substrate/stellaswap Acala: https://dune.com/substrate/acala Unique: https://dune.com/substrate/unique Phala: https://dune.com/substrate/phala Neuroweb: https://dune.com/substrate/neuroweb Manta: https://dune.com/substrate/manta Mythos: https://dune.com/substrate/mythos Collectives: https://dune.com/substrate/collectives Nodle: https://dune.com/substrate/nodle Invarch: https://dune.com/substrate/invarch ## Kusama Relay Chain Kusama Staking: https://dune.com/substrate/kusama-staking Kusama Governance: * General: https://dune.com/substrate/kusama-gov * Refs: https://dune.com/substrate/kusama-referendum ## Kusama System Chains / Other Coretime: https://dune.com/substrate/coretime\_kusama DED: https://dune.com/substrate/dotisded For the latest of ongoing ingestion status and dashboard catalyst, see this [public sheet](https://docs.google.com/spreadsheets/d/1eG14AFNpJBx8Ng46XIy19hM1R\_POcT94t5mvfWaQoFE/edit#gid=2015037)