Polkadot Storage Phase 3

Hey everyone, we are back with an improved long term plan for Polkadot Storage.

Last December, phase 3 got rejected during confirmation with a very narrow margin. Work has continued uninterrupted and we now have over 90% of phase 3 completed. We are testing the system in our private testnet and will move to a public testnet soon. Integration with mainnet has been postponed because of the JAM integration plans and we’ll continue improving tools and interfaces. For more details on our progress check the weekly updates on the forum.

During this time we have also been working with Parity and with their guidance have improved our long term approach. These ideas have been incorporated into Phase 4:

Jam Integration

Using our storage for Jam DA resupply.
Service Integration so that any Jam Service can use our long term storage.

Light weight proof system

This was first initially flagged by Lucky Friday (thank you) and we later refined it during conversations with Parity. In short, our initial proof system protects against outsourcing attacks but it's also heavy and expensive. We wanna offer an alternative lighter proof system for users who don't need guarantees like that, at a much lower price.
Merkle Proofs + zk-SNARKs
Make the network work with multiple proof systems, let power users choose the proof system/guarantees they want.

Replication

We added a naive version for now.
Move from full replication to for example Erasure coding.
Something like what Walrus uses “Twin-Code” or Reed Solomon error correction

Legal

Explore responsibilities and liabilities when dealing with toxic data.

Economic model:

Slashing - Although we have slashing implemented, we need to do some research on what the best numbers should be.
Storage costs - Modeling the costs and income model for providers.
Retrieval - Research retrieval costs to mitigate bandwidth attacks.
Pricing - Price is important for users, make sure we do everything to be competitive with centralized solutions.
Restaked DOT - Allow usage of restaked Dot so not to compete with staking rewards.

XCM

Exposing the storage so that other parachains can store data. Check usefulness after Jam integration.

Marketplace model

Thinking of this Storage layer also as a marketplace that gives access to all forms of competing centralized and decentralized storage.

Overall the plan is to turn this into a proper Storage product and build applications on top. With so much potential and attention on storage lately, it's an exciting time to build.

Links:

Code https://github.com/eigerco/polka-storage/
Book https://eigerco.github.io/polka-storage-book/
Phase 1 https://polkadot.polkassembly.io/referenda/494
Phase 2 https://polkadot.polkassembly.io/referenda/1150
Old Phase 3 https://polkadot.polkassembly.io/referenda/1313
Original proposal https://docs.google.com/document/d/16SZXvXyBRd-oEMIbgAid_vaxTuKzDlxuNc9CwSq3Lq0/edit
Thinking about JAM DA integration https://docs.google.com/document/d/1L1jlMI9-KOfxMYUMQVeq0tPNtz1fitxTXlg6gKwjXnM/edit?usp=sharing
Weekly updates https://forum.polkadot.network/t/polkadot-native-storage-updates/7021
Research on native storage we first did in 2021 but there were limitations with collators back then. We returned to this topic in 2023 and got a grant from W3F to figure out how.
Initial research pre-implementation https://github.com/eigerco/polkadot-native-storage/blob/main/doc/report/polkadot-native-storage-v1.0.0.pdf
Old forum discussion https://forum.polkadot.network/t/polkadot-native-storage/4551

Polkadot Storage has a new long-term plan to improve their storage system. They have completed 90% of phase 3 and are testing it in a private testnet. They will soon move to a public testnet and work on integrating it with the mainnet. Here are some of the new ideas they have for phase 4:

Jam Integration: They want to use their storage for Jam DA resupply and let any Jam Service use their long-term storage.
Light weight proof system: They want to offer a cheaper and lighter proof system for users who don't need strong guarantees.
Replication: They are working on moving from full replication to something like Erasure coding for better efficiency.
Legal: They are exploring responsibilities and liabilities when dealing with toxic data.
Economic model: They are researching the best numbers for slashing, storage costs, retrieval costs, and pricing to be competitive with centralized solutions. They also want to allow the use of restaked DOT.
XCM: They want to expose the storage so that other parachains can store data.
Marketplace model: They are thinking of the storage layer as a marketplace that gives access to all forms of competing centralized and decentralized storage.

Overall, they want to turn this into a proper storage product and build applications on top of it. They are excited about the potential and attention on storage lately.