QMDB Core Integration for Substrate Storage Enhancement

Treasury Proposal: QMDB Core Integration for Substrate Storage

Enhancement Proponent: 12K97oseeBxoWnyYRU3DJkxKTiDGFBh1pK4jLyuUJzYHkqdD

Date: [29.01.2025]

Requested DOT: 5 000

Short description: Initial integration of QMDB (https://github.com/LayerZero-Labs/qmdb) core functionality into Substrate (https://github.com/paritytech/polkadot-sdk). Project Category/Type: Storage and Performance Optimization

Previous Treasury Proposals:

Context of the Proposal

This proposal focuses on the first phase of QMDB integration, specifically implementing core storage functionality without iterators and prefixes with limited support or non-optimal support. This foundational work will enable future enhancements while maintaining compatibility with existing Substrate systems.

Project Details

Technology stack:

RUST language for QMDB integration with Substrate
DevOps tools for deployment and monitoring

Documentation:

Core components and architecture documentation to be developed
Integration specifications and API documentation
Performance benchmarking methodology

Research and PoC:

What this project will not provide:

Full prefix search functionality
Production-ready indexing solution
Advanced iterator implementation

Team members

Team Lead: Vas Soshnikov
Team Member: Andrew Orlov

Team's experience

Previous Polkadot grants:

Team Code Repos

Problem Statement
Current Substrate storage implementation faces several fundamental challenges that this initial integration phase aims to address: Current crate ‘sp-trie’ implementation requires optimisation Inefficient key-based operations affecting basic transaction processing Suboptimal transaction verification process

Key Limitations of QMDB

Does not support iterators and prefix searches
Non-limitation, mostly its design of QMDN, requires additional memory (RAM, Disk) for indexing (indexer)

Proposal Objectives & Solutions Objectives

Integrate QMDB as the proof storage engine within Substrate
Enable Merkle Proof-based transaction verification
Benchmark QMDB inside our DevNet (https://dev.qfnetwork.xyz)

Milestones, Timelines, and Budgets Milestone

1. Implement QMDB storage traits Working prototype integrated with Substrate
3 weeks
2 000 DOT

2. Develop a storage indexing mechanism In-memory B-Tree or sorted sparse list
2 weeks
1 500 DOT

3. Conduct full system testing All Substrate tests pass
1 week
750 DOT

4. Deploy, benchmark and optimise Live on DevNet with monitoring
1 week
750 DOT

Risks and Mitigation Risk Mitigation Strategy

QMDB may not be production-ready
Extensive DevNet testing before MainNet deployment
Substrate’s evolving architecture may cause compatibility issues
Freeze the Substrate version until integration is stable
Increased resource consumption due to indexers
Optimise RAM/disk usage and implement caching mechanisms

Resources Required

Senior Rust Developer: Expertise in Substrate and low-level systems
Senior Rust Developer: Specializing in database architecture
DevNet Environment: Dedicated nodes for rigorous testing

Success Criteria

Substrate nodes use QMDB for transaction storage
All Substrate and pallet-specific tests pass successfully
Performance benchmarks demonstrate improved transaction processing times
DevNet deployment runs stably with real-time monitoring and alerting

Payment Conditions

Payments will be made in instalments based on milestone completion. Each instalment will be subject to the successful delivery of agreed milestones. A detailed progress report will be submitted on Polkassembly/Subsquare.

Why Polkadot Network?

Polkadot’s focus on modular and scalable blockchain infrastructure makes it an ideal candidate for advanced storage optimisations like QMDB integration. By enhancing storage efficiency, this proposal aligns with Polkadot’s long-term vision of high-performance, decentralized systems.

Additional References
QMDB Technical Paper
QMDB GitHub Repository