Proposal for the Development of SubstrateSQL Server

Introduction

We, Sphere 10 Software, propose the creation of an open-source and comprehensive SQL solution, tentatively named SubstateSQL Server (or PolkadotSQL Server, final name TBD) designed specifically for Polkadot and Substrate-based chains. This open-source initiative aims to enable enterprises to fully harness the potential of Substrate-based blockchains, enhancing their operational capabilities with complete blockchain integration and data awareness via traditional SQL databases.

Demo Product (for Bitcoin)

Project Overview

SubstrateSQL Server will be a robust system that converts any Substrate-based blockchain into an automatically generated and performance-optimized SQL database. This system will not only synchronize data in real-time but will also allow for direct interaction with the blockchain via SQL, bypassing traditional JSON API interfaces. Key features include:

Auto-Schema Generation: Generates SQL schema automatically based on Substrate Pallet configuration.
Real-time Data Pump: Seamlessly exports blockchain data into the SQL database, ensuring data is current and readily accessible.
Extrinsic Dispatch: Enables the creation of extrinsics (i.e., transactions) through direct insertion of database records, which are then automatically signed and sent to the network. This bypasses the need for JSON APIs, simplifying the integration process.

Use Cases

The development of SubstrateSQL Server will unlock several critical use cases:

Analytics: Provides comprehensive access to historical and current blockchain data in structured forms, ideal for complex data analysis required by financial institutions and trading firms.
Taint Analysis: Supports advanced transaction graph analyses to identify and isolate compromised assets, crucial for compliance and law enforcement.
Integration Broker: Simplifies integration of enterprise applications with Substrate-based blockchains, enhancing operational efficiency.
Payment and Event Notifications: Utilizes database triggers for instant notifications on blockchain events, enabling reactive integrations into enterprise backends.
Low-Friction Adoption: Employs familiar technologies such as SQL and .NET, facilitating easy adoption without the need for deep blockchain expertise in the organization.

Value Proposition

Investing in the development of SubstrateSQL Server offers substantial strategic and technical benefits for the Polkadot ecosystem, providing a strong return on investment for the Polkadot treasury. Here’s why:

Open-Source Foundation:
SubstrateSQL Server will be released as an open-source project, fostering transparency, collaboration, and community-driven innovation. Decision-makers in enterprises and organizations are more likely to choose a web3 stack with an open-source foundation, as it ensures greater security, flexibility, and long-term sustainability. By supporting this project, Polkadot reinforces its commitment to open-source principles, making it a more attractive choice for enterprises.
Simplified Integration for Enterprises:
Integrating with blockchains is currently a significant challenge for enterprises, often requiring specialized technical expertise that is not readily available. SubstrateSQL Server simplifies this process by converting complex blockchain data into familiar SQL databases, allowing enterprises to integrate blockchain technology into their existing systems without needing deep blockchain expertise. This ease of integration reduces the barrier to entry and accelerates the adoption of Polkadot-based solutions in enterprise environments.
Enterprise-Grade Integration Broker:
As Polkadot aims to be utilized by enterprises around the world, having a robust Enterprise Backend -> Polkadot integration broker is essential. SubstrateSQL Server provides this critical capability by enabling seamless data synchronization and interaction between enterprise applications and Polkadot-based blockchains. This integration broker addresses a broad need across various industries, from finance and compliance to supply chain and healthcare, driving widespread adoption and usage of the Polkadot ecosystem.
Global Enterprise Adoption:
By providing a solution that bridges the gap between traditional enterprise systems and blockchain technology, SubstrateSQL Server positions Polkadot as a viable and attractive option for global enterprises. The ability to leverage existing SQL databases and technologies makes it easier for enterprises to adopt and integrate Polkadot into their operations, enhancing their capabilities and operational efficiency.

Technical Architecture

The solution will include:

A cross-platform console application allowing users to connect to local nodes (this can also run as a background service (or daemon) and also hosts a web-based front-end)
A web-based interface to query the database as required, hosted by console application.
Dynamic SQL database generation based on Substrate Pallet configurations
Plug-n-play support for multiple DBMSs (e.g., SQL Server, PostgreSQL, Oracle, etc.)
Tiered data export options tailored to specific needs, from full historical data to streamlined, app-level datasets
Real-time updates from finalized blocks and efficient dispatching of extrinsics via outbound tables
Continuous adaptation to changes in Substrate pallet configurations, ensuring database relevance and integrity.

Deliverables

Our commitment is to deliver:

An open-source codebase hosted on GitHub under the MIT license, developed in C# targeting the latest .NET framework.
Cross-platform compatibility including macOS, Linux, Windows, and ARM architectures.
Docker containers to facilitate easy deployment.
Comprehensive documentation for both system administrators and end-users.

Funding Request

We are seeking 175,000 DOT to cover research and development costs, primarily for developer salaries. In line with our fiscal strategy, we intend to retain the funds in DOT for the maximum feasible duration, converting to fiat currency only as necessary to fulfil financial commitments.

To facilitate this approach, we have established an arrangement with an offshore holding company. This entity will manage the DOT funds on behalf of the community, ensuring controlled disbursement to Sphere 10 Software Pty Ltd as operational expenses arise. While Sphere 10 Software is committed to compensating our team in DOT to the extent possible, circumstances requiring currency conversion will be handled through exchanges to USDT or USDC, as appropriate.

With this strategy, we anticipate minimal to no market impact resulting from this funding request.

Timeline

We anticipate the completion of a Minimum Viable Product (MVP) within nine months, with the entire project fully delivered by the twelve-month mark.

Maintenance and Long-term Commitment

UPDATE This proposal now includes an 18 month commitment to maintain SubstrateSQL Server to ensure it remains relevant and compatible with latest Substrate changes and that any issues/bugs are resolved. We anticipate that as the project matures, it will evolve into a self-sustaining open-source project, negating the need for further funding from the treasury.

Who We Are & Previous Experience

Sphere 10 Software Pty Ltd, the team behind this proposal, is an Australian-based proprietary limited company (ABN: 39 600 596 316) specializing in web3 development. We have previously developed a similar system called BlockchainSQL Server for Bitcoin-based UTXO blockchains. Our experience ensures the delivery of a high-quality product efficiently and effectively.

Call to Action

We invite you to review our existing products, such as the BlockchainSQL Server, to evaluate the quality and potential of the proposed solution.

Addressing Community Questions

Budget and Complexity:

It has been brought to our attention that concerns exist regarding the budget and complexity of the project, with comparisons made to other tools in the market. Without specific references to these tools, it’s challenging to make direct comparisons. However, the complexity of Substrate-based blockchains, which include diverse Pallet configurations, significantly exceeds that of simpler blockchain models like Bitcoin’s UTXO. This inherent complexity justifies the proposed budget and timeline.

Our previous experience with Bitcoin’s BlockchainSQL Server, which took six months to develop, informs our current proposal. The added complexity of Substrate-based chains means a longer development timeline, but we are leveraging existing code to achieve our goals within nine to twelve months.

Comparison to ParityDB:

ParityDB is a NoSQL DBMS used primarily as the database for blockchain-based applications, such as parachains. SubstrateSQL serves a different purpose as a secondary database that extracts and transforms data from the primary database into a SQL database for enterprise applications. This secondary database supports features like custom data transformations and integration with existing business applications, facilitating easier adoption of blockchain technology.

Relevance and Existing Tools:

Some members questioned the necessity of this proposal given the availability of tools like Dune and SubSquid. It is important to distinguish that while Dune offers robust analytics, it is not open-source and cannot be deployed on-premise, which limits its adaptability for enterprise use. SubSquid, although valuable, primarily uses GraphQL and is not fully open-source, presenting limitations for direct SQL integration. Our SubstrateSQL Server addresses these gaps by being fully open-source, providing on-premise deployment options, and focusing on SQL integration, which is widely utilized within enterprise environments.

Use of an Offshore Holding Company:

The use of an offshore holding company has been implemented to mitigate financial inefficiencies associated with taxes, which could significantly reduce the funds available for project development. This approach ensures that more of the community's investment is directed towards development while remaining compliant with Australian tax regulations.

Reputation and Experience in the Ecosystem:

We acknowledge concerns regarding our new entry into the ecosystem and the perception of our experience with open-source projects. Although we are newer to this specific ecosystem, our team's background in developing BlockchainSQL Server for Bitcoin underpins our capability. Moreover, we have openly shared nearly one million lines of code on GitHub under the MIT license, demonstrating our commitment and ability to deliver high-quality open-source software.

Licensing and Open-Source Commitment:

Some concerns were expressed about the choice of MIT license over GPL/AGPL and the potential for making the repository private post-delivery. We chose the MIT license to encourage wider adoption and flexibility. We are committed to maintaining the development repository public and active, ensuring continuous access and contribution from the community.

Conclusion:

Your feedback is invaluable, and we are committed to addressing each point raised by the community. We believe that SubstrateSQL Server will significantly enhance the operational capabilities and data accessibility for enterprises using Substrate-based blockchains, thereby driving broader adoption and integration within the Polkadot ecosystem.

Thank you for considering our proposal to revolutionize enterprise integration with Substrate-based blockchain technology through SubstrateSQL Server.

This proposal has been crafted to align with the interests and requirements of the Polkadot OpenGov community, focusing on transparency, innovation, and community engagement.