Patract Hub’s treasury report for Elara v0.2 (API Services)

lipeng
3yrs ago
0 Comments

5 weeks ago, Patract Hub applied a treasury proposal for Elara v0.2. Now, we have finished all the development work on time and recorded a Youtube demo video. Please review our design, work, and enjoy the separated workspace free services (1M requests per day) online at https://elara.patract.io. We have also provided a shared limited bandwidth node RPC service for Polkadot (wss://polkadot.elara.patract.io and https://polkadot.elara.patract.io), you can add it directly into your project and https://polkadot.js.org/apps.

Recap of the detailed design of Elara v0.2

  • Implement the account space features, supporting developers to use Github as a third-party login system to create an account space.
  • Support the creating and management of multiple projects under the developer’s account space
  • Provide developers with detailed access statistics features on the single project dimension, showing statistical indicators such as daily and weekly requests, calling methods, and source of user groups.
  • Officially launch https://elara.patract.io, providing the community with public access services to the Polkadot mainnet

Now, let's show you the design and implementation of v0.2, and how to run and verify the work. With this micro-service architecture described blow, we can provide ** 5K requests per second ** and 500M requests per day for now. In the future, we will upgrade this architecture for ** 10 times improvements. **

Architecture of Elara v0.2

We had learned from Infura's work model, so that each development account can apply for multiple Projects to divide different Workspace, and then assign different Provider to access chain nodes. Therefore, the user's smallest resource unit is a Project. We have assigned an independent PID for each project. In the index of all resources, the developer can use this PID to find the information corresponding to this Project, such as detailed information, daily or weekly access statistics, etc.

Therefore, the architecture of Elara v0.2 is roughly as follows:

elara

As can be seen from the above, in the backend management platform of Elara v0.2, the entire system is divided into 3 main modules:

  • Developer-Account: The module to manage the user's account login status.
  • Stat: The module for management and data statistics of Project.
  • API: The Route module to manage user's Requests.

In the overall design, the three modules are designed into a microservice architecture system, which can be independently extended. Because these three modules all run in independent processes after they are started, in the future, individual modules can be upgraded in parallel, such as expansion, reconstruction, and replacement. Among them, the Developer-Account module and the Stat module share data through a shared database, and the Stat and API are decoupled through the message queue kafka. In the future, it will be easy to add more complex functional modules based on this design to make Elara play a more powerful effect.

These three modules are responsible for different functions:

1. Developer-Account module

This module manages user account information and maintains login status. This module currently uses Redis to cache user information and maintain the user's login status. This login status is shared with the Stat module. Currently, this module uses Github's AccountID information to request authorization to access Github to create users.

Therefore, the external access interfaces provided by the module after running are:

  • GET /auth/login: Get login status information
  • GET /auth/logout: logout, remove the user's login information.
  • GET /auth/github: request authorization information from github
  • GET /auth/github/callback: processing of callbacks authorized to github

2. Stat module

This module is mainly responsible for the management of Project, and counts the request information initiated by users from the API module in the dimensions of Project and users. This module is Consumer of kafka Message Middleware, which accepts request information sent from the API module and performs processing statistics. At the same time, the module shares the user's login status information with Developer-Account to check whether the user is still connected to Elara's webpage. In this module, each user can apply for the allocation of multiple Projects, representing a restricted resource unit in Elara, such as limiting the amount of requests and total bandwidth that a Project can accept per day.

The access interface of the Stat module uses PID as the basic element, and the PID of a project will be used as the specific node Provider of this project in the API module. In the process of using this PID to raise a node access request to the API, the developer will consume the resources managed under the project corresponding to this PID and accumulate the statistical information of this project.

Based on the above discussion, the Stat module is designed as follows:

  • GET /project/list: List all project information under the currently logged-in user, and return:
{
   "code": 0,
   "message": "",
   "data": [
       {
           "id": "b78a79f98a2bb1a991a357504a5b04c1",
           "status": "Active",
           "chain": "substrate",
           "name": "thedao",
           "uid": "...",
           "secret": "...",
           "createtime": "1600398327",
           "lasttime": "1600398327",
           "allowlist": "false"
       },
      ...
   ]
}

The id in the returned information is the PID of a Project.

  • GET /project/<PID>: Returns the detailed information associated with this Project.
  • POST /project/create: Create a new Project based on the information of the currently logged in user

Parameters：

chain:<Polkdaot|Kusama>
name:<Validation rules /[a-zA-Z]{4,32}/>
  • GET /stat/day/<PID>: Get the statistics of a Project in one day
  • GET /stat/day/<PID>: Get the statistical information of a Project within one week
  • See the document for other interface definitions: Stat README

3. API module

After receiving requests from different users, the API module distinguishes which request to a Project according to the PID carried in the routing information, sends the corresponding information to the kafka queue, and forwards the user request to the corresponding node. Therefore, for the API module, Elara designed the following two access interfaces, among which the parameters of <chain> can currently be filled in Polkadot and Kusama.

  • HTTPS, POST /<chain>/<PID>: Accept the RPC request initiated by the user. The parameters of the request are all RPC interfaces that the node provides external services
  • Websocket /<chain>/<PID>: Accept the Websocket request initiated by the user. The parameters of the request are all RPC interfaces that the node provides external services

The above three modules work together to provide Elara's back-end components. Adding a front-end kit on top of it can become a complete Elara product to provide external services: https://elara.patract.io/ The sample screenshot is as follows:

This module is used to provide node services. It is mainly responsible to proxy user's node access request (RPC & websocket) to the node, and as the Producer of the kafka message queue, it sends the relevant information of the user request to the Stat module. In the API module, the user composes the PID created in the Stat module into the URL of the access node Provider, and then initiates a node access request to the API module.

4. Summary

The above three modules work together to provide Elara's back-end components. Adding a front-end kit on top of it can become a complete Elara product to provide external services: https://elara.patract.io/ The sample screenshot is as follows:

elara1
elara2
elara3

Run Elara

The processes corresponding to the above 3 modules need to be started in sequence in the following order to make Elara work properly: 1, Developer-Account, 2, Stat, 3.API

1. Environment preparation

Elara's operating environment requires redis and kafka support. Therefore, before running the following commands, first ensure that there is a running redis instance in the running environment and the executable file of kafka has been installed. The 3 processes of Elara are run by node, so you need to install the corresponding dependencies with the yarn command before running.

2. Start Services

  1. Start Developer-Account Service

    1. Enter elara/packages/account directory
      $ cd package/account
    1. Install dependence
      $ yarn install
    1. Modify the configuration file of the Developer-Account service, including the redis field and the github field. The default port is set to 7001, so if the port domain is not modified, the service will occupy the 7001 port after startup.
    $ vim ./config/env/dev.env.js
    1. Start the service, and the log output by the process is located in the ./logs directory.
       $ node app.js
   # or use pm2 to manage process
   $ pm2 start pm2.json --env dev

  2. Start Stat Service

    1. Enter elara/packages/stat directory（If you are in account directory）
       $ cd ../stat
    1. Install dependence
       $ yarn install
    1. Switch to the kafka directory, start the kafka directory and set the topic
      1. Start kafka and related processes. If the configuration file of kafka is not modified, port 9092 will be occupied by default.
         $ cd <the root directory of kafka>
   $ bin/zookeeper-server-start.sh config/zookeeper.properties
   $ bin/kafka-server-start.sh config/server.properties　
      1. Add elara-dev topic to kafka
         # add `elara-dev` topic to kafka
   $ bin/kafka-topics.sh --create --topic elara-dev --bootstrap-server localhost:9092
   # check whethere this topic is initted
   bin/kafka-topics.sh --describe --topic elara-dev --bootstrap-server localhost:9092
    2. Switch to elara/packages/stat directory，modify the configration file of Stat module.
       $ vim ./config/env/dev.env.js

    1. Since Stat and Developer-Account share the same redis instance, the configuration of the redis field in this configuration file should be consistent with the configuration of redis in the Developer-Account configuration file .

    2. The kafka domain here needs to be consistent with the configuration of starting kafka in the previous step.

    3. If it is only used for local testing, please modify the test:false in the default configuration to test:true. After enabling this flag, when accessing the interface provided by stat, the login status of the login account will not be verified, and all interfaces can be directly accessed. In a production environment, please ensure that the value of the test field is false.

    4. The default port domain in the configuration file is 7002, so if you do not modify this value, it will occupy the 7002 port to provide services after startup.

    5. Start Services

      1. Start the stat service. If you do not modify the port after startup, the 7002 port will be occupied. The following text uses 127.0.0.1:7002 as the service address performed by stat. The log output by the process is located in the ./logs directory.
         $ node app.js
   # or use pm2 to manage process
   $ pm2 start pm2.json --env dev
      1. Start kafka consumer service
         $ node ./kafka/consumer.js
      1. If you need to see the console information, you need to start the dashboard service
         $ node ./timer/dashboard.js

      At this point, you can open http://127.0.0.1:7002/dashboard in the browser to see the statistics information of the monitoring station.

  3. Start API Service

    1. Enter elara/packages/api directory（if you are in stat）
       $ cd ../api
    1. Install dependence
       $ yarn install
    1. Modify the configuration file of the API module
       $ vim ./config/env/dev.env.js
    • The chain field is used to configure the real chain node, where the name of the chain is part of the route provided by the api to the developer. For example, if there are the following configurations in the default, the access method of /Polkadot/<PID> is supported in the routing supported by the API.
       'Polkadot': {
       'rpc': ['****:**'], //configure as node http://IP: RPC port in step 2
       'ws': ['****:**'] //configure as node ws://IP: WS port in step 2
   }
    • The kafka field is used to configure the information of the kafka process started in the stat section. Note that the configured topic is the topic elara-dev registered to kafka when the stat part is configured. ,
    • startServer filed is used to configure the ip and port of the stat process. For example, if api and stat are on the same host and the default configuration is used (that is, stat occupies port 7002 by default), then The value here is 127.0.0.1:7002
    • port field is used to configure the port occupied by the api service process, the default configuration is 7003
    1. Start Services
       $ node app.js
   # or use pm2 to manage process
   $ pm2 start pm2.json --env dev

    At this time the service has been started, if the port is not modified, the port 7003 will be occupied. The following text uses the port 127.0.0.1:7003 as the host for user access. The log output by the process is located in the ./logs directory.

3. Call Elara

Since the configuration of test:true is used when starting stat in the above case, the process of user login is skipped in the following process, and the process of creating Project is directly started.

  1. Create Project：

    Send a request to create a project to the service address 127.0.0.1:7002 of stat: This project is a request for access to the Polkadot chain, so the value of the parameter chain is specified as Polkadot.

    POST http://127.0.0.1:7002/project/create

chain:Polkadot
name:hello

    Return：

    {
    "code": 0,
    "mssage": "",
    "data": {
        "id": "5d4ebb40b08f127652ad022f5936b9da",
        "status": "Active",
        "chain": "Polkadot",
        "name": "hello",
        "uid": "Only  For Test",
        "secret": "0b89a386965bfe4dcb19f724ecb41890",
        "createtime": "1606581366",
        "lasttime": "1606581366",
        "allowlist": "false"
    }
}

    You can get the detailed information of creating Project and PID, here is 5d4ebb40b08f127652ad022f5936b9da

  2. List all the Projects for this user

    GET http://127.0.0.1:7002/project/list

    Return:

    {
    "code": 0,
    "message": "",
    "data": [
        {
            "id": "5d4ebb40b08f127652ad022f5936b9da",
            "status": "Active",
            "chain": "Polkadot",
            "name": "hello",
            "uid": "Only  For Test",
            "secret": "0b89a386965bfe4dcb19f724ecb41890",
            "createtime": "1606581366",
            "lasttime": "1606581366",
            "allowlist": "false"
        }
    ]
}

    Since testing is currently enabled, all Project information will be listed. If the test is not turned on, only the information list of Project for the current login will be listed.

  3. Use this PID as part of the route to initiate an access request to the node.

    The access request to the node needs to be served through API, the current API service address is 127.0.0.1:7003, so we can initiate the following request: The access route is <api server host>/<project chain>/< PID> composition, so in this case

    • The access address for rpc is http://127.0.0.1:7003/Polkadot/5d4ebb40b08f127652ad022f5936b9da
    • The access address for websocket is ws://127.0.0.1:7003/Polkadot/5d4ebb40b08f127652ad022f5936b9da
    POST http://127.0.0.1:7003/Polkadot/5d4ebb40b08f127652ad022f5936b9da

{
    "id":1, 
    "jsonrpc":"2.0", 
    "method":"system_chain",
    "params":[ ]
}

    API Service is normal and return:

    {
    "jsonrpc": "2.0",
    "result": "Polkadot",
    "id": 1
}

    This request will be counted by Elara normally and recorded that project 5d4ebb40b08f127652ad022f5936b9da has one request.

  4. If you want to test whether the api service can satisfy normal node requests, you can add the websocket access link to Polkadot/Substrate Portal。

    1. Log in to the Portal web page in the browser
    2. Click on the upper left corner
    3. Click on the DEVELOPMENT column and click on the Custom column to fill in ws://127.0.0.1:7003/Polkadot/5d4ebb40b08f127652ad022f5936b9da
    4. Click "Switch"

    image

    At this point the Portal should be able to work normally, and all access requests are recorded by Elara.

  5. Access the Stat service to get statistics

    Such as listing all the visit statistics of the current project in the day

    GET http://127.0.0.1:7002/stat/day/5d4ebb40b08f127652ad022f5936b9da

    Or list the statistics of this project this week

    GET http://127.0.0.1:7002/stat/week/5d4ebb40b08f127652ad022f5936b9da

  6. Visit http://127.0.0.1:7002/dashboard and you can see the total statistics from the console.

At this point, the entire system can operate normally.

Recap of verification of v0.2

  • Developers can use Github account to log in to the developer account space from https://elara.patract.io

    Anyone can currently get access to the Polkadot chain through this link.

  • Developers can create new projects and access related API keys under the account

    The above process has introduced the concept of project, and you can log in to github with authorization and create a project at https://elara.patract.io.

  • Developers can access the Polkadot node in Elara through Http and web-socket protocols

    The API service is used to support this function, and the corresponding operations have been shown above

  • Developers can view the monitoring statistics of the projects on the day and the recent weeks

    The /stat/day/<PID> and /stat/week/<PID> interfaces of Stat service are used to support this function.

  • You can view the relevant monitoring statistics from the Elara dashboard

    You can see the corresponding information from the http://127.0.0.1:7002/dashboard console.

Up
Comments
No comments here