SRLabs conducted a security audit on the Totem Kapex pallets and identified an issue with some illegal code that impacted the Transaction Payment pallet adapted by the team. Unfortunately the team won a slot on Polkadot before the audit could have been conducted and so the issue was not discovered until the audit completed.
The standard Transaction Payment Pallet had been forked and adapted to record transaction fees into the accounting engine of the KAPEX Parachain. the original version of this code was changed by a new team developer to use a function reserved for off-chain workers instead of the original code but as the code compiled it was not spotted until the audit. It was thought that an upgrade would fix the issue.
As the Transaction Payment Pallet is called during an upgrade to reserve funds (even though the actual upgrade is free of charge), the illegal code was unfortunately also called to record this reservation of fees causing an error, preventing the upgrade from taking place and therefore not allowing the parachain to be fixed.
In addition this also meant that the XCM security vulnerability discovered in late September also cannot be fixed either at this point.
All attempts were made to try to fix this issue without having to go to the Council and Referendum process, with help provided with gratitude from the Kilt.io team and Parity team Bastian, Alejandro and Santiago.
The new Manual Parachain Lock cannot fix our chain because the unlocking mechanism does not allow for centrally controlled chains to unlock the parachain from the relaychain side.
The fix that worked was to apply a validation code substitution using paras.forceSetCurrentCode() executed on the relaychain once the parachain chain spec had been updated accordingly and applied to the collators in our test networks. This is the subject of this proposal.
Tested the fix (paras.forceSetCurrentCode()) on our Relaychain development network connected to the Lego development parachain.
Upgraded the Lego Parachain to version v1.2.0
Upgrade Lego to v1.3.0
Tested the fix on the Rococo chain connected to our Stagex production staging parachain with the help of Parity devs.
Upgraded the Stagex Parachain to version v1.2.0
Then upgrade Stagex to v1.3.0
Upgraded docker Stagex nodes to run version v1.3.0
Decide the block number for the transition in Kapex chainspec - Block number 400000
Apply the new production validation code to Kapex chainspec
Tag the code repo from the main branch in the repo as kapex-v1.1.0-codeSub
Build a new docker image with the new Kapex chainspec.
Restart the Kapex Collators with the new chainspec.
Transfer funds to request code Substitution on Polkadot
Create Preimage hash on Polkadot 0x1ca2532cbb04d421f1b216f354354524feffd8540e7ca74b0938e58d2cdfc923
Create a proposal
Create Preimage on Polkadot
We are unfortunately unable to distribute our funds from the Crowdloan until this issue is fixed, so we look forward to the community supporting this proposal through the referendum as soon as possible. Thanks in advance.
The Totem Kapex Team
Introduction: The Kapex Parachain on Polkadot has been producing blocks but cannot be upgraded due to a coding error.
Audit Findings: A security audit by SRLabs found illegal code in the Transaction Payment Pallet, which was adapted from the standard version.
Upgrade Failure: The illegal code prevents the parachain from being upgraded, as it is called during the upgrade process.
XCM Vulnerability: The parachain also cannot fix a security vulnerability discovered in September.
Fix Attempts: Efforts were made to fix the issue without involving the Council and Referendum process, with help from Kilt.io and Parity teams.
Testing the Fix: The fix was tested on development and staging networks, upgrading versions from v1.2.0 to v1.3.0.
Production Preparation: Steps were taken to prepare for the production fix, including updating the chainspec and creating a new docker image.
Referendum Needed: The community's support is needed to pass a referendum to fix the issue and allow the distribution of Crowdloan funds.