SRLabs conducted a security audit on the Totem Kapex pallets and identified an issue with some illegal code that impacted the Transaction Payment pallet adapted by the team. Unfortunately the team won a slot on Polkadot before the audit could have been conducted and so the issue was not discovered until the audit completed.
The standard Transaction Payment Pallet had been forked and adapted to record transaction fees into the accounting engine of the KAPEX Parachain. the original version of this code was changed by a new team developer to use a function reserved for off-chain workers instead of the original code but as the code compiled it was not spotted until the audit. It was thought that an upgrade would fix the issue.
As the Transaction Payment Pallet is called during an upgrade to reserve funds (even though the actual upgrade is free of charge), the illegal code was unfortunately also called to record this reservation of fees causing an error, preventing the upgrade from taking place and therefore not allowing the parachain to be fixed.
In addition this also meant that the XCM security vulnerability discovered in late September also cannot be fixed either at this point.
All attempts were made to try to fix this issue without having to go to the Council and Referendum process, with help provided with gratitude from the Kilt.io team and Parity team Bastian, Alejandro and Santiago.
The new Manual Parachain Lock cannot fix our chain because the unlocking mechanism does not allow for centrally controlled chains to unlock the parachain from the relaychain side.
The fix that worked was to apply a validation code substitution using paras.forceSetCurrentCode() executed on the relaychain once the parachain chain spec had been updated accordingly and applied to the collators in our test networks. This is the subject of this proposal.
Tested the fix (paras.forceSetCurrentCode()) on our Relaychain development network connected to the Lego development parachain.
Upgraded the Lego Parachain to version v1.2.0
Upgrade Lego to v1.3.0
Tested the fix on the Rococo chain connected to our Stagex production staging parachain with the help of Parity devs.
Upgraded the Stagex Parachain to version v1.2.0
Then upgrade Stagex to v1.3.0
Upgraded docker Stagex nodes to run version v1.3.0
Decide the block number for the transition in Kapex chainspec - Block number 400000
Apply the new production validation code to Kapex chainspec
Tag the code repo from the main branch in the repo as kapex-v1.1.0-codeSub
Build a new docker image with the new Kapex chainspec.
Restart the Kapex Collators with the new chainspec.
Transfer funds to request code Substitution on Polkadot
Create Preimage hash on Polkadot 0x1ca2532cbb04d421f1b216f354354524feffd8540e7ca74b0938e58d2cdfc923
Create a proposal
Create Preimage on Polkadot
We are unfortunately unable to distribute our funds from the Crowdloan until this issue is fixed, so we look forward to the community supporting this proposal through the referendum as soon as possible. Thanks in advance.
The Totem Kapex Team