The Kapex Parachain has been producing blocks since winning and parachain slot on Polkadot, but cannot be upgraded due to an error.
SRLabs conducted a security audit on the Totem Kapex pallets and identified an issue with some illegal code that impacted the Transaction Payment pallet adapted by the team. Unfortunately the team won a slot on Polkadot before the audit could have been conducted and so the issue was not discovered until the audit completed.
The standard Transaction Payment Pallet had been forked and adapted to record transaction fees into the accounting engine of the KAPEX Parachain. the original version of this code was changed by a new team developer to use a function reserved for off-chain workers instead of the original code but as the code compiled it was not spotted until the audit. It was thought that an upgrade would fix the issue.
As the Transaction Payment Pallet is called during an upgrade to reserve funds (even though the actual upgrade is free of charge), the illegal code was unfortunately also called to record this reservation of fees causing an error, preventing the upgrade from taking place and therefore not allowing the parachain to be fixed.
In addition this also meant that the XCM security vulnerability discovered in late September also cannot be fixed either at this point.
All attempts were made to try to fix this issue without having to go to the Council and Referendum process, with help provided with gratitude from the Kilt.io team and Parity team Bastian, Alejandro and Santiago.
The new Manual Parachain Lock cannot fix our chain because the unlocking mechanism does not allow for centrally controlled chains to unlock the parachain from the relaychain side.
The fix that worked was to apply a validation code substitution using
paras.forceSetCurrentCode() executed on the relaychain once the parachain
chain spec had been updated accordingly and applied to the collators in our test networks. This is the subject of this proposal.
Testing the fix
Tested the fix (
paras.forceSetCurrentCode()) on our Relaychain development network connected to the Lego development parachain.
Upgraded the Lego Parachain to version
Upgrade Lego to
Tested the fix on the Rococo chain connected to our Stagex production staging parachain with the help of Parity devs.
Upgraded the Stagex Parachain to version
Then upgrade Stagex to
Upgraded docker Stagex nodes to run version
Decide the block number for the transition in Kapex chainspec - Block number
Apply the new production validation code to Kapex
Tag the code repo from the main branch in the repo as
Build a new docker image with the new Kapex
Restart the Kapex Collators with the new
Transfer funds to request code Substitution on Polkadot
Create Preimage hash on Polkadot
Create a proposal
Create Preimage on Polkadot
We are unfortunately unable to distribute our funds from the Crowdloan until this issue is fixed, so we look forward to the community supporting this proposal through the referendum as soon as possible. Thanks in advance.
The Totem Kapex Team