The Kapex Parachain has been producing blocks since winning and parachain slot on Polkadot, but cannot be upgraded due to an error.
SRLabs conducted a security audit on the Totem Kapex pallets and identified an issue with some illegal code that impacted the Transaction Payment pallet adapted by the team. Unfortunately the team won a slot on Polkadot before the audit could have been conducted and so the issue was not discovered until the audit completed.
Background
The standard Transaction Payment Pallet had been forked and adapted to record transaction fees into the accounting engine of the KAPEX Parachain. the original version of this code was changed by a new team developer to use a function reserved for off-chain workers instead of the original code but as the code compiled it was not spotted until the audit. It was thought that an upgrade would fix the issue.
Upgrade failure
As the Transaction Payment Pallet is called during an upgrade to reserve funds (even though the actual upgrade is free of charge), the illegal code was unfortunately also called to record this reservation of fees causing an error, preventing the upgrade from taking place and therefore not allowing the parachain to be fixed.
In addition this also meant that the XCM security vulnerability discovered in late September also cannot be fixed either at this point.
All attempts were made to try to fix this issue without having to go to the Council and Referendum process, with help provided with gratitude from the Kilt.io team and Parity team Bastian, Alejandro and Santiago.
The new Manual Parachain Lock cannot fix our chain because the unlocking mechanism does not allow for centrally controlled chains to unlock the parachain from the relaychain side.
The fix that worked was to apply a validation code substitution using paras.forceSetCurrentCode()
executed on the relaychain once the parachain chain spec
had been updated accordingly and applied to the collators in our test networks. This is the subject of this proposal.
Testing the fix
Tested the fix (
paras.forceSetCurrentCode()
) on our Relaychain development network connected to the Lego development parachain.Upgraded the Lego Parachain to version
v1.2.0
Upgrade Lego to
v1.3.0
Tested the fix on the Rococo chain connected to our Stagex production staging parachain with the help of Parity devs.
Upgraded the Stagex Parachain to version
v1.2.0
Then upgrade Stagex to
v1.3.0
Upgraded docker Stagex nodes to run version
v1.3.0
Production preparation
Decide the block number for the transition in Kapex chainspec - Block number
400000
Apply the new production validation code to Kapex
chainspec
Tag the code repo from the main branch in the repo as
kapex-v1.1.0-codeSub
Build a new docker image with the new Kapex
chainspec
.Restart the Kapex Collators with the new
chainspec
.Transfer funds to request code Substitution on Polkadot
Create Preimage hash on Polkadot
0x1ca2532cbb04d421f1b216f354354524feffd8540e7ca74b0938e58d2cdfc923
Create a proposal
Create Preimage on Polkadot
We are unfortunately unable to distribute our funds from the Crowdloan until this issue is fixed, so we look forward to the community supporting this proposal through the referendum as soon as possible. Thanks in advance.
The Totem Kapex Team