Critical Runtime Upgrade to Prevent Further Exploitation
Executed
Content
AI Summary
Translations
Incident Overview
- The attacker maliciously upgraded the Parallel runtime, transferring over 312,185 DOT and 126,837 USDT. More than half of the stolen funds have been moved to CEX and sold for XMR. The attacker is still the sudo owner of the parachain and has the minting access, putting additional 125,688 DOT at risk. This proposal aims to prevent further DOT losses and ensure users can safely access their funds through an urgent runtime upgrade. Attacker's address (Polkadot) (Ethereum)
Timeline of Events (Please Check Detailed Link)
- 4/14/2024: The attacker initially deposited DOT and began interacting with the parachain. Link
- 10/31/2024: Proposal of Malicious Runtime Upgrade: The attacker submitted a new runtime upgrade to the democracy pallet containing a malicious code addition. Link
- 10/31/2024: Voting on Proposal: The attacker voted to approve their malicious proposal. Link
- 11/07/2024: Triggering Malicious Upgrade: The attacker executed the malicious runtime upgrade, introducing a custom "sudo" pallet, granting themselves administrative privileges on the parachain. Link
- 11/07/2024: Exploitation of Permissions: The attacker leveraged sudo privileges to manipulate proxy permissions on the relay chain. Link
- 11/22/2024: Token Minting and Subsystem Control: The attacker minted unauthorized tokens (DOT and USDT) and seized control of parachain features. Link Link2
- 11/24/2024: Through the proxy account, the attacker successfully transferred a total of 303,208+ DOT from three of the parachain’s DOT staking ledger accounts to their attack account. Link Link2 Link3
- From 11/24/2024 onward: The attacker unstaked funds on the Polkadot relay chain and sold them on exchanges for XMR (Binance, Huobi, FixedFloat, WhiteBit) (XMR addresses: 133W4GGX3WUEozsvVNMEWeptPfy8Cj8AjqQr6RXTjCT2W1cC,
14dynDKN3Bcze5R7eYfS485apRCuGrxEBg3HZXvivHv7zNuH, and many more) Subscan Account Activity - 11/30/2024: All remaining funds were moved to Ethereum, through moonbeam and Axelar bridge. Link Link2
Why Is Upgrading the Runtime Necessary?
- Despite multiple attempts to fix the parachain internally, we discovered that the attacker has disabled critical governance mechanisms, such as the democracy and technical committee modules. This makes it impossible to remove the attacker's sudo access and proxy account without assistance from the relay chain. As a result, we have no option but to request Polkadot Governance's support in restoring our parachain, both for the benefit of our users and the broader Polkadot ecosystem.
Changes Introduced:
Github Release Link Our proposal is paras -> forceSetCurrentCode(para: 2012, newCode: [our good runtime], leading to the preimage hash of 0x5df19e1463d4bc3e19ad1210e860c4ab305dcf258bffd7ce778ef7bf3c337358 and preimage length of 1525669 as shown in the following screenshot:
(1) Addition of the Sudo Pallet
- This pallet is required to enable the sudo origin; otherwise, the chain state can only be managed through the governance module, which has been disabled by the attacker.
- Code Reference
(2) Introduction of a Dummy Pallet
- A dummy pallet has been implemented to allow our parachain to set a Sudo Key (
5CLbxwBcUf8PG4zzf56w27YwwJzkyGv4ULsBNfkCBGEdRGKv) under the control of our team. This Sudo Key is guaranteed a minimum balance of 10,000 PARA. - Code Reference 1
- Ensuring Sudo Key Integrity:
- From lines 14 to 27, the code ensures that the
Sudo.keyis always set to the correct value, even if tampered with. During each block initialization (on_initialize), it compares the currentSudo.keywith the expected value. If they differ, the code usesunhashed::putto overwrite the incorrect key with the correct one. - Once updated, an event (
SudoMigrated) is triggered to log the change, ensuring transparency. This mechanism guarantees the Sudo Key remains properly set, securing administrative control of the chain. - Maintaining Sudo Key Balance:
- From lines 29 to 36, the code ensures the Sudo Key always has sufficient funds for critical actions. It checks the Sudo Key’s balance using
free_balance. If the balance falls below the predefined minimum (amount_to_add), the code usesdeposit_creatingto credit the account with the required amount. - This operation automatically creates the account if it doesn’t exist. Any system imbalance resulting from this process is safely handled using
drop(imbalance). An event (SudoBalanceDeposited) is also triggered to log the balance update, ensuring transparency. This ensures the Sudo Key remains functional at all times.
- A dummy pallet has been implemented to allow our parachain to set a Sudo Key (
(3) Maintaining the Current Spec Version
- To avoid triggering additional runtime upgrade logic, the current spec version has been maintained.
- Code Reference
(4) Removal of Migration Scripts and Minor Adjustments
- Some migration scripts were removed, and minor adjustments were made. These changes are purely for optimization and do not impact runtime logic.
- Code Reference
(5) Updating Docker Image Versions and Off-Chain Scripts
- Docker image versions were updated, and off-chain scripts (including RPC changes) were adjusted. These updates are strictly for optimization purposes.
- Code Reference
Testing:
Upgrade Testing:
- Upgrade tests were conducted in both a fork of Heiko (Parallel’s testnet on Kusama) and the real Heiko environment. The tests confirmed that the Sudo Key was successfully reset to the specified value.
Verification on Heiko Mainnet:
- The Sudo Key for Heiko was verified to be
hJFU3r4zioT39AaBiTriJCVvoepeEGViF38DAkKECUjVwsvZK, corresponding to the universal ss58 address5CLbxwBcUf8PG4zzf56w27YwwJzkyGv4ULsBNfkCBGEdRGKv. - Heiko Mainnet Link
- The Sudo Key for Heiko was verified to be
Verification on Chopsticks:
- We also conducted upgrade tests in a parallel fork environment. It works as expected: the sudo key was successfully replaced with the desired address. Even when the sudo key’s native balance was emptied, it was still able to mint some gas for the sudo key in the new block. Screenshot here:
- We also conducted upgrade tests in a parallel fork environment. It works as expected: the sudo key was successfully replaced with the desired address. Even when the sudo key’s native balance was emptied, it was still able to mint some gas for the sudo key in the new block. Screenshot here:
Action Plan After the Runtime Upgrade
Once the proposal is passed and the Sudo pallet is restored, the following actions will be taken:
- Remove Proxy Accounts: Eliminate all proxy accounts that the attacker has access to on the relay chain. Detailed steps for this process will be shared next week.
- Revoke Unauthorized Privileges: If the attacker added additional privileged access (e.g., minting admin) during the voting period, we will revoke these privileges. Similarly, if the attacker unbonded any available DOT during this period, we will rebond them.
- Monitor Suspicious Activity: Closely monitor transactions from the democracy pallet to prevent similar attacks in the future.
- Remove Asset Mint Admin Access: As previously promised, we will remove the asset mint admin access to enhance security.
How to Recover Stolen Funds?
- Exchange Blocking: After collaborating with law enforcement and security firm, we successfully engaged over 55 exchanges to actively block and freeze funds from attacker's addresses. For full transparency, we includes an analysis from SlowMist.
- Negotiation: Despite multiple attempts to communicate with the attacker, we have received no response.
- Law Enforcement: Every action the attacker takes leaves a trail, and eventually, the net will close. A generous bounty will be offered for information that results in the attacker's capture.
What’s Next?
- Regular Communication: We will keep users, partners, and the community informed about the situation’s progress and any updates to product policies or services.
- Commitment to Users:
- We are committed to maintaining the codebase and implementing robust security practices to prevent similar issues.
- We will ensure that users’ funds are protected and accessible safely and securely.
- We are dedicated to doing right by our community and users.
- Community Supervision: We encourage the community to actively supervise and provide feedback to ensure transparency and accountability throughout this process.
We are pleased to share our announcement for addressing some of the existing issues on our platform. Please know that we are fully committed to assisting every user to the best of our abilities. We also welcome your suggestions on how we can improve. Additionally, if you're interested, you can check out this AAG video where Yubo explains the proposal and answers some of the community's questions.
Thank you for your support!
Edited
- An attacker stole over 312,185 DOT and 126,837 USDT from a parachain.
- The attacker gained control by upgrading the runtime with malicious code.
- They moved most of the stolen funds to exchanges and sold them for XMR.
- The attacker still has control over the parachain, putting more DOT at risk.
- A runtime upgrade is needed to stop further losses and secure user funds.
- The upgrade will restore control and remove the attacker's privileges.
- Efforts are being made to recover stolen funds and improve security.
- The community is being kept informed and encouraged to provide feedback.
Source
Incident Overview
- The attacker maliciously upgraded the Parallel runtime, transferring over 312,185 DOT and 126,837 USDT. More than half of the stolen funds have been moved to CEX and sold for XMR. The attacker is still the sudo owner of the parachain and has the minting access, putting additional 125,688 DOT at risk. This proposal aims to prevent further DOT losses and ensure users can safely access their funds through an urgent runtime upgrade. Attacker's address (Polkadot) (Ethereum)
Timeline of Events (Please Check Detailed Link)
- 4/14/2024: The attacker initially deposited DOT and began interacting with the parachain. Link
- 10/31/2024: Proposal of Malicious Runtime Upgrade: The attacker submitted a new runtime upgrade to the democracy pallet containing a malicious code addition. Link
- 10/31/2024: Voting on Proposal: The attacker voted to approve their malicious proposal. Link
- 11/07/2024: Triggering Malicious Upgrade: The attacker executed the malicious runtime upgrade, introducing a custom "sudo" pallet, granting themselves administrative privileges on the parachain. Link
- 11/07/2024: Exploitation of Permissions: The attacker leveraged sudo privileges to manipulate proxy permissions on the relay chain. Link
- 11/22/2024: Token Minting and Subsystem Control: The attacker minted unauthorized tokens (DOT and USDT) and seized control of parachain features. Link Link2
- 11/24/2024: Through the proxy account, the attacker successfully transferred a total of 303,208+ DOT from three of the parachain’s DOT staking ledger accounts to their attack account. Link Link2 Link3
- From 11/24/2024 onward: The attacker unstaked funds on the Polkadot relay chain and sold them on exchanges for XMR (Binance, Huobi, FixedFloat, WhiteBit) (XMR addresses: 133W4GGX3WUEozsvVNMEWeptPfy8Cj8AjqQr6RXTjCT2W1cC,
14dynDKN3Bcze5R7eYfS485apRCuGrxEBg3HZXvivHv7zNuH, and many more) Subscan Account Activity - 11/30/2024: All remaining funds were moved to Ethereum, through moonbeam and Axelar bridge. Link Link2
Why Is Upgrading the Runtime Necessary?
- Despite multiple attempts to fix the parachain internally, we discovered that the attacker has disabled critical governance mechanisms, such as the democracy and technical committee modules. This makes it impossible to remove the attacker's sudo access and proxy account without assistance from the relay chain. As a result, we have no option but to request Polkadot Governance's support in restoring our parachain, both for the benefit of our users and the broader Polkadot ecosystem.
Changes Introduced:
Github Release Link Our proposal is paras -> forceSetCurrentCode(para: 2012, newCode: [our good runtime], leading to the preimage hash of 0x5df19e1463d4bc3e19ad1210e860c4ab305dcf258bffd7ce778ef7bf3c337358 and preimage length of 1525669 as shown in the following screenshot:
(1) Addition of the Sudo Pallet
- This pallet is required to enable the sudo origin; otherwise, the chain state can only be managed through the governance module, which has been disabled by the attacker.
- Code Reference
(2) Introduction of a Dummy Pallet
- A dummy pallet has been implemented to allow our parachain to set a Sudo Key (
5CLbxwBcUf8PG4zzf56w27YwwJzkyGv4ULsBNfkCBGEdRGKv) under the control of our team. This Sudo Key is guaranteed a minimum balance of 10,000 PARA. - Code Reference 1
- Ensuring Sudo Key Integrity:
- From lines 14 to 27, the code ensures that the
Sudo.keyis always set to the correct value, even if tampered with. During each block initialization (on_initialize), it compares the currentSudo.keywith the expected value. If they differ, the code usesunhashed::putto overwrite the incorrect key with the correct one. - Once updated, an event (
SudoMigrated) is triggered to log the change, ensuring transparency. This mechanism guarantees the Sudo Key remains properly set, securing administrative control of the chain. - Maintaining Sudo Key Balance:
- From lines 29 to 36, the code ensures the Sudo Key always has sufficient funds for critical actions. It checks the Sudo Key’s balance using
free_balance. If the balance falls below the predefined minimum (amount_to_add), the code usesdeposit_creatingto credit the account with the required amount. - This operation automatically creates the account if it doesn’t exist. Any system imbalance resulting from this process is safely handled using
drop(imbalance). An event (SudoBalanceDeposited) is also triggered to log the balance update, ensuring transparency. This ensures the Sudo Key remains functional at all times.
- A dummy pallet has been implemented to allow our parachain to set a Sudo Key (
(3) Maintaining the Current Spec Version
- To avoid triggering additional runtime upgrade logic, the current spec version has been maintained.
- Code Reference
(4) Removal of Migration Scripts and Minor Adjustments
- Some migration scripts were removed, and minor adjustments were made. These changes are purely for optimization and do not impact runtime logic.
- Code Reference
(5) Updating Docker Image Versions and Off-Chain Scripts
- Docker image versions were updated, and off-chain scripts (including RPC changes) were adjusted. These updates are strictly for optimization purposes.
- Code Reference
Testing:
Upgrade Testing:
- Upgrade tests were conducted in both a fork of Heiko (Parallel’s testnet on Kusama) and the real Heiko environment. The tests confirmed that the Sudo Key was successfully reset to the specified value.
Verification on Heiko Mainnet:
- The Sudo Key for Heiko was verified to be
hJFU3r4zioT39AaBiTriJCVvoepeEGViF38DAkKECUjVwsvZK, corresponding to the universal ss58 address5CLbxwBcUf8PG4zzf56w27YwwJzkyGv4ULsBNfkCBGEdRGKv. - Heiko Mainnet Link
- The Sudo Key for Heiko was verified to be
Verification on Chopsticks:
- We also conducted upgrade tests in a parallel fork environment. It works as expected: the sudo key was successfully replaced with the desired address. Even when the sudo key’s native balance was emptied, it was still able to mint some gas for the sudo key in the new block. Screenshot here:
- We also conducted upgrade tests in a parallel fork environment. It works as expected: the sudo key was successfully replaced with the desired address. Even when the sudo key’s native balance was emptied, it was still able to mint some gas for the sudo key in the new block. Screenshot here:
Action Plan After the Runtime Upgrade
Once the proposal is passed and the Sudo pallet is restored, the following actions will be taken:
- Remove Proxy Accounts: Eliminate all proxy accounts that the attacker has access to on the relay chain. Detailed steps for this process will be shared next week.
- Revoke Unauthorized Privileges: If the attacker added additional privileged access (e.g., minting admin) during the voting period, we will revoke these privileges. Similarly, if the attacker unbonded any available DOT during this period, we will rebond them.
- Monitor Suspicious Activity: Closely monitor transactions from the democracy pallet to prevent similar attacks in the future.
- Remove Asset Mint Admin Access: As previously promised, we will remove the asset mint admin access to enhance security.
How to Recover Stolen Funds?
- Exchange Blocking: After collaborating with law enforcement and security firm, we successfully engaged over 55 exchanges to actively block and freeze funds from attacker's addresses. For full transparency, we includes an analysis from SlowMist.
- Negotiation: Despite multiple attempts to communicate with the attacker, we have received no response.
- Law Enforcement: Every action the attacker takes leaves a trail, and eventually, the net will close. A generous bounty will be offered for information that results in the attacker's capture.
What’s Next?
- Regular Communication: We will keep users, partners, and the community informed about the situation’s progress and any updates to product policies or services.
- Commitment to Users:
- We are committed to maintaining the codebase and implementing robust security practices to prevent similar issues.
- We will ensure that users’ funds are protected and accessible safely and securely.
- We are dedicated to doing right by our community and users.
- Community Supervision: We encourage the community to actively supervise and provide feedback to ensure transparency and accountability throughout this process.
We are pleased to share our announcement for addressing some of the existing issues on our platform. Please know that we are fully committed to assisting every user to the best of our abilities. We also welcome your suggestions on how we can improve. Additionally, if you're interested, you can check out this AAG video where Yubo explains the proposal and answers some of the community's questions.
Thank you for your support!
Edited
Call
Metadata
Timeline6
Votes Bubble
Curves
Statistics
Timeline
Comments