Dear ZilFam, before we go into our post-mortem, we want to thank you all again for your support and confidence in us through this incident. We are fortunate to have dodged a bullet by receiving the report first-hand, and for that, we will forever be grateful for those involved. This event displayed the true spirit of the Zilliqa community and we came out stronger as a result. Most importantly, we want to assure you that your funds are safe.


What happened?

  1. At approximately 8pm SGT (12pm UTC) on 26 November, as we were retiring for the Black Friday weekend, we were abruptly notified by the Zilliqa team that a critical issue in ZilSwap's core contract has been identified by a community member.
  2. Though most of our dev team were not near their development machines, we were still able to swiftly verify the validity of the issues reported. Once verified, a war room was immediately established to facilitate discussions between the ZilSwap devs, Zilliqa team, and the reporter, @AlienVakeel, while the ZilSwap devs rushed back from their appointments.
  3. The root issue identified was that the RemoveLiquidity transition had a pathway where the user's LP balance was not properly checked. This happens when the contribution_amount input parameter is set to the entire pool's contribution. The fix was simple, involving moving a single line from L670 to 656: https://github.com/Switcheo/zilswap/commit/71b839659c6c2a44cc486a34e1892bdb2b20c584#diff-7d728eb1b68b8a9e83376e488b397ede10b255c149f00e02dd588ceab8fe34a6L670. (More on how this happened in the section below)
  4. We recognised that the loophole was not an extremely complex one and could be inadvertently exposed to bad actors if we were not careful. It was therefore established that the best course of action to safeguard our community's funds was to perform a white-hack swiftly without alerting anyone outside the war room until the funds were secured.
  5. This meant that (i) Our white-hack had to be prepared without invoking any test transactions on TestNet — which was publicly visible; (ii) No pre-operation announcements could be made; and (iii) User interfaces had to be kept operational until the very last minute.
  6. Over the next three hours, preparations were made for the white-hack operation which needed to be batched and performed in a single block with high enough fees so that the transactions are prioritised.
  7. By midnight, we had a reasonable amount of confidence in the script that was to be run and had prepared (the now-famous) white-hack hot wallet zil15p46v72tl4gvqn6d93u4zu8jvmdpzy7vf7ycsj that held LP contributions in all pools — a requirement for the operation.
  8. For safety, we needed more assurance that our solution will definitely work as we only had one shot at it, and so we spent another two hours attempting to boot up private copies of the chain with the full MainNet state as well as testing the script more locally.
  9. At block 1616133, 1:43 AM SGT (5:43pm UTC) on 27 November, we were finally ready to pull the trigger, and fired off the batch transactions which would remove LP funds from ZilSwap through the same issue identified earlier. We also coordinated a silent disablement of the ZilSwap user interface, which would minimise the chance for user transactions to inadvertently pollute the final block's state for ZilSwap. This was needed to migrate the balances into the fixed contract based on the captured seed state here: https://github.com/Switcheo/zilswap/tree/master/artifacts.
  10. The team let out a sigh of relief as we watched transactions get executed on the blockchain explorers successfully.
  11. However, we discovered only ~72 pools were rescued upon closer inspection. As the transactions were ordered such that pools with the largest ZIL balance were rescued first, the majority of funds were already secured at this point. But we were not fully out of the woods yet.
  12. We immediately reran the rescue script which automatically retried the same action for the remaining pools — but it did not work and only produced multiple failed transactions. We were in a state of confusion when we realised the reason for the failed transactions was due to the contract having insufficient ZIL balance (!) to withdraw from the remaining pools — which implied that the contract's internal balance did not match its actual balance.
  13. The differences in these values amounted to ~697,108 ZILs, which thankfully was small at 0.14% of ZilSwap's TVL. We also observed that some tokens, conversely, had a higher ZIL balance than the contract's internal balance — leaving some tokens behind even after the pool had been fully withdrawn from.
  14. Now, we realised that we have identified another possible smart contract issue but needed to focus our attention on rescuing tokens in the remaining pools first.
  15. Over the next 30 minutes, we were able to use a simple technique to rescue the remaining pools before anyone deduced what we were up to: We created a new pool with a dummy token and added liquidity with the missing ZILs, so that there will be sufficient ZILs available in the contract. Followed by re-executing the script that has been modified to avoid rescuing the dummy token pool.
  16. The team took a cursory look at what the second issue could be, but the root reason for this imbalance was not immediately obvious. At this point, it was 3am in Asia, and the team decided that they should reconvene to figure out the discrepancy after some well-needed rest. Rescued funds were flushed to a hardware wallet for safekeeping meanwhile