Mainnet Upgrade Report: v0.1.6 to v0.1.8

Mainnet Upgrade Report: v0.1.6 to v0.1.8

This article highlights lessons learnt and procedural upgrades as a response to the recent chain halt experienced during the v0.1.6 to v0.1.8 upgrade.

The SEDA team recently addressed a technical challenge during the Mainnet upgrade from v0.1.6 to v0.1.8 on February 26th, 2025. Our established procedure of rehearsing upgrades on Testnet before Mainnet implementation demonstrated its value as we navigated an unprecedented situation involving two closely timed security patches that required immediate attention. The team's proactive approach to security vulnerabilities positioned us to respond effectively to these critical patches concerning the packet forward middleware and Cosmos SDK's group module.

‍

During the upgrade execution, our monitoring systems immediately identified an issue where the upgrade handler was registered under "v0.1.7" instead of "v0.1.8," causing the chain to fail restart at block height 3249665. The SEDA team quickly diagnosed the root cause: the rapid succession of two security patches had created an unusual testing scenario where the v0.1.8 upgrade test used the same environment as the earlier v0.1.7 test. This situation led to the naming discrepancy that wasn't detected during testing. Our technical team developed a hotfix solution by creating a properly tagged fix that addressed the handler name mismatch.

‍

The experience has strengthened our upgrade protocols with valuable enhancements to both our rehearsal and failure response procedures. The team has now implemented a more robust release candidate testing process including checksum verification, independent node preparation by team members, and thorough post-upgrade feature testing. Additionally, we've established clear guidelines for responding to upgrade challenges, including a structured hotfix approach that preserves version integrity. These improvements further reinforce SEDA's commitment to maintaining network stability and security while responding rapidly to emerging vulnerabilities.

‍

Thank you to the swift response from all validators who supported in upgrading their validators speeding up the restart of the network. We look forward to maintaining our current trajectory and traction as we work around the clock towards a successful mainnet v1 upgrade event. 

‍