Details of the release process for updating smart contracts on the Celo platform.
Details of the release process for updating smart contracts on the Celo platform.
This release process is a work in progress. Many infrastructure components required to execute it are not in place, and the process itself is subject to change.
Versioning
Each deployed Celo core smart contract is versioned independently, according to semantic versioning, as described at semver.org, with the following modifications:
STORAGE version when you make incompatible storage layout changes
MAJOR version when you make incompatible ABI changes
MINOR version when you add functionality in a backwards compatible manner, and
PATCH version when you make backwards compatible bug fixes.
Changes to core smart contracts are made via on-chain Governance, approximately four times a year. When a release is made, all smart contracts from the release branch that differ from the deployed smart contracts are released, and included in the same governance proposal. Each release is identified by a unique monotonically increasing version number N, with 1 being the first release.
Core contracts
Every deployed Celo core contract has its current version number as a constant which is publicly accessible via the getVersionNumber() function, which returns the storage, major, minor, and patch versions. The version number is encoded in the Solidity source and updated as part of code changes.Celo Core Contracts deployed to a live network without the getVersionNumber() function, such as the original set of core contracts, are to be considered version 1.1.0.0.
Mixins and libraries
Mixin contracts and libraries are considered part of the contracts that consume them. When a mixin or library has changed, all contracts that consume them should be considered to have changed as well, and thus the contracts should have their version numbers incremented and should be re-deployed as part of the next smart contract release.
Initialize data
Whenever Celo Core Contracts need to be re-initialized, their initialization arguments should be checked into version control under packages/protocol/releaseData/initializationData/release${N}.json.
Release management in Git/GitHub
Github branches/tags and Github releases are used to coordinate past and ongoing releases. Ongoing smart contract development is done on the master branch (even after release branches are cut). Every smart contract release has a designated release branch, e.g. release/core-contracts/${N} in the celo-monorepo.
When a new release branch is cut:
A new release branch is created release/core-contracts/${N} with the contracts to be audited.
The latest commit on the release branch is tagged with core-contracts.v${N}.pre-audit.
On Github, a pre-release Github release should be created pointing at the latest tag on the release branch.
On master branch, .github/workflows/celo-monorepo.yml should be edited so that the variable RELEASE_TAG points to the tag core-contracts.v${N}.pre-audit so that all future changes to master are versioned against the new release.
Ongoing audit responses/fixes should continue to go into release/celo-core-contracts/${N}.
After a completed release process:
The release branch should be merged into master with a merge commit (instead of the usual squash merge strategy).
On master branch, .github/workflows/celo-monorepo.yml should be edited so that the variable RELEASE_TAG points to the tag core-contracts.v${N}
Release process
There are several scripts provided (under packages/protocol in celo-org/celo-monorepo and via celocli). The release ownersâ own reference is RELEASE_PROCESS_FOUNDRY.md, which maps each retired Truffle command to its Foundry replacement. These are for use in the release process and with contract upgrade governance proposals to give participating stakeholders increased confidence.
For these to run, you may need to set up the celo-monorepo: follow the Getting Started steps and install the Node version pinned in .nvmrc (managed with nvm). A successful yarn install and yarn build in the protocol package signal a completed setup. The release scripts also shell out to Foundry (forge, cast) and jq, so install those too.
Using these tools, a contract release candidate can be built, deployed, and proposed for upgrade automatically on a specified network. Subsequently, stakeholders can verify the release candidate against a governance upgrade proposalâs contents on the network.Typical script options:
By default, the scripts expect a celo-blockchain RPC at port 8545 locally. With -f you can specify the scripts to use a hosted forno node
By default, scripts will output verbose logs under /tmp/celo-${script-name}.log. You can change the location of the log output with -l file.log
View the tagged releases for each network
yarn tags:view
Verify the previous release on the network
release:verify-deployed:foundry is a script that allows you to assess whether the bytecode on the given network matches the source code of a particular commit. It will run through the Celo Core Contracts and verify that the contractsâ bytecodes as specified in the Registry match. Here, we will want to sanity-check that our network is running the previous releaseâs audited commit.
# Run from `packages/protocol` in the celo-monorepoN=18 # the release being preparedPREVIOUS_RELEASE="core-contracts.v$((N-1))"NETWORK="celo-sepolia" # "anvil", "celo-sepolia" or "mainnet"# A -f boolean flag can be provided to use a forno full node to connect to the provided networkyarn release:verify-deployed:foundry -n $NETWORK -b $PREVIOUS_RELEASE -f
A $NETWORK-$PREVIOUS_RELEASE-libraries.json file is written to disk describing linked library addresses. release:make:foundry requires it.
Check backward compatibility
This script performs some automatic checks to ensure that the smart contract versions in the source code have been set correctly with respect to the latest release. It is run as part of CI and helps ensure that backwards incompatibilities are not accidentally introduced by requiring that devs manually update version numbers whenever smart contract changes are made.Specifically, it compiles the latest and candidate releases and compares smart contracts:
Storage layout, to detect storage version changes
ABI, to detect major and minor version changes
Bytecode, to detect patch version changes
Finally, it checks release candidate smart contract version numbers and requires that they have been updated appropriately since the latest release by following semantic versioning as defined in the Versioning section above.The following exceptions apply:
If the STORAGE version has changed, it does not perform backward compatibility checks
If the MAJOR version has changed, it checks storage layout compatibility but not ABI compatibility
Critically, this ensures that proxied contracts do not experience storage
collisions between implementation versions. See this
article
by OpenZeppelin for a good overview of this problem and why itâs important to
check for it.The script generates a detailed report on version changes in JSON format, written to report-$PREVIOUS_RELEASE-$RELEASE_CANDIDATE.json.
# Run from `packages/protocol` in the celo-monorepoN=18PREVIOUS_RELEASE="core-contracts.v$((N-1))"RELEASE_CANDIDATE="core-contracts.v${N}"yarn release:check-versions:foundry -a $PREVIOUS_RELEASE -b $RELEASE_CANDIDATE
This should be used in tandem with release:verify-deployed:foundry -b $PREVIOUS_RELEASE -n $NETWORK to ensure the compatibility checks compare the release candidate to what is actually active on the network.
Deploy the release candidate
Use the following script to build and deploy a candidate release. This takes as input the corresponding backward compatibility report and canonical library address mapping to deploy changed contracts to the specified network. The Foundry script has no dry-run; pass -u with a local anvil forkâs RPC URL to rehearse it.
STORAGE updates are adopted by deploying a new proxy/implementation pair. This script writes a JSON contract upgrade governance proposal to proposal-$NETWORK-$RELEASE_CANDIDATE.json.
# Run from `packages/protocol` in the celo-monorepoN=18NETWORK="celo-sepolia" # "anvil", "celo-sepolia" or "mainnet"PREVIOUS_RELEASE="core-contracts.v$((N-1))"RELEASE_CANDIDATE="core-contracts.v${N}"PRIVATE_KEY="0x..." # deployer key for $NETWORKyarn release:make:foundry -b $RELEASE_CANDIDATE -n $NETWORK -r "report-$PREVIOUS_RELEASE-$RELEASE_CANDIDATE.json" -i "releaseData/initializationData/release${N}.json" -l "$NETWORK-$PREVIOUS_RELEASE-libraries.json" -k "$PRIVATE_KEY"
-k "$PRIVATE_KEY" puts the deployer key on the command line, where it is readable by other processes on the machine (for example via /proc on Linux) and is written to your shell history. Run it from a dedicated deploy machine, and clear the key from the environment and history afterwards.
The proposal encodes STORAGE updates by repointing the Registry to the new proxy. Storage compatible upgrades are encoded by repointing the existing proxyâs implementation.
Submit upgrade proposal
Submit the autogenerated upgrade proposal to the Governance contract for review by voters, outputting a unique identifier.
# resultant proposal ID should be communicated publiclyN=18NETWORK="celo-sepolia" # "anvil", "celo-sepolia" or "mainnet"RELEASE_CANDIDATE="core-contracts.v${N}"YOUR_ADDRESS="0x..." # proposer account; must hold the depositcelocli governance:propose --deposit 100e18 --from $YOUR_ADDRESS --jsonTransactions "proposal-$NETWORK-$RELEASE_CANDIDATE.json" --descriptionURL https://github.com/celo-org/governance/blob/main/CGPs/cgp-0055.md
Fetch upgrade proposal
Fetch the upgrade proposal and output the JSON encoded proposal contents.
# Make sure you run at least celocli 0.0.60celocli governance:show --proposalID <proposalId> --jsonTransactions "upgrade_proposal.json"
Verify proposed release candidate
This script serves the same purpose as release:verify-deployed:foundry but for a not-yet
accepted contract upgrade (in the form of the upgrade_proposal.json you fetched in the step prior). It gives you the confidence that the branch specified in the -b flag in (same as release:check-versions:foundry) will be the resulting network state of the proposal if executed. It does so by going over all Celo Core Contracts and determining updates to the Registry pointers, proxy or implementation contracts and verifying their implied bytecode against the compiled source code.Additionally, include initialization_data.json from the CGP if any of the contracts have to be initialized.
# Run from `packages/protocol` in the celo-monorepoN=18RELEASE_CANDIDATE="core-contracts.v${N}"NETWORK="celo-sepolia" # "anvil", "celo-sepolia" or "mainnet"# A -f boolean flag can be provided to use a forno full node to connect to the provided networkyarn release:verify-deployed:foundry -p "upgrade_proposal.json" -b $RELEASE_CANDIDATE -n $NETWORK -f -i initialization_data.json
Verify executed release
After a release executes via Governance, you can use release:verify-deployed:foundry again to check that the resulting network state does indeed reflect the tagged release candidate:
# Run from `packages/protocol` in the celo-monorepoN=18RELEASE="core-contracts.v${N}"NETWORK="celo-sepolia" # "anvil", "celo-sepolia" or "mainnet"yarn release:verify-deployed:foundry -n $NETWORK -b $RELEASE -f
Testing
All releases should be evaluated according to the following tests.
Unit tests
All changes since the last release should be covered by unit tests. Unit test coverage should be enforced by automated checks run on every commit.
Manual checklist
After a successful release execution on a testnet, the resulting network state should be spot-checked to ensure that no regressions have been caused by the release. Flows to test include:
A ceiling on the gas consumption for all common operations should be defined and enforced by automated checks run on every commit.For troubleshooting please see Readme.md of protocol package.
Backwards compatibility
Automated checks should ensure that any new commit to master does not introduce a breaking change to storage layout, ABI, or other common backward compatibility issues unless the STORAGE or MAJOR version numbers are incremented.Backwards compatibility tests will also be run before every release to confirm that no breaking changes exist between the pending release and deployed smart contracts.
Audits
All changes since the last release should be audited by a reputable third party auditor.
Emergency patches
If patches need to be applied before the next scheduled smart contract release, they should be cherry-picked to a new release branch, branched from the latest deployed release branch.
Promotion process
Deploying a new contract release should occur with the following process. On-chain governance proposals should be submitted on Tuesdays for consistency and predictability.
Date
Action
T
Create a Github issue tracking all these checklist items as an audit
log
Let the community know about the upcoming release proposal by posting
details to the Governance category on https://forum.celo.org and cross
post in the
Discord #governance channel.
See the âCommunication guidelinesâ section below for information on what
your post should contain.
T+2w
Confirm all contracts working as intended on an Anvil testnet.
Run the smart contract release scriptin order to to deploy the contracts to Celo Sepolia as well as submit a
governance proposal.
Update your forum post with the Celo Sepolia PROPOSAL_ID,
updated timings (if any changes), and notify the community in the
Discord #governance channel.
T+3w
Confirm all contracts working as intended on Celo Sepolia.
Confirm audit is complete and make the release notes and forum post
contain a link to it.
On Tuesday: Run the smart contract release scriptin order to to deploy the contracts to Mainnet as well as submit a
governance proposal.
Update the corresponding governance proposal with the updated on-chain
PROPOSAL_ID and mark CGP status as âPROPOSEDâ.
Update your forum post with the Mainnet PROPOSAL_ID,
updated timings (if any changes), and notify the community in the
Discord #governance channel.
At this point all stakeholders are encouraged to
verify the proposed contracts
deployed match the contracts from the release branch.
Currently the governance process should take approximately 1 week:
24 hours for the dequeue process, 24 hours for the approval
process, and 5 days for the referendum process. After which, the
proposal is either declined or is ready to be executed within 3
days.
For updated timeframes, use the celocli:
celocli network:parameters
T+5w
If the proposal passed:
Confirm all contracts working as intended on Mainnet.
Update your forum post with the Mainnet governance outcome (
Passed or Rejected) and notify the
community in the Discord #governance channel.
Change corresponding CGP status to EXECUTED.
Merge the release branch into master with a merge
commit
If the proposal failed:
Change corresponding CGP status to EXPIRED.
If the contents of the release (i.e. source Git commit) change at any point after the release has been tagged in Git, the process should increment the release identifier, and process should start again from the beginning. If the changes are small or do not introduce new code (e.g. reverting a contract to a previous version) the audit step may be accelerated.
Communication guidelines
Communicating the upcoming governance proposal to the community is critical and may help getting it approved.Each smart contract release governance proposal should be accompanied by a Governance category forum post that contains the following information:
Name of proposer (individual contributor or organization).
Background information.
Link to the release on Github.
Link to the audit report(s).
Anticipated timings for the Celo Sepolia testnet and Mainnet.
Make sure to keep the post up to date. All updates (excluding fixing typos) should be communicated to the community in the Discord#governance channel.
Emergency patches
Work in progress
Vulnerability disclosure
Vulnerabilities in smart contract releases should be disclosed according to the security policy.