Rolling back to a previous version

The following steps describe how to roll back Prefect Customer-Managed to a previous version.

This process includes two components: the database migrations and the Prefect Customer-Managed chart version, which includes the application image tag.

In most cases, rolling back to a previous version of Prefect only requires installing a previous version of the Helm chart.

Considerations

Please note that rolling back the application should only be required in rare cases. If this process requires downgrading the database migrations as well, it is critical to take backups of the databases before proceeding because the downgrades can cause data loss if schemas have changed significantly.

We also highly recommend testing rollbacks in non-production environments first.

Downgrading database migrations

Downgrading database migrations involves retrieving the current database revision and the target database revision, and then running the downgrade command.

The following instructions will show you how to find the latest revision for each database for posterity. This process can also be used to list all available revisions. With that in mind, please consider reaching out to Prefect support before attempting to downgrade database migrations.

First, retrieve the current nebula, orion, and events migration revisions. These commands should be run from within a pod that has the Prefect Customer-Managed source code, such as any pod running the server image.

# Nebula and Orion
for db in nebula orion; do
  path="/prefect_cloud/src/prefect_cloud/$db/migrations/versions"
  grep '^revision' $(find "${path}" | sort | tail -n 1) | sed "s;^;$db: ;"
done

# Events
path=/prefect_cloud/src/prefect_cloud/events/storage/postgres/migrations/versions
grep '^revision' $(find "${path}" | sort | tail -n 1) | sed "s;^;events: ;"

This will print out the current revision for each database. For example:

nebula: revision = "3fab1acc05c0"
orion: revision = "2a2090e47993"
events: revision = "3b5402bd348f"

Make a note of these versions for future reference.

Next, create a Job manifest to run the database downgrades. You can use the Helm chart to retrieve the configuration:

helm template <helm_release_name> <chart_repo_path> \
  -f <your_values_file> \
  --show-only templates/db-migration-job.yaml \
  > migration-downgrade-job.yaml

Next, edit migration-downgrade-job.yaml to replace each container's args field from prefect <database> upgrade to the following:

# Nebula
prefect-cloud nebula database downgrade -r <nebula revision>

# Orion
prefect-cloud orion database downgrade -r <orion revision>

# Events
prefect-cloud events database downgrade -r <events revision>

Next, apply the Job manifest:

kubectl apply -f migration-downgrade-job.yaml

Watch the logs for each of the containers to ensure they all succeed.

Once the downgrades are complete, you can move on to adjusting the Customer-Managed chart version.

Prefect Customer-Managed chart version

To install a previous version of the Customer-Managed Helm chart, use the helm CLI or your preferred installation tooling to install a specified version:

helm upgrade --install \
  <chart_name> \
  <chart_location> \
  --version=$TARGET_VERSION \
  -f <values_file>

This will install the Helm chart at the specified version, which includes application image tags configured as default values.