🔧 How to Migrate From Terraform Cloud
Nachrichtenbereich: 🔧 Programmierung
🔗 Quelle: dev.to
When it comes to changing products, the biggest obstacle is often migration. Yes, the product looks amazing, but once you start thinking about all the work involved migrating from the old one to the new one, you abandon the idea. A seamless migration experience would stop you missing out. Think of how Apple enables painless device upgrades: You simply keep your old phone next to your new one for less than an hour, and everything migrates.
Terraform Cloud (TFC) recently switched to a RUM (Resources Under Management) pricing model. But this model is unsustainable for many customers, as it will be tough to predict what their bill will look like at the end of the month. You can read more about the changes here.
Migrating from Terraform Cloud is easier than enduring their new pricing model, so at Spacelift, we developed a way to speed up the migration process: We call it the Spacelift Migration Kit.
Getting started with Spacelift Migration Kit
To migrate from Terraform Cloud and move your workspaces in bulk, take the following steps:
- Install Python3.10 or newer and Poetry
- Clone the Spacelift Migration Kit.
- Install other dependencies and the spacemk command in a Python virtual environment
- Create your config.yml
- [Optional] Audit your current setup
- Export the configuration
- Generate the Terraform/OpenTofu configuration
- Create a repository based on the generated code
- Create a stack based on the repository
- State Migration
- [Optional] Customization
- Useful information
Let's cover these steps in detail.
How to migrate from Terraform Cloud
1. Install prerequisites
We need to ensure that a version newer than python3.10 is installed and also that poetry is installed.
Installing poetry on MacOS using pipx
brew install pipx
pipx install poetry
pipx ensurepath
To install it on any other operating system, check out this tutorial.
2. Clone the Spacelift migration kit
Cloning the repository is a fairly simple process. Simply go to Spacelift Migration Kit, and select either the ssh or http links and run git clone.
git clone [email protected]:spacelift-io/spacelift-migration-kit.git
3. Install spacemk
Go to the Spacelift migration kit directory, and run the following:
poetry install
Creating virtualenv spacemk-liMdtQjr-py3.12 in /Users/flaviuscdinu/Library/Caches/pypoetry/virtualenvs
Installing dependencies from lock file
Now after you have installed spacemk, activate the virtual environment:
poetry shell
4. Create your config.yml
Inside the repository, there is a config.yml.example file, containing the values you need to configure for the migration:
# These are all the possible properties for the configuration files.
# The values included in this file are the default values for the property they are associated with.
exporter:
name: # Valid values: terraform
settings:
# Specific to the Terraform exporter (exporter.name: terraform)
api_endpoint: https://app.terraform.io
api_token:
include:
workspaces: ^example-.*$
generator:
extra_vars:
foo: bar # "{{ extra_vars.foo }}" in a template will be replaced by "bar"
github:
api_token:
endpoint: https://api.github.com
spacelift:
api:
api_key_endpoint: https://<ACCOUNT NAME>>.app.spacelift.io/graphql
api_key_id:
api_key_secret:
Rename this file, or create a new one that should be called "config.yml".
In this file, you must add the login details to your Terraform Cloud account and your Spacelift account. You can also use environment variables inside this file, to avoid using credentials inside the file (these variables should be prefixed with the "$" sign).
Here is an example that uses environment variables:
# These are all the possible properties for the configuration files.
# The values included in this file are the default values for the property they are associated with.
exporter:
name: terraform
settings:
# Specific to the Terraform exporter (exporter.name: terraform)
api_endpoint: https://app.terraform.io
api_token: $TFC_TOKEN
generator:
extra_vars:
foo: bar # "{{ extra_vars.foo }}" in a template will be replaced by "bar"
github:
api_token: $GITHUB_TOKEN
endpoint: https://api.github.com
spacelift:
api:
api_key_endpoint: https://saturnhead.app.spacelift.io/graphql
api_key_id: $SPACELIFT_API_KEY_ID
api_key_secret: $SPACELIFT_API_KEY_SECRET
You will need to export these values inside of your terminal (e.g.):
export TFC_TOKEN=tfctoken
Based on the values above, we will include all workspaces in the migration.
If you don't already have a Spacelift API key, you can go to Settings → API keys and generate one:
After you click Add Key, a file will be downloaded that contains the API key secret. The API Key ID can be found in the previous screen.
To create an API key for Terraform Cloud check this tutorial, and to create an GitHub token, check this tutorial.
5. [Optional] Audit your current setup
To audit your current setup, you can simply run the spacemk audit command:
Agent Pools: 1 (including 1 with warnings)
Modules: 1
Organizations: 1
Policies: 1 (including 1 with warnings)
Policy Sets: 1
Projects: 4
Providers: 0
Tasks: 0
Teams: 601
Variable Set Variables: 0
Variable Sets: 0
Workspace Variables: 0
Workspaces: 1
This will give you all the information about your Terraform Cloud account, and everything that can be exported.
6. Export your configuration
To export your configuration, simply run spacemk export. This command will generate a json file in the tmp directory, called data.json.
In this file, you will see information about everything that will be migrated and how they are mapping out to Spacelift resources.
{
"context_variables": [],
"contexts": [],
"modules": [
{
"_migration_id": "aks",
"_relationships": {
"space": {
"_migration_id": "saturnhead",
"_source_id": "saturnhead",
"name": "saturnhead",
"requires_terraform_workflow_tool": false
}
},
"_source_id": "...",
"name": "aks",
"status": "setup_complete",
"terraform_provider": "az",
"vcs": {
"branch": "main",
"namespace": "flavius-dinu",
"provider": "github_custom",
"repository": "terraform-az-aks"
},
"visibility": "private"
}
],
"spaces": [
{
"_migration_id": "saturnhead",
"_relationships": {},
"_source_id": "saturnhead",
"name": "saturnhead",
"requires_terraform_workflow_tool": false
},
{
"_migration_id": "default_project",
"_relationships": {},
"_source_id": "...",
"name": "Default Project",
"requires_terraform_workflow_tool": false
},
...
{
"_migration_id": "sdsd",
"_relationships": {},
"_source_id": "...",
"name": "sdsd",
"requires_terraform_workflow_tool": false
}
],
"stack_variables": [],
"stacks": [
{
"_migration_id": "blog_examples",
"_relationships": {
"space": {
"_migration_id": "default_project",
"_source_id": "...",
"name": "Default Project",
"requires_terraform_workflow_tool": false
}
},
"_source_id": "...",
"autodeploy": false,
"description": null,
"has_secret_variables_with_invalid_name": false,
"has_variables_with_invalid_name": false,
"name": "blog-examples",
"slug": "blog-examples",
"terraform": {
"version": "1.5.5",
"workflow_tool": "TERRAFORM_FOSS"
},
"vcs": {
"branch": "main",
"namespace": "flavius-dinu",
"project_root": "migration",
"provider": "github_custom",
"repository": "blog-examples"
}
}
]
}
This file can be reviewed and modified to accommodate your needs.
💡 You might also like:
- DevOps Tech Stack : How to Choose the Right Tools
- Top 8 GitOps Tools You Should Know
- What is Developer Self-Service?
7. Generate the Terraform/OpenTofu configuration
To generate the Terraform/OpenTofu configuration, you will need to use the spacemk generate command.
terraform {
required_providers {
spacelift = {
source = "spacelift-io/spacelift"
version = "~> 1.0"
}
}
}
resource "spacelift_space" "saturnhead" {
inherit_entities = true
name = "saturnhead"
parent_space_id = "root"
}
resource "spacelift_stack" "default_project_blog_examples" {
branch = "main"
manage_state = false
name = "blog-examples"
project_root = "migration"
repository = "blog-examples"
space_id = spacelift_space.default_project.id
terraform_version = "1.5.5"
github_enterprise {
namespace = "flavius-dinu"
}
}
...
resource "spacelift_module" "saturnhead_aks" {
branch = "main"
name = "aks"
repository = "terraform-az-aks"
space_id = spacelift_space.saturnhead.id
terraform_provider = "az"
github_enterprise {
namespace = "flavius-dinu"
}
}
This will generate the Spacelift configuration required for the migration. Again, the code can be modified to accommodate your needs.
If you are not using the default VCS integration, you will also need to specify the id property in the vcs block (in my case, I will add an id to the github_enterprise block):
github_enterprise {
namespace = "flavius-dinu"
id = "flavius-dinu"
}
8. Create a repository based on this code
In your VCS, simply create a repository with the exported code. Ensure the vcs provider you are using is configured in Spacelift, and the code repositories you've exported from TFC are available in this vcs provider.
To configure a new VCS provider inside your Spacelift provider, follow the steps from this tutorial for your VCS.
Create the repo with the exported code, this is how it looks like for my GitHub configuration:
9. Create a stack based on the repository
Go to your Spacelift account, select Stacks, and the Create Stack option:
Add a name to your stack, select a space, and add optional labels and description.
In the next step, select the repository containing your migration code as shown above, and specify the branch you want to use.
Next, you can choose your vendor, it can be either Terraform, or OpenTofu, so I will accept the default Terraform FOSS workflow.
Ensure that the Administrative option is toggled on in the define behavior tab, and accept all other defaults to create the stack.
After the stack is created, trigger a run:
After reaching the unconfirmed state, you will see all the resources that it will create, in my case, there are 5 spaces, 1 stack, and 1 module.
You can easily see that all resources have been migrated successfully.
10. State Migration
If we go to our stack and select resources, we will see that there are no resources in the state:
In TFC, however, we have a couple of resources created:
In our local environment, when we've done the export, the state files were also downloaded:
To upload our state files, we must run the following command: spacemk import-state-files-to-spacelift.
Note: if you have changed the name of the stack in your terraform configuration, you will also need to modify the data.json file, to reflect the stack name, when importing the state.
After running the command, if we go back to our stack, we can see the state has been migrated successfully:
11. [Optional] Customization
While the Migration Kit does a great job when it comes to migrating to Spacelift, we understand that every migration is different and you may need to customize the functionality of the automation. To do that, you can easily implement your own commands and even your own exporter. More details about this can be found here.
Key points
This automation makes it easier to migrate from one vendor to Spacelift by getting the existing configuration and state and mapping it to Spacelift resources. We know that Terraform Cloud's new pricing model is causing problems, so we really wanted to build something that will make it quicker for you to migrate to a product that offers more for less.
To learn more about the advantages of migrating from Terraform Cloud to Spacelift, check out our Terraform Cloud alternative page.
This is not a one-size-fits-all solution. You will probably need to tweak it to adapt it to your requirements. Nonetheless, it saves considerable time and makes the overall process leaner.
Written by Flavius Dinu
...