I have created EKS cluster using terraform-aws-modules/vpc/aws with Terraform, I use one VPC with 3 private subnets on each AZs in Frankfurt. I've created two services (tomcat and psql) and deployment which are exposed via LoadBalancer and accessible via internet. It looks fine so far. but the problem is that it's only one environment (DEV). I would like to create multiple environments like stage,test and more inside one VPC and inside one cluster, how to do it using terraform? should I create new files per environment? It would not make sense but nothing comes to my mind... I was considering also workspaces but the problem is that new workspace requires new state - it means that I need to create new VPC with new cluster per one workspace! maybe I should divide my terraform files to have something like "general" workspace and there would be a configuration to VPC and cluster, and create new workspaces for each of the environments? do you have any ideas or better solutions?
VPC - 172.26.0.0/16
+----------------------+----------------------------------+
| |
| |
| KUBERNETES CLUSTER |
| +-------------------------------------------------+ |
| | | |
| | | |
| | | |
| | +------------------+ +-----------------+ | |
| | | | | | | |
| | | TEST ENV | | DEV ENV | | |
| | | +------+ +-----+ | | +-----+ +-----+ | | |
| | | |tomcat| |psql | | | |tomcat |psql | | | |
| | | +------+ +-----+ | | +-----+ +-----+ | | |
| | | | | | | |
| | +------------------+ +-----------------+ | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| +-------------------------------------------------+ |
| |
+---------------------------------------------------------+
It is possible to create multiple environments in a single K8s cluster. You could use namespace for this. To access the different environments from outside the cluster, you can use a different domain name for each environment.
For example dev.abc.com to access the development environment and test.abc.com to access the test environment.
You can "separate the vpc" in its own state file. And then have a workspace for each EKS cluster. For the EKS you can pull the VPC info one of two ways, either from AWS data source by tag or from the state file.
Your tree structure would look something like this:
├── vpc
│ ├── main.tf
│ └── outputs.tf
└── eks
└── main.tf
Add the following to the backend settings in vpc/main.tf:
terraform {
backend "s3" {
...
key = "vpc/terraform.tfstate"
workspace_key_prefix = "vpc"
...
}
}
and eks/main.tf:
terraform {
backend "s3" {
...
key = "eks/terraform.tfstate"
workspace_key_prefix = "eks"
...
}
}
Passing the VPC to the EKS section:
Option 1 (pull from aws data source by name, ref https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/vpc):
data "aws_vpc" "selected" {
filter {
...
}
}
Option 2 (pull from state file):
data "terraform_remote_state" "vpc" {
backend = "s3"
config = {
...
key = "vpc/terraform.tfstate"
workspace_key_prefix = "vpc"
...
}
}
It's not a good practice to manage your applications inside terraform, you can use terraform just to create your cluster (infra) EC2, EKS, VPC.... but what inside the cluster, you can use helm/kubectl.... to manage your pods, for example you can have two repositories, one for terraform iac and the other for projects, then you can manage your environments ( dev, staging, prod...) by namespaces...