Debug School

Cover image for Disaster Recovery with AWS EKS on EFS
Suyash Sambhare
Suyash Sambhare

Posted on

Disaster Recovery with AWS EKS on EFS

#aws #eks #dr #devops

Start by cloning the samples.

git clone https://github.com/aws-samples/dr-with-eks-efs.git
cd dr-with-eks-efs/
chmod +x ~/dr-with-eks-efs/workshop/startup.sh
~/dr-with-eks-efs/workshop/startup.sh
Enter fullscreen mode Exit fullscreen mode

Finally, let's make the environment variables persistent across our sessions by running the below command:

source ~/.bashrc
echo 'source ~/.bashrc' >> ~/.bash_profile
echo "AWS_DEFAULT_REGION : $AWS_DEFAULT_REGION"
echo "PRI_REGION         : $PRI_REGION"
echo "DR_REGION          : $DR_REGION"
echo "PRI_CLUSTER_NAME   : $PRI_CLUSTER_NAME"
echo "DR_CLUSTER_NAME    : $DR_CLUSTER_NAME"
echo "ACCOUNT_ID         : $(aws sts get-caller-identity --query Account --output text)"
Enter fullscreen mode Exit fullscreen mode

Each line should print a non-empty value. If any variable is blank, re-run the setup steps above and then source ~/.bashrc.

EKS

Confirm connectivity to the primary EKS cluster:
aws eks update-kubeconfig --name $PRI_CLUSTER_NAME --region $PRI_REGION
kubectl get nodes

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap
  fileSystemId: fs-92107410
  directoryPerms: "700"
  gidRangeStart: "1000" # optional
  gidRangeEnd: "2000" # optional
  basePath: "/dynamic_provisioning" # optional
  subPathPattern: "${.PVC.namespace}/${.PVC.name}" # optional
  ensureUniqueDirectory: "false" # optional
  reuseAccessPoint: "false" # optional
Enter fullscreen mode Exit fullscreen mode

Step 2 - Define the environment variables for the primary region

We will use a few variables during the next steps. Please configure these values with your choice. AWS region codes are listed

export PRI_REGION=<Replace with your choice>
export PRI_CFN_NAME=<Replace with your choice>
export PRI_CLUSTER_NAME=<Replace with your choice>
Enter fullscreen mode Exit fullscreen mode

Create CloudFormation stack in the primary region : 

aws cloudformation create-stack --stack-name $PRI_CFN_NAME --template-body file://config_files/pri_region_cfn.yaml --region $PRI_REGION
Enter fullscreen mode Exit fullscreen mode

Check the status of the CloudFormation stack in the primary region 

watch aws cloudformation describe-stacks --stack-name $PRI_CFN_NAME --query "Stacks[].StackStatus" --output text --region $PRI_REGION
Enter fullscreen mode Exit fullscreen mode

Once the output shows CREATE_COMPLETE you can move on to the next step. Exit using CTRL + C.

Create the EKS cluster in the primary region

Have a look at the cluster configuration manifest file (pri_region_cluster.yaml). We specify Kubernetes version 1.28 and EFS CSI driver as an EKS managed addon.

Set and embed additional variables into manifest file and deploy the cluster to the primary region.

source config_files/pri_region_env.sh
envsubst < config_files/pri_region_eksctl_template.yaml | eksctl create cluster -f -
Enter fullscreen mode Exit fullscreen mode

EKS cluster creation process completes in about 15 minutes. Once it completes update your kubeconfig file to access the cluster by doing aws eks update-kubeconfig --name $PRI_CLUSTER_NAME --region $PRI_REGION.

Verify that the worker nodes status is Ready by doing kubectl get nodes.

[!NOTE]

You can either wait for cluster creation or you can open a separate terminal and move on to deploy the infrastructure in the DR region. If you use a separate terminal then keep in mind that the variables you created in Step 1 and 5 above wont be available in that new terminal. You need to define them again as you will need to have all variables defined in the same terminal when configuring EFS replication at a later step.

Define the environment variables for the DR region

We will use a few variables during the next steps. Please configure the values of your choice below. AWS region codes are listed here.

export DR_REGION=<Replace with your choice>
export DR_CFN_NAME=<Replace with your choice>
export DR_CLUSTER_NAME=<Replace with your choice>
Enter fullscreen mode Exit fullscreen mode

Set and embed additional variables into the eksctl cluster config file for the DR region

Have a look at the cluster configuration manifest file (dr_region_cluster.yaml). We specify Kubernetes version 1.28 and EFS CSI driver as an EKS managed addon.

Set and embed additional variables into manifest file and deploy the cluster to the primary region.

source config_files/dr_region_env.sh
envsubst < config_files/dr_region_eksctl_template.yaml | eksctl create cluster -f -
Enter fullscreen mode Exit fullscreen mode

EKS cluster creation process completes in about 15 minutes. Once it completes update your kubeconfig file to access the cluster by doing aws eks update-kubeconfig --name $DR_CLUSTER_NAME --region $DR_REGION.

Verify that the worker nodes status is Ready by doing kubectl get nodes.

Enable EFS replication

Configure replication from primary to DR region.

aws efs update-file-system-protection --file-system-id $DR_EFS_ID --replication-overwrite-protection DISABLED --region $DR_REGION
aws efs create-replication-configuration --source-file-system-id $PRI_EFS_ID --destinations Region=$DR_REGION,FileSystemId=$DR_EFS_ID --region $PRI_REGION
Enter fullscreen mode Exit fullscreen mode

You can check the status of the replication by watch aws efs describe-replication-configurations --file-system-id $PRI_EFS_ID --region $PRI_REGION --query 'Replications[].Destinations[].Status' --output text. It takes ~15 minutes for the initial replication to complete. Once you see the Status as Enabled you can then move on to the next step.

Ref: https://github.com/aws-samples/dr-with-eks-efs
Ref: https://aws.amazon.com/blogs/containers/multi-region-disaster-recovery-with-amazon-eks-and-amazon-efs-for-stateful-workloads/

Top comments (0)