Running and Optimizing Analytics Workloads on Amazon EKS
This advanced 400-level workshop demonstrates how to optimize Apache Spark workloads on Amazon EKS Auto Mode through hands-on exercises focusing on performance, and cost optimization using various pricing options (EC2 Spot ), processor architecture (AWS Graviton processor based instances), Amazon EC2 instance types (Memory Optimized and Compute Optimized families), and Amazon EC2 storage configurations (NVMe storage based instances). This workshop, designed to be completed in under two hours, provides you with practical experience in setting up observability tools, leveraging various processor choices, computes families, and storage options to achieve optimal performance and cost-effectiveness for your Spark jobs on Amazon EKS.
Provision the Stack
First, clone the spark-workshop branch from the data-on-eks repository:
sudo apt install terraform
git clone -b spark-workshop https://github.com/awslabs/data-on-eks.git
cd data-on-eks/analytics/terraform/spark-k8s-operator
./install.sh
The script will sequentially provision:
- VPC and networking components
- EKS cluster with AutoMode
- Karpenter node classes and pools
- EBS CSI driver and storage classes
- EKS Blueprints addons
Update Kubeconfig
Once the deployment is complete, update your local kubeconfig to access the Kubernetes cluster:
aws eks update-kubeconfig --name spark-on-eks --region us-west-2
kubectl get nodes
NAME STATUS ROLES AGE VERSION
i-02ffb325c17f97793 Ready <none> 15m v1.33.7-eks-3c60543
i-054759a89b32ca877 Ready <none> 15m v1.33.7-eks-3c60543
i-0cad804d029dcd457 Ready <none> 6m44s v1.33.7-eks-3c60543
kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
amazon-cloudwatch amazon-cloudwatch-observability-controller-manager-bbffd6cdtkbh 1/1 Running 0 16m
amazon-cloudwatch cloudwatch-agent-fvnxt 1/1 Running 0 16m
amazon-cloudwatch cloudwatch-agent-h97jm 1/1 Running 1 (7m36s ago) 7m49s
amazon-cloudwatch cloudwatch-agent-ltrdn 1/1 Running 0 16m
amazon-cloudwatch fluent-bit-5fj54 1/1 Running 0 16m
amazon-cloudwatch fluent-bit-f6jf9 1/1 Running 0 7m49s
amazon-cloudwatch fluent-bit-nd9jv 1/1 Running 0 16m
ingress-nginx ingress-nginx-controller-6f449f6b9d-52mfg 1/1 Running 0 8m54s
kube-prometheus-stack kube-prometheus-stack-grafana-b8644d6bb-9hr49 3/3 Running 0 8m48s
kube-prometheus-stack kube-prometheus-stack-kube-state-metrics-787d55fc86-zcx2g 1/1 Running 0 8m48s
kube-prometheus-stack kube-prometheus-stack-operator-79df675c88-hzgmk 1/1 Running 0 8m48s
kube-prometheus-stack kube-prometheus-stack-prometheus-node-exporter-bqdkw 1/1 Running 0 8m48s
kube-prometheus-stack kube-prometheus-stack-prometheus-node-exporter-lmwj5 1/1 Running 0 7m49s
kube-prometheus-stack kube-prometheus-stack-prometheus-node-exporter-rdrbm 1/1 Running 0 8m48s
kube-prometheus-stack prometheus-kube-prometheus-stack-prometheus-0 2/2 Running 0 8m41s
kube-system aws-for-fluent-bit-2qzqf 1/1 Running 0 8m40s
kube-system aws-for-fluent-bit-tkwlt 1/1 Running 0 8m40s
kube-system aws-for-fluent-bit-zsvb5 1/1 Running 0 7m47s
kube-system coredns-7bf648ff5d-kszkh 1/1 Running 0 16m
kube-system coredns-7bf648ff5d-vw5b8 1/1 Running 0 16m
kube-system ebs-csi-controller-6554fb87b4-2cj2k 6/6 Running 0 16m
kube-system ebs-csi-controller-6554fb87b4-9kjbk 6/6 Running 0 16m
kube-system metrics-server-5b9d857696-2mthf 1/1 Running 0 16m
kube-system metrics-server-5b9d857696-d9tvl 1/1 Running 0 16m
kube-system s3-csi-controller-5c9697d6fd-hlkpd 1/1 Running 0 16m
kube-system s3-csi-node-5k56b 3/3 Running 0 16m
kube-system s3-csi-node-jtb2p 3/3 Running 0 16m
kube-system s3-csi-node-k2drl 3/3 Running 0 7m49s
kubecost kubecost-cost-analyzer-7b5cd5cdc6-rblrm 4/4 Running 0 8m6s
kubecost kubecost-forecasting-7b75bc5bbb-8nhfr 1/1 Running 0 8m6s
kubecost kubecost-network-costs-b2cmq 1/1 Running 0 8m6s
kubecost kubecost-network-costs-hjsv2 1/1 Running 0 8m6s
kubecost kubecost-network-costs-rz25g 1/1 Running 0 7m48s
spark-history-server spark-history-server-0 1/1 Running 0 8m8s
spark-operator spark-operator-controller-85d96b88fc-mr4qd 1/1 Running 0 8m7s
spark-operator spark-operator-webhook-66c6b99949-pdh6z 1/1 Running 0 8m7s
cd data-on-eks/analytics/scripts
python3 data-gen.py
Gather Key Terraform Output Values
Set the repository's local home path as an environment variable:
export REPO_HOME=$(pwd)/data-on-eks
cd $REPO_HOME/analytics/terraform/spark-k8s-operator
#Value for S3 Logs Bucket name
export s3_logs_bucket_name=$(terraform output -raw s3_bucket_id_spark_history_server)
#Value for S3 Logs Bucket region
export s3_logs_bucket_region=$(terraform output -raw s3_bucket_region_spark_history_server)
cd $REPO_HOME/analytics/scripts
bash order-execute.sh ${s3_logs_bucket_name} ${s3_logs_bucket_region}
Deploy and Monitor Spark Job
- Configure the Spark job
cd $REPO_HOME/analytics/terraform/spark-k8s-operator/examples/karpenter
sed -i -e "s|<S3_BUCKET>|$s3_logs_bucket_name|g" spark-app-ondemand.yaml
kubectl apply -f spark-app-ondemand.yaml
Once you apply the Spark Job configuration the following happens:
Spark Driver Provisioning:
- Spark Driver pod requests: 2 CPU cores, 6GB memory (4GB + 2GB overhead)
- Auto Mode identifies no matching nodes with
NodeGroupType=SparkComputeGeneral - Auto Mode provisions a new node matching requirements and AZ constraints
- Spark Driver pod schedules and initializes
Spark Executor Setup:
- Spark Driver requests 4 Spark Executor pods
- Each Spark Executor needs: 2 CPU cores, 6GB memory
- Total Spark Executor requirements: 8 cores, 24GB memory
- Auto Mode provisions 1-2 nodes based on bin-packing efficiency
- All nodes must support the requirements, capacity requirements, and be in correct AZ
Spark Job Execution:
- Processes data from
s3a://spark-on-eks/order/input/ - Writes to
s3a://spark-on-eks/order/output/ondemand/
AWS_REGION=$AWS_REGION eks-node-viewer -resources cpu,memory -extra-labels "karpenter.sh/nodepool,topology.kubernetes.io/zone"
kubectl get sparkapplication -n spark-team-a
NAME STATUS ATTEMPTS START FINISH AGE
order-ondemand SUBMITTED 1 2025-11-21T00:42:07Z <no value> 47s
kubectl get pod -n spark-team-a -l spark-role=driver
NAME READY STATUS RESTARTS AGE
order-ondemand-driver 1/1 Running 0 84s
kubectl get pod -n spark-team-a -l spark-role=executor
NAME READY STATUS RESTARTS AGE
order-ondemand-exec-1 1/1 Running 0 57s
order-ondemand-exec-2 1/1 Running 0 57s
order-ondemand-exec-3 1/1 Running 0 56s
order-ondemand-exec-4 1/1 Running 0 56s
kubectl describe sparkapplication order-ondemand -n spark-team-a
Name: order-ondemand
Namespace: spark-team-a
Labels: app=order-ondemand
queue=root.test
Annotations: <none>
API Version: sparkoperator.k8s.io/v1beta2
Kind: SparkApplication
Metadata:
Creation Timestamp: 2025-11-21T00:42:07Z
Generation: 1
Resource Version: 440830
UID: 660e1012-30a6-479c-92eb-6fa46b768029
Spec:
Arguments:
s3a://spark-on-eks-spark-logs-20251120082140604400000004/order/input/
s3a://spark-on-eks-spark-logs-20251120082140604400000004/order/output/ondemand/
Deps:
Driver:
Annotations:
karpenter.sh/do-not-disrupt: true
Cores: 2
Memory: 4g
Memory Overhead: 2g
Node Selector:
Node Group Type: SparkComputeGeneral
karpenter.sh/capacity-type: on-demand
Service Account: spark-team-a
Executor:
Affinity:
Pod Affinity:
Required During Scheduling Ignored During Execution:
Label Selector:
Match Expressions:
Key: app
Operator: In
Values:
order-ondemand
Topology Key: topology.kubernetes.io/zone
Annotations:
karpenter.sh/do-not-disrupt: true
Cores: 2
Instances: 4
Memory: 4g
Memory Overhead: 2g
Node Selector:
Node Group Type: SparkComputeGeneral
karpenter.sh/capacity-type: on-demand
Service Account: spark-team-a
Image: public.ecr.aws/data-on-eks/spark:4.0.1-scala2.13-java21-python3-r-ubuntu
Image Pull Policy: IfNotPresent
Main Application File: s3a://spark-on-eks-spark-logs-20251120082140604400000004/scripts/pyspark-order.py
Mode: cluster
Python Version: 3
Restart Policy:
On Failure Retries: 3
On Failure Retry Interval: 10
On Submission Failure Retries: 3
On Submission Failure Retry Interval: 20
Type: OnFailure
Spark Conf:
spark.app.name: order-ondemand
spark.eventLog.dir: s3a://spark-on-eks-spark-logs-20251120082140604400000004/spark-event-logs
spark.eventLog.enabled: true
spark.eventLog.rolling.enabled: true
spark.eventLog.rolling.maxFileSize: 64m
spark.executor.processTreeMetrics.enabled: true
spark.hadoop.fs.s3.impl: org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.aws.credentials.provider: software.amazon.awssdk.auth.credentials.ContainerCredentialsProvider,software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider
spark.hadoop.fs.s3a.connection.maximum: 200
spark.hadoop.fs.s3a.connection.timeout: 1200000
spark.hadoop.fs.s3a.fast.upload: true
spark.hadoop.fs.s3a.input.fadvise: random
spark.hadoop.fs.s3a.path.style.access: true
spark.hadoop.fs.s3a.readahead.range: 256K
spark.kubernetes.driver.pod.name: order-ondemand-driver
spark.kubernetes.executor.podNamePrefix: order-ondemand
spark.metrics.conf.*.sink.prometheusServlet.class: org.apache.spark.metrics.sink.PrometheusServlet
spark.metrics.conf.driver.sink.prometheusServlet.path: /metrics/driver/prometheus/
spark.metrics.conf.executor.sink.prometheusServlet.path: /metrics/executors/prometheus/
spark.ui.prometheus.enabled: true
Spark Version: 4.0.1
Type: Python
Status:
Application State:
State: COMPLETED
Driver Info:
Pod Name: order-ondemand-driver
Web UI Address: 172.20.177.132:4040
Web UI Port: 4040
Web UI Service Name: order-ondemand-ui-svc
Execution Attempts: 1
Executor State:
order-ondemand-exec-1: COMPLETED
order-ondemand-exec-2: COMPLETED
order-ondemand-exec-3: COMPLETED
order-ondemand-exec-4: COMPLETED
Last Submission Attempt Time: 2025-11-21T00:42:07Z
Spark Application Id: spark-279d190d2f8744d293baa49a9d8e2f82
Submission Attempts: 1
Submission ID: de37b837-657b-4795-a7f7-ddc2d65972e7
Termination Time: 2025-11-21T00:45:01Z
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SparkApplicationSubmitted 2m57s spark-application-controller SparkApplication order-ondemand was submitted successfully
Normal SparkDriverRunning 115s spark-application-controller Driver order-ondemand-driver is running
Normal SparkExecutorPending 105s spark-application-controller Executor order-ondemand-exec-1 is pending
Normal SparkExecutorPending 105s spark-application-controller Executor order-ondemand-exec-2 is pending
Normal SparkExecutorPending 104s spark-application-controller Executor order-ondemand-exec-3 is pending
Normal SparkExecutorPending 104s spark-application-controller Executor order-ondemand-exec-4 is pending
Normal SparkExecutorRunning 68s spark-application-controller Executor order-ondemand-exec-3 is running
Normal SparkExecutorRunning 68s spark-application-controller Executor order-ondemand-exec-2 is running
Normal SparkExecutorRunning 68s (x2 over 68s) spark-application-controller Executor order-ondemand-exec-1 is running
Normal SparkExecutorRunning 68s (x3 over 68s) spark-application-controller Executor order-ondemand-exec-4 is running
Normal SparkExecutorCompleted 11s spark-application-controller Executor order-ondemand-exec-2 completed
Normal SparkExecutorCompleted 11s spark-application-controller Executor order-ondemand-exec-1 completed
Normal SparkExecutorCompleted 9s spark-application-controller Executor order-ondemand-exec-4 completed
Normal SparkExecutorCompleted 9s spark-application-controller Executor order-ondemand-exec-3 completed
Normal SparkDriverCompleted 6s spark-application-controller Driver order-ondemand-driver completed
Top comments (0)