Debug School: Suyash Sambhare

test

Suyash Sambhare — Thu, 27 Aug 2026 05:32:25 +0000

test

MSRC

Suyash Sambhare — Tue, 18 Aug 2026 05:36:20 +0000

In this PowerShell module, you will see script functions that show how to interact with the API, as well
as functions that organize data from the industry-standard CVRF document format to show some reporting scenarios.

You can clone this repo, or download the latest modules directly from the Microsoft Powershell Gallery (Powershell v3 and higher).

Installing directly from Microsoft Powershell Gallery

Must be running Powershell v5.1 or higher
With this solution, you do not need to download code from this Github repository.
Instead, you can download the directly from the Microsoft Powershell Gallery.
The Powershell version can be verified by checking the Major variable in the $PSVersionTable.PSVersion variable:

$PSVersionTable.PSVersion

Major  Minor  Build  Revision
-----  -----  -----  --------
5      1      14393  693

If you're running as an administrator you can install the module in your program files:

Install-Module MSRCSecurityUpdates -Force 
Import-Module MSRCSecurityUpdates

If you're not running as an administrator, you can install the module in your profile by providing an additional parameter about the Scope:

Install-Module MSRCSecurityUpdates -Force -Scope CurrentUser
Import-Module MSRCSecurityUpdates

Generating a HTML document of Monthly Updates

In this common scenario, the Get-MsrcCvrfDocument and Get-MsrcVulnerabilityReportHtml can be pipelined together to generate a HTML document with out-file:

### Install the module from the PowerShell Gallery (must be run as Admin)
Install-Module -Name msrcsecurityupdates -force
Import-module MsrcSecurityUpdates
$monthOfInterest = '2017-Apr'

Get-MsrcCvrfDocument -ID $monthOfInterest -Verbose | 
Get-MsrcSecurityBulletinHtml -Verbose | 
Out-File c:\temp\MSRCAprilSecurityUpdates.html

You can also build a modified object to pass into Get-MsrcVulnerabilityReportHtml. This allows more customization of the report being generated. In this example, the generated report will only have the wanted CVE's included:

Install-Module -Name MsrcSecurityUpdates -Force
Import-Module -Name MsrcSecurityUpdates -Force

$monthOfInterest = "2017-Mar"

$CVEsWanted = @(
        "CVE-2017-0001", 
        "CVE-2017-0005"
        )
$Output_Location = "C:\your\path\here"

$CVRFDoc = Get-MsrcCvrfDocument -ID $monthOfInterest -Verbose
$CVRFHtmlProperties = @{
    Vulnerability = $CVRFDoc.Vulnerability | Where-Object {$_.CVE -in $CVEsWanted}
    ProductTree = $CVRFDoc.ProductTree
}

Get-MsrcVulnerabilityReportHtml @CVRFHtmlProperties -Verbose | Out-File $Output_Location

An alternate HTML template is also avalible which generates reports which are grouped into catagories rather than by each CVE. This report may be more helpful if you are interested in all the vulnerabilities that affect a certain product and platform. This report can be ran exactly like above, however if you are filtering based on CVE's (or not piping the output from Get-MsrcCvrfDocument) two aditional fields are required. the above examples are replicated here for the different report type.

Building a report that contains all CVE's:

### Install the module from the PowerShell Gallery (must be run as Admin)
Install-Module -Name MsrcSecurityUpdates -Force
Import-module MsrcSecurityUpdates
$monthOfInterest = '2017-Apr'

Get-MsrcCvrfDocument -ID $monthOfInterest -Verbose | Get-MsrcSecurityBulletinHtml -Verbose | Out-File c:\temp\MSRCAprilSecurityUpdates.html

Using powershell to filter the report to your liking:

Install-Module -Name MsrcSecurityUpdates -Force
Import-Module -Name MsrcSecurityUpdates -Force

$monthOfInterest = "2017-Mar"

$CVEsWanted = @(
        "CVE-2017-0001", 
        "CVE-2017-0005"
        )
$Output_Location = "C:\your\path\here"

$CVRFDoc = Get-MsrcCvrfDocument -ID $monthOfInterest -Verbose
$CVRFHtmlProperties = @{
    Vulnerability = $CVRFDoc.Vulnerability | Where-Object {$_.CVE -in $CVEsWanted}
    ProductTree = $CVRFDoc.ProductTree
    DocumentTracking = $CVRFDoc.DocumentTracking
    DocumentTitle = $CVRFDoc.DocumentTitle
}

Get-MsrcSecurityBulletinHtml @CVRFHtmlProperties -Verbose | Out-File $Output_Location

Finding Mitigations and Workarounds

In this scenario, you can use the Get-MsrcCvrfDocument and extract the migrations & remediations from each vulnerability.

### Install the module from the PowerShell Gallery (must be run as Admin)
Install-Module -Name MsrcSecurityUpdates

### Download the March CVRF as an object
$cvrfDoc = Get-MsrcCvrfDocument -ID 2017-Mar

### Get the Remediations of Type 'Workaround' (0)
$cvrfDoc.Vulnerability.Remediations | Where Type -EQ 0

### Get the Remediations of Type 'Mitigation' (1)
$cvrfDoc.Vulnerability.Remediations | Where Type -EQ 1

### Get the Remediations of Type 'VendorFix' (2)
$cvrfDoc.Vulnerability.Remediations | Where Type -EQ 2

Help!

You'll find help via Get-Help for each cmdlet:

Get-Help Get-MsrcSecurityUpdate

NAME
    Get-MsrcSecurityUpdate

SYNOPSIS
    Get MSRC security updates


SYNTAX
    Get-MsrcSecurityUpdate [<CommonParameters>]

    Get-MsrcSecurityUpdate [-After <DateTime>] [-Before <DateTime>] [<CommonParameters>]

    Get-MsrcSecurityUpdate -Year <Int32> [<CommonParameters>]

    Get-MsrcSecurityUpdate -Vulnerability <String> [<CommonParameters>]


DESCRIPTION
    Calls the CVRF Update API to get a list of security updates


RELATED LINKS

REMARKS
    To see the examples, type: "get-help Get-MsrcSecurityUpdate -examples".
    For more information, type: "get-help Get-MsrcSecurityUpdate -detailed".
    For technical information, type: "get-help Get-MsrcSecurityUpdate -full".

... as well as sample code with --examples:

Get-Help Get-MsrcSecurityUpdate -examples

NAME
    Get-MsrcSecurityUpdate

SYNOPSIS
    Get MSRC security updates

    -------------------------- EXAMPLE 1 --------------------------

    PS C:\>Get-MsrcSecurityUpdate


    Get all the updates




    -------------------------- EXAMPLE 2 --------------------------

    PS C:\>Get-MsrcSecurityUpdate -Vulnerability CVE-2017-0003


    Get all the updates containing Vulnerability CVE-2017-003




    -------------------------- EXAMPLE 3 --------------------------

    PS C:\>Get-MsrcSecurityUpdate -Year 2017


    Get all the updates for the year 2017




    -------------------------- EXAMPLE 4 --------------------------

    PS C:\>Get-MsrcSecurityUpdate -Cvrf 2017-Jan


    Get all the updates for the CVRF document with ID of 2017-Jan




    -------------------------- EXAMPLE 5 --------------------------

    PS C:\>Get-MsrcSecurityUpdate -Before 2017-01-01


    Get all the updates before January 1st, 2017




    -------------------------- EXAMPLE 6 --------------------------

    PS C:\>Get-MsrcSecurityUpdate -After 2017-01-01


    Get all the updates after January 1st, 2017




    -------------------------- EXAMPLE 7 --------------------------

    PS C:\>Get-MsrcSecurityUpdate -Before 2017-01-01 -After 2016-10-01


    Get all the updates before January 1st, 2017 and after October 1st, 2016




    -------------------------- EXAMPLE 8 --------------------------

    PS C:\>Get-MsrcSecurityUpdate -After (Get-Date).AddDays(-60) -Before (Get-Date)


    Get all updates between now and the last 60 days

Ref: https://msrc.microsoft.com/update-guide

Database references

Suyash Sambhare — Mon, 10 Aug 2026 05:31:07 +0000

Database references in SQL projects enable you to incorporate objects that aren't included in a project by linking to another project, .dacpac file, or published NuGet package. The database objects you add to a project can be part of the same database, a different database on the same server, or a different database on a different server. For SQL Server development, use database references to link to another database on the same server for three-part naming, or to link to a different database on a different server for cross-database queries. For databases with a large number of objects in distinct groups, use database references to break up a database into smaller, more manageable projects. Smaller project size can help improve performance and reduce the time required to build a project during iterative local development.

SQL project file sample and syntax

Include database references in a project through entries in the .sqlproj file, similar to C# projects. Use SQLCMD syntax to reference the database name in SQL project objects. When a database reference points to a different database on the same server, include a <DatabaseSqlCmdVariable> element in the project reference. When a database reference points to a different database on a different server, also include a <ServerSqlCmdVariable> element in the project reference. Database references to the same database don't include <ServerSqlCmdVariable> or <DatabaseSqlCmdVariable> elements.

To include a specific reference to the database reference in the SQL scripts, use SQLCMD variables named in the project file to specify the database name. For example, the following SQL script references a table in the Warehouse database:

SELECT ProductId,
       StorageLocation,
       BinNumber
FROM [$(Warehouse)].[Production].[ProductInventory];

UPDATE [$(DatabaseName)].[SalesLT].[Customer]
SET [SalesPerson] = 'John Doe',
    [ModifiedDate] = GETDATE()
WHERE [CustomerId] = @CustomerId;

Build with project references

Building a SQL project with database references might require extra configuration to ensure that the referenced objects are available during the build process. For example, if you're building a project in a continuous integration (CI) pipeline, you need to set up the build agent environment similarly to the local development environment.

.dacpac references in the SQL project require that the .dacpac be present on the build agent at the same relative file path as specified in the project file.

Project references in the SQL project require that the referenced project be present on the build agent at the same relative file path as specified in the project file and be able to build successfully on the build agent.
System database references created in original SQL projects in Visual Studio require that the build agent have Visual Studio installed.
NuGet package references in the SQL project require the package be published to a NuGet feed that is also set as a package source for the build agent.

Publish with project references

Publishing a .dacpac built from a project with database references requires no extra steps. The .dacpac file contains the referenced objects and the SQLCMD variables specified in the project file.

For database references to objects in the same database, the objects from the referenced project are included in the .dacpac file but aren't included in the deployment by default. To include the objects in the deployment, use the /p:IncludeCompositeObjects=true option in the SqlPackage command line tool. For example, the following command deploys the AdventureWorks project with the /p:IncludeCompositeObjects=true option to include the objects from database references to AdventureWorks:

Ref: https://learn.microsoft.com/en-gb/sql/tools/sql-database-projects/concepts/database-references

Disaster Recovery with AWS EKS on EFS

Suyash Sambhare — Fri, 07 Aug 2026 03:35:02 +0000

Start by cloning the samples.

git clone https://github.com/aws-samples/dr-with-eks-efs.git
cd dr-with-eks-efs/
chmod +x ~/dr-with-eks-efs/workshop/startup.sh
~/dr-with-eks-efs/workshop/startup.sh

Finally, let's make the environment variables persistent across our sessions by running the below command:

source ~/.bashrc
echo 'source ~/.bashrc' >> ~/.bash_profile
echo "AWS_DEFAULT_REGION : $AWS_DEFAULT_REGION"
echo "PRI_REGION         : $PRI_REGION"
echo "DR_REGION          : $DR_REGION"
echo "PRI_CLUSTER_NAME   : $PRI_CLUSTER_NAME"
echo "DR_CLUSTER_NAME    : $DR_CLUSTER_NAME"
echo "ACCOUNT_ID         : $(aws sts get-caller-identity --query Account --output text)"

Each line should print a non-empty value. If any variable is blank, re-run the setup steps above and then source ~/.bashrc.

Confirm connectivity to the primary EKS cluster:
aws eks update-kubeconfig --name $PRI_CLUSTER_NAME --region $PRI_REGION kubectl get nodes

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap
  fileSystemId: fs-92107410
  directoryPerms: "700"
  gidRangeStart: "1000" # optional
  gidRangeEnd: "2000" # optional
  basePath: "/dynamic_provisioning" # optional
  subPathPattern: "${.PVC.namespace}/${.PVC.name}" # optional
  ensureUniqueDirectory: "false" # optional
  reuseAccessPoint: "false" # optional

Step 2 - Define the environment variables for the primary region

We will use a few variables during the next steps. Please configure these values with your choice. AWS region codes are listed

export PRI_REGION=<Replace with your choice>
export PRI_CFN_NAME=<Replace with your choice>
export PRI_CLUSTER_NAME=<Replace with your choice>

Create CloudFormation stack in the primary region :

aws cloudformation create-stack --stack-name $PRI_CFN_NAME --template-body file://config_files/pri_region_cfn.yaml --region $PRI_REGION

Check the status of the CloudFormation stack in the primary region

watch aws cloudformation describe-stacks --stack-name $PRI_CFN_NAME --query "Stacks[].StackStatus" --output text --region $PRI_REGION

Once the output shows CREATE_COMPLETE you can move on to the next step. Exit using CTRL + C.

Create the EKS cluster in the primary region

Have a look at the cluster configuration manifest file (pri_region_cluster.yaml). We specify Kubernetes version 1.28 and EFS CSI driver as an EKS managed addon.

Set and embed additional variables into manifest file and deploy the cluster to the primary region.

source config_files/pri_region_env.sh
envsubst < config_files/pri_region_eksctl_template.yaml | eksctl create cluster -f -

EKS cluster creation process completes in about 15 minutes. Once it completes update your kubeconfig file to access the cluster by doing aws eks update-kubeconfig --name $PRI_CLUSTER_NAME --region $PRI_REGION.

Verify that the worker nodes status is Ready by doing kubectl get nodes.

[!NOTE]

You can either wait for cluster creation or you can open a separate terminal and move on to deploy the infrastructure in the DR region. If you use a separate terminal then keep in mind that the variables you created in Step 1 and 5 above wont be available in that new terminal. You need to define them again as you will need to have all variables defined in the same terminal when configuring EFS replication at a later step.

Define the environment variables for the DR region

We will use a few variables during the next steps. Please configure the values of your choice below. AWS region codes are listed here.

export DR_REGION=<Replace with your choice>
export DR_CFN_NAME=<Replace with your choice>
export DR_CLUSTER_NAME=<Replace with your choice>

Set and embed additional variables into the eksctl cluster config file for the DR region

Have a look at the cluster configuration manifest file (dr_region_cluster.yaml). We specify Kubernetes version 1.28 and EFS CSI driver as an EKS managed addon.

Set and embed additional variables into manifest file and deploy the cluster to the primary region.

source config_files/dr_region_env.sh
envsubst < config_files/dr_region_eksctl_template.yaml | eksctl create cluster -f -

EKS cluster creation process completes in about 15 minutes. Once it completes update your kubeconfig file to access the cluster by doing aws eks update-kubeconfig --name $DR_CLUSTER_NAME --region $DR_REGION.

Verify that the worker nodes status is Ready by doing kubectl get nodes.

Enable EFS replication

Configure replication from primary to DR region.

aws efs update-file-system-protection --file-system-id $DR_EFS_ID --replication-overwrite-protection DISABLED --region $DR_REGION
aws efs create-replication-configuration --source-file-system-id $PRI_EFS_ID --destinations Region=$DR_REGION,FileSystemId=$DR_EFS_ID --region $PRI_REGION

You can check the status of the replication by watch aws efs describe-replication-configurations --file-system-id $PRI_EFS_ID --region $PRI_REGION --query 'Replications[].Destinations[].Status' --output text. It takes ~15 minutes for the initial replication to complete. Once you see the Status as Enabled you can then move on to the next step.

Ref: https://github.com/aws-samples/dr-with-eks-efs
Ref: https://aws.amazon.com/blogs/containers/multi-region-disaster-recovery-with-amazon-eks-and-amazon-efs-for-stateful-workloads/

Spark on EKS

Suyash Sambhare — Wed, 08 Jul 2026 02:31:15 +0000

Running and Optimizing Analytics Workloads on Amazon EKS

This advanced 400-level workshop demonstrates how to optimize Apache Spark workloads on Amazon EKS Auto Mode through hands-on exercises focusing on performance, and cost optimization using various pricing options (EC2 Spot ), processor architecture (AWS Graviton processor based instances), Amazon EC2 instance types (Memory Optimized and Compute Optimized families), and Amazon EC2 storage configurations (NVMe storage based instances). This workshop, designed to be completed in under two hours, provides you with practical experience in setting up observability tools, leveraging various processor choices, computes families, and storage options to achieve optimal performance and cost-effectiveness for your Spark jobs on Amazon EKS.

Provision the Stack

First, clone the spark-workshop branch from the data-on-eks repository:

sudo apt install terraform
git clone -b spark-workshop https://github.com/awslabs/data-on-eks.git
cd data-on-eks/analytics/terraform/spark-k8s-operator
./install.sh

The script will sequentially provision:

VPC and networking components
EKS cluster with AutoMode
Karpenter node classes and pools
EBS CSI driver and storage classes
EKS Blueprints addons

Update Kubeconfig

Once the deployment is complete, update your local kubeconfig to access the Kubernetes cluster:

aws eks update-kubeconfig --name spark-on-eks --region us-west-2
kubectl get nodes
NAME                  STATUS   ROLES    AGE     VERSION
i-02ffb325c17f97793   Ready    <none>   15m     v1.33.7-eks-3c60543
i-054759a89b32ca877   Ready    <none>   15m     v1.33.7-eks-3c60543
i-0cad804d029dcd457   Ready    <none>   6m44s   v1.33.7-eks-3c60543

kubectl get pods -A
NAMESPACE               NAME                                                              READY   STATUS    RESTARTS        AGE
amazon-cloudwatch       amazon-cloudwatch-observability-controller-manager-bbffd6cdtkbh   1/1     Running   0               16m
amazon-cloudwatch       cloudwatch-agent-fvnxt                                            1/1     Running   0               16m
amazon-cloudwatch       cloudwatch-agent-h97jm                                            1/1     Running   1 (7m36s ago)   7m49s
amazon-cloudwatch       cloudwatch-agent-ltrdn                                            1/1     Running   0               16m
amazon-cloudwatch       fluent-bit-5fj54                                                  1/1     Running   0               16m
amazon-cloudwatch       fluent-bit-f6jf9                                                  1/1     Running   0               7m49s
amazon-cloudwatch       fluent-bit-nd9jv                                                  1/1     Running   0               16m
ingress-nginx           ingress-nginx-controller-6f449f6b9d-52mfg                         1/1     Running   0               8m54s
kube-prometheus-stack   kube-prometheus-stack-grafana-b8644d6bb-9hr49                     3/3     Running   0               8m48s
kube-prometheus-stack   kube-prometheus-stack-kube-state-metrics-787d55fc86-zcx2g         1/1     Running   0               8m48s
kube-prometheus-stack   kube-prometheus-stack-operator-79df675c88-hzgmk                   1/1     Running   0               8m48s
kube-prometheus-stack   kube-prometheus-stack-prometheus-node-exporter-bqdkw              1/1     Running   0               8m48s
kube-prometheus-stack   kube-prometheus-stack-prometheus-node-exporter-lmwj5              1/1     Running   0               7m49s
kube-prometheus-stack   kube-prometheus-stack-prometheus-node-exporter-rdrbm              1/1     Running   0               8m48s
kube-prometheus-stack   prometheus-kube-prometheus-stack-prometheus-0                     2/2     Running   0               8m41s
kube-system             aws-for-fluent-bit-2qzqf                                          1/1     Running   0               8m40s
kube-system             aws-for-fluent-bit-tkwlt                                          1/1     Running   0               8m40s
kube-system             aws-for-fluent-bit-zsvb5                                          1/1     Running   0               7m47s
kube-system             coredns-7bf648ff5d-kszkh                                          1/1     Running   0               16m
kube-system             coredns-7bf648ff5d-vw5b8                                          1/1     Running   0               16m
kube-system             ebs-csi-controller-6554fb87b4-2cj2k                               6/6     Running   0               16m
kube-system             ebs-csi-controller-6554fb87b4-9kjbk                               6/6     Running   0               16m
kube-system             metrics-server-5b9d857696-2mthf                                   1/1     Running   0               16m
kube-system             metrics-server-5b9d857696-d9tvl                                   1/1     Running   0               16m
kube-system             s3-csi-controller-5c9697d6fd-hlkpd                                1/1     Running   0               16m
kube-system             s3-csi-node-5k56b                                                 3/3     Running   0               16m
kube-system             s3-csi-node-jtb2p                                                 3/3     Running   0               16m
kube-system             s3-csi-node-k2drl                                                 3/3     Running   0               7m49s
kubecost                kubecost-cost-analyzer-7b5cd5cdc6-rblrm                           4/4     Running   0               8m6s
kubecost                kubecost-forecasting-7b75bc5bbb-8nhfr                             1/1     Running   0               8m6s
kubecost                kubecost-network-costs-b2cmq                                      1/1     Running   0               8m6s
kubecost                kubecost-network-costs-hjsv2                                      1/1     Running   0               8m6s
kubecost                kubecost-network-costs-rz25g                                      1/1     Running   0               7m48s
spark-history-server    spark-history-server-0                                            1/1     Running   0               8m8s
spark-operator          spark-operator-controller-85d96b88fc-mr4qd                        1/1     Running   0               8m7s
spark-operator          spark-operator-webhook-66c6b99949-pdh6z                           1/1     Running   0               8m7s

cd data-on-eks/analytics/scripts
python3 data-gen.py

Gather Key Terraform Output Values

Set the repository's local home path as an environment variable:

export REPO_HOME=$(pwd)/data-on-eks
cd $REPO_HOME/analytics/terraform/spark-k8s-operator  

#Value for S3 Logs Bucket name
export s3_logs_bucket_name=$(terraform output -raw s3_bucket_id_spark_history_server)

#Value for S3 Logs Bucket region
export s3_logs_bucket_region=$(terraform output -raw s3_bucket_region_spark_history_server)

cd $REPO_HOME/analytics/scripts
bash order-execute.sh ${s3_logs_bucket_name} ${s3_logs_bucket_region}

Deploy and Monitor Spark Job

Configure the Spark job

cd $REPO_HOME/analytics/terraform/spark-k8s-operator/examples/karpenter
sed -i -e "s|<S3_BUCKET>|$s3_logs_bucket_name|g" spark-app-ondemand.yaml
kubectl apply -f spark-app-ondemand.yaml

Once you apply the Spark Job configuration the following happens:

Spark Driver Provisioning:

Spark Driver pod requests: 2 CPU cores, 6GB memory (4GB + 2GB overhead)
Auto Mode identifies no matching nodes with NodeGroupType=SparkComputeGeneral
Auto Mode provisions a new node matching requirements and AZ constraints
Spark Driver pod schedules and initializes

Spark Executor Setup:

Spark Driver requests 4 Spark Executor pods
Each Spark Executor needs: 2 CPU cores, 6GB memory
Total Spark Executor requirements: 8 cores, 24GB memory
Auto Mode provisions 1-2 nodes based on bin-packing efficiency
All nodes must support the requirements, capacity requirements, and be in correct AZ

Spark Job Execution:

Processes data from s3a://spark-on-eks/order/input/
Writes to s3a://spark-on-eks/order/output/ondemand/

AWS_REGION=$AWS_REGION eks-node-viewer -resources cpu,memory -extra-labels "karpenter.sh/nodepool,topology.kubernetes.io/zone"
kubectl get sparkapplication -n spark-team-a
NAME             STATUS      ATTEMPTS   START                  FINISH       AGE
order-ondemand   SUBMITTED   1          2025-11-21T00:42:07Z   <no value>   47s

kubectl get pod -n spark-team-a -l spark-role=driver
NAME                    READY   STATUS    RESTARTS   AGE
order-ondemand-driver   1/1     Running   0          84s

kubectl get pod -n spark-team-a -l spark-role=executor
NAME                    READY   STATUS    RESTARTS   AGE
order-ondemand-exec-1   1/1     Running   0          57s
order-ondemand-exec-2   1/1     Running   0          57s
order-ondemand-exec-3   1/1     Running   0          56s
order-ondemand-exec-4   1/1     Running   0          56s

kubectl describe sparkapplication order-ondemand -n spark-team-a
Name:         order-ondemand
Namespace:    spark-team-a
Labels:       app=order-ondemand
              queue=root.test
Annotations:  <none>
API Version:  sparkoperator.k8s.io/v1beta2
Kind:         SparkApplication
Metadata:
  Creation Timestamp:  2025-11-21T00:42:07Z
  Generation:          1
  Resource Version:    440830
  UID:                 660e1012-30a6-479c-92eb-6fa46b768029
Spec:
  Arguments:
    s3a://spark-on-eks-spark-logs-20251120082140604400000004/order/input/
    s3a://spark-on-eks-spark-logs-20251120082140604400000004/order/output/ondemand/
  Deps:
  Driver:
    Annotations:
      karpenter.sh/do-not-disrupt:  true
    Cores:                          2
    Memory:                         4g
    Memory Overhead:                2g
    Node Selector:
      Node Group Type:             SparkComputeGeneral
      karpenter.sh/capacity-type:  on-demand
    Service Account:               spark-team-a
  Executor:
    Affinity:
      Pod Affinity:
        Required During Scheduling Ignored During Execution:
          Label Selector:
            Match Expressions:
              Key:       app
              Operator:  In
              Values:
                order-ondemand
          Topology Key:  topology.kubernetes.io/zone
    Annotations:
      karpenter.sh/do-not-disrupt:  true
    Cores:                          2
    Instances:                      4
    Memory:                         4g
    Memory Overhead:                2g
    Node Selector:
      Node Group Type:             SparkComputeGeneral
      karpenter.sh/capacity-type:  on-demand
    Service Account:               spark-team-a
  Image:                           public.ecr.aws/data-on-eks/spark:4.0.1-scala2.13-java21-python3-r-ubuntu
  Image Pull Policy:               IfNotPresent
  Main Application File:           s3a://spark-on-eks-spark-logs-20251120082140604400000004/scripts/pyspark-order.py
  Mode:                            cluster
  Python Version:                  3
  Restart Policy:
    On Failure Retries:                    3
    On Failure Retry Interval:             10
    On Submission Failure Retries:         3
    On Submission Failure Retry Interval:  20
    Type:                                  OnFailure
  Spark Conf:
    spark.app.name:                                           order-ondemand
    spark.eventLog.dir:                                       s3a://spark-on-eks-spark-logs-20251120082140604400000004/spark-event-logs
    spark.eventLog.enabled:                                   true
    spark.eventLog.rolling.enabled:                           true
    spark.eventLog.rolling.maxFileSize:                       64m
    spark.executor.processTreeMetrics.enabled:                true
    spark.hadoop.fs.s3.impl:                                  org.apache.hadoop.fs.s3a.S3AFileSystem
    spark.hadoop.fs.s3a.aws.credentials.provider:             software.amazon.awssdk.auth.credentials.ContainerCredentialsProvider,software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider
    spark.hadoop.fs.s3a.connection.maximum:                   200
    spark.hadoop.fs.s3a.connection.timeout:                   1200000
    spark.hadoop.fs.s3a.fast.upload:                          true
    spark.hadoop.fs.s3a.input.fadvise:                        random
    spark.hadoop.fs.s3a.path.style.access:                    true
    spark.hadoop.fs.s3a.readahead.range:                      256K
    spark.kubernetes.driver.pod.name:                         order-ondemand-driver
    spark.kubernetes.executor.podNamePrefix:                  order-ondemand
    spark.metrics.conf.*.sink.prometheusServlet.class:        org.apache.spark.metrics.sink.PrometheusServlet
    spark.metrics.conf.driver.sink.prometheusServlet.path:    /metrics/driver/prometheus/
    spark.metrics.conf.executor.sink.prometheusServlet.path:  /metrics/executors/prometheus/
    spark.ui.prometheus.enabled:                              true
  Spark Version:                                              4.0.1
  Type:                                                       Python
Status:
  Application State:
    State:  COMPLETED
  Driver Info:
    Pod Name:             order-ondemand-driver
    Web UI Address:       172.20.177.132:4040
    Web UI Port:          4040
    Web UI Service Name:  order-ondemand-ui-svc
  Execution Attempts:     1
  Executor State:
    order-ondemand-exec-1:       COMPLETED
    order-ondemand-exec-2:       COMPLETED
    order-ondemand-exec-3:       COMPLETED
    order-ondemand-exec-4:       COMPLETED
  Last Submission Attempt Time:  2025-11-21T00:42:07Z
  Spark Application Id:          spark-279d190d2f8744d293baa49a9d8e2f82
  Submission Attempts:           1
  Submission ID:                 de37b837-657b-4795-a7f7-ddc2d65972e7
  Termination Time:              2025-11-21T00:45:01Z
Events:
  Type    Reason                     Age                From                          Message
  ----    ------                     ----               ----                          -------
  Normal  SparkApplicationSubmitted  2m57s              spark-application-controller  SparkApplication order-ondemand was submitted successfully
  Normal  SparkDriverRunning         115s               spark-application-controller  Driver order-ondemand-driver is running
  Normal  SparkExecutorPending       105s               spark-application-controller  Executor order-ondemand-exec-1 is pending
  Normal  SparkExecutorPending       105s               spark-application-controller  Executor order-ondemand-exec-2 is pending
  Normal  SparkExecutorPending       104s               spark-application-controller  Executor order-ondemand-exec-3 is pending
  Normal  SparkExecutorPending       104s               spark-application-controller  Executor order-ondemand-exec-4 is pending
  Normal  SparkExecutorRunning       68s                spark-application-controller  Executor order-ondemand-exec-3 is running
  Normal  SparkExecutorRunning       68s                spark-application-controller  Executor order-ondemand-exec-2 is running
  Normal  SparkExecutorRunning       68s (x2 over 68s)  spark-application-controller  Executor order-ondemand-exec-1 is running
  Normal  SparkExecutorRunning       68s (x3 over 68s)  spark-application-controller  Executor order-ondemand-exec-4 is running
  Normal  SparkExecutorCompleted     11s                spark-application-controller  Executor order-ondemand-exec-2 completed
  Normal  SparkExecutorCompleted     11s                spark-application-controller  Executor order-ondemand-exec-1 completed
  Normal  SparkExecutorCompleted     9s                 spark-application-controller  Executor order-ondemand-exec-4 completed
  Normal  SparkExecutorCompleted     9s                 spark-application-controller  Executor order-ondemand-exec-3 completed
  Normal  SparkDriverCompleted       6s                 spark-application-controller  Driver order-ondemand-driver completed

Ref: http://join.workshops.aws/8b34-065ee4-a5

ECDSA vs RSA

Suyash Sambhare — Wed, 08 Jul 2026 02:11:45 +0000

Elliptic Curve Digital Signature Algorithm vs Rivest, Shamir, and Adleman

ECDSA, or Elliptic Curve Digital Signature Algorithm, is a cryptographic method that uses the mathematics of elliptic curves to create digital signatures. It is known for its efficiency and strong security with smaller key sizes. This makes it particularly suitable for environments where computational power and storage are limited, such as mobile devices and Internet of Things (IoT) gadgets.

RSA is named after its inventors: Rivest, Shamir, and Adleman. It is one of the oldest and most widely adopted cryptographic algorithms. RSA uses the mathematical properties of large prime numbers to encrypt data and create digital signatures. While highly secure when using large key sizes, RSA requires more computational resources compared to ECDSA.

Asymmetric Encryption Algorithms

Asymmetric encryption algorithms are cryptographic methods that use a pair of keys—a public key and a private key—to secure data. The public key is used to encrypt information, and the private key is used to decrypt it

Anyone can use the public key to encrypt a message intended for a specific recipient since the public key is openly shared. However, only the person who holds the corresponding private key can decrypt the message and access its original content. This structure keeps the private key secret while receiving secure, encrypted messages.

ECDSA (elliptic curve digital signature algorithm), or ECC (elliptic curve cryptography) as it’s sometimes known, is the successor of the digital signature algorithm (DSA). ECDSA was born when two mathematicians named Neal Koblitz and Victor S. Miller proposed the use of elliptical curves in cryptography. However, it took almost two decades for the ECDSA algorithm to become standardized.

ECDSA is an asymmetric cryptography algorithm that’s constructed around elliptical curves and an underlying function that’s known as a “trapdoor function.” An elliptic curve represents the set of points that satisfy a mathematical equation $y^{2} = x^{3} + a x + b$ .

If compatibility is your top concern, go with RSA. It’s supported everywhere and doesn’t require changes to your infrastructure. It’s perfect for websites, email servers, and internal tools that rely on well-established cryptographic libraries.

If you care more about performance and efficiency, ECDSA is the clear winner. It’s faster, requires less memory, and is easier on power usage, ideal for mobile apps, connected devices, or blockchain platforms.

Some setups use both. For example, a site might use RSA for TLS while using ECDSA for internal API authentication. The two aren’t mutually exclusive.

Future Proofing against Quantum Computing

As encryption standards evolve, it’s not just about what works now, but also about what survives the next wave of change. That brings us to the threat both RSA and ECDSA can’t ignore.

Quantum computing is a real danger to current encryption systems. Algorithms like RSA and ECDSA rely on problems that a quantum computer running Shor’s algorithm could eventually solve. That would break both prime factorization and discrete logarithm-based systems.

Neither RSA nor ECDSA is safe in a fully quantum future. That’s why researchers are working on post-quantum security tools designed to resist quantum attacks. The shift is already in motion.

Ref:

Native messaging

Suyash Sambhare — Sun, 28 Jun 2026 23:50:35 +0000

To communicate with a native Win32 app that's installed on a user's device, an extension uses an API that's similar to the other message passing APIs. The native app host sends and receives messages with extensions using standard input and standard output.

Extensions that use native messaging are installed in Microsoft Edge similar to any other extension. However, native apps aren't installed or managed by Microsoft Edge.

To acquire the extension and native app host, there are two different distribution models:

Package your extension and the host together. When a user installs the package, both the extension and the host are installed.
Install your extension from Microsoft Edge Add-ons, and your extension prompts users to install the host.

To create your extension to send and receive messages with native app hosts, do the following steps.

Add permissions to the extension manifest

Add the nativeMessaging permission to the manifest.json file of the extension.

This is the extension manifest file, not the native messaging host manifest file, which is covered in later sections.

{
    "name": "Native Messaging Example",
    "version": "1.0",
    "manifest_version": 3,
    "description": "Send a message to a native app.",
    "app": {
        "launch": {
            "local_path": "main.html"
        }
    },
    "icons": {
        "128": "icon-128.png"
    },
    "permissions": ["nativeMessaging"]
}

Create your native messaging host manifest file

Native apps must provide a native messaging host manifest file. A native messaging host manifest file contains the following information:

The path to the native messaging host runtime.

The method of communication with the extension.

A list of allowed extensions to which it communicates.

The browser reads and validates the native messaging host manifest. The browser doesn't install or manage the native messaging host manifest file.

The native messaging host manifest file is distinct from the Manifest V3 or V2 file that is part of the Microsoft Edge extension.

Example of a native messaging host manifest file:

{
    "name": "com.my_company.my_app",
    "description": "My App",
    "path": "C:\\Program Files\\My App\\chrome_native_messaging_host.exe",
    "type": "stdio",
    "allowed_origins": [
        "chrome-extension://knldjmfmopnpolahpmmgbagdohdnhkik/"
    ]
}

The native messaging host manifest file must be a valid JSON file that contains the following keys:

name: Specifies the name of the native messaging host. Clients pass the string to runtime.connectNative or runtime.sendNativeMessage. The value must only contain lowercase alphanumeric characters, underscores, and dots. The value must not start or end with a dot (a period), and a dot must not be followed by another dot.
description: Describes the app.
path: Specifies the path to the native messaging host binary. On Windows devices, you can use relative paths to the directory that contains the native messaging host manifest file. On macOS and Linux, the path must be absolute. The host process starts with the current directory set to the directory that contains the host binary. For example (Windows), if the parameter is set to C:\App\nm_host.exe, the binary is started using the current directory (C:\App\).
type: Specifies the type of the interface used to communicate with the native messaging host. The value instructs Microsoft Edge to use stdin and stdout to communicate with the host. The only acceptable value is stdio.
allowed_origins: Specifies the list of extensions that have access to the native messaging host. To turn on your app to identify and communicate with an extension, in your native messaging host manifest file, set the following value: "allowed_origins": ["chrome-extension://{microsoft_catalog_extension_id}"]

Sideload

Sideload your extension to test native messaging with the host. To sideload your extension during development and retrieve microsoft_catalog_extension_id:

Go to edge://extensions, and then turn on the Developer mode toggle button.
Select Load unpacked, and then select your extension package to sideload.
Click OK.
Go to the edge://extensions page and verify that your extension is listed.
Copy the key from microsoft_catalog_extension_id (ID) from the extension listing on the page.

When you're ready to distribute your extension to users, publish your extension at Microsoft Edge Add-ons. The extension ID of the published extension might differ from the ID that's used while sideloading your extension. If the ID changed, update allowed_origins in the native messaging host manifest file with the ID of your published extension.

Copy the native messaging host manifest file to your system

The final step involves copying the native messaging host manifest file to your computer, and making sure that this manifest file is correctly configured. To ensure your native messaging host manifest file is placed in the expected location, do the following steps. The location varies by platform.
The native messaging host manifest file may be located anywhere in the file system. The app installer must create a registry key and set the default value of the key to the full path of the native messaging host manifest file.

The following locations are examples of registry keys:

HKEY_CURRENT_USER\SOFTWARE\Microsoft\Edge\NativeMessagingHosts\com.my_company.my_app
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Edge\NativeMessagingHosts\com.my_company.my_app

To add a registry key to the directory with the manifest key run a command in the command prompt:

REG ADD "HKCU\Software\Microsoft\Edge\NativeMessagingHosts\com.my_company.my_app" /ve /t REG_SZ /d "C:\path\to\nmh-manifest.json" /f

Microsoft Edge queries the HKEY_CURRENT_USER root key, followed by HKEY_LOCAL_MACHINE. In both of the keys, the 32-bit registry is searched first, and then the 64-bit registry is searched to identify native messaging hosts. The registry key specifies the location of the native messaging host manifest file.

If the registry entries for Microsoft Edge don't have the location of the native messaging host manifest file, the Chromium and Chrome registry locations are used as fallback options.

If Microsoft Edge finds the registry key at any of the previously listed locations, it doesn't query the locations that are listed in the following code snippet.

The search order for the registry locations is:

HKEY_CURRENT_USER\SOFTWARE\Microsoft\Edge\NativeMessagingHosts\
HKEY_CURRENT_USER\SOFTWARE\Chromium\NativeMessagingHosts\
HKEY_CURRENT_USER\SOFTWARE\Google\Chrome\NativeMessagingHosts\

HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\Edge\NativeMessagingHosts\
HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Chromium\NativeMessagingHosts\
HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Google\Chrome\NativeMessagingHosts\

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Edge\NativeMessagingHosts\
HKEY_LOCAL_MACHINE\SOFTWARE\Chromium\NativeMessagingHosts\
HKEY_LOCAL_MACHINE\SOFTWARE\Google\Chrome\NativeMessagingHosts\

Native messaging protocol

Microsoft Edge starts each native messaging host in a separate process and communicates with it using standard input (stdin) and standard output (stdout). The same format is used to send messages in both directions; each message is serialized using JSON, UTF-8 encoded and is preceded with 32-bit message length in native byte order. The maximum size of a single message from the native messaging host is 1 MB, mainly to protect Microsoft Edge from misbehaving native applications. The maximum size of the message sent to the native messaging host is 4 GB.

The first argument to the native messaging host is the origin of the caller, usually chrome-extension://[ID of allowed extension]. This allows native messaging hosts to identify the source of the message when multiple extensions are specified in the allowed_origins key in the native messaging host manifest; see Step 2: Create your native messaging host manifest file, above.

On Windows, the native messaging host is also passed a command line argument with a handle to the calling Microsoft Edge native window: --parent-window=. This lets the native messaging host create native UI windows that are correctly parented. This value will be 0 if the calling context is a service worker.

When a messaging port is created by using runtime.connectNative, Microsoft Edge starts a native messaging host process and keeps it running until the port is destroyed. On the other hand, when a message is sent by using runtime.sendNativeMessage, without creating a messaging port, Microsoft Edge starts a new native messaging host process for each message. In that case, the first message that's generated by the host process is handled as a response to the original request, and Microsoft Edge will pass it to the response callback specified when runtime.sendNativeMessage is called. All other messages generated by the native messaging host in that case are ignored.

Connecting to a native application

Sending and receiving messages to and from a native application is very similar to cross-extension messaging. The main difference is that runtime.connectNative is used instead of runtime.connect, and runtime.sendNativeMessage is used instead of runtime.sendMessage.

To use these methods, the nativeMessaging permission must be declared in your extensions's manifest file; see Step 1: Add permissions to the extension manifest, above.

These methods are not available inside content scripts, only inside your extension's pages and service worker. If you want to communicate from a content script to the native application, send the message to your service worker to pass it along to the native application.

The following example creates a runtime.Port object that's connected to native messaging host com.my_company.my_application, starts listening for messages from that port and sends one outgoing message:

var port = chrome.runtime.connectNative('com.my_company.my_application');
port.onMessage.addListener(function (msg) {
  console.log('Received' + msg);
});
port.onDisconnect.addListener(function () {
  console.log('Disconnected');
});
port.postMessage({text: 'Hello, my_application'});
Use runtime.sendNativeMessage to send a message to the native application without creating a port; for example:

chrome.runtime.sendNativeMessage(
  'com.my_company.my_application',
  {text: 'Hello'},
  function (response) {
    console.log('Received ' + response);
  }
);

Ref: https://learn.microsoft.com/en-us/microsoft-edge/extensions/developer-guide/native-messaging

Windows 11 context menu

Suyash Sambhare — Fri, 08 May 2026 05:15:07 +0000

Ye Olde Right Click menu

Design engineer in Microsoft probably thinking, "Hmm, the context menu when you right-click on a file is quite cluttered, lets fix that by moving everything to an overflow option called Show more options"

Le New Right Click Menu

Much better!

Win Updates via PS1

Suyash Sambhare — Wed, 22 Apr 2026 03:07:02 +0000

PowerShell file

cd C:\
sfc /scannow
chkdsk C: /scan /perf
winget upgrade --all
install-Module PSWindowsUpdate -Force
import-Module PSWindowsUpdate
get-WindowsUpdate -Install -AcceptAll
tree
shutdown /r
exit

Requires reboot

AWS Secrets Manager in GitLab CI/CD

Suyash Sambhare — Wed, 15 Apr 2026 05:45:02 +0000

Using AWS Secrets Manager in GitLab CI/CD

GitLab lets you fetch secrets directly from AWS Secrets Manager at job runtime, instead of hard‑coding or manually syncing secrets into GitLab CI variables.

This gives you:

Centralized secret management (AWS)
Short‑lived credentials (OIDC)
No secrets stored permanently in GitLab

How the Integration Works (Architecture)

GitLab Runner authenticates to AWS
- Via IAM Role attached to the runner host
- Or via OpenID Connect (OIDC) using GitLab ID tokens
During the “Resolving secrets” phase:
- Runner calls AWS Secrets Manager
- Fetches the secret value
GitLab:
- Writes the secret to a temporary file
- Exposes the file path as an environment variable
Your job uses the secret
Temp files are removed when the job finishes

Secrets never appear in the job logs unless you echo them.

Authentication Methods

1. IAM Role

variables:
  AWS_REGION: us-east-1

No tokens or role assumptions needed.

With Kubernetes runners, the IAM role must be on the runner manager, not just the pod.

2. OpenID Connect

GitLab issues a short‑lived OIDC token
AWS STS exchanges it for temporary credentials
AWS_ROLE_ARN defines what role is assumed

id_tokens:
  AWS_ID_TOKEN:
    aud: sts.amazonaws.com

variables:
  AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-secrets-role
  AWS_REGION: us-east-1

Defining Secrets in Jobs

Standard Form

secrets:
  DATABASE_PASSWORD:
    aws_secrets_manager:
      secret_id: app-secrets/database
      field: password
    file: false

password field is extracted from JSON
Value is exposed as $DATABASE_PASSWORD
file: false → variable, not file path

secrets:
  API_KEY:
    aws_secrets_manager: app-secrets/api#api_key
    file: false

Format:
secret-name[#json-field]

Working with Full JSON Secrets

If you don’t specify field, GitLab retrieves the entire value.

secrets:
  FULL_SECRET:
    aws_secrets_manager: app-secrets/api

Then:

cat "$FULL_SECRET" | jq -r '.api_key'

Secret Versioning

You can pin secrets to:

A version stage (recommended)
A specific version ID

version_stage: AWSCURRENT

version_id: 01234567-89ab-cdef-0123-456789abcdef

You cannot specify both.

Cross‑Account Access

To read secrets from another AWS account:

Use OIDC
Use the full secret ARN

secret_id: arn:aws:secretsmanager:us-east-1:987654321098:secret:shared-api-keys-AbCdEf

IAM role must trust:

GitLab’s OIDC provider
The specific GitLab project/group

Per‑Secret Overrides (Advanced)

You can override AWS settings per secret:

aws_secrets_manager:
  secret_id: eu-app-secrets/database
  region: eu-west-1
  role_arn: arn:aws:iam::123456789012:role/eu-role

File vs Variable (`file: true | false`)

Default behavior:

GitLab writes secret to a temp file
Environment variable contains file path

cat "$DATABASE_PASSWORD"

If you want the raw value:

file: false

Then:

echo "$DATABASE_PASSWORD"

File mode is safer for binaries & large secrets.

Use AWS Secrets Manager secrets in GitLab CI/CD

You can use secrets stored in AWS Secrets Manager
in your GitLab CI/CD pipelines.

Prerequisites:

Have access to AWS Secrets Manager in your AWS account.
Configure authentication using one of the following methods:
- IAM Role: Use the IAM role assigned to your GitLab Runner instance.
- OpenID Connect: Configure OpenID Connect in AWS to retrieve temporary credentials.
Add CI/CD variables to your project to provide details about your AWS configuration:
- AWS_REGION: The AWS region where your secrets are stored.
- AWS_ROLE_ARN: The ARN of the AWS IAM role to assume (required when using OpenID Connect).
- AWS_ROLE_SESSION_NAME: Optional. Custom session name for the assumed role.

Use AWS Secrets Manager secrets in a CI/CD job

With IAM Role authentication

You can use a secret stored in AWS Secrets Manager in a job by defining it with the
aws_secrets_manager keyword.

This method uses the IAM role assigned to your GitLab Runner instance. When using the
Kubernetes executor or autoscaling,
make sure the IAM role is applied to your runner manager.

Prerequisites:

GitLab Runner 18.3 or later.

For example:

variables:
  AWS_REGION: us-east-1

database-migration:
  secrets:
    DATABASE_PASSWORD:
      aws_secrets_manager:
        secret_id: app-secrets/database
        field: 'password'
      file: false
  stage: deploy
  script:
    - echo "Running database migration..."
    - mysql -h $DB_HOST -u $DB_USER -p$DATABASE_PASSWORD < migration.sql
    - echo "Migration completed successfully."

With OpenID Connect authentication

For enhanced security, you can use OpenID Connect to authenticate with AWS and assume a specific IAM role.
By default, the runner looks for an ID token named AWS_ID_TOKEN. For example:

variables:
  AWS_REGION: us-east-1
  AWS_ROLE_ARN: 'arn:aws:iam::123456789012:role/gitlab-secrets-role'

database-migration:
  id_tokens:
    AWS_ID_TOKEN:
      aud: 'sts.amazonaws.com'
  secrets:
    DATABASE_PASSWORD:
      aws_secrets_manager:
        secret_id: app-secrets/database
        field: 'password'
      file: false
  stage: deploy
  script:
    - echo "Connecting to production database..."
    - psql postgresql://$DB_USER:$DATABASE_PASSWORD@$DB_HOST:5432/$DB_NAME -c "SELECT version();"
    - echo "Database connection successful."

You can also specify a custom token using the token option. For example:

variables:
  AWS_REGION: us-east-1
  AWS_ROLE_ARN: 'arn:aws:iam::123456789012:role/gitlab-secrets-role'

database-migration:
  id_tokens:
    CUSTOM_AWS_TOKEN:
      aud: 'sts.amazonaws.com'
  secrets:
    DATABASE_PASSWORD:
      aws_secrets_manager:
        secret_id: app-secrets/database
        field: 'password'
      token: $CUSTOM_AWS_TOKEN
      file: false
  stage: deploy
  script:
    - echo "Connecting to production database with custom token..."
    - psql postgresql://$DB_USER:$DATABASE_PASSWORD@$DB_HOST:5432/$DB_NAME -c "SELECT version();"
    - echo "Database connection successful."

Short form syntax

You can use a simplified syntax by specifying the secret ID as a string.
You can optionally specify a field by separating it with a # character.
For example:

variables:
  AWS_REGION: us-east-1

api-deployment:
  secrets:
    API_KEY:
      aws_secrets_manager: 'app-secrets/api#api_key'
      file: false
    FULL_SECRET:
      aws_secrets_manager: 'app-secrets/api'
      file: false
  stage: deploy
  script:
    - echo "Deploying API with specific field..."
    - curl --header "Authorization: Bearer $API_KEY" https://api.example.com/deploy
    - echo "Using full secret..."
    - curl --header "Authorization: Bearer $(cat $FULL_SECRET | jq --raw-output '.api_key')" https://api.example.com/status

Secret versioning

AWS Secrets Manager supports multiple versions of secrets. You can specify a particular version
using either version_id or version_stage. For example:

variables:
  AWS_REGION: us-east-1

production-deployment:
  secrets:
    DATABASE_PASSWORD:
      aws_secrets_manager:
        secret_id: prod-app-secrets/database
        field: 'password'
        version_stage: 'AWSCURRENT'
      file: false
    STAGING_DATABASE_PASSWORD:
      aws_secrets_manager:
        secret_id: prod-app-secrets/database
        field: 'password'
        version_id: '01234567-89ab-cdef-0123-456789abcdef'
      file: false
  stage: deploy
  script:
    - echo "Deploying to production with current secret version..."
    - deploy-prod.sh --db-password $DATABASE_PASSWORD
    - echo "Testing with specific secret version..."
    - test-with-version.sh --db-password $STAGING_DATABASE_PASSWORD

Cross-account secret access

To retrieve secrets from another AWS account, you must use the full ARN.
For example:

variables:
  AWS_REGION: us-east-1
  AWS_ROLE_ARN: 'arn:aws:iam::123456789012:role/cross-account-secrets-role'

cross-account-deployment:
  id_tokens:
    AWS_ID_TOKEN:
      aud: 'sts.amazonaws.com'
  secrets:
    SHARED_API_KEY:
      aws_secrets_manager:
        secret_id: 'arn:aws:secretsmanager:us-east-1:987654321098:secret:shared-api-keys-AbCdEf'
        field: 'production_key'
      file: false
  stage: deploy
  script:
    - echo "Accessing shared secret from another account..."
    - curl --header "Authorization: Bearer $SHARED_API_KEY" https://shared-api.example.com/deploy

Per-secret configuration overrides

You can override global AWS settings on a per-secret basis. For example:

variables:
  AWS_REGION: us-east-1
  AWS_ROLE_ARN: 'arn:aws:iam::123456789012:role/default-role'

multi-region-deployment:
  id_tokens:
    AWS_ID_TOKEN:
      aud: 'sts.amazonaws.com'
    EU_AWS_TOKEN:
      aud: 'sts.amazonaws.com'
  secrets:
    EU_DATABASE_PASSWORD:
      aws_secrets_manager:
        secret_id: eu-app-secrets/database
        field: 'password'
        region: 'eu-west-1'
        role_arn: 'arn:aws:iam::123456789012:role/eu-deployment-role'
        role_session_name: 'gitlab-eu-deployment'
      token: $EU_AWS_TOKEN
      file: false
    US_DATABASE_PASSWORD:
      aws_secrets_manager:
        secret_id: us-app-secrets/database
        field: 'password'
      file: false
  stage: deploy
  script:
    - echo "Deploying to EU region..."
    - deploy-to-eu.sh --db-password $EU_DATABASE_PASSWORD
    - echo "Deploying to US region..."
    - deploy-to-us.sh --db-password $US_DATABASE_PASSWORD

In these examples:

aud: The audience, which must match the audience used when creating the federated identity credentials.
secret_id: The name or ARN of the secret in AWS Secrets Manager. To retrieve a secret from another account, you must use an ARN.
field: Is the specific key in the JSON secret to retrieve. If not specified, the entire secret is retrieved. Field access is only supported for flat JSON secrets (top-level keys only) and supports string, number, and boolean values. For example:
- password: Accesses the password field.
- api_key: Accesses the api_key field.
- token: Specifies which ID token to use for authentication. If not specified, the runner looks for a token named AWS_ID_TOKEN.
version_id: Is the unique identifier of a specific version of the secret. If you don't specify either version_id or version_stage, AWS Secrets Manager returns the AWSCURRENT version.
version_stage: The staging label of the version of the secret to retrieve (such as AWSCURRENT or AWSPENDING). You cannot specify both version_id and version_stage for the same secret.
region: Overrides the global AWS_REGION for this specific secret.
role_arn: Overrides the global AWS_ROLE_ARN for this specific secret.
role_session_name: Overrides the global AWS_ROLE_SESSION_NAME for this specific secret.
GitLab fetches the secret from AWS Secrets Manager and stores the value in a temporary file. The path to this file is stored in a CI/CD variable, similar to file type CI/CD variables.

Troubleshooting

Refer to OIDC for AWS troubleshooting for general
problems when setting up OIDC with AWS.

Error: `no EC2 IMDS role found`

The following error might happen if both of these conditions are true:

The CI/CD job is configured to use IAM role authentication.
The job is executed by a runner with the Kubernetes executor hosted on AWS EKS.

Resolving secrets
Resolving secret "MY_AWS_SECRET"...
Using "aws_secrets_manager" secret resolver...
ERROR: Job failed (system failure): resolving secrets: operation error Secrets Manager: GetSecretValue, get identity: get credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, canceled, context deadline exceeded

The Resolving secrets step is handled by the runner manager. This step accesses IAM credentials
cached in EC2 IMDS.
If the IAM role has not been applied to the runner manager, the Resolving secrets step fails.

To address this error, apply the correct IAM role to the runner manager.

Applying the IAM role to the runner pods that are spawned and managed by the runner manager does not resolve this issue.

Ref: https://docs.gitlab.com/ci/secrets/aws_secrets_manager/

Fix invalid checkpoint record in Postgres

Suyash Sambhare — Tue, 31 Mar 2026 22:10:21 +0000

When PostgreSQL is unable to detect a valid checkpoint from which to begin the recovery procedure, it displays the error "PANIC: could not locate a valid checkpoint record." This may occur if the PostgreSQL container is not restarted safely.

Prevent data corruption

For stateful applications like databases, the first method is to use Stateful Sets rather than Deployments; this is a Kubernetes best practice. To preserve the stability and integrity of your database, Stateful Sets offer assurances regarding the ordering and uniqueness of pods. Additionally, stateful sets will manage restarts more effectively.

The second method is to safely shut down your PostgreSQL instance by using a preStop hook in Kubernetes.
Before a pod is terminated in Kubernetes, the preStop hook is invoked. Once the preStop hook is finished, the pod's termination process starts.

Apply the preStop hook in a Kubernetes configuration for PostgreSQL:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  replicas: 1
  serviceName: "postgres"
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:latest
        ports:
        - containerPort: 5432
        volumeMounts:
        - name: postgres-data
          mountPath: /var/lib/postgresql/data
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "pg_ctl -D /var/lib/postgres/data -w -t 60 -m fast stop"]
  volumeClaimTemplates:
  - metadata:
      name: postgres-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi

This preStop hook runs the pg_ctl stop command to shut down the PostgreSQL server.

-D /var/lib/postgres/data option specifies the directory where the data base files live,
-w waits until the server shuts down,
-t 60 specifies the wait timeout in seconds, and
-m fast means to do a "fast" shutdown, which rolls back all active transactions, disconnects clients immediately and shuts down.

For AWS ECS deployments you can set the Min and max running tasks respectively: 0% min and 100% max

Recover from an error state

This can be fixed by running pg_resetwal command by connecting to the container.
However, since the pod is in crashLoopBackOff state, we will not be able to connect to the container.
We must first make the pod to be in stable state to execute the command.

Please follow the steps below for manual execution of the command pg_resetwal in a container.
This process requires the pod to be in a stable state, and not in a crash loop.
You can achieve this by introducing a delay in the postgres-deployment.yaml file.

Scale down the deployment by reducing the replica count of the deployment to zero using the following command: kubectl scale deployment postgres - replicas=0
Introduce a delay: Modify the postgres-deployment.yaml file to include a sleep command that delays the initialization process by 600 seconds, keeping the pod in the initializing state.

Here is an example of how to add it:

spec:
      containers:
      - name: postgres
        image: postgres:latest
        command: [ "/bin/bash", "-c", " - " ] 
        args: [ "sleep 600;" ]

Scale up the deployment: Bring the deployment back up by increasing the replica count to one using the following command: kubectl scale deployment postgres - replicas=1
Execute pg_resetwal in the pod: Once the pod reaches the initializing state, run the pg_resetwal command by executing into the pod. Here is how you can do this:

kubectl exec -it postgres - /bin/bash
su postgres
pg_resetwal /var/lib/postgres/data

Proceed despite the warning: If you do receive the warning, continue the process by forcing the pg_resetwal command as follows: pg_resetwal /var/lib/posgtgres/data -f

After running this command, you should receive a confirmation message stating “Write-ahead log reset.”
After you have successfully executed the pg_resetwal command, you should be able to restart the PostgreSQL server.
Remember that the pg_resetwal command is a measure of last resort and carries the risk of data loss or inconsistency.
Always make sure to maintain regular backups of your PostgreSQL databases and consider setting up high availability and replication solutions for your production databases.

Ref: https://medium.com/@adnanitdev/fix-for-error-panic-could-not-locate-a-valid-checkpoint-record-in-postgres-or-citus-running-in-b03d8341a258

VMWare vExpert Program

Suyash Sambhare — Thu, 29 Jan 2026 04:35:52 +0000

Announcement: vExpert 2025 Recognition

I am pleased to share that I have been recognised as a vExpert.

Directory Listing: https://vexpert.vmware.com/directory/9629

vExpert Badge

The vExpert program highlights individuals who demonstrate a strong passion for VMware technologies and who dedicate their time to sharing knowledge through blogs, forums, community groups, and various online or in‑person events. I am honoured to be part of such a dynamic and collaborative community, where ideas continually evolve and opportunities to engage with experts from around the world are abundant.

Importance of the vExpert Program

The VMware by Broadcom vExpert Program is a global community recognition initiative that acknowledges individuals who have made meaningful contributions to the VMware ecosystem. The program welcomes anyone with a deep interest in VMware products and services and seeks to recognise those who actively share their expertise, insights, and experiences with the broader community.

I am particularly grateful for the wealth of technical blogs, articles, and resources produced by fellow vExperts. Their contributions not only support other vExperts but also provide invaluable guidance to VMware professionals worldwide. These resources span a wide range of topics, including best practices, troubleshooting, automation, networking, and advanced virtualisation concepts.

Ways to Qualify for vExpert

The vExpert award recognises contributions across multiple categories, including:

Enterprise/Internal Influencer – Leading internal knowledge‑sharing sessions such as workshops or webinars on VMware technologies.
Blogging – Publishing educational content on VMware products or events like VMware Explore.
Code Sharing – Providing tools, scripts, or code samples through websites, apps, VMware {code}, or internal platforms.
Event Speaking – Presenting at VMUG meetings, VMware Explore, or other industry conferences.
Podcasting – Hosting or co‑hosting podcasts focused on VMware topics.
Video Content – Creating tutorials, demonstrations, or technical walkthroughs on platforms such as YouTube.
Online Forums – Supporting the community by answering questions on VMTN or VMware‑related Reddit communities.
VMUG Leadership Support – Assisting local VMUG chapters in leadership or organisational roles.

Value of the vExpert Program

Becoming a vExpert provides numerous benefits that extend well beyond recognition:

Content Promotion – vExpert content is amplified through VMware’s official social channels, LinkedIn, Broadcom Advocacy, and the VMware Blog RSS feed, increasing visibility and engagement.
Personal Brand Development – The vExpert title enhances professional credibility and strengthens your reputation within the technology community.
Professional Recognition – Inclusion in the vExpert Directory and on LinkedIn validates your community contributions and helps distinguish you in your career.
Complimentary VMUG Advantage Membership – Includes access to valuable resources and discounts.
Free VMware Licenses – vExperts receive VCF Home Lab licenses through VMUG Advantage (subject to passing the VCP‑VCF exam), enabling hands‑on learning and skill development.
Exclusive Networking Opportunities – Access to private vExpert communication channels and industry experts.
Early Access & NDA Briefings – Participation in NDA sessions and early access to VMware roadmap materials.
VMware Explore Blogger Passes – Complimentary blogger access to VMware Explore events (limited availability).

Pathways to Becoming a vExpert

There are several recognised avenues for contributing to the VMware community:

Enterprise / Internal Influencers
Bloggers
Code Contributors
VMUG Leaders
Event Speakers
Podcasters
Online Forum Supporters
Authors
VCDX Certification Holders

Beyond being a symbol of achievement, the vExpert program empowers individuals to expand their professional network, strengthen their personal brand, and gain access to exclusive resources and events.

Closing Thoughts

Being part of the vExpert community is not solely about holding a title—it is about the collective impact we make on the VMware ecosystem. One of the most valuable aspects of this community is the wealth of knowledge shared by its members, which continues to inspire and support professionals across the globe.

If you found this article helpful, a share on LinkedIn or X would be greatly appreciated.

vExpert Suyash Sambhare

Directory Listing: https://vexpert.vmware.com/directory/9629

For more information, visit the official vExpert website: https://vexpert.vmware.com

Debug School: Suyash Sambhare

test

MSRC

Installing directly from Microsoft Powershell Gallery

Generating a HTML document of Monthly Updates

Finding Mitigations and Workarounds

Help!

Database references

SQL project file sample and syntax

Build with project references

Publish with project references

Disaster Recovery with AWS EKS on EFS

Step 2 - Define the environment variables for the primary region

Create CloudFormation stack in the primary region :

Check the status of the CloudFormation stack in the primary region

Create the EKS cluster in the primary region

Define the environment variables for the DR region

Enable EFS replication

Spark on EKS

Running and Optimizing Analytics Workloads on Amazon EKS

Provision the Stack

Update Kubeconfig

Gather Key Terraform Output Values

Deploy and Monitor Spark Job

ECDSA vs RSA

Asymmetric Encryption Algorithms

Future Proofing against Quantum Computing

Native messaging

Add permissions to the extension manifest

Create your native messaging host manifest file

Sideload

Copy the native messaging host manifest file to your system

Native messaging protocol

Connecting to a native application

Windows 11 context menu

Ye Olde Right Click menu

Le New Right Click Menu

Win Updates via PS1

AWS Secrets Manager in GitLab CI/CD

Using AWS Secrets Manager in GitLab CI/CD

How the Integration Works (Architecture)

Authentication Methods

1. IAM Role

2. OpenID Connect

Defining Secrets in Jobs

Standard Form

Working with Full JSON Secrets

Secret Versioning

Cross‑Account Access

Per‑Secret Overrides (Advanced)

File vs Variable (file: true | false)

Use AWS Secrets Manager secrets in GitLab CI/CD

Use AWS Secrets Manager secrets in a CI/CD job

With IAM Role authentication

With OpenID Connect authentication

Short form syntax

Secret versioning

Cross-account secret access

Per-secret configuration overrides

Troubleshooting

Error: no EC2 IMDS role found

Fix invalid checkpoint record in Postgres

Prevent data corruption

Recover from an error state

VMWare vExpert Program

Announcement: vExpert 2025 Recognition

vExpert Badge

Importance of the vExpert Program

Ways to Qualify for vExpert

Value of the vExpert Program

Pathways to Becoming a vExpert

Closing Thoughts

File vs Variable (`file: true | false`)

Error: `no EC2 IMDS role found`