Debug School

Cover image for AWS Secrets Manager in GitLab CI/CD
Suyash Sambhare
Suyash Sambhare

Posted on

AWS Secrets Manager in GitLab CI/CD

Using AWS Secrets Manager in GitLab CI/CD

GitLab lets you fetch secrets directly from AWS Secrets Manager at job runtime, instead of hard‑coding or manually syncing secrets into GitLab CI variables.

This gives you:

  • Centralized secret management (AWS)
  • Short‑lived credentials (OIDC)
  • No secrets stored permanently in GitLab

How the Integration Works (Architecture)

  1. GitLab Runner authenticates to AWS
    • Via IAM Role attached to the runner host
    • Or via OpenID Connect (OIDC) using GitLab ID tokens
  2. During the “Resolving secrets” phase:
    • Runner calls AWS Secrets Manager
    • Fetches the secret value
  3. GitLab:
    • Writes the secret to a temporary file
    • Exposes the file path as an environment variable
  4. Your job uses the secret
  5. Temp files are removed when the job finishes

Secrets never appear in the job logs unless you echo them.

Authentication Methods

1. IAM Role 

variables:
  AWS_REGION: us-east-1
Enter fullscreen mode Exit fullscreen mode

No tokens or role assumptions needed.

With Kubernetes runners, the IAM role must be on the runner manager, not just the pod.

2. OpenID Connect

  • GitLab issues a short‑lived OIDC token
  • AWS STS exchanges it for temporary credentials
  • AWS_ROLE_ARN defines what role is assumed 
id_tokens:
  AWS_ID_TOKEN:
    aud: sts.amazonaws.com
Enter fullscreen mode Exit fullscreen mode 
variables:
  AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-secrets-role
  AWS_REGION: us-east-1
Enter fullscreen mode Exit fullscreen mode

Defining Secrets in Jobs

Standard Form 

secrets:
  DATABASE_PASSWORD:
    aws_secrets_manager:
      secret_id: app-secrets/database
      field: password
    file: false
Enter fullscreen mode Exit fullscreen mode
  • password field is extracted from JSON
  • Value is exposed as $DATABASE_PASSWORD
  • file: false → variable, not file path
secrets:
  API_KEY:
    aws_secrets_manager: app-secrets/api#api_key
    file: false
Enter fullscreen mode Exit fullscreen mode

Format:
secret-name[#json-field]

Working with Full JSON Secrets

If you don’t specify field, GitLab retrieves the entire value.

secrets:
  FULL_SECRET:
    aws_secrets_manager: app-secrets/api
Enter fullscreen mode Exit fullscreen mode

Then:

cat "$FULL_SECRET" | jq -r '.api_key'
Enter fullscreen mode Exit fullscreen mode

Secret Versioning

You can pin secrets to:

  • A version stage (recommended)
  • A specific version ID 
version_stage: AWSCURRENT
Enter fullscreen mode Exit fullscreen mode

or

version_id: 01234567-89ab-cdef-0123-456789abcdef
Enter fullscreen mode Exit fullscreen mode

You cannot specify both.

Cross‑Account Access

To read secrets from another AWS account:

  • Use OIDC
  • Use the full secret ARN 
secret_id: arn:aws:secretsmanager:us-east-1:987654321098:secret:shared-api-keys-AbCdEf
Enter fullscreen mode Exit fullscreen mode

IAM role must trust:

  • GitLab’s OIDC provider
  • The specific GitLab project/group

Per‑Secret Overrides (Advanced)

You can override AWS settings per secret:

aws_secrets_manager:
  secret_id: eu-app-secrets/database
  region: eu-west-1
  role_arn: arn:aws:iam::123456789012:role/eu-role
Enter fullscreen mode Exit fullscreen mode

File vs Variable (file: true | false)

Default behavior:

  • GitLab writes secret to a temp file
  • Environment variable contains file path 
cat "$DATABASE_PASSWORD"
Enter fullscreen mode Exit fullscreen mode

If you want the raw value:

file: false
Enter fullscreen mode Exit fullscreen mode

Then:

echo "$DATABASE_PASSWORD"
Enter fullscreen mode Exit fullscreen mode

File mode is safer for binaries & large secrets.

AWS

Use AWS Secrets Manager secrets in GitLab CI/CD

You can use secrets stored in AWS Secrets Manager
in your GitLab CI/CD pipelines.

Prerequisites:

  • Have access to AWS Secrets Manager in your AWS account.
  • Configure authentication using one of the following methods:
  • Add CI/CD variables to your project to provide details about your AWS configuration:
    • AWS_REGION: The AWS region where your secrets are stored.
    • AWS_ROLE_ARN: The ARN of the AWS IAM role to assume (required when using OpenID Connect).
    • AWS_ROLE_SESSION_NAME: Optional. Custom session name for the assumed role.

Use AWS Secrets Manager secrets in a CI/CD job

With IAM Role authentication

You can use a secret stored in AWS Secrets Manager in a job by defining it with the
aws_secrets_manager keyword.

This method uses the IAM role assigned to your GitLab Runner instance. When using the
Kubernetes executor or autoscaling,
make sure the IAM role is applied to your runner manager.

Prerequisites:

  • GitLab Runner 18.3 or later.

For example:

variables:
  AWS_REGION: us-east-1

database-migration:
  secrets:
    DATABASE_PASSWORD:
      aws_secrets_manager:
        secret_id: app-secrets/database
        field: 'password'
      file: false
  stage: deploy
  script:
    - echo "Running database migration..."
    - mysql -h $DB_HOST -u $DB_USER -p$DATABASE_PASSWORD < migration.sql
    - echo "Migration completed successfully."
Enter fullscreen mode Exit fullscreen mode

With OpenID Connect authentication

For enhanced security, you can use OpenID Connect to authenticate with AWS and assume a specific IAM role.
By default, the runner looks for an ID token named AWS_ID_TOKEN. For example:

variables:
  AWS_REGION: us-east-1
  AWS_ROLE_ARN: 'arn:aws:iam::123456789012:role/gitlab-secrets-role'

database-migration:
  id_tokens:
    AWS_ID_TOKEN:
      aud: 'sts.amazonaws.com'
  secrets:
    DATABASE_PASSWORD:
      aws_secrets_manager:
        secret_id: app-secrets/database
        field: 'password'
      file: false
  stage: deploy
  script:
    - echo "Connecting to production database..."
    - psql postgresql://$DB_USER:$DATABASE_PASSWORD@$DB_HOST:5432/$DB_NAME -c "SELECT version();"
    - echo "Database connection successful."
Enter fullscreen mode Exit fullscreen mode

You can also specify a custom token using the token option. For example:

variables:
  AWS_REGION: us-east-1
  AWS_ROLE_ARN: 'arn:aws:iam::123456789012:role/gitlab-secrets-role'

database-migration:
  id_tokens:
    CUSTOM_AWS_TOKEN:
      aud: 'sts.amazonaws.com'
  secrets:
    DATABASE_PASSWORD:
      aws_secrets_manager:
        secret_id: app-secrets/database
        field: 'password'
      token: $CUSTOM_AWS_TOKEN
      file: false
  stage: deploy
  script:
    - echo "Connecting to production database with custom token..."
    - psql postgresql://$DB_USER:$DATABASE_PASSWORD@$DB_HOST:5432/$DB_NAME -c "SELECT version();"
    - echo "Database connection successful."
Enter fullscreen mode Exit fullscreen mode

Short form syntax

You can use a simplified syntax by specifying the secret ID as a string.
You can optionally specify a field by separating it with a # character.
For example:

variables:
  AWS_REGION: us-east-1

api-deployment:
  secrets:
    API_KEY:
      aws_secrets_manager: 'app-secrets/api#api_key'
      file: false
    FULL_SECRET:
      aws_secrets_manager: 'app-secrets/api'
      file: false
  stage: deploy
  script:
    - echo "Deploying API with specific field..."
    - curl --header "Authorization: Bearer $API_KEY" https://api.example.com/deploy
    - echo "Using full secret..."
    - curl --header "Authorization: Bearer $(cat $FULL_SECRET | jq --raw-output '.api_key')" https://api.example.com/status
Enter fullscreen mode Exit fullscreen mode

Secret versioning

AWS Secrets Manager supports multiple versions of secrets. You can specify a particular version
using either version_id or version_stage. For example:

variables:
  AWS_REGION: us-east-1

production-deployment:
  secrets:
    DATABASE_PASSWORD:
      aws_secrets_manager:
        secret_id: prod-app-secrets/database
        field: 'password'
        version_stage: 'AWSCURRENT'
      file: false
    STAGING_DATABASE_PASSWORD:
      aws_secrets_manager:
        secret_id: prod-app-secrets/database
        field: 'password'
        version_id: '01234567-89ab-cdef-0123-456789abcdef'
      file: false
  stage: deploy
  script:
    - echo "Deploying to production with current secret version..."
    - deploy-prod.sh --db-password $DATABASE_PASSWORD
    - echo "Testing with specific secret version..."
    - test-with-version.sh --db-password $STAGING_DATABASE_PASSWORD
Enter fullscreen mode Exit fullscreen mode

Cross-account secret access

To retrieve secrets from another AWS account, you must use the full ARN.
For example:

variables:
  AWS_REGION: us-east-1
  AWS_ROLE_ARN: 'arn:aws:iam::123456789012:role/cross-account-secrets-role'

cross-account-deployment:
  id_tokens:
    AWS_ID_TOKEN:
      aud: 'sts.amazonaws.com'
  secrets:
    SHARED_API_KEY:
      aws_secrets_manager:
        secret_id: 'arn:aws:secretsmanager:us-east-1:987654321098:secret:shared-api-keys-AbCdEf'
        field: 'production_key'
      file: false
  stage: deploy
  script:
    - echo "Accessing shared secret from another account..."
    - curl --header "Authorization: Bearer $SHARED_API_KEY" https://shared-api.example.com/deploy
Enter fullscreen mode Exit fullscreen mode

Per-secret configuration overrides

You can override global AWS settings on a per-secret basis. For example:

variables:
  AWS_REGION: us-east-1
  AWS_ROLE_ARN: 'arn:aws:iam::123456789012:role/default-role'

multi-region-deployment:
  id_tokens:
    AWS_ID_TOKEN:
      aud: 'sts.amazonaws.com'
    EU_AWS_TOKEN:
      aud: 'sts.amazonaws.com'
  secrets:
    EU_DATABASE_PASSWORD:
      aws_secrets_manager:
        secret_id: eu-app-secrets/database
        field: 'password'
        region: 'eu-west-1'
        role_arn: 'arn:aws:iam::123456789012:role/eu-deployment-role'
        role_session_name: 'gitlab-eu-deployment'
      token: $EU_AWS_TOKEN
      file: false
    US_DATABASE_PASSWORD:
      aws_secrets_manager:
        secret_id: us-app-secrets/database
        field: 'password'
      file: false
  stage: deploy
  script:
    - echo "Deploying to EU region..."
    - deploy-to-eu.sh --db-password $EU_DATABASE_PASSWORD
    - echo "Deploying to US region..."
    - deploy-to-us.sh --db-password $US_DATABASE_PASSWORD
Enter fullscreen mode Exit fullscreen mode

In these examples:

  • aud: The audience, which must match the audience used when creating the federated identity credentials.
  • secret_id: The name or ARN of the secret in AWS Secrets Manager. To retrieve a secret from another account, you must use an ARN.
  • field: Is the specific key in the JSON secret to retrieve. If not specified, the entire secret is retrieved. Field access is only supported for flat JSON secrets (top-level keys only) and supports string, number, and boolean values. For example:
    • password: Accesses the password field.
    • api_key: Accesses the api_key field.
    • token: Specifies which ID token to use for authentication. If not specified, the runner looks for a token named AWS_ID_TOKEN.
  • version_id: Is the unique identifier of a specific version of the secret. If you don't specify either version_id or version_stage, AWS Secrets Manager returns the AWSCURRENT version.
  • version_stage: The staging label of the version of the secret to retrieve (such as AWSCURRENT or AWSPENDING). You cannot specify both version_id and version_stage for the same secret.
  • region: Overrides the global AWS_REGION for this specific secret.
  • role_arn: Overrides the global AWS_ROLE_ARN for this specific secret.
  • role_session_name: Overrides the global AWS_ROLE_SESSION_NAME for this specific secret.
  • GitLab fetches the secret from AWS Secrets Manager and stores the value in a temporary file. The path to this file is stored in a CI/CD variable, similar to file type CI/CD variables.

Troubleshooting

Refer to OIDC for AWS troubleshooting for general
problems when setting up OIDC with AWS.

Error: no EC2 IMDS role found

The following error might happen if both of these conditions are true:

Resolving secrets
Resolving secret "MY_AWS_SECRET"...
Using "aws_secrets_manager" secret resolver...
ERROR: Job failed (system failure): resolving secrets: operation error Secrets Manager: GetSecretValue, get identity: get credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, canceled, context deadline exceeded
Enter fullscreen mode Exit fullscreen mode

The Resolving secrets step is handled by the runner manager. This step accesses IAM credentials
cached in EC2 IMDS.
If the IAM role has not been applied to the runner manager, the Resolving secrets step fails.

To address this error, apply the correct IAM role to the runner manager.

Applying the IAM role to the runner pods that are spawned and managed by the runner manager does not resolve this issue.

Ref: https://docs.gitlab.com/ci/secrets/aws_secrets_manager/

Top comments (0)