Debug School

Cover image for EFS Utils
Suyash Sambhare
Suyash Sambhare

Posted on

EFS Utils

#devops #aws #efs #utils

Utilities for Amazon Elastic File System

AWS EFS (Elastic File System) Utils is a mount helper for using Amazon EFS file systems used for various tasks related to managing and using Amazon EFS services. Here’s a breakdown of their primary functionalities:

Purpose of AWS EFS Utils

1. Mounting EFS File Systems

AWS EFS Utils provide easy-to-use commands for mounting EFS file systems to EC2 instances, allowing you to access files stored in EFS from your applications.

2. Performance Management

The tools include options to configure mount settings that can optimize performance based on your specific workload needs, such as choosing throughput modes.

3. Automatic DNS Resolution

EFS Utils help automatically resolve the correct DNS name for the EFS file system, simplifying the mounting process.

4. Health Monitoring

These utils include commands that can be used to check the health and status of your EFS file systems, ensuring they are functioning correctly.

5. Security Configuration

AWS EFS Utils help with configuring security options like AWS Identity and Access Management (IAM) policies and mounted instances' security groups, enhancing security during access.

6. Command-Line Interface

The utilities provide a command-line interface for performing various administrative tasks related to EFS, making it easier to integrate EFS with your scripts or applications.

7. Troubleshooting

They also include features for debugging and monitoring, allowing users to diagnose and troubleshoot issues that may arise during usage.

Summary

AWS EFS Utils streamline the process of using Amazon EFS by providing tools for mounting, managing, and optimizing EFS interactions, making it a vital component for applications that require scalable file storage. If you have specific scenarios in mind for using EFS, let me know!

The efs-utils package has been verified against the following Linux distributions:

Distribution Package Type init System
Amazon Linux 2 rpm systemd
Amazon Linux 2023 rpm systemd
RHEL 8 rpm systemd
RHEL 9 rpm systemd
Ubuntu 20.04 deb systemd
Ubuntu 22.04 deb systemd
Ubuntu 24.04 deb systemd
OpenSUSE Leap rpm systemd
SLES 15 rpm systemd

To install AWS EFS (Elastic File System) Utils on Ubuntu, follow these steps:

Note: Building from source requires Rust 1.70+, Cargo, Go 1.17.13+, CMake 3.0+, GCC/G++, and Perl.

Installing AWS EFS Utils on Ubuntu

Step 1: Update the Package Index

Open your terminal and update the package index:

sudo apt update
Enter fullscreen mode Exit fullscreen mode

Step 2: Install Required Packages

Install the necessary packages for building the EFS Utils:

sudo apt install -y git make gcc
Enter fullscreen mode Exit fullscreen mode

Step 3: Clone the EFS Utils Repository

Clone the AWS EFS Utils GitHub repository:

git clone https://github.com/aws/efs-utils
Enter fullscreen mode Exit fullscreen mode

Step 4: Change Directory

Navigate into the cloned directory:

cd efs-utils
Enter fullscreen mode Exit fullscreen mode

Step 5: Install EFS Utils

Run the make command to compile and install:

sudo make install
Enter fullscreen mode Exit fullscreen mode

Step 6: Verify Installation

To ensure that the EFS Utils are installed correctly, you can check the version:

sudo efs-utils --version
Enter fullscreen mode Exit fullscreen mode

If you see the version number, the installation was successful.

Detailed Installation Information

Building efs-utils from Source

This guide provides detailed instructions for building efs-utils from source on various Linux distributions.

Build Prerequisites

Building efs-utils v2.0+ requires the following dependencies:

  • rust 1.70+
  • cargo
  • go 1.17.13+
  • perl
  • cmake 3.0+
  • gcc and g++ (or gcc-c++)
  • make
  • git

Recommended Resource Size: minimum 2 vCPUs, 4GB RAM to ensure sufficient resources for compilation. In AWS EC2, use t3.medium or larger.

Installing Rust and Cargo

If your distribution doesn't provide a rust or cargo package, or it provides versions
that are older than 1.70, then you can install rust and cargo through rustup:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
. "$HOME/.cargo/env"
Enter fullscreen mode Exit fullscreen mode

Installing Go

Ensure you have Go 1.17.13 or later is installed and configured on your system.
Some distributions provide Go packages through package manager, but they may have outdated versions.

# DEB-based
sudo apt-get update  
sudo apt-get -y install golang

# Verify Go 1.17.13 or later is installed
go version
Enter fullscreen mode Exit fullscreen mode

GCC Version Requirements 

# Install GCC 13 (if not already installed)
# For Debian 13
sudo apt-get install -y gcc-13 g++-13

# Set GCC 13 as the compiler for the build
export CC=gcc-13
export CXX=g++-13
# Then proceed with the normal build steps
Enter fullscreen mode Exit fullscreen mode

Ubuntu 20.04, upgrade to use gcc-10 and g++-10

# Install GCC 10
sudo apt-get -y install gcc-10 g++-10

# Set GCC 10 as the compiler for the build
export CC=gcc-10
export CXX=g++-10
Enter fullscreen mode Exit fullscreen mode

Note: Alternatively, you can set the system default compiler using update-alternatives (requires sudo and affects all applications)

CMake version requirement

Building AWS-LC requires CMake 3.0 or later. CMake is typically available through the standard packager manager.

DEB-based Distributions

Debian/Ubuntu 

sudo apt-get update
sudo apt-get -y install git binutils rustc cargo libssl-dev pkg-config gettext make gcc g++ cmake wget perl # remove gcc g++ here if you already installed a compatible version following GCC Version Requirements instruction
git clone https://github.com/aws/efs-utils
cd efs-utils
./build-deb.sh
sudo apt-get -y install ./build/amazon-efs-utils*deb
Enter fullscreen mode Exit fullscreen mode

Running Tests

After building from source, you can run the test suite:

  1. Set up a virtualenv: 
virtualenv ~/.envs/efs-utils
source ~/.envs/efs-utils/bin/activate
pip install -r requirements.txt
Enter fullscreen mode Exit fullscreen mode
  1. Run tests: 
make test
Enter fullscreen mode Exit fullscreen mode

Verifying Installation

After installation, verify efs-utils is working:

mount.efs --version
/usr/sbin/mount.efs Version: 2.2.0

mount.efs --help
Usage: mount.efs [--version] [-h|--help] <fsname> <mountpoint> [-o <options>]
Enter fullscreen mode Exit fullscreen mode

Additional Notes

  • Make sure you have nfs-common installed, as it is required for mounting EFS file systems. You can install it with: 
sudo apt install nfs-common
Enter fullscreen mode Exit fullscreen mode

Efs Utils

Usage

mount.efs

efs-utils includes a mount helper utility, mount.efs, that simplifies and improves the performance of EFS file system mounts.

mount.efs launches a proxy process that forwards NFS traffic from the kernel's NFS client to EFS.
This proxy is responsible for TLS encryption, and for providing improved throughput performance.

To mount with the recommended default options, simply run:

sudo mount -t efs file-system-id efs-mount-point/
Enter fullscreen mode Exit fullscreen mode

To mount file system to a specific mount target of the file system, run:

sudo mount -t efs -o mounttargetip=mount-target-ip-address file-system-id efs-mount-point/
Enter fullscreen mode Exit fullscreen mode

To mount file system within a given network namespace, run:

sudo mount -t efs -o netns=netns-path file-system-id efs-mount-point/
Enter fullscreen mode Exit fullscreen mode

To mount file system to the mount target in a specific availability zone (e.g. us-east-1a), run:

sudo mount -t efs -o az=az-name file-system-id efs-mount-point/
Enter fullscreen mode Exit fullscreen mode

To mount file system to the mount target in a specific region (e.g. us-east-1), run:

sudo mount -t efs -o region=region-name file-system-id efs-mount-point/
Enter fullscreen mode Exit fullscreen mode

To mount the filesystem mount target in the same physical availability zone ID (e.g. use1-az1) as the client instance over cross-AWS-account mounts, run:

sudo mount -t efs -o crossaccount file-system-id efs-mount-point/
Enter fullscreen mode Exit fullscreen mode

To mount over TLS, simply add the tls option:

sudo mount -t efs -o tls file-system-id efs-mount-point/
Enter fullscreen mode Exit fullscreen mode

To authenticate with EFS using the system’s IAM identity, add the iam option. This option requires the tls option.

sudo mount -t efs -o tls,iam file-system-id efs-mount-point/
Enter fullscreen mode Exit fullscreen mode

To mount using an access point, use the accesspoint= option. This option requires the tls option.
The access point must be in the "available" state before it can be used to mount EFS.

sudo mount -t efs -o tls,accesspoint=access-point-id file-system-id efs-mount-point/
Enter fullscreen mode Exit fullscreen mode

To mount your file system automatically with any of the options above, you can add entries to /efs/fstab like:

file-system-id efs-mount-point efs _netdev,tls,iam,accesspoint=access-point-id 0 0
Enter fullscreen mode Exit fullscreen mode

For more information on mounting with the mount helper, see the manual page:

man mount.efs
Enter fullscreen mode Exit fullscreen mode

mount.efs launches a proxy process that forwards NFS traffic from the kernel's NFS client to EFS. This proxy
is responsible for TLS encryption, and for providing improved throughput performance.

fs-id-or-dns-name has to be of one of the following two forms:

  • An EFS filesystem ID in the form of "fs-abcd1234", generated when the file system is created.

  • A domain name that has a resolvable DNS-CNAME record, which in turn points to a fully-qualified EFS DNS name in the form of "fs-abcd1234.efs.us-east-1.amazonaws.com" or "us-east-1a.fs-abcd1234.efs.us-east-1.amazonaws.com".
    mount-point is the local directory on which the file system will be mounted.
    mount.efs automatically applies the following NFS options:

  • nfsvers=4.1

  • rsize=1048576

  • wsize=1048576

  • hard

  • timeo=600

  • retrans=2

  • noresvport

  • tls (for Mac distributions)

By default, when using the Amazon EFS mount helper with Transport Layer Security (TLS), the mount helper enforces the certificate hostname checking and disables the use of Online Certificate Status Protocol (OCSP). These options can be configured in the config file located at /etc/amazon/efs/efs-utils.conf.

Additionally, the Amazon EFS mount helper has built-in logging for troubleshooting purposes. These logs are located at /var/log/amazon/efs.

It is possible to configure your Amazon EC2 instance to automatically remount your Amazon EFS file system when it reboots.

Options

-o, Options are specified with a -o flag followed by a comma separated string of options. All of the options specified in nfs are available, in addition to the following EFS-specific options:

  • tls: Mounts the EFS file system over TLS. For EC2 instances using Mac distributions, this option is by default passed and the EFS file system is mounted over TLS.
  • notls: Mounts the EFS file system without TLS, applies for Mac distributions only.
  • region: Mounts the EFS file system from the specified region, overriding any config file value.
  • tlsport=n: Configures the proxy process to listen for connections from the NFS client on the specified port. This is applicable to both non-tls and tls mounts. By default, the tlsport is chosen randomly from port range defined in the config file located at /etc/amazon/efs/efs-utils.conf.
  • verify=n: Verify TLS certificates using the specified stunnel verify level.
  • ocsp / noocsp: Selects whether to perform OCSP validation on TLS certificates, overriding /etc/amazon/efs/efs-utils.conf. By default OCSP is disabled. The ocsp mount option is incompatible with the efs-proxy process, and will revert efs-utils to the legacy stunnel mode, which does not support improved per-client throughput performance.
  • iam: Use the system's IAM identity to authenticate with EFS. The mount helper will try to retrieve the required IAM credentials from the following locations: the aws credentials URI passed by mount option, the AWS CLI credentials file (~/.aws/credentials), and the AWS CLI config file (~/.aws/config), the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variable, the AssumeRoleWithWebIdentity, the EC2 instance profile. The first location that has credentials will be used. This option requires the tls option.
  • rolearn: Role ARN for IAM authentication with AssumeRoleWithWebIdentity API.
  • jwtpath: Identity token for IAM authentication with AssumeRoleWithWebIdentity API.
  • accesspoint: Mount the EFS file system using the specified access point. This option requires the tls option. The access point must be in the "available" state before it can be used to mount EFS.
  • awsprofile: Use the named profile used to lookup IAM credentials in the AWS CLI credentials file (~/.aws/credentials) or AWS CLI config file (~/.aws/config). If botocore is installed, assume the named profile and use the credentials of the assumed profile. If "awsprofile" is not specified, the "default" profile is used.
  • awscredsuri: Use the relative uri to lookup IAM credentials from ecs task metadata endpoint.
  • cafile: Use the cafile as the stunnel certificate authority file.
  • netns: Mount the EFS file system to the specified network namespace.
  • az: Mount the EFS file system to the specified availability zone mount target.
  • mountport: Use the port 2049 to bypass portmapper daemon on EC2 Mac instances running macOS Big Sur.
  • mounttargetip: Mount the EFS file system to the specified mount target ip address.
  • stunnel: Forward NFS traffic from the local NFS client to EFS using stunnel instead of efs-proxy. This will enable compatibility with the ocsp mount option, but will not deliver the increased throughput performance provided by efs-proxy. This option is enabled by default for Mac clients.

Ref: https://github.com/aws/efs-utils

Top comments (0)