Utilities for Amazon Elastic File System (EFS)
The efs-utils
package has been verified against the following Linux distributions:
Distribution | Package Type | init System |
---|---|---|
Amazon Linux 2017.09 | rpm |
upstart |
Amazon Linux 2 | rpm |
systemd |
CentOS 7 | rpm |
systemd |
CentOS 8 | rpm |
systemd |
RHEL 7 | rpm |
systemd |
RHEL 8 | rpm |
systemd |
Fedora 28 | rpm |
systemd |
Fedora 29 | rpm |
systemd |
Fedora 30 | rpm |
systemd |
Fedora 31 | rpm |
systemd |
Fedora 32 | rpm |
systemd |
Debian 9 | deb |
systemd |
Debian 10 | deb |
systemd |
Ubuntu 16.04 | deb |
systemd |
Ubuntu 18.04 | deb |
systemd |
Ubuntu 20.04 | deb |
systemd |
OpenSUSE Leap | rpm |
systemd |
OpenSUSE Tumbleweed | rpm |
systemd |
SLES 12 | rpm |
systemd |
SLES 15 | rpm |
systemd |
The efs-utils
package has been verified against the following MacOS distributions:
Distribution | init System |
---|---|
MacOS Big Sur | launchd |
nfs-utils
(RHEL/CentOS/Amazon Linux/Fedora) ornfs-common
(Debian/Ubuntu)- OpenSSL 1.0.2+
- Python 3.4+
stunnel
4.56+
botocore
1.12.0+
For those using Amazon Linux or Amazon Linux 2, the easiest way to install efs-utils
is from Amazon's repositories:
$ sudo yum -y install amazon-efs-utils
You can now use AWS Systems Manage Distributor to automatically install or update amazon-efs-utils
.
Please refer to Using AWS Systems Manager to automatically install or update Amazon EFS clients for more guidance.
The following are prerequisites for using AWS Systems Manager Distributor to install or update amazon-efs-utils
:
-
AWS Systems Manager agent is installed on the distribution (For
Amazon Linux
andUbuntu
, AWS Systems Manager agent is pre-installed, for other distributions, please refer to install AWS Systems Manager agent on Linux EC2 instance for more guidance.) -
Instance is attached with IAM role with AWS managed policy
AmazonElasticFileSystemsUtils
, this policy will enable your instance to be managed by AWS Systems Manager agent, also it contains permissions to support specific features.
Other distributions require building the package from source and installing it.
- To build and install an RPM:
If the distribution is not OpenSUSE or SLES
$ sudo yum -y install git rpm-build make
$ git clone https://github.com/aws/efs-utils
$ cd efs-utils
$ make rpm
$ sudo yum -y install build/amazon-efs-utils*rpm
Otherwise
$ sudo zypper refresh
$ sudo zypper install -y git rpm-build make
$ git clone https://github.com/aws/efs-utils
$ cd efs-utils
$ make rpm
$ sudo zypper --no-gpg-checks install -y build/amazon-efs-utils*rpm
On OpenSUSE, if you see error like File './suse/noarch/bash-completion-2.11-2.1.noarch.rpm' not found on medium 'https://download.opensuse.org/tumbleweed/repo/oss/'
during installation of git
, run the following commands to re-add repo OSS and NON-OSS, then run the install script above again.
sudo zypper ar -f -n OSS https://download.opensuse.org/tumbleweed/repo/oss/ OSS
sudo zypper ar -f -n NON-OSS https://download.opensuse.org/tumbleweed/repo/non-oss/ NON-OSS
sudo zypper refresh
- To build and install a Debian package:
$ sudo apt-get update
$ sudo apt-get -y install git binutils
$ git clone https://github.com/aws/efs-utils
$ cd efs-utils
$ ./build-deb.sh
$ sudo apt-get -y install ./build/amazon-efs-utils*deb
For EC2 Mac instances running macOS Big Sur, you can install amazon-efs-utils from the homebrew-aws respository.
brew install amazon-efs-utils
This will install amazon-efs-utils on your EC2 Mac Instance running macOS Big Sur in the directory /usr/local/Cellar/amazon-efs-utils
. At the end of the installation, it will print a set of commands that must be executed in order to start using efs-utils. The instructions that are printed after amazon-efs-utils and must be executed are:
Perform below actions to start using efs:
sudo mkdir -p /Library/Filesystems/efs.fs/Contents/Resources
sudo ln -s /usr/local/bin/mount.efs /Library/Filesystems/efs.fs/Contents/Resources/mount_efs
Perform below actions to stop using efs:
sudo rm /Library/Filesystems/efs.fs/Contents/Resources/mount_efs
To enable watchdog for using TLS mounts:
sudo cp /usr/local/Cellar/amazon-efs-utils/<version>/libexec/amazon-efs-mount-watchdog.plist /Library/LaunchAgents
sudo launchctl load /Library/LaunchAgents/amazon-efs-mount-watchdog.plist
To disable watchdog for using TLS mounts:
sudo launchctl unload /Library/LaunchAgents/amazon-efs-mount-watchdog.plist
- Set up a virtualenv for efs-utils
$ virtualenv ~/.envs/efs-utils
$ source ~/.envs/efs-utils/bin/activate
$ pip install -r requirements.txt
- Run tests
$ make test
efs-utils
includes a mount helper utility to simplify mounting and using EFS file systems.
To mount with the recommended default options, simply run:
$ sudo mount -t efs file-system-id efs-mount-point/
To mount file system to a specific mount target of the file system, run:
$ sudo mount -t efs -o mounttargetip=mount-target-ip-address file-system-id efs-mount-point/
To mount file system within a given network namespace, run:
$ sudo mount -t efs -o netns=netns-path file-system-id efs-mount-point/
To mount file system to the mount target in specific availability zone (e.g. us-east-1a), run:
$ sudo mount -t efs -o az=az-name file-system-id efs-mount-point/
To mount over TLS, simply add the tls
option:
$ sudo mount -t efs -o tls file-system-id efs-mount-point/
To authenticate with EFS using the system’s IAM identity, add the iam
option. This option requires the tls
option.
$ sudo mount -t efs -o tls,iam file-system-id efs-mount-point/
To mount using an access point, use the accesspoint=
option. This option requires the tls
option.
The access point must be in the "available" state before it can be used to mount EFS.
$ sudo mount -t efs -o tls,accesspoint=access-point-id file-system-id efs-mount-point/
To mount your file system automatically with any of the options above, you can add entries to /efs/fstab
like:
file-system-id efs-mount-point efs _netdev,tls,iam,accesspoint=access-point-id 0 0
For more information on mounting with the mount helper, see the manual page:
man mount.efs
or refer to the documentation.
For EC2 instances using Mac distribution, the recommended default options will perform a tls mount:
$ sudo mount -t efs file-system-id efs-mount-point/
or
$ sudo mount -t efs -o tls file-system-id efs-mount-point/
To mount without TLS, simply add the notls
option:
$ sudo mount -t efs -o notls file-system-id efs-mount-point/
efs-utils
contains a watchdog process to monitor the health of TLS mounts. This process is managed by either upstart
or systemd
depending on your Linux distribution and launchd
on Mac distribution, and is started automatically the first time an EFS file system is mounted over TLS.
By default, when using the EFS mount helper with TLS, it enforces certificate hostname checking. The EFS mount helper uses the stunnel
program for its TLS functionality. Please note that some versions of Linux do not include a version of stunnel
that supports TLS features by default. When using such a Linux version, mounting an EFS file system using TLS will fail.
Once you’ve installed the amazon-efs-utils
package, to upgrade your system’s version of stunnel
, see Upgrading Stunnel.
Run the following commands and follow the output hint of zypper package manager to upgrade the stunnel on your SLES12 instance
sudo zypper addrepo https://download.opensuse.org/repositories/security:Stunnel/SLE_12_SP5/security:Stunnel.repo
sudo zypper refresh
sudo zypper install -y stunnel
The installation installs latest stunnel available in brew repository. You can also upgrade the version of stunnel on your instance using the command below:
brew upgrade stunnel
efs-utils
uses botocore to interact with other AWS services. Please note the package type from the above table and install
botocore based on that info. If botocore is already installed and does not meet the minimum required version,
you can upgrade the botocore by following the upgrade botocore section.
- Download the
get-pip.py
script
sudo yum -y install wget
if [[ "$(python3 -V 2>&1)" =~ ^(Python 3.5.*) ]]; then
sudo wget https://bootstrap.pypa.io/3.5/get-pip.py -O /tmp/get-pip.py
elif [[ "$(python3 -V 2>&1)" =~ ^(Python 3.4.*) ]]; then
sudo wget https://bootstrap.pypa.io/3.4/get-pip.py -O /tmp/get-pip.py
else
sudo wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py
fi
sudo apt-get update
sudo apt-get -y install wget
if echo $(python3 -V 2>&1) | grep -e "Python 3.5"; then
sudo wget https://bootstrap.pypa.io/3.5/get-pip.py -O /tmp/get-pip.py
elif echo $(python3 -V 2>&1) | grep -e "Python 3.4"; then
sudo wget https://bootstrap.pypa.io/3.4/get-pip.py -O /tmp/get-pip.py
else
sudo apt-get -y install python3-distutils
sudo wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py
fi
- To install botocore on RPM
sudo python3 /tmp/get-pip.py
sudo pip3 install botocore || sudo /usr/local/bin/pip3 install botocore
- To install botocore on DEB
sudo python3 /tmp/get-pip.py
sudo pip3 install botocore || sudo /usr/local/bin/pip3 install botocore
sudo python3 /tmp/get-pip.py
sudo pip3 install --target /usr/lib/python3/dist-packages botocore || sudo /usr/local/bin/pip3 install --target /usr/lib/python3/dist-packages botocore
sudo pip3 install botocore
Pass --upgrade
to the corresponding installation scripts above based on system platform and distribution
sudo pip3 install botocore --upgrade
efs-utils
now support publishing mount success/failure logs to CloudWatch log. By default, this feature is disabled. There are three
steps you must follow to enable and use this feature:
Follow install botocore section
sudo sed -i -e '/\[cloudwatch-log\]/{N;s/# enabled = true/enabled = true/}' /etc/amazon/efs/efs-utils.conf
- For MacOS:
sudo sed -i -e '/\[cloudwatch-log\]/{N;s/# enabled = true/enabled = true/;}' /usr/local/Cellar/amazon-efs-utils/<version>/etc/amazon/efs/efs-utils.conf
You can also configure CloudWatch log group name and log retention days in the config file.
Attach AWS managed policy AmazonElasticFileSystemsUtils
to the iam role you attached to the instance, or the aws credentials
configured on your instance.
After completing the three prerequisite steps, you will be able to see mount status notifications in CloudWatch Logs.
A change in the Linux kernel 5.4+ results a throughput regression on NFS client. With patch, starting from 5.4.*, Kernels containing this patch now set the default read_ahead_kb size to 128 KB instead of the previous 15 MB. This read_ahead_kb is used by the Linux kernel to optimize performance on NFS read requests by defining the maximum amount of data an NFS client can pre-fetch in a read call. With the reduced value, an NFS client has to make more read calls to the file system, resulting in reduced performance.
To avoid above throughput regression, efs-utils will modify read_ahead_kb to 15 * rsize (could be configured via mount option, 1MB by default) after mount success on Linux 5.4+. (not support on MacOS)
This optimization will be enabled by default. To disable this optimization:
sed -i "s/optimize_readahead = false/optimize_readahead = true/" /etc/amazon/efs/efs-utils.conf
To re-enable this optimization
sed -i "s/optimize_readahead = true/optimize_readahead = false/" /etc/amazon/efs/efs-utils.conf
You can mount file system with a given rsize, run:
$ sudo mount -t efs -o rsize=rsize-value-in-bytes file-system-id efs-mount-point/
You can also manually chose a value of read_ahead_kb to optimize read throughput on Linux 5.4+ after mount.
$ sudo bash -c "echo read-ahead-value-in-kb > /sys/class/bdi/0:$(stat -c '%d' efs-mount-point)/read_ahead_kb"
efs-utils
now supports using botocore to retrieve mount target ip address when dns name cannot be resolved, e.g.
when user is mounting a file system in another VPC. There are two prerequisites to use this feature:
Follow install botocore section
Allow the elasticfilesystem:DescribeMountTargets
and ec2:DescribeAvailabilityZones
action in your policy attached to
the iam role you attached to the instance, or the aws credentials configured on your instance. We recommend you attach
AWS managed policy AmazonElasticFileSystemsUtils
.
This feature will be enabled by default. To disable this feature:
sed -i "s/fall_back_to_mount_target_ip_address_enabled = true/fall_back_to_mount_target_ip_address_enabled = false/" /etc/amazon/efs/efs-utils.conf
If you decide that you do not want to use this feature, but need to mount a cross-VPC file system, you can use the mounttargetip option to do so, using the desired mount target ip address in the mount command.
efs-utils
by default uses IMDSv2, which is a session-oriented method used to access instance metadata. If you don't want to use
IMDSv2, you can disable the token fetching feature by running the following command:
sed -i "s/disable_fetch_ec2_metadata_token = false/disable_fetch_ec2_metadata_token = true/" /etc/amazon/efs/efs-utils.conf
To authenticate with EFS using the system’s IAM identity of an awsprofile, add the iam
option and pass the profile name to
awsprofile
option. These options require the tls
option.
$ sudo mount -t efs -o tls,iam,aws-profile=test-profile file-system-id efs-mount-point/
To configure the named profile, see the Named Profiles doc
and Support Config File Settings doc
for more details. If the credentials (e.g. aws_access_key_id) are not configured in /root/.aws/credentials
or /root/.aws/config
(note that the path prefix may vary based on the root path of sudo), efs-utils will use botocore to assume the named profile.
This will require botocore is pre-installed, please follow install botocore section to install botocore first.
Normally you will need to configure your profile IAM policy to make the assume works. For example, if you want to perform a cross-account mounting, suppose you have established vpc-peering-connections between your vpcs, next step you need to do is giving permission to account B so that it can assume a role in account A and then mount the file system that belongs to account A. You can see IAM doc for more details.
After the IAM identity is setup, you can configure your awsprofile credentials or config. You can refer to
sdk settings. For example you can define
the profile to use the credentials of profile default
to assume role in account A by defining the source_profile
.
# /root/.aws/credentials
[default]
aws_access_key_id = AKIAIOSFODNN7EXAMPLE
aws_secret_access_key_id =wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
# /root/.aws/config
[default]
...
[profile test-profile]
role_arn = <role-arn-in-account-A>
source_profile = default
Or you can use the credentials from IAM role attached to instance to assume the named profile, e.g.
# /root/.aws/config
[profile test-profile]
role_arn = <role-arn-in-account-A>
credential_source = Ec2InstanceMetadata
This code is made available under the MIT license.