How to mount S3 bucket in Linux server

---

Learn to mount S3 bucket in RHEL, ubuntu, CentOS Linux server. Understand how to debug issues while mounting S3 bucket.

Mount S3 bucket in Linux

---

In this article we will walk through process by which you can mount S3 bucket on Linux server. S3 bucket is storage container of S3 (simple storage service) AWS service. As all traditional datacenters are moving to cloud computing, it is necessary to know how to interconnect cloud and traditional services. Lets dive into process to mount S3 bucket in RHEL, ubuntu, CentOS Linux. Complete process can be done in below 3 steps:

1. Install fuse and s3fs packages
2. Configure access keys of your AWS account
3. Mount S3 bucket

For this tutorial, we are assuming you have S3 bucket ready in your AWS account with proper permission setup. If not, follow this tutorial to create S3 bucket in AWS.

Install fuse and s3fs packages

These packages can be found here : fuse & s3fs. You have to download them on Linux server  with tools like wget and compile them. Make sure you have their dependencies installed before you try to compile them.

Dependencies are :

for RedHat based : automake gcc gcc-c++ git libcurl-devel libxml2-devel make openssl-devel mailcap curl-devel libstdc++-devel

for Debian based : automake autotools-dev g++ git libcurl4-gnutls-dev libfuse-dev libssl-dev libxml2-dev make pkg-config

Install all these packages and follow below steps to configure fuse and s3fs. Make sure there is no package named fuse of s3fs exist in server before you proceed. This is to avoid conflicts in installation.

Fuse installation –

Use below commands. Use latest Fuse download link used in wget command. You can obtain it from Github here.

Shell

1 2 3 4 5 6 7 8 9 10 11

# cd /usr/src/ # wget https://github.com/libfuse/libfuse/releases/download/fuse-3.1.0/fuse-3.1.0.tar.gz # tar -zxf fuse-3.1.0.tar.gz # cd fuse-3.1.0 # ./configure -prefix=/usr/local # make && make install # export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig # ldconfig # modprobe fuse

Fuse installation output is below for your reference. Click and expand if you want to view it.

Click To Expand Code

s3fs installation-

Use below commands. We are cloning git here so no need to check latest release from GitHub website.

Shell

1 2 3 4 5 6 7 8

# git clone https://github.com/s3fs-fuse/s3fs-fuse.git # cd s3fs-fuse # ./autogen.sh # ./configure # make # make install

s3fs installation output is below for your reference. Click and expand if you want to view it.

Click To Expand Code

---

Configure access keys of your AWS account

Now you need to configure your AWS account security keys in s3fs utility. For that you need to visit your AWS account’s IAM (Identity and Access Management) page and get those keys under ‘Access Keys (Access Key ID and Secret Access Key)‘. If you havnt created any till time, then you can create new key pair and use it.

There are two keys : Access Key ID and Secret Access Key. Add those keys in file /etc/s3fs-keys separated by colon. You can keep them in any file name and path of your choice. I prefer to keep it in /etc where other OS imp files resides.

Shell

1 2 3 4 5

# cat /etc/s3fs-keys AKIAJORAC6V66ML22F6Q:gsR2eul5mpvRYUQu4r15YhuxWKdKZQVeBbGwoOUw # chmod 600 /etc/s3fs-keys

Remove others permission to read this key file. If you dont set permission 0 to others then s3fs command utility will warn you about it like below . For extra security you can make this file hidden by adding . in filename.

Shell

1 2 3

s3fs: credentials file /etc/s3fs-keys should not have others permissions.

Now you are ready to mount your bucket.

---

Mount S3 bucket

Now run s3fs utility with bucket name you want to mount followed by directory on which you want to mount it. -o switch to be used to specify key file path. If you used another filename and path than /etc/s3fs-keys then use it accordingly. There are many other options which can be supplied with this command to control cache and permission on Linux server (OS side) which I havnt mentioned here.

Shell

1 2 3

# s3fs kerneltalks /my_s3_bucket/ -o passwd_file=/etc/passwd-s3fs

Here –

• kerneltalks.bucket is my bucket name in S3
• /my_s3_bucket is directory on server on which I mounted S3 bucket
• passwd_file is path where I kept my AWS account keys

Your bucket is mounted on your server! You can check it in df output. You will see filesystem as s3fs.

Shell

1 2 3 4 5 6 7 8 9 10 11 12 13 14

# df -h Filesystem      Size  Used Avail Use% Mounted on udev            488M     0  488M   0% /dev tmpfs           100M  3.1M   97M   4% /run /dev/xvda1      7.7G  1.3G  6.5G  17% / s3fs            256T     0  256T   0% /my_s3_bucket   # ll /my_s3_bucket total 5 drwxrwxrwx 1 root root 0 Jan 1 1970 ./ drwxr-xr-x 25 root root 4096 Jul 11 06:19 ../ d--------- 1 root root 0 Jul 11 06:33 test/

You can perform all file and directory operations like you do on normal mount point. Observe size it showing – 256T ! Thats huge, thats S3 – almost unlimited storage!

You can add entry in /etc/fstab to mount your bucket on boot too. Use below entry :

Shell

1 2 3

s3fs#bucket_name /mountpoint fuse _netdev,allow_other 0 0

---

Bonus tip :

If your bucket name includes . then s3fs may fail to mount your bucket. Curl will exist the mount operation as SSL on Amazon wont be matched as certificate wild card matching fails due to extra . in your bucket name. To understand, if I have bucket name kerneltalks.bucket then SSL wildcard mismatch will happen as indicated by below error –

SSL: certificate subject name (*.s3.amazonaws.com) does not match target host name ‘kerneltalks.bucket.s3.amazonaws.com’

To see errors, warning during mount operation you can run s3fs command with debug on. Append below switches to the end of your s3fs mount command:

Shell

1 2 3

-d -d -f -o f2 -o curldbg

This will help you troubleshooting any issues you faced during S3 bucket mount.

This is how we mount S3 bucket in Linux servers like Redhat, ubuntu, Debian etc. Let us know if you have any feedback/comments/suggestions in below comment box.