How to Mount S3 Bucket on Linux (AWS EC2 Instance) Use Case:

Use Case:

AWS S3 is awesome resource for cloud object storage and the consumption of S3 is varies from customer, very common use cases were:

• Backup and Storage – Provide data backup and storage services for others.
  
• Application Hosting – Provide services that deploy, install, and manage web applications.
  
• Media Hosting – Build a redundant, scalable, and highly available infrastructure that hosts video, photo, or music uploads and downloads.
  
• Software Delivery – Host your software applications that customers can download. 

Here I will explain how we implemented S3 in one of our customer.

Customer Requirement:

Customer have an application index logs in AWS EC2 application cluster which have 4 instance as part of the cluster, from each application server logs need to stored centrally and need to accessible (Read-Only) frequently from all server.

We can consider NFS sort of solution, even now we have EFS from Amazon but it costly and even the same data were using for their analytics solution. So we thought to use S3 to satisfy both the requirement.

Solution:

We have mount S3 on all required Linux EC2 instance using S3fs, so that all required instances have access to logs at the same time their analytic solution also can read data using s3api.

Filesystem in Userspace (FUSE) is a simple interface for userspace programs to export a virtual file-system to the Linux kernel. It also aims to provide a secure method for non privileged users to create and mount their own file-system implementations.

S3fs-fuse project is written in python backed by Amazons Simple Storage Service. Amazon offers an open API to build applications on top of this service, which several companies have done, using a variety of interfaces (web, rsync, fuse, etc).

Below are the pre-requisites to install and setup S3fs:

• EC2 instance with root access or sudo access to install S3fs and mount volume
• IAM user which have S3 Full access (For Upload/Download)
• Download latest S3fs package from http://code.google.com/p/s3fs/downloads/list 
• update your system to latest using

         yum update all (for CentOS)

         apt-get update (for Ubuntu)

• Install below dependencies before installing S3fs package

For CentOS 

yum install gcc libstdc++-devel gcc-c++ fuse fuse-devel curl-devel libxml2-devel openssl-devel mailcap 

For Ubuntu

apt-get install build-essential gcc libfuse-dev libcurl4-openssl-dev libxml2-dev mime-support pkg-config libxml++2.6-dev libssl-dev

Follow the below steps to mount your S3 bucket to your Linux Instance:

 Step 1: Download latest s3fs package and extract:

wget https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/s3fs/s3fs-1.74.tar.gz

tar -xvzf s3fs-1.74.tar.gz

Step 2: Update OS and install dependencies as mentioned in above pre-req.

Step 3:  Now change to extracted directory, compile and install s3fs source code.

cd s3fs-1.74

./configure --prefix=/usr

make

make install

Step 4: Use below command to check where s3fs command is placed in O.S. It will also confirm whether installation is ok:

which s3fs

Step 5: Get IAM user Access and secret key which have appropriate permissions (e.g. S3 Full access), You can get the same from AWS IAM console

Step 6: Create a new file in /etc with the name passwd-s3fs and Paste the access key and secret key in the below format and change the permission for the file:

echo "AccessKey:SecretKey" > /etc/passwd-s3fs

chmod 640 /etc/passwd-s3fs

Note: Replace AcessKey and SecretKey with original keys.

Step 7: Now create a directory and mount S3bucket in it. Here, Provide your S3 bucket name in place of “your_bucketname”

mkdir /sravancloudarch

s3fs your_bucketname -o use_cache=/tmp -o allow_other -o multireq_max=5 /mys3bucket

Replace your_bucket = S3 bucket name which you want to mount
Replace /sravancloudarchS3 = Directory name which you want to mount

/usr/bin/s3fs sravancloudarch -o use_cache=/tmp -o allow_other -o multireq_max=5 /sravancloudarch

You can validate whether it is mounted using below command:

[root@ip-172-31-49-68 ~]# df -h

Filesystem      Size  Used Avail Use% Mounted on

devtmpfs        488M   56K  488M   1% /dev

tmpfs           497M     0  497M   0% /dev/shm

/dev/xvda1      7.8G  1.1G  6.6G  15% /

s3fs            256T     0  256T   0% /mys3bucket

s3fs            256T     0  256T   0% /sravancloudarch

[root@ip-172-31-49-68 ~]# 

Note: At any given point you can unmount this volume using below command:

umount /sravancloudarch

Now this volume is Non-Persistent i.e; once you reboot your system this mount point wont exists, to make it persistent and automatically mount for every reboot we need to add below entries to /etc/rc.local

nano /etc/rc.local

Add below Line and save the file:

/usr/bin/s3fs sravancloudarch -o use_cache=/tmp -o allow_other -o multireq_max=5 /sravancloudarch

Now you should be able to read and write filed to S3 (Considering you have S3 full access).

touch /sravancloudarchS3/sravan

[root@ip-172-31-49-68 ~]# ls -lrt /sravancloudarch/

total 1

-rw-r--r-- 1 root root 0 Jun  7 10:11 sravan

[root@ip-172-31-49-68 ~]# 

Now you have successfully mounted your S3 bucket as a Volume in your EC2 instance, any files which you write in /sravancloudarch directory will be replicated to your Amazon S3 bucket.

Hopefully you enjoyed this post, Here is the relevant hands on video