Use Case : Design openstack networking to meet inhouse email marketing server

Note: This post will cover info more about openstack networking configuration assuming customer already have an email marketing server.

Background:

At present email is playing a vital role in this communication world, so when we need to send emails we may not considering to have our own email setup rather we use public free email providers like Gmail/Hotmail/Yahoo..etc.; but when we need to send bulk emails for any of our promotion or marketing activity even then can we use this free email providers ? It is completely depends up on what is the number we are referring the term "bulk" !

For some people they may have requirement to send emails 10's , 100's , for others may be 1000's it all depends on how big is our lead list/source, So for all our needs we may not be able to use free email providers, here is the comprehensive list of email providers and their limitations.

For our bulk email requirements we may choose any marketing agency our we may out SMTP service providers , rather if we want to build our own email server for marketing purpose, we may feel what is difference between general email server and marketing email server, In general regular email setup is easy by using Microsoft exchange or even we can opensource sendmail to setup but for marketing email server need little more specific config, because while you are sending 1000's of email at same time from same public IP then there is lot of chance to become your public IP black listed then most of the email scanners will mark your email as spam. and we know how many people read spam emails 👶

               

Now we know why most of our inboxes have lot of spam ✋

Solution Overview

For one of my Techclyde (Where I am working) customer who is into Digital marketing services, where their existing digital marketing Applications and email servers were in their our private datacenter powered by VMware, to align with their cloud roadmap and other factors (open secret "cost") Customer decided to migrate to private cloud powered by Openstack.

Techclyde had involved in in its own delivery model to provide cutting edge solution and services to the customer.

After successful migration then customer had a new requirement for their new business line which is email marketing, then customer requested us to evaluate possibilities to set up email marketing server with all the best practices, we were able to cover all the required best practices apart from the requirement to have multiple public IP address for single instance.

This is where we have given different solutions for our customer:

As we know in openstack we cannot directly assign multiple floating IP's to single interface.

Reason :Floating ip addresses are implemented as NAT rules on the neutron router, and it doesn't make sense to have multiple NAT rules with the same fixed ip address but different floating addresses.

So we have to add another fixed ip to our instance to assign other floating IP, even here to have isolated environment from existing applications we have created separate project in openstack and created multiple private networks in it and associated each one of the private IP to an instance, so that our instance will have different private networks associated with different floating IP's. By this now our customer is rocking on their new business line with effective cost along with more security and scalability.

Technical Details:

Who are all interested on technical details about How? please find below:

Step 1: Launch your instance with one private IP and one Floating IP using below commands

nova boot --image centos-6 --flavor m1.small --nic net-id=3ff9b098-e732-1908-f98t-dgh8f16123456 --key-name lars emailserver

Create Floating IP using below command

neutron floatingip-create public 

(above public is referring your network name you can get the same by using neutron net-list)

Add floating ip to an instance

neutron floatingip-create public

nova add-floating-ip emailserver 124.123.65.4

Step 2:  Now Create one more floating IP but you cannot directly assign to same interface as mentioned reason above.

Add another fixed IP 
nova add-fixed-ip emailserver 3ff9b903-e921-4752-a26f-cba8f1433992

Unfortunately, here our instance isn't going to know about this new address; DHCP will only pick up a single address , and with two fixed addresses, you don't know which address you're going to get by default. Because that will cause all kinds of grief, you're going to have to configure networking statically on your host.

So how we add here depends on which OS distribution we are using as instance:

For Ubuntu:

$ cd /etc/network/interfaces.d

$ cp eth0.cfg eth1.cfg

Now in the editor of your choice, change any references of eth0 to eth1 in the new eth1.cfg file.  There should only be a couple. Then bring up eth1:

$ sudo ifup eth1

Now the routing can be done via IP rules.

# 10.10.2.1 = gateway for the secondary interface

# 10.10.2.11 = ip address for the secondary interface eth1

$ ip route add default via 10.10.2.1 dev eth1 table 10 

$ ip rule add from 10.10.2.11 table 10

Step 3: Now we can associate the second floating ip to second interface

nova add-floating-ip --fixed-address 10.0.0.5 emailserver 103.46.74.85

Now we should be able to ping and access the instance from both Floating IP's

Hope this post helped.

Unable to delete openstack instance stuck at Error state

When you attempt to delete instance from either openstack CLI or openstack dashboard and if it get stuck at error state as follows:

GUI:

CLI:

`nova list` will show:

  | 1541a197-9f80-4ee5-a7d6-08e591aa8lokin | sravan0 | ERROR | deleting | NOSTATE  

There may be so many reasons for this error state like attached volume may not able to reclaim, or unable to reclaim associated port by neutron and so on.

Sometimes we can try from CLI after resetting state "nova reset-state <instance>" using "nova delete $instanceid"

When above workaround is not working along with any other options, at last if you want to mark instance as deleted in nova database then we can follow below approach:

Note: Please use this when you hit the wall other wise not recommended:

Step 1: Get instance details using "nova show "$instanceid""

[root@controller ~]# nova show 92b9c565-d908-4dd0-a169-f58b3ce71428

Property	Value
OS-DCF:diskConfig	AUTO
OS-EXT-AZ:availability_zone	nova
OS-EXT-SRV-ATTR:host	kvm-02
OS-EXT-SRV-ATTR:hypervisor_hostname	kvm-02
OS-EXT-SRV-ATTR:instance_name	instance-0000005b
OS-EXT-STS:power_state	1
OS-EXT-STS:task_state	-
OS-EXT-STS:vm_state	active
OS-SRV-USG:launched_at	2016-05-16T09:35:14.000000
OS-SRV-USG:terminated_at	-
accessIPv4
accessIPv6
config_drive
created	2016-05-16T09:35:09Z
flavor	Standard 5 (b6be6358-120a-4fb9-987c-f10a7e82cb09)
hostId	b8b17a365b15d16f31860029f971348e8c8af7f2092652c5e7c36b83
id	92b9c565-d908-4dd0-a169-f58b3ce71428
image	CentOS-7-x86_64-GenericCloud-1603.qcow2 (7e78fc90-4eed-41a5-8a79-9fef27371278)
key_name	suresh
metadata	{}
name	Jenkins
net-int network	10.10.10.82, 202.65.153.24
os-extended-volumes:volumes_attached	[]
progress	0
security_groups	default
status	ACTIVE
tenant_id	87725975eafe46ebb8c54e02acf0b654
updated	2016-05-16T09:36:04Z
user_id	670959f49dde4aa582d99afbf6835387

 Step 2: Connect to mysql database

[root@controller ~]# mysql -u root -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 241348
Server version: 5.5.47-MariaDB MariaDB Server

Copyright (c) 2000, 2015, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| cinder             |
| glance             |
| keystone           |
| mysql              |
| neutron            |
| nova               |
| performance_schema |
| test               |
+--------------------+

9 rows in set (0.00 sec)

#Now change the database to use nova:

MariaDB [(none)]> use nova;

Reading table information for completion of table and column names

You can turn off this feature to get a quicker startup with -A

Database changed

MariaDB [nova]>

Step 3: Get the current status of instance from database

MariaDB [nova]> select uuid, hostname, vm_state, task_state, power_state, deleted from instances where uuid="92b9c565-d908-4dd0-a169-f58b3ce71428";

+--------------------------------------+----------+----------+------------+-------------+---------+

| uuid                                 | hostname | vm_state | task_state | power_state | deleted |

+--------------------------------------+----------+----------+------------+-------------+---------+

| 92b9c565-d908-4dd0-a169-f58b3ce71428 | jenkins  | active   | NULL       |           1 |       0 |

+--------------------------------------+----------+----------+------------+-------------+---------+

1 row in set (0.00 sec)

##Mark instance deleted state to 1 using below query

MariaDB [nova]> update instances set deleted='1', vm_state='deleted', deleted_at='now()' where uuid='77beaec2-636e-4e3e-9867-5df057fc94be' and project_id='87725975eafe46ebb8c54e02acf0b654';

validate the state again by running above status query:

MariaDB [nova]> select uuid, hostname, vm_state, task_state, power_state, deleted from instances where uuid="92b9c565-d908-4dd0-a169-f58b3ce71428";

+--------------------------------------+----------+----------+------------+-------------+---------+

| uuid                                 | hostname | vm_state | task_state | power_state | deleted |

+--------------------------------------+----------+----------+------------+-------------+---------+

| 92b9c565-d908-4dd0-a169-f58b3ce71428 | jenkins  | deleted| NULL       |           0 |       1 |

+--------------------------------------+----------+----------+------------+-------------+---------+

1 row in set (0.00 sec)

Now you can see your power state and deleted db columns set to 0,1 respectively even this instance will be removed from openstack dashboard also.

Hope this post helped...

All About AWS CLI

As a AWS solution engineers, very frequently we will use AWS CLI or at least for bulk operations, audit/inventory operations. So here I am going to consolidate the most of AWS CLI details as:

• Installation
• Configuration
• Consumption (Examples)

AWS CLI Installation:

We can download AWS CLI for windows here for Mac and Linux we can install using pip.

For Windows based systems:

• We can download from above portal based your operating system architecture (64/32-bit).
• We have run the downloaded file using administrator privileges and install with default options.

For Linux based systems:

• As pre-req mentioned in the AWS CLI official page we should have python 2.6.5 or higher already in our systems if not install it before going to next step.
• We can install awscli using pip install awscli

After installation you can validate as below by running

aws --version

aws-cli/1.11.44 Python/2.7.10 Darwin/16.4.0 botocore/1.5.7

Now we can confirm our AWSCLI installation was successfully completed.

AWSCLI Configuration:

For our general use aws configure command will help us to quickly setup our AWS installation,  

aws configure

AWS Access Key ID [****************UMIA]: 

AWS Secret Access Key [****************gkYc]: 

Default region name [us-west-2]: 

Default output format [None]: text

Once we ran and give "AWS Access Key ID,  AWS Secret Access Key, Default region & Default output format (Optional) Note: Default output format can be either json, text, or table. If you don't specify an output format, json will be used" 

For the first time once we configure below files will be created on our user profile:

Windows Location:

C:\Users\Username\.aws\credentials

C:\Users\Username\.aws\config

Linux Location

/home/Users/username/.aws/credentials

/home/Users/username/.aws/config

Here I have documented available regions as of now which you can follow while passing region name parameter.

When we have multiple user profiles to use AWSCLI then we can --profile option :

aws configure --profile cloud

AWS Access Key ID [None]: ****************UMIA

AWS Secret Access Key [None]: ****************gkYc

Default region name [None]: us-west-2

Default output format [None]: table

or else we can directly edit/add credentials and config details to respective profile files as below:

cat .aws/credentials 

[default]

aws_access_key_id = AKIijjhbbGK564123567888MIA

aws_secret_access_key = bZ6uNU6PahjfajfkaflalfllWR8PD0R5GjOVktgkYc

[cloud]

aws_access_key_id = ****************UMIA

aws_secret_access_key = ****************gkYc

cat .aws/config 

[default]

region = us-west-2

output = text

[profile cloud]

output = table

region = us-west-2

Note: The AWS credentials file uses a different naming format than the CLI config file for named profiles. Do not include the 'profile ' prefix when configuring a named profile in the AWS credentials file.

Consumption (Examples)

Using Profiles with the AWS CLI

To use a named profile, add the --profile option to your command. The following example lists running instances using the cloud profile from the previous section.

$ aws ec2 describe-instances --profile cloud

If you are going to use a named profile for multiple commands, you can avoid specifying the profile in every command by setting the AWS_DEFAULT_PROFILE environment variable at the command line:

Linux, macOS, or Unix

$ export AWS_DEFAULT_PROFILE=cloud

Windows

> set AWS_DEFAULT_PROFILE=cloud

Note:Setting the environment variable changes the default profile until the end of your shell session, or until you set the variable to a different value. More on variables in the next section.

The following environment variables are supported by the AWS CLI

AWS_ACCESS_KEY_ID – AWS access key.

AWS_SECRET_ACCESS_KEY – AWS secret key. Access and secret key variables override credentials stored in credential and config files.

AWS_SESSION_TOKEN – session token. A session token is only required if you are using temporary security credentials.

AWS_DEFAULT_REGION – AWS region. This variable overrides the default region of the in-use profile, if set.

AWS_DEFAULT_PROFILE – name of the CLI profile to use. This can be the name of a profile stored in a credential or config file, or default to use the default profile.

AWS_CONFIG_FILE – path to a CLI config file.

Command Line Options:

The AWS CLI uses GNU-style long command line options preceded by two hyphens. Command line options can be used to override default configuration settings for a single operation, but cannot be used to specify credentials.

The following settings can be specified at the command line.

--profile – name of a profile to use, or "default" to use the default profile.

--region – AWS region to call.

--output – output format.

--endpoint-url – The endpoint to make the call against. The endpoint can be the address of a proxy or an endpoint URL for the in-use AWS region. Specifying an endpoint is not required for normal use as the AWS CLI determines which endpoint to call based on the in-use region.

Note:The above options override the corresponding profile settings for a single operation. Each takes a string argument with a space or equals sign ("=") separating the argument from the option name. Quotes around the argument are not required unless the argument string contains a space.

Sample e.g. 

aws ec2 describe-instances --output table --region us-east-1

-------------------

|DescribeInstances|

+-----------------+

aws s3 ls --output json

2016-10-15 13:31:49 adithya34

2016-11-29 12:08:09 aruprcciit

2016-11-29 12:07:52 blazee

2016-06-09 13:56:12 mockert

Miscellaneous Tip (Command Auto Completion):

On Unix-like systems, the AWS CLI includes a command-completion feature that enables you to use the TAB key to complete a partially typed command. This feature is not automatically installed so you need to configure it manually.

add the below line in your /etc/bashrc file :

complete -C aws_completer aws

After enabling command completion, type in a partial command and press tab to see the available commands.

$ aws sTAB
s3              ses             sqs             sts             swf
s3api           sns             storagegateway  support

                

Hope this topic helps you start learning and working on AWSCLI.

How to Delete Multiple AWS S3 buckets which are in different region

We usually can delete multiple buckets in same region, but S3 is not region centric and sometimes we may need to delete buckets from different regions at same time, So I consider to have below online shell script to help for the same:

Step 1: Place required buckets names to deleted in a file e.g. list.txt, you can get the same using aws s3 ls --output text using AWSCLI

Step 2: You can use below one liner to delete buckets which are mentioned in above file from shell command prompt where you installed AWSCLI for linux.

for i in `cat list.txt`;do j="$(aws s3api get-bucket-location --bucket $i --output text)"; aws s3 rb s3://$i --region $j --force;done

Above liner explanation:

Shell script related:
Declared 2 variables 
i = To store bucket name from file called list.txt
j = region name of the bucket
Usual for loop

AWS related commands:
aws s3api get-bucket-location  -- will help us to get bucket location(aka region)
aws s3 rb                                   -- will help us to delete bucket even though bucket is not
                                                      empty with --force option