Intro
aws cli is a strong yet easy way to automate simple repeatable tasks covering aws resources as it’s a direct, human-readable doorway to aws cloud platform.
This tutorial will not just demonstrate how to provision a website in awscloud, but will hopefully trigger your curiosity on the numerous possibilities the Command line tool can enrich your IaC experience with. We will see how it can amplify your productivity when combined with powerful shell scripts.
Here’s a direct link to my Github repo with all the scripts : https://github.com/brokedba/aws-cli-examples
Here’s also a gif demo if your curious : https://bit.ly/3hvTwVu
I. Shell scripting to the rescue
After a similar cli experience done on Oracle (see my oci-cli post) I decided to take a shot at awscli. The journey started with a simple instance launch script and ended up with an automated website provisioning on not one, but 6 different Operating systems 🙂 (RHEL, CENTOS, Ubuntu, amazon-linux2, SUSE, and Windows server). 5 interactive BASH scripts & userdata files were crafted. I also had to learn how auto-install and configure nginx in each mentioned OS’ (windows was the nastiest). On top of that, the code had to ensure that required network components were created if missing (route table,security group…).
It took me few days to pull out but I am glad I did as I learned more about what’s under the API hood than before. Hoping this would also give a solid intro to aws-cli for beginners.
Bonus: If you like Latino music and Dj snake you’ll be in for a treat !! (complete the lab to know more ;))
Topology
The following figure shows the layers involved between our workstation an AWS while running the cli commands along with the instance attributes we will be provisioning (values might change).
CLI setup and assumptions
If your cli environment is not configured yet, go check my previous post (aws-cli installation) before continuing.
I will assume that the below elements are present/configured in your workstation:
- AWSCLI default profile configured with your aws credentials (Access keys). Refer to my Blog post for more details
$ aws configure list Name Value Type Location ---- --------- ---- -------- profile <not set> None None access_key ****************J2WA shared-credentials-file region us-east-1 config-file ~/.aws/config
$ ssh-keygen -P "" -t rsa -b 2048 -m pem -f ~/id_rsa_aws
Generating public/private rsa key pair.
Your identification has been saved in /home/brokedba/id_rsa_aws.
Your public key has been saved in /home/brokedba/id_rsa_aws.pub.
$ aws ec2 describe-regions --filters "Name=region-name,Values=us-west-2" --query 'Regions[]' +-----------------------------+--------------------+-------------------+ | Endpoint | OptInStatus | RegionName | +-----------------------------+--------------------+-------------------+ | ec2.us-west-2.amazonaws.com | opt-in-not-required| us-west-2 | +-----------------------------+--------------------+-------------------+
II. Clone the repository
- Pick an area on your file system and issue the following command.
$ git clone https://github.com/brokedba/aws-cli-examples.git
- Below are the files you will retrieve after cloning the directory (including the description)
$ cd ./aws-cli-examples $ tree |-- check_image.sh ---> Displays most recent AMI details per OS type |-- create_igateway.sh ---> Create an Internet Gateway to link to a route table |-- create_instance.sh ---> Launch an instance (requires a vpc and subnet) |-- create_route.sh ---> Create a route table & add a route to internet |-- create_subnet.sh ---> Create a subnet within the VPC |-- create_vpc.sh ---> Create a VPC with dedicated security group |-- cloud-init ---> Subfolder containing bootstrap code for each OS |-- amzl_userdata.txt ---> userdata script for amazon linux 2 instance |-- el_userdata.txt ---> userdata script for CENTOS and RHEL7 instance |-- sles_userdata.txt ---> userdata script for SUSE 15 instance |-- ubto_userdata.txt ---> userdata script for Ubuntu instance |-- Win_userdata.txt ---> userdata script for Windows server instance
III. Deployment
- Although awscli leverages JMSPATH to query the JSON based cloud resource data , it can get hard to avoid mistakes with IDs and lookups syntax when running these commands. That’s where “Shell scripts” come handy
- Note : My profile’s default region is us-east-1 and the default output is table (cue the tabular display).
-
CREATE A VPC
- As said before, all these scripts are interactive. Here, you can change or choose default values for the VPC Name and CIDR block. I also added checks on the IP/CIDR format. Scripts are also clickable so you can see the content.
- You will notice a security group menu that offers to open few pots (22,80) needed for our instance launch
brokedba@ ./create_vpc.sh Enter the VPC name you wish to create [CLI-VPC]: selected VPC name : CLI-VCN Enter the VPC CIDR to assign '/16-To-/28' [192.168.0.0/16]: 192.168.0.0/16 == VCN information === VPC name = CLI-VPC CIDR = 192.168.0.0/16 ==== Created VPC details ==== +----------------+--------+----------------------+-------------+ | CIDR | Name | VPC_ID | association | +----------------+--------+----------------------+-------------+ | 192.168.0.0/20 | CLI-VPC| vpc-0091141e28608813c| associated | +----------------+--------+----------------------+-------------+ Note : the last octet is always zeroed even if you specify a non zero value ************ Security Group ! ************ 1) SSH port Only 3) SSH ,HTTP,RDP, and HTTPS 2) SSH, HTTP, and HTTPS Select a security group ingress rule and press Enter: 2 ******************* Security Group detail ****************** +---------------+----------------------------+-----------------------------+ | Description | Protocol | SourceCIDR | ToPort | fromport | +-----------------------+-----------+-------------+---------+------------+ | Inbound HTTP access | tcp | 0.0.0.0/0 | 80 | 80 | | Inbound SSH access | tcp | 0.0.0.0/0 | 22 | 22 | +-----------------------+-----------+-------------+---------+------------+ SG delete command ==> aws ec2 delete-security-group --group-id sg-a38937a4d75361 VPC delete command ==> aws ec2 delete-vpc --vpc-id vpc-0091141e28608813c
Tip: click on the script to see its cli .
A delete command is also included in each script so you could destroy/repeat without using the Web Console.
-
CREATE A SUBNET WITHIN THE VPC
- You can pick the right VPC name then choose both a new subnet name and CIDR Block.
- This will also Auto-assign a public IP address to future instance launched in this subnet
- My CIDR checker won’t validate any complex subnetting. If you want to choose a custom CIDR block make sure it’s contained in the VPC and doesn’t overlap with any existing subnet (of the same VPC).
brokedba@ ./create_subnet.sh +----------------+--------+----------------------+-------------+ | CIDR | Name | VPC_ID | association | +----------------+--------+----------------------+-------------+ | 192.168.0.0/16 | CLI-VPC| vpc-0091141e28608813c| associated | +----------------+--------+----------------------+-------------+ Select the VPC Name you wish to attach your subnet to []: CLI-VPC Selected VCN name : CLI-VPC Enter the subnet name you wish to create [CLI-SUB]: CLI-SUB Selected SUBNET name : CLI-SUB ============ SUBNET CIDR ========================== Subnet CIDR must be contained in its VPC CIDR block 192.168.0.0/16 =================================================== Enter the VCN network CIDR to assign [192.168.10.0/24]: 192.168.10.0/24 == Subnet information === VPC name = CLI-VPC VPC CIDR = 192.168.0.0/16 SUBNET name = CLI-SUB SUBNET CIDR = 192.168.10.0/24 ==== Created SUBNET details ==== ------------------------------------------ | DescribeSubnets | +-----------+----------------------------+ | AZ | us-east-1f | | AutoIP | False | | CIDR | 192.168.10.0/24 | | IP_COUNT | 251 | | Name | CLI-SUB | | SUB_id | subnet-017a7233422dbec22 | | VPC_id | vpc-0091141e28608813c | +-----------+----------------------------+ --> Auto-assign Public IP enabled for CLI-SUB Delete command ==> aws ec2 delete-subnet --subnet-id subnet-017a7233422dbec22
Note: The script still checks if the entered subnet CIDR has a prefix that is between the VPC’s CIDR prefix and /28.
-
CREATE AN INTERNET GATEWAY
- This is pretty straightforward but I managed to tuck in a little check that exits if a gateway already exists.
brokedba@ ./create_igateway.sh +----------------+--------+----------------------+-------------+ | CIDR | Name | VPC_ID | association | +----------------+--------+----------------------+-------------+ | 192.168.0.0/16 | CLI-VPC| vpc-0091141e28608813c| associated | +----------------+--------+----------------------+-------------+ select the VPC you wish to add the I-Gateway to []: CLI-VPC selected VPC name : CLI-VPC Creating a New Internet gateway ... Enter the Internet gateway name you wish to create [CLI-IGW]: CLI-IGW ==== Created Internet gateway Details ==== +------------------------+----------+------------+-------------------------+ | Igw_id | Name | State | Vpc_id | +------------------------+----------+------------+-------------------------+ | igw-0003fea32eb918944 | CLI-IGW | available | vpc-0091141e28608813c | +------------------------+----------+------------+-------------------------+ Detach command ==> aws ec2 detach-internet-gateway --internet-gateway-id igw-00xx --vpc-id vpc-0091xxx Delete command ==> aws ec2 delete-internet-gateway --internet-gateway-id igw-00xx
-
SETUP A ROUTE TABLE
- A VPC list prompted at the beginning of the output(omitted here) will let you choose the VPC before asking to enter the subnet. If both subnet and internet gateway exist the route table is then set.
brokedba@ ./create_route.sh Select the VPC you wish to set the route table for []: CLI-VPC selected VPC name : CLI-VPC Internet gateway exists => checking the subnet availability ... ------------------------------------------ | DescribeSubnets | +-----------+----------------------------+ | AZ | us-east-1f | | CIDR | 192.168.10.0/24 | | IP_COUNT | 251 | | Name | CLI-SUB | | SUB_id | subnet-017a7233422dbec22 | | VPC_id | vpc-0091141e28608813c | +-----------+----------------------------+ select the subnet Name you wish to set the route table for []: CLI-SUB Internet gateway and subnet exist => Setting up the new Route table ... Create Route Table Create new route to Internet Gateway for Route Table ID 'rtb-062ff0c3c7069c885'. ... Associate 'CLI-SUB' Subnet with the Route Table. ...
==== Custom Route table entries for CLI-VPC ==== ----------------------------------------------------------------------------------- | DescribeRouteTables | +--------+---------------+---------------------------+----------------------------+ | Main | Name | Vpc_id | rt_id | +--------+---------------+---------------------------+----------------------------+ | False | rt_CLI-SUB | vpc-0091141e28608813c | rtb-062ff0c3c7069c885 | +--------+---------------+---------------------------+----------------------------+ || Routes || |+----------------------+-------------------------+--------------------+---------+| || DestinationCidrBlock | GatewayId | Origin | State || |+----------------------+-------------------------+--------------------+---------+| || 192.168.0.0/16 | local | CreateRouteTable | active || || 0.0.0.0/0 | igw-0003fea32eb918944 | CreateRoute | active || |+----------------------+-------------------------+--------------------+---------+| Detach route command ==> aws ec2 disassociate-route-table --association-id rtbassoc- Delete route command ==> aws ec2 delete-route --route-table-id rtb-69c885 --destination-cidr-block 0.0.0.0/0 delete route-table command ==> aws ec2 delete-route-table --route-table-id rtb-062
Delete commands are added at the end to get familiar with the delete syntax and know more about the underlying tasks.
-
LIST IMAGES
- Below menu returns the last available image per OS type. This will help you decide which OS to choose for your instance
brokedba@ ./check_image.sh ******* AWS Image Selecta ! ************ Choose your Destiny ||{**}|| 1) Oracle-Linux 3) Amazon Linux 2 5) Windows 7) Exit? 2) CentOS 4) Ubuntu 6) Suse Select an option and press Enter: 5 ------------------------------------------------------------------------------------- | DescribeImages | +--------------+--------------------+---------------------------------------+-------+ | Ami | Created | Name | SizeGb| +--------------+--------------------+---------------------------------------+-------+ | ami-06f6f331| 2020-09-09T06:26:58| Windows_Server-2016-English-Full-Base-| 30 | +--------------+--------------------+---------------------------------------+-------+
-
CREATE INSTANCE
- Awesome, all the network resources are set, we can now launch a our new instance.The script’s default instance type is free tier eligible (t2.micro) and there are 6 different OS’ to pick from .You’ll be asked to retry if the VPC and Subnet don’t match.
- I chose Windows in this example because it has additional requirements that make it tricky but you can spin any of the available options
brokedba@ ./create_instance.sh ******* Oci instance launch ! ************ Choose your Shape ||{**}|| Note: t2.Micro is the default FreeTier elligible instance type used here Enter the Path of your ssh key [~/id_rsa_aws.pub]: Enter the name of your new Instance [Demo-Cli-Instance]: Demo-Win2016 ----- selected Instance name : Demo-Win2016 selected public key: /home/brokedba/id_rsa_aws.pub The Instance Type will be the most recent FreeTier Elligible : t2.micro ------------------------------------------ | DescribeInstanceTypes | +-----+------------+---------+-----------+ | Ghz | Instance | Memory | VirType | +-----+------------+---------+-----------+ | 2.5| t2.micro | 1024 | hvm | +-----+------------+---------+-----------+ ********** Network *********** ----------------------------------------------------------------------- | DescribeVpcs | +-----------------+----------+-------------------------+--------------+ | CIDR | Name | VPCID | association | +-----------------+----------+-------------------------+--------------+ | 192.168.0.0/16 | CLI-VPC | vpc-0091141e28608813c | associated | +-----------------+----------+-------------------------+--------------+ select the VPC Name for your new instance []: CLI-VPC selected VPC name : CLI-VPC 1. Internet gateway check Internet gateway exists => checking the subnet availability ... ------------------------------------------------------------------------------------- | DescribeSubnets | +--------+---------+------------------+-----------+----------+----------------------- | AZ | AutoIP | CIDR | IP_COUNT | Name | SUB_id | +------------+---------+------------------+-----------+----------+------------------- | us-east-1f| True | 192.168.10.0/24 | 251 | CLI-SUB | subnet-2dbec22 | +------------+---------+------------------+-----------+----------+------------------- Select The Subnet for your new instance []: CLI-SUB selected subnet name : CLI-SUB Internet gateway and subnet exist => checking the Route table ... ...Route Table check The vpc has a route table with a route across an internet gateway. checking the association with CLI-SUB subnet. ... 2. Route is associated with CLI-SUB subnet. Checking the Security Group ... ... Checking the availability of a security Group with SSH/HTTP ingress rule . 3. dedicated security Group ingress rules exists PORT (22,80). ... Creating the instance with the below SG . ---------------------------------------------------------------------------- | DescribeSecurityGroups | +---------------+----------------------------+-----------------------------+ | Name | SG_id | Vpc_id | +---------------+----------------------------+-----------------------------+ | sg_CLI-VPC | sg-0e8a38937a4d75361 | vpc-0091141e28608813c | +---------------+----------------------------+-----------------------------+ || Rules || |+-----------------------+-----------+-------------+---------+------------+| || Description | Protocol | SourceCIDR | ToPort | fromport || |+-----------------------+-----------+-------------+---------+------------+| || Inbound HTTP access | tcp | 0.0.0.0/0 | 80 | 80 || || Inbound SSH access | tcp | 0.0.0.0/0 | 22 | 22 || |+-----------------------+-----------+-------------+---------+------------+| 4. Choose your Image ||{**}|| 1) RHEL 3) amazon Linux 2 5) Windows 7) Exit? 2) CentOS 4) Ubuntu 6) Suse Select an option and press Enter: 5 ------------------------------------------------------------------------------------- | DescribeImages | +-------------+------------------+---------------------------------------+----------+ | Ami | Created | Name | SizeGb | +-------------+------------------+---------------------------------------+----------+ | ami-06f6f3*| 2020-09-09T06:26 |Windows_Server-2016-English-Full-Base | 30 | +-------------+------------------+---------------------------------------+----------+ >> opening port 3389 ===== Instance Deployment Detail ======== selected Subnet name : CLI-SUB selected Instance name : Demo-Win2016 selected instance Type: t2.micro selected public key: /home/brokedba/id_rsa_aws.pub selected Security Group: sg-0e8a38937a4d75361 selected OS : Windows ... Importing/checking the key pair to/from AWS key-pair exists .. ==================================== Check the status of the new Instance ==================================== The compute instance is being created. This will take few minutes ... ------------------------------------------- | DescribeInstances | +------------+----------------------------+ | AZ | us-east-1f | | Hostname | | | ID | i-04b8144695dd402b0 | | Name | Demo-Win2016 | | Platform | windows | | PrivIP | 192.168.10.156 | | Public_IP | 3.235.253.148 | | Subnet | subnet-017a7233422dbec22 | | Type | t2.micro | | VPCID | vpc-0091141e28608813c | | image | ami-06f6f33114d2db0b1 | | status | running | +------------+----------------------------+ Password is being generated... please wait Windows User = Administrator Password => y=L6J&SrQ;kMnNB2z66Od(oYZtgyaZca Generated password can be retreived using : aws ec2 get-password-data --instance-id xxx --priv-launch-key ~/id_rsa_aws Your website is ready at this IP :) : http://3.235.253.148 Termination command ==> aws ec2 terminate-instances --instance-ids i-04b8144695dd402b0
- – Below is the instance view on your AWS console once the instance is provisioned.
-
CONNECTION TO THE INSTANCE AND CHECK THE PUBLIC WEBPAGE
- Grab the public IP and password displayed above and connect to the windows instance using MSTSC as admin
- Type the Public IP in your browser & check the Homepage we customized with our user-data script during bootstrap
and voilà! => http://3.235.253.148
- Note: All other Linux based instances must be accessed using ssh as ec2-user (except for ubuntu which uses “ubuntu” and CENTOS which has “centos” as default user).
$ ssh –i ~/id_rsa_aws ec2-user@Public_ip
BONUS
Now what if I’m lazy (like most of us :p) and don’t want to pre-create the route table, internet gateway or other security group rules for my instance?! Well In that case, the create_instance script got you covered as it does just that for you. All you need to create is the VPC and Subnet, nothing else. From there, the instance script will check and create any missing network piece for you (internet gateway, Security group, opening HTTP ports,routes ..etc) .
CONCLUSION
- AWSCLI and bash can perfectly fit in with Cloud practitioners’ daily activities, so long as we take the time to refine and strengthen the logic of our scripts. Moreover, the painful handling of long IDs and JMSPATH query syntax errors are here eliminated.
- This took some time to build but it’s definitely the most direct and simple way to interact with your AWS infrastructure (no SDK). You could even use the learned awscli practice and port it to other SDK supported languages later.
- Feel free to fork the repository and adapt your own version of these scripts
- Improvement : Add additional menus to choose among resources with duplicate names (using the id)
Thanks for reading!