AWS Cloudformation is a service that allows you to define your infrastructure on AWS in a template. This template it’s just a json file where you can define all the resources you need to create on your infrastructure, this is really useful to keep track of all your changes on the infrastructure under a version control of your choice, rollback changes and replicate your environment in other places in question of minutes.
When you define a template, you can think about the definition of one stack where is a set of logical resources you’ll need to provide a service. For example imagine a typical architecture for a web application, is composed basically by a web layer and database layer. Depending on the size of the project, we’ll need more than one web server to serve the content to the clients so we’ll need a load balancer to distribute the load to the web servers. The web server layer can be setup under an auto scaling groups to scale up or scale down the number of servers depending on the load of our web servers. As far we’ve our basic web stack defined:
– Web server instances.
– Database server instance.
– Auto scaling group for web servers.
– Load balancer.
So based on the example of the web application, cloudformation allows us to define all these resources in a json file creating a stack and cloudformation will be responsible to create automatically all the resources for us. After the creation of the stack you can update, add or delete more resources modifying the template and updating our stack, it’s possible to protect some resources to be modified or deleted if they are critical for our service creating a stack policy. Now let’s see the basic anatomy of a cloudformation template:
{
"AWSTemplateFormatVersion" : "version date",
"Description" : "JSON string",
"Parameters" : {
set of parameters
},
"Mappings" : {
set of mappings
},
"Conditions" : {
set of conditions
},
"Resources" : {
set of resources
},
"Outputs" : {
set of outputs
}
}
– AWSTemplateFormatVersion: Cloudformation template format used, most of the time is used the last one “2010-09-09”.
– Description: A small explanation about our stack and all the resources.
– Parameters: All the parameters passed to our resources at the creation time of the stack, for example the administrator user and password of the database instance, the number of initial instances to launch, and elastic ip to associate to an ec2 instance, etc…
– Mappings: It’s a kind of lookup table where you can store definitions of key:value and retrieve the value using the internal function Fn::FindInMap. This is useful for example in cases we need to launch ec2 instances using different AMIs based on the region the stack is created.
– Conditions: Includes statements to conditionally create or associate resources in our stack. For example imagine we’ve a stack definition for a test and a production environment and conditionally creates t2.small ec2 instances for our testing environment or m3.xlarge size for the production environment.
– Resources: This is the central part of our template, here are defined all the resources of the stack such as s3 buckets, ec2 instances, load balancers, etc…
– Outputs: The values returned by the different resources created, for example the URL of a S3 bucket, the dns record of a load balancer, the elastic ip address associated to an EC2 instance, etc…
The documentation of cloudformation it’s well documented, so understanding the anatomy of a template and following the documentation and some examples is enough to start working with cloudformation. I’ll leave link to my github repository where I’ve defined a small stack for my environment: https://github.com/opentodonet/cloudformation-templates/blob/master/WebServer-Stack.json
Basically this stack creates an EC2 instance with an elastic IP associated, a RDS database instance, two S3 buckets to store backups and logs with a lifecycle, a couple of security groups associated to the web server instance and the rds instance and an IAM role including policies to grant access to EC2, S3 buckets and cloudformation resources to the new EC2 instance. Let’s see with a bit more detail the different resources defined on this stack:
– AWS::EC2::SecurityGroup: Creates two security groups, one is for the EC2 instance to give access to http, https and ssh services and the security group for the RDS with access to the entire internal subnet to the port 3306. See how parameters are referenced using the internal function {“Ref” : “VpcId”}, where associates the security groups to the VPC id passed.
"WebServerSecurityGroup" : {
"Type" : "AWS::EC2::SecurityGroup",
"Properties" : {
"GroupDescription" : "Enable HTTP, HTTPS and SSH access",
"VpcId" : {"Ref" : "VpcId"},
"SecurityGroupIngress" : [
{"IpProtocol" : "tcp", "FromPort" : "80", "ToPort" : "80", "CidrIp" : "0.0.0.0/0"},
{"IpProtocol" : "tcp", "FromPort" : "443", "ToPort" : "443", "CidrIp" : "0.0.0.0/0"},
{"IpProtocol" : "tcp", "FromPort" : "22", "ToPort" : "22", "CidrIp" : "0.0.0.0/0"}
]
}
},
"DBEC2SecurityGroup": {
"Type": "AWS::EC2::SecurityGroup",
"Properties" : {
"GroupDescription" : "Frontend Access",
"VpcId" : {"Ref" : "VpcId"},
"SecurityGroupIngress" : [{
"IpProtocol" : "tcp",
"FromPort" : { "Ref" : "DBPort" },
"ToPort" : { "Ref" : "DBPort" },
"CidrIp" : "172.16.0.0/16"
}]
}
},
– AWS::S3::Bucket: Here are defined the two buckets for logs and backups. The DeletionPolicy ensures if the stack is deleted the s3 buckets will be preserved. AccessControl defines the ACL to access on this bucket, in that case both are private. LifecycleConfiguration allows you to create a lifecycle policy to apply on the bucket, in that case both buckets will remove the files older than 15 or 30 days, but here you can setup to archive the files to AWS Glacier for example.
"S3BackupBucket" : {
"Type" : "AWS::S3::Bucket",
"DeletionPolicy" : "Retain",
"Properties" : {
"AccessControl" : "Private",
"BucketName" : "opentodo-backups",
"LifecycleConfiguration" : {
"Rules" : [ {
"ExpirationInDays" : 15,
"Status" : "Enabled"
} ]
}
}
},
"S3LogBucket" : {
"Type" : "AWS::S3::Bucket",
"DeletionPolicy" : "Retain",
"Properties" : {
"AccessControl" : "Private",
"BucketName" : "opentodo-logs",
"LifecycleConfiguration" : {
"Rules" : [ {
"ExpirationInDays" : 30,
"Status" : "Enabled"
} ]
}
}
}
– AWS::IAM::Role: Allows to make API requests to AWS services without using an access and secret keys, using Temporary Security Credentials. This role creates different policies to give access to S3 buckets backups and logs, ec2 access and cloudformation resources.
"WebServerRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version" : "2012-10-17",
"Statement": [ {
"Effect": "Allow",
"Principal": {
"Service": [ "ec2.amazonaws.com" ]
},
"Action": [ "sts:AssumeRole" ]
} ]
},
"Path": "/",
"Policies": [
{ "PolicyName": "EC2Access",
"PolicyDocument": {
"Version" : "2012-10-17",
"Statement": [ {
"Effect": "Allow",
"Action": ["ec2:*","autoscaling:*"],
"Resource": "*"
} ]
}
},
{ "PolicyName": "S3Access",
"PolicyDocument": {
"Version" : "2012-10-17",
"Statement": [ {
"Effect": "Allow",
"Action": "s3:*",
"Resource": ["arn:aws:s3:::opentodo-backups","arn:aws:s3:::opentodo-backups/*","arn:aws:s3:::opentodo-logs","arn:aws:s3:::opentodo-logs/*"]
} ]
}
},
{ "PolicyName": "CfnAccess",
"PolicyDocument": {
"Version" : "2012-10-17",
"Statement": [ {
"Effect": "Allow",
"Action": ["cloudformation:DescribeStackResource"],
"Resource": "*"
} ]
}
}
]
}
},
– AWS::IAM::InstanceProfile: references to the IAM role, this is just a container for the IAM role and this allows to assign the role to an EC2 instance.
"WebServerInstanceProfile": {
"Type": "AWS::IAM::InstanceProfile",
"Properties": {
"Path": "/",
"Roles": [ {
"Ref": "WebServerRole"
} ]
}
},
– AWS::EC2::Instance: Creates the EC2 instance using the AMI id ami-df1696a8, and assigns the InstanceProfile defined before, in the subnet id subnet-7d59d518 and the instance size and key pairs passed as parameters. The property UserData allows to setup scripts to run in the startup process. This commands passed on the user-data are run by the cloud-init service, which is included on the public AMIs provided by AWS. This user-data setup here installs the package python-setuptools and installs CloudFormation Helper Scripts, which are a set of python scripts to install packages, run commands, create files or start services as part of the cloudformation stack on the EC2 instances. The cfn-init command gets the cloudformation metadata to check what tasks has to run the instance (that’s why we include the policy access to cloudformation:DescribeStackResource on the IAM role before). The cloudformation metadata is setup on the AWS::CloudFormation::Init key, where basically installs some packages including the awscli tool and creates a couple of files, the /root/.my.cnf to access to the RDS instance which is filled using the attributes got after create the RDS instance, and the file /etc/bash_completion.d/awscli for the awscli auto completion. The cfn-signal command on user-data is used to indicate if the EC2 instance have been successfully created or updated, which is handled by the CreationPolicy attribute to wait until the cf-init command has finished, with a timeout of 5 minutes.
"WebServerEc2Instance" : {
"Type" : "AWS::EC2::Instance",
"Metadata" : {
"AWS::CloudFormation::Init" : {
"config" : {
"packages" : {
"apt" : {
"nginx" : [],
"php5-fpm" : [],
"git" : [],
"etckeeper" : [],
"fail2ban" : [],
"mysql-client" : []
},
"python" : {
"awscli" : []
}
},
"files" : {
"/root/.my.cnf" : {
"content" : { "Fn::Join" : ["", [
"[client]\n",
"user=", { "Ref" : "DBUser" }, "\n",
"password=", { "Ref" : "DBPassword" }, "\n",
"host=", { "Fn::GetAtt" : [ "DBInstance", "Endpoint.Address" ] }, "\n",
"port=", { "Fn::GetAtt" : [ "DBInstance", "Endpoint.Port" ] }, "\n"
] ] },
"mode" : "000600",
"owner" : "root",
"group" : "root"
},
"/etc/bash_completion.d/awscli" : {
"content" : { "Fn::Join" : ["", [
"complete -C aws_completer aws\n"
] ] },
"mode" : "000644",
"owner" : "root",
"group" : "root"
}
}
}
}
},
"Properties" : {
"ImageId" : "ami-df1696a8",
"InstanceType" : { "Ref" : "InstanceType" },
"SecurityGroupIds" : [ {"Ref" : "WebServerSecurityGroup"} ],
"KeyName" : { "Ref" : "KeyPair" },
"IamInstanceProfile" : { "Ref" : "WebServerInstanceProfile" },
"SubnetId" : "subnet-7d59d518",
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash\n",
"aptitude update\n",
"aptitude -y install python-setuptools\n",
"easy_install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz\n",
"# Install the files and packages from the metadata\n",
"cfn-init --stack ", { "Ref" : "AWS::StackName" }," --resource WebServerEc2Instance --region ", { "Ref" : "AWS::Region" }, "\n",
"# Signal the status from cfn-init\n",
"cfn-signal -e $? ","--stack ", { "Ref" : "AWS::StackName" }," --resource WebServerEc2Instance --region ", { "Ref" : "AWS::Region" }, "\n"
]
]
}
}
},
"CreationPolicy" : {
"ResourceSignal" : {
"Timeout" : "PT5M"
}
}
}
– AWS::EC2::EIPAssociation: Associates the elastic IP passed as parameter to the EC2 instance. The elastic IP must be allocated before on AWS.
"EIPAssociation" : {
"Type" : "AWS::EC2::EIPAssociation",
"Properties" : {
"InstanceId" : {"Ref" : "WebServerEc2Instance"},
"EIP" : {"Ref" : "ElasticIP"}
}
},
– AWS::RDS::DBSubnetGroup: Creates a DB subnet group using the subnet ids defined where the RDS instance will be setup.
"DBSubnetGroup" : {
"Type" : "AWS::RDS::DBSubnetGroup",
"Properties" : {
"DBSubnetGroupDescription" : "WebServer DB subnet group",
"SubnetIds" : [ "subnet-058c0560", "subnet-2072c457" ]
}
},
– AWS::RDS::DBInstance: Creates the RDS instance on the subnet group created before with some properties passed as parameter.
"DBInstance" : {
"Type": "AWS::RDS::DBInstance",
"Properties": {
"DBInstanceIdentifier" : "WebServerRDS",
"Engine" : "MySQL",
"MultiAZ" : { "Ref": "MultiAZDatabase" },
"MasterUsername" : { "Ref" : "DBUser" },
"MasterUserPassword": { "Ref" : "DBPassword" },
"DBInstanceClass" : { "Ref" : "DBClass" },
"AllocatedStorage" : { "Ref" : "DBAllocatedStorage" },
"DBSubnetGroupName" : { "Ref" : "DBSubnetGroup" },
"Port" : { "Ref" : "DBPort" },
"StorageType" : "gp2",
"AutoMinorVersionUpgrade" : "true",
"BackupRetentionPeriod" : 5,
"PreferredBackupWindow" : "02:30-03:30",
"PreferredMaintenanceWindow" : "sun:04:30-sun:05:30",
"VPCSecurityGroups": [ { "Fn::GetAtt": [ "DBEC2SecurityGroup", "GroupId" ] } ]
}
},
As I said AWS has a very well documentation, so all that you need you can find on his doc pages and find very useful examples:
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-sample-templates.html
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-init.html
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-init.html
– Github repository with the full template:
https://github.com/opentodonet/cloudformation-templates/
Pingback:Snort Amazon Aws | Cloud Computing Security