Scenario: A third party security agency hired by your boss or client wishes to monitor the SSH logs on the bastion host of your application. The bastion host is setup on an AWS EC2 instance. Instead of giving direct access to the bastion host for pulling the SSH logs in /var/log/auth.log, we can forward the SSH logs to AWS CloudWatch Logs and just give access to that particular log group (and not all the log groups).
The steps below are typed out in Markdown format as I’m too lazy to type out the HTML tags for the nested lists 😛
## Steps
- Create an IAM role for the EC2 instance and allow it to write to CloudWatch Logs.
If the role already exists, just add the permission.
+ AWS Console > EC2 > Roles > Create role
* Select trusted entity
- Trusted entity type: AWS service
- Use case: EC2
* Add permission
- AWS managed policy: `AmazonAPIGatewayPushToCloudWatchLogs`
```
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:DescribeLogGroups",
"logs:DescribeLogStreams",
"logs:PutLogEvents",
"logs:GetLogEvents",
"logs:FilterLogEvents"
],
"Resource": "*"
}
]
}
```
* Name, review, create
- Role name: `iam-role-ec2-bastion`
- Description: Allows bastion EC2 instance to call AWS services on your behalf.
- Step 1: Select trusted entitites
```
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"sts:AssumeRole"
],
"Principal": {
"Service": [
"ec2.amazonaws.com"
]
}
}
]
}
```
- Step 2: Add permissions
+ AWS managed permissions policy: `AmazonAPIGatewayPushToCloudWatchLogs`
* Create role
- Attach IAM role to EC2 instance, assumed with name `ec2-bastion`.
+ AWS Console > EC2 > Instances > ec2-bastion > Actions > Security > Modify IAM role
* Set IAM role to `iam-role-ec2-bastion`.
- Setup the CloudWatch agent on the EC2 instance, assumed to be running the Ubuntu operating system.
+ SSH into EC2 instance.
+ Install the CloudWatch agent as per
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/download-cloudwatch-agent-commandline.html
```
bash
cd ~
wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb
sudo dpkg -i -E ./amazon-cloudwatch-agent.deb
sudo apt install collectd # required later when starting CloudWatch agent
```
+ Run `sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard` to
create the agent configuration file using the configuration wizard as per
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-cloudwatch-agent-configuration-file.html
* On which OS are you planning to use the agent? linux
* Are you using EC2 or On-Premises hosts? EC2
* Which user are you planning to run the agent? cwagent
* Do you want to turn on StatsD daemon? no
* Do you want to monitor metrics from CollectD? yes
* Do you want to monitor any host metrics? yes
* Do you want to monitor cpu metrics per core? yes
* Do you want to add ec2 dimensions (ImageId, InstanceId, InstanceType, AutoScalingGroupName)
into all of your metrics if the info is available? yes
* Do you want to aggregate ec2 dimensions (InstanceId)? yes
* Would you like to collect your metrics at high resolution (sub-minute resolution)? 60s
* Which default metrics config do you want? Basic
* Are you satisfied with the above config? yes
* Do you have any existing CloudWatch Log Agent configuration file to import for migration? no
* Do you want to monitor any log files? yes
- Log file path: `/var/log/auth.log` (SSH logs)
- Log group name: `ec2-bastion-log-group`
- Log stream name: `[{instance_id}]`
- Log Group Retention in days: -1
* Do you want to specify any additional log files to monitor? no
* Do you want to store the config in the SSM parameter store? no
* The config file is located at `/opt/aws/amazon-cloudwatch-agent/bin/config.json`.
+ Start the CloudWatch agent as per
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Agent-on-EC2-Instance-fleet.html#start-CloudWatch-Agent-EC2-fleet
```
# This needs to be run everytime the configuration is changed also
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json
```
+ To stop the CloudWatch agent: `sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a stop`
- Create an IAM user account for the third party security agency. Skip this if the user account
already exists.
+ AWS Console > IAM > Users > Add users
* Specify user details
- User name: `third-party`
- Provide user access to the AWS Management Console: No
* Set permissions
- Skip as we will be attaching an inline policy later.
* Review and create
* Create user
- Allow the IAM user account to access the CloudWatch Logs log group for
the SSH logs forwarded from the EC2 instance.
+ AWS Console > IAM > Users > `third-party` > Permissions >
Add permissions > Create inline policy
* Create policy > JSON. The `*` at the end of the `Resource` key is to
allow access to all log streams in the log group.
```
{
"Version":"2012-10-17",
"Statement":[
{
"Action": [
"logs:Describe*",
"logs:Get*",
"logs:List*",
"logs:StartQuery",
"logs:StopQuery",
"logs:TestMetricFilter",
"logs:FilterLogEvents"
],
"Effect": "Allow",
"Resource": "arn:aws:logs:ap-southeast-1:1234:log-group:ec2-bastion-log-group:*"
}
]
}
```
- Phew, done!