Cloud Watch Logs
Let's move on to CW logs and how it works in relation to IAM and AWS Organizations
Its a public service hosted in the AWS public zone. It’s accessible from AWS VPCs , on-premises environment or other cloud platforms provided you have network connectivity and AWS permissions.
The CW Logs product allows to store, monitor and access logging data
Logging data – is basically – piece of information and timestamp-DDMMYYhhmmss TZ format
It has integrations with AWS services like EC2, VPC Flow Logs, Lambda, CloudTrail, R53, etc..that means these services can store logging data in CW Logs' The security is provided using IAM roles or service roles
For anything outside AWS or for logging custom application or OS logs on EC2 – CW agent
There is a third way i.e the dev kits for AWS and implement logging into CW Logs directly from the application
CW Logs can take the logging data and generate a metric from it – known as a metric filter
Let’s suppose there’s a linux instance which has an OS log file which captures any failed ssh connection attempts
If that log from an instance is fed into CW Logs service, then a metric filter can scan the log data constantly and update the metric and an alarm can be setup based on the metric
CW Architecture
It is also a regional service
Starting point – Logging source which can include AWS products and services, external computer services , virtual or physical servers db, external APIs
These source feed data into CW Logs as log events
Log events have a time stamp and message block.
CW Logs treats it as a raw block of data but the data can be interpreted and columns and fields can be defined from that.
Log events are stored inside log streams and log streams – which are sequence of log events from same source
One log stream for one instance and one type of log
/var/log/messages for one instance is one log stream for system diagnostics in linux-based systems. Each log stream is an ordered set of log events for a specific source for a specific thing.
Log groups – container for multiple log streams for same type of logging data
Log groups – also stores configuration settings like retention settings and permissions
Retention settings and permissions are also defined on log groups – applicable to all log streams inside it
Metric filters will also be configured for log groups. Metric filters constantly review log events for any log streams in that log group looking for certain patterns, when found, the metric filters increment a metric and these metric can have alarms which would notify admins or AWS or external systems to take action. So it is a very useful products
