In the previous article, I explained how to set up live tail logging in AWS CloudWatch. The setup of log group live tail logging was completed using AWS Console.
In this article, I will demonstrate another way to get live tail for the selected cloud log group using a programmatic approach as oppose to using the AWS Console. I will use Python and Boto3 to write a command line python program that will publish the live tail based on the cloud watch log group and filter pattern provided.
If you’re a Linux user, you’re likely familiar with the tail command for viewing live logs. CloudWatch’s live tail feature works similarly, allowing you to monitor logs in real time as your functions are executed.
This feature is incredibly helpful for various use cases, including debugging issues, monitoring deployments, command center oversight, identifying performance issues, and more.
If you are interested in prior articles on the cloud watch logs topics, Please refer to the links below:
Link:
Link:
Let's see the steps to create this solution using AWS CloudWatch live tail and Lambda logging.
Few points about CloudWatch Live Tail
CloudWatch Live Tail provides options to refine and target specific log data:
- Use Highlight to flag events that match your specified terms (up to 5 terms).
- Use Filter to limit the events displayed in the session.
You can select up to 10 log groups to include in the live tail session.
AWS may update this limit so please refer to AWS documentation for the latest limit.
Note: CloudWatch Live Tail is charged on a per-minute basis during the session, making it a flexible, on-demand solution for troubleshooting and monitoring.
Review the CloudWatch Log Group
For this CloudWatch Logs live tail demonstration, I will use the log group /aws/lambda/MeetingNotesSummarizer. This log group is already in place for an existing Lambda function, and all logs generated by this function are stored here. This setup allows us to monitor and analyze the log data produced by this Lambda function in real time.
Review the Lambda Function Description
The Lambda function we’re using is pre-configured with logging capabilities. This function supports a Generative AI use case and includes guardrails to ensure data security and compliance. For example, if Personally Identifiable Information (PII) is detected in the data processed by this function, a guardrail log entry will be generated in the CloudWatch log group to flag the occurrence. This helps maintain data safety by providing insights into sensitive data handling during AI operations.
Review the Python Program
This python program is for a command line utility that monitors the AWS CloudWatch logs in real time (live tail) with the option to filter the logs. Here’s a simple breakdown of its main parts:
The script uses a config.json file to get the following settings:
log_group: The CloudWatch log group you want to monitor.
filter_pattern: The text used to filter log messages.
The functions in this program are as below:
read_config(): Reads the settings from the config.json file.
get_log_events(): Fetches log entries from a specific log group and log stream.
get_log_streams(): Retrieves log streams and sorts them by the most recent ones.
Example function:
Run the Python Program and start the live tail session!
Monitor the Live tail for Filtered logs
When lambda function is invoked, logs will be created and live tail will show it as well!
𝒢𝒾𝓇𝒾𝓈𝒽 ℬ𝒽𝒶𝓉𝒾𝒶
𝘈𝘞𝘚 𝘊𝘦𝘳𝘵𝘪𝘧𝘪𝘦𝘥 𝘚𝘰𝘭𝘶𝘵𝘪𝘰𝘯 𝘈𝘳𝘤𝘩𝘪𝘵𝘦𝘤𝘵 & 𝘋𝘦𝘷𝘦𝘭𝘰𝘱𝘦𝘳 𝘈𝘴𝘴𝘰𝘤𝘪𝘢𝘵𝘦
𝘊𝘭𝘰𝘶𝘥 𝘛𝘦𝘤𝘩𝘯𝘰𝘭𝘰𝘨𝘺 𝘌𝘯𝘵𝘩𝘶𝘴𝘪𝘢𝘴𝘵
SOCIAL SHARE CARD GENERATOR