Collecting S3 logs using the Lambda Function
S3 Buckets acts as scalable containers in which large volumes of data can be stored. Site24x7 uses the Lambda Function to look for new logs added in the S3 Buckets and sends it to Site24x7 for indexing. Learn more about log management with Site24x7.
You can also configure your logs to be collected from S3 buckets using SQS. To avoid the overhead of configuring SQS permissions, you can use Lambda Functions to collect your logs as describ
ed below.
Define the Log Type
A Log Type is a clear definition of the format in which an application writes logs. Different applications (such as IIS, Cassandra, Apache, MySQL) may write logs in different formats. Defining them as Log Types groups logs from different applications to simplify access and assist in efficient searching.
To create a log type, go to Admin > AppLogs > Log Types > Add Log Type.
You can choose from the list of AWS log types supported out-of-the-box:
| AWS API Gateway Access logs | CloudWatch logs |
| VPC Flow logs | CloudFront logs |
| Amazon ECS logs | CloudTrail logs |
| Lambda Runtime logs | Application Load Balancer logs |
You can also define a custom log type to group logs from various applications for easier access and more efficient searching.
- If you choose Custom Log Type, enter a display name.
- Provide sample log lines to discover the log pattern.
- Save the log type.
Once you define a Log Type for your logs stored in your S3 bucket, list it under a Log Profile and start managing your logs by performing search queries.
Create a Log Profile
A Log Profile enables you to associate log types to a particular log source.
To create a Log Profile, navigate to Admin > AppLogs > Log Profile > Add Log Profile, and follow the instructions below:
- Profile Name: Enter a name for your Log Profile.
- Choose the Log Type: Choose the Log Type of the S3 logs you would like to associate with this profile.
- Log Source: Choose Amazon Lambda.
- Timezone: Select a timezone for your logs.
- Click Save.
- Configure the Lambda function as described here.
Configure the Lambda Function
- Sign in to Manage AWS Resources - AWS Management Console
https://us-east-2.console.aws.amazon.com/lambda/home?region=us-east-2#/functions - Choose Lambda from the Services drop-down list, and choose Create function.
- Select Author from scratch, define a name for the function, and and choose Python 3.12 (the latest available) as the Runtime.
- Permissions: You can either choose an existing IAM role, or Create a new role from the AWS policy template.
Method 1: If you choose an existing IAM role, select the relevant role from the drop down menu shown.
Method 2: If you choose to Create a new role from AWS policy template, enter the Role name and select Amazon S3 Object read-only permissions from the Policy templates drop-down. Click Create function.
- Add triggers: Click Add trigger.
Select the source as S3 bucket. Any log file added to the S3 bucket will be sent to Site24x7 by the Lambda function.
- Configure Triggers
- Bucket:Enter the name of the S3 bucket from which logs will be collected.
- Event type: Choose All object create events.
- Acknowledge and Add.
- In the window that opens, go to Code. Scroll to the editor and place the code provided in the link below:
https://github.com/site24x7/applogs-aws-lambda/blob/master/s3/s3-sender.py
- After entering the code, go to the Site24x7 web client. Navigate to Admin > Applogs > Log Profile, select the created Log Profile, and copy the displayed code for the logTypeConfig.
- Back in the AWS Management Console
- Go to Configuration.
- Select Environment variables.
- Click Edit.
- Enter the key as logTypeConfig. Enter the value you copied from the Site24x7 Log Profile page. Click Save.
- After saving the environment variables, go to Code. Click Deploy.
- After a few minutes, you can go to the Site24x7 web console and select AppLogs from the left menu. On the AppLogs Search page, you can view the logs by searching with the log type as shown below:
logtype="ELB Application Log"
