Forward logs to an AWS S3 bucket

Amazon S3 (Simple Storage Service) is an object storage service that you can use to retain a copy of your AppLogs outside the monitoring console for compliance, archival, or downstream processing purposes.

By forwarding logs to an S3 bucket in your AWS account, you can retain records and feed them into your own SIEM tool, data lake, or analytics pipeline.

Key capabilities of the integration

This Site24x7 integration with AWS S3 provides the following capabilities:

• Forwards AppLogs to Amazon S3 buckets swiftly.
• Supports server-side encryption on the destination bucket.
• Delivers logs as gzip-compressed objects in a NDJSON, JSON, or CSV format.
• Authenticates through the identity and access management (IAM) role or access keys of an existing AWS cloud monitor; no separate credentials are needed to manage.

Prerequisites for Amazon S3 log forwarding

1. An AWS Cloud account needs to be configured in Site24x7 for your chosen destination. Log forwarding reuses the credentials from your existing cloud integration, so you don't re-enter secret keys.
   • If you haven't configured an AWS monitor, add an AWS account under Admin > Inventory > Add Monitor > AWS.
2. The monitor's IAM role or user needs write access (s3:PutObject) on the target bucket. Clicking Test Connection verifies this this by writing a sample file. Only object creation permission is required. List and delete permissions are not needed.
3. The target S3 bucket must already exist in the selected region. Objects are written under the optional path prefix you configure.

Configuring AWS S3 as a log forwarding destination

1. Log in to Site24x7.
2. Navigate to Admin > Third-Party Integrations.
3. Click on the plus icon + next to the Log Forwarding to open the supported destination list.
4. Select AWS S3 and click Integrate Now.

This will open the integration page for the AWS S3 destination. Enter the following details:

1. Basic Information: Provide the details of the destination to which the logs will be forwarded.
   • Name: Enter a label to identify this configuration.
   • Enable Forwarding: Toggle to Enabled to activate the configuration once saved.
2. Destinations: Choose where logs should be forwarded and configure destination credentials.
   • Destination Type: By default, AWS S3 is selected.
   • AWS Monitor: Select the existing AWS cloud monitor whose credentials are used to write to the bucket.
   • Bucket Name: Select or enter the destination S3 bucket. The bucket must already exist.
   • Path Prefix: Optionally, enter a folder prefix prepended to every object key, for example site24x7/logs.
   • File Format: Select the file format NDJSON (default), JSON, or CSV. Learn more about the file formats.
     • Click Test Connection to test the connection and confirm the bucket is reachable and writable.
3. Filtering: Restrict which logs are forwarded by log type and query.
   • Log Types: Optionally, you can select the 100+ default log types whose data you want to forward, and enter a Filter Query to forward only matching records. You can select more than one log type.
     • Click the X icon on the respective log type to remove the query.
4. Schedule & Retry: Control how often logs are forwarded and retry behavior occurs upon failure.
   • Forwarding Frequency: Select how often batches are written to the bucket; you can select any one from the following list:
     • Every 1 minute
     • Every 5 minutes
     • Every 15 minutes
     • Every 30 minutes
     • Every 60 minutes
   • Batch Size: Set the maximum number of records per object. Default is 1,000. Learn more about the batching and delivery options.
   • Retry on Failure: Toggle to Enable to automatically retry failed deliveries.
   • Max Retires: Enter a number of maximum entries the forward can fail.
5. Click Save. Forwarding begins with the next matching batch of incoming logs.
   
   

Managing the AWS S3 log forwarding configuration

1. Navigate to Admin > Third-Party Integrations > Log Forwarding.
2. Locate the configuration on the listing page. To make any changes, click the configured forwarding entry to open the Edit Configuration page.
3. Click the hamburger icon on the right corner of each configured forwarding entry to perform the below actions:
   1. Click Disable forwarding to temporarily stop forwarding.
   2. Click Enable forwarding to resume the forwarding.
   3. Click Delete to remove it permanently.
   

Related articles

• AWS API Gateway Access logs
• Amazon ECS logs
• AWS Managed Microsoft AD logs
• CloudFront logs
• CloudTrail logs
• CloudWatch logs
• Lambda Runtime logs
• S3 logs using Lambda Function
• S3 logs using SQS
• VPC Flow logs