AWS S3
Introduction
Amazon Simple Storage Service (Amazon S3) is a scalable object storage service offered by Amazon Web Services (AWS). It is designed to store and retrieve any amount of data from anywhere on the web.
Getting Started
Compatibility
vuSmartMaps supports the monitoring of the flow of traffic in your AWS S3.
Data Collection Method
vuSmartMaps collects health and performance data for AWS S3 using VuNet's Internal Data Collector.
Prerequisites
Dependent Configuration
To configure this O11ySource, create a 'credential' of type 'aws' under the 'Definition' tab.
Inputs for Configuring Data Source
- Data Source Name: The AWS S3 data source that will uniqly identify the source.
- AWS Region: AWS Region where the instance of this component is running. For eg: Asia Pacific (Mumbai), the region would be ap-south-1.
- AWS Credential: AWS credential that provides Access key and Secret key to access Cloudwatch.
- Period (in minutes): Specifies the interval in minutes at which data is collected. Data collection occurs once every specified period. The period should be between 1 - 60 minutes.
Firewall Requirement
To collect data from this O11ySource, ensure the following ports are opened:
| Source IP | Destination IP | Destination Port | Protocol | Direction |
|---|---|---|---|---|
| vuSmartMaps IP | AWS CloudWatch IPs | 443* | TCP | Outbound |
*Before providing the firewall requirements, please update the port based on the customer environment.
Configuring the Target
Health and Performance metrics of AWS S3 is collected through AWS CloudWatch service. So AWS CloudWatch services must be enabled in your AWS account. By default, AWS S3 provides a set of basic metrics for each bucket at no additional cost, known as storage metrics. However, for more detailed request-level metrics, you need to enable them manually.
An IAM role or user with the following permissions to access CloudWatch metrics:
- cloudwatch:GetMetricData
- cloudwatch:ListMetrics
- s3:ListBucket
- s3:GetBucketLocation
The following configuration changes are required in CloudWatch / S3 to enable the required metrics:
- Login to the AWS Management Console and navigate to the S3 service.
- Select the bucket for which you want to enable metrics.
- Go to the Metrics tab.
- Click on 'Create metric filter' under the Request metrics section.
- Configure the filter as per your needs. You can filter metrics for the whole bucket or specific object prefixes.
- Save the filter.
Configuration Steps
Enablethe O11ySource.- Select the sources tab and press the
+button to add a new instance that has to be monitored. - Provide the required configurations:
- *Data Source Name
- *AWS Region
- *Credential
- *Period
- Click
Saveto close the data source window.
Metrics Collected
| Name | Description | Data Type |
|---|---|---|
| BucketSizeBytes | The amount of data in bytes that is stored in a bucket. | Float64 |
| NumberOfObjects | The total number of objects stored in a general purpose bucket for all storage classes. This value is calculated by counting all objects in the bucket, including current and noncurrent objects, delete markers, and incomplete multipart uploads. | UInt64 |
| All Requests Volume | The total number of HTTP requests made to an Amazon S3 bucket, regardless of type. | UInt64 |
| Total Bytes Downloaded | The number of bytes downloaded for requests made to an Amazon S3 bucket. | Float64 |
| Bytes Downloaded per Request | The average number of bytes downloaded per request. | Float64 |
| Total Bytes Uploaded | The number of bytes uploaded that contain a request body, made to an Amazon S3 bucket. | Float64 |
| Bytes Uploaded per Request | The average number of bytes uploaded per request. | Float64 |
| Get Requests Volume | The number of HTTP GET requests made for objects in an Amazon S3 bucket (doesn't include list operations). | UInt64 |
| Post Requests Volume | The number of HTTP POST requests made to an Amazon S3 bucket. | UInt64 |
| Put Requests Volume | The number of HTTP PUT requests made for objects in an Amazon S3 bucket. | UInt64 |
| Head Requests Volume | The number of HTTP HEAD requests made to an Amazon S3 bucket. | UInt64 |
| Delete Requests Volume | The number of HTTP DELETE requests made for objects in an Amazon S3 bucket (includes Delete multiple objects requests). | UInt64 |
| List Requests Volume | The number of HTTP requests that list the contents of a bucket. | UInt64 |
| Select Requests Volume | The number of Amazon S3 SelectObjectContent requests made for objects in an Amazon S3 bucket. | UInt64 |
| First Byte Latency | Per request, the time from when the Amazon S3 bucket receives the complete request to when the response starts to be returned. | Float64 |
| Total Select Bytes Returned | The number of bytes of data returned with Amazon S3 SelectObjectContent requests in an Amazon S3 bucket. | Float64 |
| Select Bytes Returned per Request | The average number of bytes returned per Amazon S3 SelectObjectContent request. | Float64 |
| Total Select Bytes Scanned | The number of bytes of data scanned with Amazon S3 SelectObjectContent requests in an Amazon S3 bucket. | Float64 |
| Select Bytes Scanned per Request | The average number of bytes scanned per Amazon S3 SelectObjectContent request. | Float64 |
| Total Request Latency | The time per request, starting from the first byte received to the last byte sent to an Amazon S3 bucket. This includes the time taken to receive the request body and send the response body. | Float64 |
| Count of 4xx Errors | The number of HTTP 4xx client error status code requests made to an Amazon S3 bucket with a value of either 0 or 1. The average statistic shows the error rate, and the sum statistic shows the count of that type of error, during each period. | UInt64 |
| Count of 5xx Errors | The number of HTTP 5xx server error status code requests made to an Amazon S3 bucket with a value of either 0 or 1. The average statistic shows the error rate, and the sum statistic shows the count of that type of error, during each period. | UInt64 |
| S3 Error Percentage | Percentage of errors. | UInt8 |
| S3 Select Efficiency Percentage | Percentage of efficiency of S3 SelectObjectContent requests. Higher numbers are better. | UInt8 |
| Storage Type | Type of the AWS S3 Storage. | String |
| Bucket Name | Name of the AWS S3 Storage. | LowCardinality(String) |
| Region | AWS Region | String |
| Type | Document type to identify the data collected | String |
| Interval | Interval at which the data has been collected. | String |
| Target | Target server | String |
| Host | Host server | String |
| Timestamp | Timestamp of the data | DateTime64 |
| Tenand ID | Tenant Id | LowCardinality(String) |
| BU Id | BU Id | LowCardinality(String) |
