Traces

Introduction

Collects the Distributed Traces telemetry from the applications to enable observability.

Getting Started

Compatibility

Traces is a comprehensive application observability solution that offers deep visibility into the application performance with the standard applications performance monitoring dashboards. Traces O11ySources is currently compatible for applications in the following languages: Java, .NET Core, .NET Framework

Data Collection Method

For collecting Traces data, we have to do the instrumentation on the application.

Prerequisites

Inputs for Configuring Data Source

• Applications Group: Name of application group.
• SSL required: SSL allowed or not. TLS will be turned on if selected Yes.

Application Services

• Language: Select application language.
• Application Name: Name of the target application.
• Service Name: Service name in the application.Typically in monolithic applications, the service name and application name are the same. In the microservices application, there would be one application that can have one or more services.

Firewall Requirement

To collect data from this O11ySource, ensure the following ports are opened:

Source IP	Destination IP	Destination Port	Protocol	Direction
Application Server	vuSmartMaps Data Collector IP	4319*	TCP	Inbound

*Before providing the firewall requirements, please update the port based on the customer environment.

Configuring the Target

Sampling Policies/Sampling Sub Policies Description:

• Allow All
• *Meaning: This represents the sampling policy which captures all incoming traces. It does not discard any data. Each and every trace that flows in is kept, with no filtering or selective sampling applied.
• Allow based on Latency
• *Meaning: This policy type focuses on sampling data based on the processing time or latency.
• *Attributes: threshold_ms
• *Usage: Data exceeding this specified threshold in terms of processing time or latency will be sampled for further analysis or processing.
• Allow based on Numeric Attributes
• *Meaning: This sampling policy involves filtering or sampling based on numeric attribute values falling within specified ranges.
• *Attributes: key, min_value, max_value
• *Usage: Sampling only data where the specified attribute key values fall within the defined range aids in proactive maintenance or alerting.
• Allow a Percentage
• *Meaning: This policy type allows sampling based on a defined probability or percentage.
• *Attributes: sampling_percentage
• *Usage: Collecting a specific percentage of incoming data enables analysis without overloading the system with excessive data, providing a representative subset for insights.
• Allow based on Status
• *Meaning: Sampling based on specific HTTP status codes.
• *Attributes: status_codes
• *Usage: Sampling data associated with particular HTTP error codes helps focus on error instances for detailed investigation and troubleshooting.
• Allow based on String Attributes
• *Meaning: Sampling based on specific string attribute values.
• *Attributes: key, values, enabled_regex_matching, invert_match
• *Usage: Sampling data related to specific string attribute values aids in monitoring and analyzing critical endpoints' performance and usage.
• Limit on Rate
• *Meaning: This policy limits the rate at which data is sampled based on a defined frequency.
• *Attributes: spans_per_second
• *Usage: Limiting the number of spans processed per second helps prevent system overload during traffic spikes, ensuring downstream systems aren't inundated with excessive data.

Configuration Steps

• Turn on the O11ySources, by clicking the Enable button.
• Select the sources tab and press the + button to add a new services to be monitored.
• Fill up the following details in the wizard
• *Specify Applications Group.
• *Define Application Services.
• *Configure Sampling Policies.
• *Configure Sampling Sub Policies.
• Download the client instrumentation package for the selected language from the agent configuration wizard.
• Follow the instructions in the client instrumentation package.

Metrics Collected

Name	Description	Data Type
timestamp	The exact timestamp when the log was recorded, with nanosecond precision.	DateTime64(9)
application	Name of the application associated with the trace or span.	String
service_name	Name of the service generating the span.	String
span_name	Name or description of the span representing a specific operation.	String
span_kind	Type of span, indicating its role (e.g., client, server, internal).	String
host_name	The name of the host or machine generating the span.	String
http_status_code	HTTP status code for the request, if applicable.	Nullable(Int32)
status_code	Status code for the span operation (e.g., OK, ERROR).	String
start_time	The start time of the span, with nanosecond precision.	DateTime64(9)
call_count	Total number of calls associated with the span.	UInt64
error_count	Total number of errors encountered during the span.	UInt64
duration_sum	Total sum of the durations of calls in the span, typically in milliseconds.	Float64
duration_bucket_counts	Histogram bucket counts representing the distribution of call durations.	Array(UInt64)
duration_explicit_bounds	Explicit bounds for the duration histogram buckets.	Array(Float64)
timestamp	The precise timestamp when the span was recorded, with microsecond precision.	DateTime64(6)
traceId	Unique identifier for the trace.	String
spanId	Unique identifier for the span.	String
span_parentSpanId	Identifier for the parent span of the current span.	String
span_name	Name of the span, representing the operation or request being traced.	LowCardinality(String)
span_kind	Role of the span (e.g., server, client, internal).	LowCardinality(String)
span_start_time	Start time of the span, with microsecond precision.	DateTime64(6)
span_end_time	End time of the span, with microsecond precision.	DateTime64(6)
is_root_span	Indicates whether the span is the root of the trace.	boolean
status_code	Status of the span operation (e.g., 0 = OK, 1 = ERROR).	UInt8
span_durationNano	Duration of the span in nanoseconds.	UInt64
span_attributes_site	Site associated with the span.	String
span_attributes_location	Geographic or logical location associated with the span.	LowCardinality(String)
span_attributes_library	Name of the library or module generating the span.	String
span_attributes_destination	Destination of the operation represented by the span.	String
span_attributes_destination_url	URL of the destination for the span operation.	String
span_attributes_instrumentation_library_name	Name of the library used for instrumentation.	String
span_attributes_net_peer_ip	IP address of the peer involved in the span operation.	String
span_attributes_net_peer_name	Hostname of the peer involved in the span operation.	String
span_attributes_thread_name	Name of the thread executing the span operation.	String
span_attributes_code	Code identifier for the operation.	String
span_attributes_code_function	Function name associated with the span.	String
span_attributes_code_namespace	Namespace of the code being executed.	String
resource_attributes_application	Application associated with the span resource.	LowCardinality(String)
resource_attributes_host_ip	IP address of the host executing the span.	LowCardinality(String)
resource_attributes_host_name	Name of the host executing the span.	LowCardinality(String)
resource_attributes_service_name	Name of the service associated with the span resource.	LowCardinality(String)
span_attributes_service_name	Name of the service involved in the span operation.	LowCardinality(String)
resource_attributes_k8s_node_name	Kubernetes node name associated with the resource.	LowCardinality(String)
resource_attributes_k8s_deployment_name	Kubernetes deployment name associated with the resource.	LowCardinality(String)
span_attributes_transaction	Transaction identifier for the span.	String
span_attributes_http_status_code	HTTP status code for the request associated with the span.	UInt32
span_attributes_http_host	Hostname for the HTTP request.	String
span_attributes_http_target	Target URL for the HTTP request.	String
span_attributes_http_route	Route or path for the HTTP request.	String
span_attributes_http_method	HTTP method used (e.g., GET, POST).	LowCardinality(String)
span_attributes_http_status_text	Text description of the HTTP status.	String
span_attributes_http_url	Full URL for the HTTP request.	String
span_attributes_http_user_agent	User-agent string from the HTTP request.	String
span_attributes_db_name	Name of the database involved in the span operation.	LowCardinality(String)
span_attributes_db_operation	Type of database operation (e.g., SELECT, INSERT).	LowCardinality(String)
span_attributes_db_statement	SQL statement executed in the span.	LowCardinality(String)
span_attributes_db_system	Type of database system (e.g., MySQL, PostgreSQL).	LowCardinality(String)
span_attributes_db_sql	SQL query string executed.	LowCardinality(String)
span_attributes_messaging_destination	Destination of the messaging system involved in the span.	LowCardinality(String)
span_attributes_messaging_destination_kind	Kind of messaging destination (e.g., queue, topic).	LowCardinality(String)
span_attributes_messaging_operation	Operation performed on the messaging system (e.g., SEND, RECEIVE).	LowCardinality(String)
span_attributes_messaging_system	Name of the messaging system involved.	LowCardinality(String)
span_errorSpan	Indicates whether the span contains an error.	boolean
span_exception_message	Error message associated with the span exception.	String
span_exception_stacktrace	Stack trace of the exception in the span.	String
span_exception_time	Timestamp of the exception occurrence, with microsecond precision.	DateTime64(6)
span_exception_type	Type of exception encountered in the span.	LowCardinality(String)
resource_attributes_env	Environment associated with the span (e.g., dev, prod).	LowCardinality(String)
bucket	The time bucket for aggregating data, with high precision.	DateTime64
timestamp	The precise timestamp when the data was recorded.	DateTime64
resource_attributes_application	Name of the application associated with the resource.	LowCardinality(String)
resource_attributes_service_name	Name of the service associated with the resource.	LowCardinality(String)
span_attributes_location	Geographic or logical location associated with the span.	String
resource_attributes_host_name	Name of the host associated with the resource.	LowCardinality(String)
resource_attributes_host_ip	IP address of the host associated with the resource.	LowCardinality(String)
span_attributes_http_route	Route or path accessed in the HTTP request.	String
span_attributes_http_status_code	HTTP status code for the request (e.g., 200, 404).	UInt32
origin	Origin of the request or span (e.g., client IP or identifier).	LowCardinality(String)
origin_type	Type of origin (e.g., client, service).	LowCardinality(String)
span_attributes_destination	Destination of the operation represented by the span.	String
destination_type	Type of destination (e.g., service, database).	LowCardinality(String)
latency	90th percentile latency of requests or spans in nanoseconds.	AggregateFunction(quantileExact(0.90), UInt64)
total_requests	Total number of requests recorded in the bucket.	AggregateFunction(count)
errors	Total number of errors recorded in the bucket.	AggregateFunction(count)
spanId	Unique identifier for the span.	String
resource_attributes_env	Environment associated with the resource (e.g., dev, prod).	LowCardinality(String)
bucket	The time bucket for aggregating data, with high precision.	DateTime64
resource_attributes_application	Name of the application associated with the resource.	LowCardinality(String)
resource_attributes_service_name	Name of the service associated with the resource.	LowCardinality(String)
span_attributes_location	Geographic or logical location associated with the span.	String
resource_attributes_host_name	Name of the host associated with the resource.	LowCardinality(String)
resource_attributes_host_ip	IP address of the host associated with the resource.	LowCardinality(String)
span_attributes_transaction	Identifier or name of the transaction related to the span.	String
span_attributes_http_status_code	HTTP status code for the request (e.g., 200, 404).	UInt32
num_requests	Total number of requests recorded within the bucket.	AggregateFunction(count)
num_service_requests	Total number of service-specific requests recorded within the bucket.	AggregateFunction(count)
green	Count of requests or spans categorized as "green" (e.g., good health/performance).	AggregateFunction(count)
light_green	Count of requests or spans categorized as "light green" (e.g., acceptable health/performance).	AggregateFunction(count)
yellow	Count of requests or spans categorized as "yellow" (e.g., warning or degraded state).	AggregateFunction(count)
red	Count of requests or spans categorized as "red" (e.g., critical health/performance).	AggregateFunction(count)
latency	90th percentile latency of requests or spans in nanoseconds.	AggregateFunction(quantileExact(0.90), UInt64)
max_latency	Maximum latency of requests or spans in nanoseconds.	AggregateFunction(max, UInt64)
num_errors	Count of errors based on a specific condition within the bucket.	AggregateFunction(countIf, UInt8)
spanId	Unique identifier for the span.	String
num_service_errors	Count of errors specific to the service within the bucket.	AggregateFunction(count, UInt64)
resource_attributes_env	Environment associated with the resource (e.g., dev, prod).	LowCardinality(String)
resource_attributes_application	The name of the application associated with the resource.	LowCardinality(String)
resource_attributes_service_name	The name of the service associated with the resource.	LowCardinality(String)
span_attributes_transaction	Identifier or name of the transaction related to the span.	String
span_attributes_http_status_code	HTTP status code for the transaction or span (e.g., 200 for success, 404 for not found).	UInt32
resource_attributes_host_name	The hostname of the resource or system involved in the transaction.	LowCardinality(String)
bucket	The time bucket for aggregating metrics.	DateTime
timestamp	The specific timestamp of the record.	DateTime
resource_attributes_application	The name of the application associated with the resource.	LowCardinality(String)
resource_attributes_service_name	The name of the service associated with the resource.	LowCardinality(String)
span_attributes_location	The geographic or logical location of the span.	LowCardinality(String)
resource_attributes_host_name	The hostname of the resource involved in the transaction.	LowCardinality(String)
resource_attributes_host_ip	The IP address of the resource involved in the transaction.	LowCardinality(String)
span_attributes_transaction	Identifier or name of the transaction related to the span.	String
span_attributes_http_status_code	HTTP status code associated with the span (e.g., 200 for success, 404 for not found).	UInt32
resource_attributes_env	The environment in which the resource is operating (e.g., production, staging).	LowCardinality(String)
num_requests	The total number of requests made in the bucket.	AggregateFunction(count)
num_service_requests	The total number of service-level requests in the bucket.	AggregateFunction(count)
green	Count of requests classified under the "green" performance threshold.	AggregateFunction(count)
light_green	Count of requests classified under the "light green" performance threshold.	AggregateFunction(count)
yellow	Count of requests classified under the "yellow" performance threshold.	AggregateFunction(count)
red	Count of requests classified under the "red" performance threshold.	AggregateFunction(count)
latency	The 90th percentile latency of the requests in the bucket.	AggregateFunction(quantile(0.9), UInt64)
latency50	The 50th percentile (median) latency of the requests in the bucket.	AggregateFunction(quantile(0.50), UInt64)
latency75	The 75th percentile latency of the requests in the bucket.	AggregateFunction(quantile(0.75), UInt64)
latency95	The 95th percentile latency of the requests in the bucket.	AggregateFunction(quantile(0.95), UInt64)
latency99	The 99th percentile latency of the requests in the bucket.	AggregateFunction(quantile(0.99), UInt64)
max_latency	The maximum latency observed in the bucket.	SimpleAggregateFunction(max, UInt64)
num_errors	The total number of errors encountered in the bucket.	AggregateFunction(count, UInt64)
num_service_errors	The total number of service-level errors encountered in the bucket.	AggregateFunction(count, UInt64)
latency50_rootspan	The 50th percentile (median) latency of the root span.	AggregateFunction(quantile(0.5), UInt64)
latency75_rootspan	The 75th percentile latency of the root span.	AggregateFunction(quantile(0.75), UInt64)
latency90_rootspan	The 90th percentile latency of the root span.	AggregateFunction(quantile(0.9), UInt64)
latency95_rootspan	The 95th percentile latency of the root span.	AggregateFunction(quantile(0.95), UInt64)
latency99_rootspan	The 99th percentile latency of the root span.	AggregateFunction(quantile(0.9), UInt64)
spanId	Unique identifier of the span associated with the metrics.	String