Skip to main content
Version: NG-2.14

Managing ContextStreams

Manage I/O Streams​

View I/O Streams​

Click on the I/O stream name to view the particular I/O stream details.

There are 3 tabs under the 'View' option (i.e., I/O Streams Info, Readers, Metrics).

I/O Streams Info​

This tab provides comprehensive information about your I/O streams. By clicking on it, you can access details such as stream configurations, data sources, and any transformations applied.

Readers​

The 'Readers' tab lists configured readers associated with the respective I/O Stream. It offers insights into how data is consumed from this I/O stream. For a detailed visual guide, check the below snapshot.

Metrics​

Metrics are vital for performance analysis and optimization. Navigate to the 'Metrics' tab to access real-time performance metrics, ensuring efficient operation of your I/O streams. The tab displays crucial performance indicators for a specified 30-minute period, including 'Data In' and 'Data Out' metrics, offering insights into data flow within your streams. Additionally, the 'Total Compressed Data' provides a snapshot of compression efficiency.

'Data In' reflects the volume of incoming data ingested into the I/O Stream in the last 30 minutes, indicating the data ingestion rate. 'Data Out' represents the amount of data processed by the stream in the same timeframe, signifying data processing efficiency. Monitoring fluctuations in both metrics aids in assessing overall data flow dynamics and system performance.

  • If 'Data In' is 0, it indicates no data ingestion.
  • If 'Data Out' is 0, no streamed data is being sent to the output.
  • If 'Data In' and 'Data Out' differ, it could imply streaming issues.

The 'Total Compressed Data' represents the cumulative size of compressed data processed by the streams, offering insights into compression effectiveness.

During debugging, the 'Refresh' button fetches real-time metric updates, providing an updated view of current performance and aiding in promptly identifying any issues.

Edit I/O Streams​

Click on the Edit button to modify any details associated with the added I/O stream.

After clicking the Edit button, you will be directed to the editing page where you can make desired changes to the I/O stream configuration. Update the fields as needed, such as the description, partition settings, retention period, and other relevant parameters.

note

For more detailed instructions on parameters to be provided in I/O Streams, you can refer here.

Once you've made the necessary adjustments, click the Save button to save the changes.

Delete I/O Streams​

Select the I/O streams that you want to delete and click on the Delete button. Confirm by clicking on the Delete button. The selected I/O stream will be removed from the list.

Manage Data Pipelines​

View Pipeline​

Click on the pipeline name to view the particular pipeline details.

There are 4 tabs under the 'View' option. You can get the details by clicking on each of them. Following are the tabs:

  1. Flows
  2. Settings
  3. Metrics
  4. View Logs

Flows​

Under Flows, the pipeline flow chart is visible, and you can perform the following action

  • Start: Start the pipeline.
  • Stop: Stop the pipeline.
  • Restart: Restart the pipeline.

Settings​

You can view the Pipeline information from here.

For more detailed instructions on parameters in Data Pipeline Settings, you can refer here.

Metrics​

In the Metrics tab, you can find a list of essential metrics providing insights into the performance of the data pipeline, particularly focusing on the data flow.

Metrics are vital for performance analysis and optimization. Navigate to the 'Metrics' tab to access real-time performance metrics, ensuring efficient operation of the data pipeline. The tab displays crucial performance indicators for a specified 30-minute period, including 'Data In' and 'Data Out' metrics, offering insights into data flow within the pipeline.

'Data In' reflects the volume of incoming data ingested into the data pipeline in the last 30 minutes, indicating the data ingestion rate. 'Data Out' represents the amount of data processed by the pipeline in the same timeframe, signifying data processing efficiency. Monitoring fluctuations in both metrics aids in assessing overall data flow dynamics and system performance.

  • If 'Data In' is 0, it indicates no data ingestion.
  • If 'Data Out' is 0, no streamed data is being sent to the output.
  • If 'Data In' and 'Data Out' differ, it could imply streaming issues. Indeed, a thorough analysis is required to determine if the discrepancy between 'Data In' and 'Data Out' is a result of intentional transformations or if there's an underlying issue that needs attention.

During debugging, the 'Refresh' button fetches real-time metric updates, providing an updated view of current performance and aiding in promptly identifying any issues.

View Logs​

This tab displays logs related to the pipeline's activities. To debug for deeper analysis, logs can be analyzed here. These logs provide a comprehensive view of the pipeline's execution, helping identify and address issues effectively.

Edit Pipeline​

To Edit the pipeline, click on the Edit button located under the Actions column. When you click on it, you will be redirected to the following page.

Working with the Pipeline Editor is explained in detail here.

Pipelines Info​

Another tab, while you are editing the pipeline, is the “Pipeline Info” under the edit section. You can configure the advanced pipeline settings from here. For more detailed instructions on parameters in Data Pipeline Settings, you can refer here.

Delete Pipeline​

Select the pipeline you want to delete and click on the Delete button. You can delete the pipeline either using the Delete button at the top or under the Action column.

Click on the Delete button to confirm the delete action. The selected pipeline will be deleted and removed from the list.

Manage DataStore Connectors​

There are 3 actions for all the connectors listed: View, Edit, and Delete.

View DataStore Connector​

By clicking on the Datastore connector name, it shows the connector details where you can also see the total Data (messages read and sent). There are two tabs under the view option:

  1. DataStore Connectors Info
  2. Metrics

DataStore Connectors Info​

Here, the Datastore Connector details are visible, and you can also configure the advanced settings options from here.

You can perform the following action:

  • Start: Start the connector.
  • Stop: Stop the connector.
  • Restart: Restart the connector.

Metrics​

Navigate to the 'Metrics' tab to access a comprehensive list of metrics for your DataStore Connectors.

Metrics are vital for performance analysis and optimization. Navigate to the 'Metrics' tab to access real-time performance metrics, ensuring efficient operation of the DataStore Connector. The tab displays crucial performance indicators for a specified 30-minute period, including 'Data In' and 'Data Out' metrics, offering insights into data flow within the streams.

'Data In' reflects the volume of incoming data ingested into the connector in the last 30 minutes, indicating the data ingestion rate. 'Data Out' represents the amount of data processed by the connector in the same timeframe, signifying data processing efficiency. Monitoring fluctuations in both metrics aids in assessing overall data flow dynamics and system performance.

  • If 'Data In' is 0, it indicates no data ingestion.
  • If 'Data Out' is 0, no streamed data is being sent to the output.
  • If 'Data In' and 'Data Out' differ, it could imply streaming issues.

During debugging, the 'Refresh' button fetches real-time metric updates, providing an updated view of current performance and aiding in promptly identifying any issues.

Edit DataStore Connector​

The 'Edit Connector' option enables you to configure specific settings and parameters for your connector. Make the desired changes to the configurations and click Save to apply them.

For detailed information on the fields and their configurations, please refer to the steps mentioned during the creation of the connector (ES Sink Connector, and JDBC Sink Connector).

Delete DataStore Connector​

Select the Datasource Connector you want to delete and click on the Delete icon. Click on the Delete button to confirm the delete action. The selected DataSource Connector will be deleted and removed from the list.