Data Retention

Introduction

In vuSmartMaps, administrators can efficiently manage data retention settings through a user-friendly graphical interface. This comprehensive guide provides an overview of how to define retention durations for Hot, Warm, and Archived storage storages, optimizing system performance, compliance, and data management.

Data Retention Hierarchies

vuSmartMaps supports a two-storageed data storage strategy, subdivided into three retention categories:

Live Data

Hot Storage

Hot Storage serves immediate and frequent access needs for recent and critical data. This storage offers the fastest access speeds, making it ideal for live queries, active analytics, and real-time operations.

Configurable Retention: Administrators can set the number of days data remains in Hot Storage before being transitioned to the next storage, ensuring optimal performance and availability.

Warm Storage

Warm Storage provides an intermediate storage for data that is less frequently accessed than Hot Storage but remains relevant and needs moderate-speed access. This storage is suitable for data that is still in use but not as frequently queried or analyzed as Hot data.

Configurable Retention: You can specify how long data stays in Warm Storage before being moved to the Archive storage.

Archived Data

Archived Storage is used for long-term retention of historical or infrequently accessed data. Data in this storage is not immediately available for live queries but is retained for reference, compliance, and long-term storage needs. Administrators can bring archived data back to the Warm storage when necessary using the restore functionality.

Archival Strategy Configuration

Administrators have the flexibility to determine when data transitions from Live (Hot/Warm) storage to the Archive storage. This can be achieved through the "Configure Archival Strategy" option. Clicking on this button opens a modal that provides two options for defining the archival strategy:

Archive Data Daily

In this strategy, data is archived daily starting from the previous day of ingestion into Live Storage (Hot/Warm). For example, if today is the 25th, data starting from the 24th will be archived. Data from the current day will not be archived. It remains in the Archive storage for a user-defined period before deletion, ensuring a continuous and incremental movement of data to the Archive for timely retention and deletion.

note

The retention period for this strategy begins from the day after the data is ingested into Hot or Warm storage. Data will be archived daily and retained in the Archive storage for the specified number of days before being permanently deleted.

Archive on Live Data Retention Period Expiry

This strategy archives data once its configured retention period in Live Storage (Hot/Warm) has ended. From that point, the data is retained in the Archive storage for a user-defined duration before deletion.

note

1. The retention period for this strategy is calculated from the day data is configured to be removed from Live Storage (Hot or Warm). To ensure data continuity, archiving starts a few days earlier, based on a configurable environment variable (DAYS_TO_SYNC_COLD_STORE). By default, this value is set to 5 days.

   For example, if today is the 25th and the retention period for Hot/Warm storage is configured to end on the 20th, data from the 20th will start being archived on the 15th (5 days earlier). From the 20th, the data will be retained in the Archive storage for the configured duration before it is deleted.

2. Every day at 10:30 PM, data is automatically moved from Hot/Warm to Archived storage. Older data is then deleted from Archived storage based on the retention settings.

Retention Periods for Data storages

You can configure data retention periods for each storage to suit your organization's needs:

1. Hot Storage (Days): Specifies the number of days data remains in Hot Storage for fast and frequent access.
2. Warm Storage (Days): Defines the duration for data retention in Warm Storage, which offers moderate-speed access before being moved to Archived Storage.
3. Archived Storage (Days): Indicates how long data stays in Archived Storage after being moved from Live Storage. This storage provides long-term storage and retrieval capabilities.

Example Retention Configuration

Example 1

Consider a configuration with the following retention periods:

• Hot Storage: 10 days
• Warm Storage: 20 days
• Archived Storage: 30 days

Explanation:

• Data is retained in Hot Storage for 10 days from the time it is ingested.
• After 10 days, it transitions to Warm Storage and stays there for the next 20 days from the initial ingestion date.
• Once the Warm Storage retention period ends, the data moves to the Archived Storage storage and is retained for 30 additional days before deletion. It depends on the global archival settings.

Example 2

Consider a configuration with the following retention periods:

• Hot Storage: 10 days
• Warm Storage: 5 days
• Archived Storage: 20 days

And today is 21st, the system calculates:

1. Data ingestion cutoff as:
   Today−(Hot+Warm)=21−15=6
   Data from 6th to 8th will be archived (3 days earlier, as per DAYS_TO_SYNC_COLD_STORE).
2. Archived data remains in Cold storage for 20 days.

By categorizing data into Hot, Warm, and Archived Storage, vuSmartMaps enables administrators to strategically align their data management with the nature and frequency of access, thus optimizing storage efficiency. Data retention settings are designed to manage data across these states effectively. Optimizing data storage is crucial, as it directly influences costs. The configuration of data retention settings is flexible, allowing configurations to suit user-specific requirements.