Data Retention
Introduction
In vuSmartMaps, administrators can efficiently manage data retention settings through a user-friendly graphical interface. This comprehensive guide provides an overview of how to define retention durations for Hot, Warm, and Archived storage storages, optimizing system performance, compliance, and data management.
Data Retention Hierarchies
vuSmartMaps supports the data storage strategy, subdivided into three retention categories - Hot, Warm, and Archived.
Live Data
Hot Storage
Hot Storage serves immediate and frequent access needs for recent and critical data. This storage offers the fastest access speeds, making it ideal for live queries, active analytics, and real-time operations. Administrators can set the number of days data remains in Hot Storage before being transitioned to the next storage, ensuring optimal performance and availability.
Warm Storage
Warm Storage provides an intermediate storage for data that is less frequently accessed than Hot Storage but remains relevant and needs moderate-speed access. This storage is suitable for data that is still in use but not as frequently queried or analyzed as Hot data. You can specify how long data stays in Warm Storage before being moved to the Archive storage.
Archived Data
Archived Storage is used for long-term retention of historical or infrequently accessed data. Data in this storage is not immediately available for live queries but is retained for reference, compliance, and long-term storage needs. Administrators can bring archived data back to the Warm storage when necessary using the restore functionality.
Every day at 10:30 PM, data schedule runs to move from Hot/Warm to Archived storage. Older data is then deleted from Archived storage based on the retention settings.
The configuration of data retention settings is flexible, allowing configurations to suit user-specific requirements.
Archival Strategy Configuration
Administrators can configure how data transitions between Live Storage (Hot and Warm) to Archived Storage using two archival strategies: Archive Data Daily and Archive on Live Data Retention Period Expiry. These strategies define how data is backed up, moved, and retained across the different storage tiers. This can be achieved through the "Configure Archival Strategy" option. Clicking on this button opens a modal that provides two options for defining the archival strategy:
Archive Data Daily
In this strategy, data is archived daily, starting from the previous day (N-1). Data in Hot and Warm storage is backed up to Archive Storage according to the retention periods configured for each storage tier.
For example: If the retention periods are configured as:
- Hot: 3 days
- Warm: 1 day
- Archive: 4 days
Here’s how the data flows:
- Data will reside in Hot Storage for 3 days and is backed up to Archive Storage during this time.
- After 3 days, the data moves from Hot Storage to Warm Storage and stays there for 1 day.
- Once the data's 1-day retention in Warm Storage ends, it moves to Archive Storage, where it is retained for 4 more days.
Example Timeline (assuming today is the 20th):
- Hot Storage will have data from the 17th to 19th (3 days).
- Warm Storage will have data from the 16th (1 day).
- Archive Storage will have data from the 12th to 19th (4 extra days in Archive).
In total, the data will be retained for 3 + 1 + 4 = 8 days across these storage tiers, with 4 days of backup stored in Archive.
The retention period for each storage (Hot, Warm, Archived) is calculated separately, and the backup is created in the Archive from Hot and Warm before the data moves. The Archive storage ensures the preservation of data for additional days beyond the live data retention.
Archive on Live Data Retention Period Expiry
In this strategy, there is no backup maintained in the Archive. Once the configured retention period for Hot and Warm data has expired, the data moves directly to the Archive storage.
For example: If the retention periods are:
- Hot: 3 days
- Warm: 1 day
- Archive: 4 days
The data will move as follows:
- Data will stay in Hot Storage for 3 days.
- After 3 days, it will move to Warm Storage for 1 day.
- Once the Warm Storage retention period expires, the data will directly move to Archive Storage for 4 days.
This strategy does not involve the backup step. Data simply moves between Hot, Warm, and Archive based on the configured retention periods.
The retention period for this strategy is calculated from the day the data is configured to be removed from Hot or Warm storage. No additional backup is stored in the Archive. Data will be retained in Archive only for the configured retention period.
Edge Case: Zero Retention Period for Archival
If the Archive Retention Period is set to 0, no archival functionality is applied. Here’s how this scenario works:
Example:
- Hot retention: 3 days
- Warm retention: 1 day
- Archive retention: 0 days
For the "Archive Data Daily" strategy, 4 days of backup will still be maintained, but no extra days will be retained in Archive Storage. Essentially, it will only duplicate the data for the backup period, and once archived, it will not stay in Archive.
For the "Archive on Live Data Retention Period Expiry" strategy, nothing will be moved to Archive Storage. Data will only remain in Hot for 3 days and Warm for 1 day, and once those retention periods expire, the data is deleted.
