Data Storage

Introduction

Efficient data storage management is crucial for ensuring seamless operations and optimal performance within vuSmartMaps. With a dedicated module for HyperScale Data Store, administrators gain comprehensive control over organizing and accessing data.

The HyperScale Data Store offers a tiered approach, dividing data into Hot, Warm, and Archived storage states, each serving distinct purposes in data management and access. From restoring archived data to archiving and retrieving indices, vuSmartMaps empowers users with intuitive tools and functionalities for streamlined data management, facilitating enhanced decision-making and productivity.

In the HyperScale Data Store, data is organized into three distinct tiers: Hot, Warm, and Archived Data Storage.

1. Hot Data Storage: This tier is dedicated to storing recently accessed and critical data, providing high-speed access for immediate retrieval. Data in this state is maintained as per policy defined in data retention settings, ensuring efficient data management.
2. Warm Data Storage: Intermediate storage for data accessed less frequently but remains relevant. It offers moderate-speed access to accommodate less frequent yet still pertinent data. Data is regularly archived from Hot/Warm to Archived state as per data retention policies.
3. Archived Data Storage: Archived Data Storage provides a long-term archival and storage solution for historical or rarely accessed data. While access speed is lower compared to Hot and Warm storage, Archived Storage ensures the preservation of critical historical information. Data stored in the Archived state cannot be queried directly and must be restored before it can be accessed. However, the data can be restored if required through the Data Storage module, offering a comprehensive data management solution that ensures data availability when needed while optimizing storage costs.

Managing Data Storage

The Data Storage page can be accessed from the platform left navigation menu by navigating to Data Management > Data Storage.

The landing page on clicking the Data Storage looks like the following:

In the HyperScale Data Store, three dedicated tabs are available for Hot, Warm, and Archived Data Storage.

In the Hot, Warm, and Archived data storage tabs, you can find a list of available data tables in the Hot state, showing their name, size, and last modified time. Additionally, for each storage tab, memory utilization is displayed at the top, showing both the actual memory used and its percentage of the total available storage. This memory utilization is color-coded for clarity, with different thresholds for each storage state:

1. Hot & Warm Storage:
   1. Warning Threshold: 50% of the total memory
   2. Critical Threshold: 80% of the total memory
2. Archived Storage:
   1. Warning Threshold: 80% of the total memory
   2. Critical Threshold: 90% of the total memory These thresholds are visually represented with color coding to alert administrators of the current memory status in each storage state.

note

1. Hot storage is mandatory and cannot be disabled.
2. If the Warm or Archive storage is not enabled, a message stating, "Disk is disabled in this system" will be shown.
3. If storage is enabled but data is inaccessible, the actual error message will be displayed, providing more detailed information.

Additionally, you can view System Tables by selecting the Show System Tables radio button. By default, system tables are hidden as they are used internally by the platform. Enabling this option will display system tables alongside regular data tables, providing a complete view of the data storage in use.

Restoring Archived Data

Archived data can be restored to a warm state if necessary, making the table queryable. It's important to note that the restored data will reside in the warm state for the next 7 days, after which it will be permanently deleted, regardless of the retention settings. The restored data will be listed on the Warm tab with the restored- prefix. A restored table would look like this restored-<table-name>.

note

As part of the nightly schedule, the Time-to-Live (TTL) settings of the corresponding table are updated. This update ensures that data is retained for the next 7 days from the point of restoration, providing sufficient time for archival continuity and retention adherence. The default TTL is set to 7 days, but it is configurable in the data retention module. A default data retention policy for Restored Tables is available, which users can update according to their needs.

To restore archived data, follow these steps:

1. Use the Restore button under the Actions column of the respective row in the Archive tab.
2. Clicking on the Restore button will open a pop-up window. Click on Initiate Restore in the pop-up to restore the data from the archived state to the warm state.
3. Similarly, for bulk restoration of data tables, select the respective rows, and use the Restore button located at the top of the table.

note

Restrictions on Initiating Restore: Users will be unable to initiate a restore operation for a file or set of files under the following conditions:

1. Restore Already in Progress: If a restore operation is currently in progress according to system information.
2. Data Already Present in Hot/Warm: If the data in the table is already present in Hot or Warm storage tiers. This is determined by checking for the presence of a table with the same name in Hot or Warm storage. Additionally, if the data is within the retention period specified in the data retention settings and is already stored in Hot or Warm storage.
3. Insufficient Warm Storage Space: If initiating the restore operation would result in only 5% of the Warm storage being available as free space after restoration.