Unstructured data is growing at breakneck speed, presenting opportunities for faster and smarter decision making. In fact, industry experts predict that unstructured data will account for 80% of all data in the next two to three years. From multimedia files, customer transactions, internet of things (IoT) sensor data to data from emerging technologies such as autonomous vehicles and deep learning models, this presents a wealth of next-generation business intelligence (BI) and operational intelligence (OI) waiting to be unearthed.
Data volumes and complexity will only continue to grow as organisations ramp up investments in digital capabilities. The trouble arises, however, when they are not equipped to store and manage this data efficiently.
The challenge of managing unstructured data
Given the sheer volume, variety and velocity of unstructured data, the challenge of managing this data is two-fold: ensuring both accuracy and speed of insights.
In a study by O’Reilly, too many data sources and inconsistent data
were cited by 60 per cent of IT and data professionals as the most common data quality issues. If the storage architecture cannot cope with the speed at which data is growing, this can result in bottlenecks, slowing down time-to-insights. Data stored in silos due to legacy processes and the multitude of data sources may also affect data quality.
Organisations will need sufficient storage and processing power to extract insights from the constantly growing data. But it’s not just simply about adding storage capacity, which may empower the unbridled, inefficient storage of data. This is unsustainable in the long run, both from the business and environmental perspectives. Not only would this rack up expenses, it would also be tricky to rein in the energy costs of running the infrastructure.
Unravelling the complexity of unstructured data
The good news: there are ways to cope with the demands of unstructured data.
Organisations will need to take a closer look at their data management strategy to ensure they are well-positioned to support smarter, data-driven decision-making. It’s important not to ignore the potential of unstructured data simply because their current IT infrastructure cannot support it. As the pandemic has shown, businesses that are able to adapt and pivot rapidly are more resilient, and data will be important in driving insights for productivity improvements and innovation.
IT leaders can start with auditing their data and storage infrastructure. It is critical to have flexible, efficient data architecture that makes it simple to consolidate and access data, and scale when needed, to analyse unstructured data effectively. This may require a migration away from legacy storage technologies, such as mechanical disk and tape, as these cannot cope with the velocity and variety of modern unstructured data. Organisations will also have to ensure that their data infrastructure is equipped to meet business priorities, such as sustainability and data protection.
For example, in 2021, Nanyang Technological University’s High Performance Computing Centre (HPCC) was able to expand its supercomputing capabilities
to meet data- and computationally-intensive requirements, while upholding its commitment to sustainability, after adopting a unified fast file and object storage (UFFO) platform backed by high-speed, ultra-dense flash technology.
For organisations with data stored across on-premise, private and public cloud environments, consolidating data, infrastructure and applications across different cloud platforms into a single management interface can help reduce the threat of information chaos rising from data silos. A robust multi-cloud strategy is essential at a time when speed is everything. This would enable organisations to leverage the flexibility and scalability of cloud, without compromising time-to-insights.
Lastly, the responsibility of ensuring data quality should not just fall on the shoulders of data scientists. It will require a cultural shift across the organisation, to ensure that all employees understand the importance of data and are actively involved in ensuring “clean” data for accurate analysis.
The catalyst for business innovation
In a digital world, organisations that are able to leverage the untapped potential of unstructured data will be better positioned to improve their operations and pursue new business opportunities. The right data management frameworks can allow for greater efficiency in processes and relevant insights. By equipping themselves with flexible, efficient tools, and inculcating a data-first mindset across employees, organisations will be well-equipped to unlock the infinite possibilities unstructured data has to offer.
Wei Meng Ng, is the Country Manager, Singapore, Pure Storage