How technology can transform dark data into actionable insights
It was British mathematician and consumer insights guru Clive Humby who coined the phrase ‘data is the new oil’. Since he first uttered those prophetic words back in 2006, data has become one of the most valuable commodities available to businesses worldwide. So much so that Indian billionaire industrialist Mukesh Ambani and others have likened data now to be more like oxygen than oil. No longer simply a fuel, it has become an existential necessity for organisations.
Its value is illustrated by some truly incredible numbers. The year 2020 saw the amount of data created reach a new high, with that expected to grow to more than 180 zettabytes by 2025. To put that figure into context, just one zettabyte is enough to store 30 billion 4K movies, 60 billion video games, or 7.5 trillion MP3s.
From the sophisticated algorithms of tech giants like Google and Facebook, to the more modest special offers and questionnaires being emailed out by the local coffee shop, acquiring and interpreting data has become a vital component of the way organisations operate.
It stands to reason then, that with so much to be gained by acquiring data, that it would be a monumental waste of time, opportunity and money for an organisation to let it slip through their fingers. But when it comes to dark data, that is exactly what is happening.
Data vs actionable information
Information is vital to organisations for a whole range of reasons, from marketing and customer retention, to compliance, resource monitoring, fraud prevention and performance monitoring – but it has to be made actionable.
Data, in essence, is simply raw information – often something as simple as numbers – and it is harvested like never before. But it is useless unless it is collected and stored in a way that enables it to be interpreted further down the line. Only by interpreting that data can it be used to inform an organisation’s decisions.
But first that data needs to be collected. Organisations often invest heavily in acquiring that data, yet they are often sitting on a goldmine of untapped data resources which they may not know they have.
With advances in technology such as digital twins, artificial intelligence and machine learning moving from the lab into mainstream use, the need to harness that unutilised operational data is becoming even more vital to business success.
What is dark data?
Much as physicists believe most of the universe is made up of invisible ‘dark matter’, so is most of the data in an organisation made up of the equally invisible ‘dark data’.
Dark data is essentially the information an organisation acquires during its regular, day-to-day activities but does not utilise for other purposes, for example, analysing customer behaviour or informing direct marketing campaigns.
The figures are huge. Much like most of an iceberg is below the water, most of an organisation’s data is dark. IBM estimates it to be as high as 80 per cent.
The Impact of dark data
In a world where every bit of information is precious, not utilising any of it can have an impact on an organisation in terms of lost insights and revenue, while proving a barrier to innovation and the refinement of existing processes.
Gaps in data are a source of concern to companies. IDC found that even though companies value data discovery and integrity, between 30%-50% of organizations said they were not satisfied with the way they currently used data.
This is having a big impact on existing industries as well as nascent ones which are due for significant growth, but could be heavily encumbered by a lack of data.
A report by McKinsey for example, found that most Internet of Things (IoT) data is not used and gave the example of an oil rig that has 30,000 sensors, but on average only 1% of that data is examined. This is because the information is mostly used to detect and control anomalies rather than for optimization and prediction.
For an industry with such growth potential – the same report predicts IoT could account for up to $11.1 trillion (11%) of the world economy by 2025 – the potential implications for lost revenue and innovation, as well as gaps in safety and compliance, are huge.
The importance of ESG data
It is not just operational effectiveness, safety and revenue which data plays a crucial role in, but also compliance and eSG – environmental, social and corporate governance (ESG). It has become increasingly vital in recent years and that importance has only been sped up during the Covid pandemic and the heightened focus on the environment.
A company may want to do the right thing – it may already be doing the right thing – but it also needs the data to prove it, because how an organisation is able to report on and prove its adherence to ESG, impacts on investment opportunities and consumer choice.
According to research by Ernst & Young, 90% of investors surveyed said that since the pandemic they attach greater importance to corporates’ ESG performance when it comes to their investment strategy and decision-making. A total of 86% percent of investors surveyed said that a corporate having a strong ESG programme and performance would have a ‘significant and direct impact’ on analyst recommendations.
This was further reinforced at the recent 26th United Nations Conference of the Parties (COP26) the IFRS Foundation – which oversees accounting standards in more than 140 countries – announced the formation of an International Sustainability Standards Board (ISSB).
ISSB will sit alongside International Accounting Standards Board (IASB), but whereas the latter oversees financial transparency, ISSB is responsible for transparency of sustainability practices. In essence, the move gave equal weight to the importance of being able to demonstrate both financial and sustainability credentials. The IFRS said international investors with global portfolios wanted transparent and reliable reporting by companies on matters relating to the environment, society, climate, and governance. Countries such as the UK have already said it will put ISSB at the centre of its own sustainability reporting requirements.
Consumers too are also watching ESG closely. A report from The Economist Intelligence Unit, commissioned by WWF, revealed a 71% rise in global online searches for sustainable goods over the past five years.
As well as the loss of data which can assist in climate change compliance, the storing of dark data has a more urgent, real-world impact on the environment. Veritas estimated that 6.4m tons of CO2 will be unnecessarily pumped into the atmosphere as a result of powering storage of dark data.
The implications are clear, being able to gather ESG data is crucial both now and in the future. The idea of up to 80 per cent of it being contained on company servers but not actionable shows the huge potential for waste.
Solutions
So what are the solutions for transforming useless dark data into actionable insights that enable organisations to thrive and get ahead? Technology is key, with innovations such as digital twinning and graph technology playing a central role in turning data into actionable insights.
Graph technology is predicted by Gartner to be one of the five most impactful, emerging technologies of the coming year and will be a critical enabler to smart cities and improved productivity.
Graph data technologies enable the relationship between data to be examined, rather than simply the data itself which can become siloed and fall into the trap of becoming dark. It enables an organisation to make sense of how data connects different organisations, transactions or people, providing models which allow that connectedness of data to be explored in detail.
One example is social media marketing. Graph data can provide insights on how influencers ‘influence’ through social media platforms by analysing how data moves between them and their followers. While the theory underpinning the technology has been around for decades, the computational power needed to make it work effectively is only now being realised, making it a potential game changer for the problem of dark data and its application in scientific and engineering fields – those industries that will shape the future of our health and environment in the years to come.
Barriers
So if there are solutions, like graph technology, to solving the problem of dark data, why are organisations reticent when it comes to taking action?
A study by Ernst and Young found the principal barriers were:
- Too much data
- Lack of skills
- Lack of collaboration and sharing
Their research found that 32% of the corporate respondents they surveyed admitted to being overwhelmed by data, while 50% said they had concerns about poor data, while the same percentage also had concerns over ROI.
More broadly, one of the problems is that, despite the vital importance of data science, so few organisations have dedicated data science teams which can be vital to interpreting complex information. Without these inhouse skills, traditional more complex data science approaches and tools cannot be deployed widely across the organisation.
These data science experts are in high demand and can command large salaries, recruiting and retaining them can pose challenging for some companies.
There are also both cultural and operational issues to consider around data sharing and collaboration. Sharing data has been anathema to some businesses, not only externally but internally too, with siloed ways of working has prevented cross-collaboration between departments. Sharing externally has also brought concerns about privacy and data loss.
But emerging technology such as Blockchain mean that companies will be able to better manage how they share data with much greater confidence.
Awareness of the solutions is also a potential barrier. Technology such as graph data and digital twinning are not always on the radar of some organisations, despite the major impact they can have.
Conclusions
It is clear that despite the soaring importance of data to revenue, operations, compliance and reputation, many organisations are suffering due to a lack of being able to harness what they already have right at their fingertips.
Data is collected all the time but if it is simply sitting there on servers doing nothing, it is not only contributing to climate change and waste, but also depriving the organisation or potential opportunities to advance and innovate – and grow.
New technological innovations which help make sense of that dark data and turn it into actionable, insightful information without the need for whole departments of data scientists are the key to eliminating this wasted potential.
Article provided orginally by Wevolver: Bridging The Time And Intelligence Chasm (wevolver.com)