What are some of the biggest challenges companies face when it comes to data governance?
One of the biggest challenges companies face today is balancing the breadth and depth of secure data. On one hand, organizations want to generate and share as much data as possible with their clients; However, this ambition comes with significant risks.
- Cost Management: Managing and storing large volumes of data can be financially burdensome, especially given the variable pricing models of many cloud service providers. These costs can adversely affect a company’s bottom line.
- Performance Issues: A vast amount of data stored on a suboptimal management platform can lead to slow data retrieval, negatively impacting user experience.
- Security Risks: More data often means more confidential and sensitive information. If a company’s infrastructure is compromised or if data is accidentally shared with unauthorized users, the risks increase significantly.
- Enhanced Due Diligence: Enhanced due diligence on data sources will be even more important as we move forward. Companies will have to know dig into to understand the genesis of external data to make sure their data is accurate and appropriately sourced, how we do this will evolve, especially as more firm use AI to create or enhance data sets. The knee jerk reaction is to use AI to validate the incoming data, which causes a recursive paradox.
Ultimately, the challenge lies in making informed decisions about how much data to manage, what to prioritize, how long to retain data, when and how to expose it, and ensuring that the data is accurate and secure. Finding the right answers to these questions is often more complex than it seems.
How do you think data governance will evolve in the next decade?
I believe we will see significant changes in data governance over the next 10 years:
- Data Utilization from Source Systems: Legacy systems have struggled to expose their data via APIs, but with the rapid advancement of artificial intelligence, particularly generative AI, it will become easier to transform and utilize data directly from source systems. This shift will reduce the number of places data is stored, thereby minimizing exposure. However, it will also necessitate enhanced security for traditionally “back-of-house” systems.
- Evolving Regulations: As data breaches become more frequent, regulations will need to adapt quickly. Tools that enhance data utilization can also be exploited by bad actors, making it crucial for regulatory agencies to balance the need for safety with the promotion of innovation. This may be an opportune moment for a global set of regulations within specific industries to streamline efforts and reduce redundancy.
- Consumer Rights: There will likely be a continued push for greater consumer rights regarding data management. The concept of the “right to forget” may become more complex as data, metadata, and the relationships between them blur, leading to ongoing challenges in defining the boundaries of customer data.
- Necessity of Data Dictionaries: With expanding data sets and access, comprehensive “data dictionaries” will become essential. This could take the form of extensive documentation or innovative visualizations of data lineage and architecture, likely leveraging the same AI technologies.
- Emergence of New Roles: We can expect the creation of various new data governance roles beyond just IT. New types of business analysts, data scientists, and product owners will emerge as companies increasingly treat data as a product. This trend will shift how organizations market themselves, with a focus on their data offerings followed by additional services.
- Increased Costs and Opportunities: While data governance will likely become more expensive, savvy companies will find ways to monetize their data in innovative ways, offsetting these costs and enhancing their competitive edge.