Data chiefs in the Asia-Pacific region are seriously looking at deploying artificial intelligence, according to a international survey of 600 global data leaders of Informatics. India leads the way in the region: 75% of respondents have already adopted generative AI.
However, APAC respondents face obstacles around data management for AI. These include data fragmentation amid an increasing number of data sources, the quality of data available for AI and the incorporation of data governance that is robust enough for the AI challenge.
Richard Scott, senior vice president of Asia-Pacific and Japan at Informatica, said data literacy is important to support organizational data management. Scott recommended implementing a cloud data architecture from the beginning and focusing on people, processes and technology.
AI is driving a parallel approach to data management
APAC data leaders said the ability to deliver reliable and consistent data suitable for generative AI (40%) was the top priority of the 2024 data strategy, along with improving data governance and processes ( 40%). This indicates that AI is driving a mutual approach to data management.
SEE: Top 10 benefits of better data quality for your organization.
The intimate connection between AI and data was also reflected in investment intentions. Three in four (78%) APAC data chiefs predicted their data investments would increase in 2024. None of the respondents did not plan to invest in data management capabilities in some form.
Regional investment in key data capabilities is increasing
Various data management capabilities are being invested in line with data strategy priorities. Data privacy and protection ranked first (45%), reflecting the need to keep data private and secure amid a surge in a rapidly changing cybersecurity environment.
This was followed by data quality and observability (42%) and data integration and engineering (40%).
“We are seeing an increase in data quality as a focus area and data governance,” Scott said. “So AI will really drive a kind of new wave of cleaning up data sets.”
SEE: How IBM's Matthew Candy sees Australia's pursuit of generative AI scale in 2024.
AI poses many challenges in data management
According to Informatica's global survey results, which came from data leaders at organizations with revenues greater than $500 million, nearly all (99%) data leaders had encountered obstacles in their AI journey, including those in APAC.
Data fragmentation and data growth.
APAC data leaders expect data fragmentation and complexity to worsen in 2024. Informatica found that 56% of data leaders were struggling to balance more than 1,000 data sources. Additionally, 78% of APAC data leaders expect the number of data sources to increase this calendar year.
“Last year alone, Informatica processed about 86 billion cloud transactions per month, up 60% from the previous year,” Scott explained. “So, as organizations try to get their data center in order, data continues to explode; “We are seeing this really explosive growth,” he stated.
Data quality and AI model bias
Data quality was considered the biggest challenge for generative AI by 42% of global respondents. The possibility of bias was highlighted as a particular concern in APAC due to large language models; 53% of Australian respondents said avoiding prejudice was their biggest concern (Figure A).
“In the age of analytics, if you had poor databases, you would make the wrong decision faster,” Scott said. “Likewise, if you have a poor data management environment, you'll get an answer from generative AI, but it can lead you in the wrong direction.”
Data Literacy Outside of the Data Estate
Organizational data literacy is holding back progress in AI, according to data leaders surveyed. For example, 98% of global data leaders said they had experienced non-technical organizational barriers to better data management, such as lack of leadership support.
Improving data-driven culture and data literacy was named by 39% of global data leaders as a top priority for 2024. Improving data literacy was the second most important measure (42%) of effectiveness of data strategy, second only to data preparation for AI. and analytics initiatives.
“Our Informatica CEO talks a lot about the fact that, as companies outsource applications, buildings, and so many other aspects of a business, for many companies their only asset is data. So it has to be a really high priority for the executive team and the board,” Scott said.
A growth in data management tools.
The number of data management tools is growing. Two-thirds (60%) of APAC leaders say they will need five or more data management tools to support priorities and manage data sets, an increase from the number of data chiefs who needed this many tools in 2023 (55%).
Data governance and democratization
40% of regional data leaders named improving data and process governance as a top data strategy priority for 2024. APAC data leaders also placed the greatest emphasis (67%) in enabling greater democratization of data across their organization when using generative AI.
This is driving providers to offer governance services and tools. Informatica recently launched an integrated cloud data access management tool following the acquisition of Privitar, which helps support the management, sharing and use of data in jurisdictions around the world.
SEE: Data governance will be a renewed focus in IT for Australian organizations in 2024.
Informatica also offers a self-service data marketplace designed to “democratize” data access. Users can request and access data based on permissions. Data is delivered with data relevance and quality ratings and tracked so data stewards understand how it is used.
Fundamental architecture to meet the data challenge
Informatica's Richard Scott advised regional data leaders to implement the right cloud architecture to support scale and focus on people and processes as well as technology.
Get started with the right cloud architecture
Organizations should start by ensuring their cloud architecture is robust, Scott said, as getting it right from the start will support future scaling efforts.
“When you're scaling and you don't have the right kind of data management architecture that's when you get into real trouble,” Scott said.
Scott added that getting a cloud architecture from the start is also cheaper.
“Enterprises with multiple cloud contracts pay a lot of money in inter-cloud ingress and egress costs,” Scott said. “The wrong cloud architecture not only results in an environment that may not be able to support generative AI, but it is also very costly.”
Informatica customer NRMA, one of Australia's oldest membership organisations, is successfully working with over 3,000 data sets. Organizations that strive to get the architecture right can control data and have a material impact on their data estate, Scott said.
Look at people, processes and technology.
The nature of the data challenge means organizations must more holistically consider people, processes and technology. Scott said that for data leaders in organizations trying to fix problems as they arise, it can feel like “putting your finger in the dike to stop a flood.”
“What will happen is that if we just plug every little hole in the dam by getting a new application or writing some code, we will end up with a very fragmented environment, which will be very fragile. You need to look at people, processes and technology and have a clear understanding of where it is going; “then you can bring in technology that will integrate incredibly well and give you the ability to transport data across your environment.”
