Enhanced data and technology infrastructure no longer a nice-to-have

By Eric Chng, State Street

Published: 18 September 2023

Data and technology play a crucial role in improving efficiency and standardisation in private market operations, and assessing environmental, social, governance (ESG) risks.

The recent market and economic turmoil sparked by high inflation and rising interest rates in response have forced private markets managers and investors into a reckoning with their process efficiency. However, lack of streamlined data management processes and disparate data sources are major hindrances to growth.

Data management in the sphere of private equity and real assets has long been both complex, piecemeal, and inconsistently applied, leaving market participants frequently unable to access vital information reliably or in a timely fashion.

Owing to the slow-paced changes in the economic landscape, investors have not addressed the concerns that a fragmented data management strategy brings. Additionally, wide availability of attractive investment opportunities has resulted in better internal capacity for analysing deals that have not given a sufficient advantage to make the necessary operational investment worthwhile.

However, in the last two years markets have witnessed highest inflation in decades. This has shrunk the pool of available deals from which investors can expect above inflationary returns or yields. Meanwhile, interest rates have risen throughout the economically developed world to their highest levels since before the 2008 financial crisis, increasing the cost of borrowing money and impacting institutions’ abilities to raise cash for leveraged private markets investments. Higher interest rates also mean higher yields on traditional fixed income and cash, which puts further pressure on expected returns for private markets investments. 

Against this backdrop, it not surprising that the majority of respondents to the State Street 2023 Future of Private Markets Study1 described “manual processes and outdated systems” for managing data as a “considerable” waste of their time and resources. 

On the other hand, a similar majority said improving their data management and analysis capabilities conferred a “competitive advantage” on their organisation.

However, respondents did not, for the most part, want to reduce their private markets exposure; more than two thirds (68%) said they planned to continue with their existing allocation targets, despite the difficult environment. They simply acknowledged that they need to improve their ability to assess opportunities and, in the case of managers, communicate essential information about investments to asset owners, who want more frequent, verifiable and standardised data about various aspects (for instance return, risk, ESG) of their private market investments.

The impact of technology on data operations

Survey respondents said that the heightened focus on deal scrutiny and the data essential to better understand private markets holdings went hand-in-hand with investment in technology infrastructure aimed at improving their abilities to view and analyse disparate types of data from various sources.

Our data shows significant technology investment is being directed towards private markets operations across the spectrum of institutional investment organisations. More than three quarters (77%) said that at least 10%of their overall technology budget was being allocated to their private markets operations. Additionally, more than half of the respondents (52%) were spending more than 20%of their technology expenditure on private markets.

From an institutional investors’ perspective, most of these investments are centred around data collection, aggregation, and harmonisation across disparate information sources, which are highly manual and complex. As the number of investment managers increase, the lack of standardisation across different reports adds to the complexities.

In particular, aligning disparate technology systems from across different areas of their organisations, operationally and geographically, scored highly when respondents were asked to assign importance to various areas of their private markets data management processes. However, the same options received low scores when respondents gave assessments of their competence in these areas.

Cloud technology was a priority direction for institutions’ technology investments, with 71% saying they were spending there, due to the simultaneous importance and underdevelopment of data systems’ interoperability, data lakes, and warehousing. The next most popular form of technology was artificial intelligence, which approximately a third (36%) were investing in. This is an interesting emerging technology for data management in particular, as it is increasingly essential to extracting and analysing information from large cloud-based ‘lakes’ of unstructured, raw data.

ESG data in private markets

Another area of high importance but relatively low competence for institutions’ private markets data operations was assessing ESG characteristics of investments, particularly from a risk perspective. ESG data has been a thorny issue for the investment industry for a long time, with a lack of consistent standards, reliable, directly comparable, and affordable sources of data. 

In private markets, these challenges are accentuated by two trends. Firstly, private companies and real assets are generally less transparent than their public counterparts, adding to the difficulty of finding reliable ESG information. In addition, the level of transparency across different asset classes differs due to the disparate nature of such investments.

Secondly – there is a lack of standardisation when it comes to private markets. ESG data varies with each asset class and generally, metrics are hard to define in a consistent manner between asset types. A real estate exposure and the ESG metrics associated with that asset class is significantly different with a Private Equity portfolio company. The increase in regulatory focus on ESG will continue and this will lead to standardisation across each asset type and the way metrics are defined and adopted. 

A trend that’s not going away

Investment institutions’ new focus on making fundamental improvements to their private markets data operations is a product of external, macro-economic headwinds. However, it is not likely to dissipate with those conditions. Over a quarter of respondents (26%) believed the current inflationary environment is likely to be short lived, or that the old market conditions will return as inflation subsides, compared to nearly half (47%). Forward thinking organisations, making the right amount of investments in sophisticated data management technology will benefit from the advantages conferred by better processes, enabling them to make informed investment decisions and help gain alpha even in a low yield environment.



1 Towards the end of 2022, we conducted a survey of nearly 500 investment institutions, including 120 insurers, across North America, Latin America, Europe and Asia Pacific (APAC) to understand the outlook on private markets of generalist asset managers, private markets managers, insurers and asset owners. This has formed the basis of the State Street 2023 Private Markets Study, in which we analyse the macro environment and its relation to private markets, what is driving demand for private markets assets among retail investors, and the role of emerging technologies and new fund types. Additionally, we took a deep dive into the role of technology in addressing data management, efficiency and transparency.