There is confusion about the objectives of ESG ratings and any regulation needs to be clear about what it is trying to achieve and for whom

Key points

  • ESG ratings are under renewed scrutiny and in regulators’ sights
  • The European Commission recently sought views on the ESG ratings market and the consideration of ESG factors in credit ratings
  • A lot of fuss is made about poor correlations between raters but comparability rather than consistency of methodologies is important 
  • ESG ratings themselves may be the wrong focus for some of the ire about ESG investing

Environmental, social and governance (ESG) ratings are under renewed scrutiny. Lively debate has been triggered by developments such as Tesla being booted out of the S&P 500 ESG stock index over concerns about worker conditions and automated driver safety, while ExxonMobil stayed put. 

A lack of consistency between ratings agencies has been in the spotlight for a while, and has again been highlighted by the publication of Aggregate Confusion, a research paper, in the academic journal Review of Finance. 

Julian Kölbel, an economist, environmental scientist and a BMW Foundation Fellow at the Center for Sustainable Finance and Private Wealth (CSP) at the University of Zurich, is one of the authors. Introducing the paper, he said: “As the discourse about ESG is intensifying, my hope is that the paper encourages a nuanced debate.” He expressed delight at the coverage about the disagreement among ESG raters, but also pointed out that he and his co-authors had written that “ESG rating divergence does not imply that measuring ESG performance is a futile exercise. It highlights that measuring ESG performance is challenging, that attention to the underlying data is essential, and that the use of ESG ratings and metrics must be carefully considered for each application.”

In other words, Kölbel said: “Know thy data.”


“The most important thing is that people understand what it is that is being measured and why those criteria have been chosen”

Sarah Wilson

One size does not fit all

Sarah Wilson, CEO proxy adviser Minerva Analytics, one of the smaller ESG rating agencies, echoes this, saying that the market is based on caveat emptor and that one-size-fits-all thinking does not help.

“Nobody who produces an ESG rating system has ever said they are the answer; we’ve always said they are a way of looking at things,” she says. “So yes, rating systems diverge, but that is healthy. The most important thing is that people understand what it is that is being measured and why those criteria have been chosen.”

This is on regulators’ minds, too. At a conference in early June, Verena Ross, chair of the European Securities and Markets Authority (ESMA), said the watchdog’s view was that “different ESG ratings can have different measurement objectives, and these can be helpful in view of the various purposes and needs users may have”.

She added: “What is important, is that whatever the objective, this is clear to the user, and the methodology followed is transparent.”

Earlier this year ESMA carried out a call for evidence on ESG ratings, with the European Commission launching its own consultation a few months later. Policy actions to address the prominent role played by ESG rating providers are also under consideration in the UK. 

The European Sustainable Investment Forum (Eurosif) believes that the role of regulation with regard to ESG ratings is to require providers to disclose information that is understandable and useful for professional investors. 

“The information should help them to better navigate the differences between methodologies rather than minimising those differences,” says Federica Casarsa, policy officer at Eurosif in Brussels.

“Asset managers generally like the fact that different providers have different methodologies, and they usually dig into the details of the methodologies and the full report attached to the ESG rating assigned to each company. What they appreciate is exactly the fact that different providers provide different perspectives…. to evaluate a company.”

Victor van Hoorn

“In most cases it’s about the lack of consensus view on what ‘considering environmental and/or social characteristics’ means”

Victor van Hoorn

Reliability concerns

In its feedback to the Commission consultation on ESG ratings and ESG factors in credit ratings, PensionsEurope said low levels of correlations between various types of ESG ratings can cause reliability concerns, “but they are not causing huge problems for investment decisions if they can be explained with the use of different methodologies”. 

“However, it is important that the methodologies are well disclosed, and it is needed to have common ground and standards at the company data level to ensure that the ratings reflect the analysis of reliable data related to materiality and correct assessment of ESG factors.”

This view is echoed by Impact-Cubed, an ESG analytics and investment solutions provider. In an opinion article in March, its chief executive officer Libby Bernick said standardisation of ESG ratings is not a silver bullet, with the 2008 financial crisis “a stark reminder of what happens when rating agencies are in lock step”.

ESG ratings are like equity analyst recommendations, she said, and “smart investors understand that analyst opinions are just one piece of information to inform their own research to determine a good investment”.

“Pushes for standardisation in reporting by companies on their ESG performance, which are in motion, are far more important than standardising ESG rating agencies,” Bernick wrote.

This is not to say there aren’t problems linked to ESG ratings, but resolving them may require looking elsewhere, or at least beyond the ratings. 

Measuring risk or impact?

At asset manager Ninety One, Stephanie Niven, portfolio manager in the sustainable equity team, says she doesn’t have the sense that everyone has grasped that ESG raters tend to evaluate companies through the lens of risk mitigation rather than how a company impacts the world.

Eurosif_Federica Casasa

“What asset managers appreciate is exactly the fact that different providers provide different perspectives… to evaluate a company”

Federica Casarsa

In May, think tank 2° Investing Initiative (2DII) carried out a market ‘pulse’ survey about ESG ratings, gathering 169 responses from a range of stakeholders: finance ESG professionals, finance non-ESG professionals, academics and researchers, non-governmental organisations, and the public sector.

The 2DII survey found that there is significant disagreement among stakeholders on whether ESG ratings should represent scores focused on “sustainability risks” or “sustainability footprint”. 

Survey respondents also disagreed with each other about what ESG ratings should measure, with the majority indicating that ESG ratings in practice measure the wrong thing relative to what they should measure in theory. 

Reporting the findings, Jakob Thomä, managing director of 2DII, said: “We can’t agree on what ESG ratings should measure.”

In its feedback to the EC, Eurosif notes that several investment products use risk-focused ESG ratings as a core element of the integration of ESG factors into their investment process while claiming to contribute to achieving sustainability objectives, at a time when the causality between the consideration of ESG risks and real-world impacts is hard to substantiate.

Lack of consensus

Victor van Hoorn, executive director of Eurosif, believes the problem is more about the interpretation and use of ESG ratings by financial market participants to qualify products as “sustainable” under the EU’s sustainable finance disclosures regulation (SFDR). 

“In most cases it’s not about ESG ratings not being transparent about their objectives, but rather about the lack of consensus view on what ‘considering environmental and/or social characteristics’ means in order to classify as an Article 8 SFDR product,” Hoorn adds.

Divergence digested: an academic view

‘Aggregate Confusion: The Divergence of ESG Ratings’, an academic paper, has been around in working paper form for a few years but gained renewed prominence when it was published in the Review of Finance in May following peer review.

Although it documented divergence between the ESG ratings of six providers (correlations ranging from 0.38 to 0.71), this disagreement had already been shown in previous research and the authors say the divergence “does not imply that measuring ESG performance is a futile exercise”.

Instead, the authors say their work’s contribution is threefold: a decomposition of ESG ratings divergence (into the elements of scope, weight and measurement); a methodology that facilitates dealing with the divergence; and the discovery of a rater effect, suggesting that measurement divergence is not randomly distributed noise but follows rater and firm-specific patterns.

The authors say their results have important implications for researchers, investors, companies, rating agencies, and regulators. For investors, the idea is that their work shows them how to disentangle various sources of divergence and trace a result to specific categories.

“For instance, investors could reduce the discrepancy between ratings by obtaining indicator-level data from several raters and then imposing their own scope and weight,” the authors write. “The remaining measurement divergence could be traced to the indicators that are driving the discrepancy, guiding an investor’s additional research.”

Regulators, meanwhile, could address the issue of ESG rating divergence, the authors say. Harmonising ESG disclosure by firms would provide a foundation of reliable and freely accessible data for all ESG ratings. Regulators could require ESG raters to map their data to a common taxonomy to make comparisons simpler and could spur competition. 

Recommendations for companies include that they work with rating agencies to establish appropriate metrics that ensure recognition of any achievement of firms’ underlying goals, while the companies should also ensure that the data they disclose is publicly accessible.

“Regarding rating agencies, our results call for greater transparency,” the authors write, adding that raters should also seek to understand what drives the rater effect to avoid potential biases.

Aggregate confusion

His colleague Casarsa says retail investors’ information needs are important, but that, in Eurosif’s opinion, these are more focused on the investment strategies of funds.

“What should be understandable to retail investors are the characteristics or strategies of the funds, rather than the methodologies for the ESG ratings that those funds may use,” she says.

In its consultation submission, Eurosif argued: “[Professional investors] are and will also remain the overwhelming group of users of ESG ratings in the future. We would caution against any idea that ESG ratings should or will become mainstream for retail investors.”

However, Ninety One contends that the investment industry relies too heavily on ESG ratings. According to a survey conducted for the asset manager at the FundForum conference in May, 92% of 130 respondents said they expected to increase their use of ESG ratings, which Niven says “doesn’t seem like necessarily the right path”. 

Referring to the Commission’s consultation on ESG ratings, she adds that “to some extent that’s based on a premise that ESG ratings are the correct way to think about sustainable and ESG investing, while we think we need to evolve far beyond that”.

Eurosif’s feedback to the Commission consultation suggests there could be a split in the market in future, whereby from some asset managers with more sophisticated investments strategies the demand for ESG raw data and ESG data products will grow more significantly than the demand for ESG ratings, while ESG ratings might continue to play a prominent role in some investment strategies such as passive index-based strategies.