Ex post evaluation of competition enforcement at DG Competition

Ex post or “retrospective” economic evaluation of competition enforcement essentially consists in assessing with the benefit of hindsight:

• whether a given past competition enforcement action such as a merger control decision or cartel decision was from an economic point of view sound [2]: typical sub-questions include whether it was based on a sound theory of harm; whether it was based on sound economic evidence supporting the theory; whether it correctly assessed the respective likelihood of different scenarios; and whether it reached the correct conclusions from an economic point of view;
• whether the enforcement action in question has achieved its economic objectives in particular where the evaluation concerns the effectiveness of remedies [3]; and
• what economic impact the enforcement action in question has had at the microeconomic [4] and/or the macroeconomic level. [5]

Contrary to judicial review the focus of such retrospective evaluation is not just on whether the decision complied with the law, but whether it was from an economic point of view correct and what its economic impact has been. The objective is also not so much to establish who was right or wrong at the time of adoption of the decision, but one of using the benefit of a retrospective analysis to learn lessons for future enforcement actions.

While ex post evaluation of enforcement pursues similar aims as the evaluation of regulatory acts under the Commission's Better Regulation policy [6], the focus is on the evaluation of individual enforcement measures and not on the evaluation of policy measures.

Ex post evaluation of enforcement can be conducted for one individual enforcement decision [7] or for several decisions of a given kind [8] or in a given sector [9] at the same time.

Ex post evaluation of enforcement can be based on qualitative methods such as surveys or interviews or quantitative methods such as Difference-in-Differences assessments or simulation models or a combination of both. [10]

Moreover, a competition authority can conduct such ex post evaluations in-house [11] or seek the input of outside experts. [12]

For a competition enforcement authority such as the European Commission there are strong reasons to invest some time and effort in ex post evaluation of competition enforcement.

Firstly, ex post evaluation of enforcement can make an important contribution to improving the enforcement performance of the authority. It must be borne in mind that competition enforcement (1) is by nature a highly complex activity: fact intensive and at the frontier of economics and law; (2) often involves high economic and societal stakes; and (3) is also by its nature “work in progress” as any competition enforcer needs to constantly adapt and improve its practices in line with advances in economic and legal thinking. If a competition authority wants to improve its enforcement performance it needs therefore to ask itself regularly “are we getting it right?” and “what has been our impact?”.

To remain successful and relevant in a rapidly evolving economic and societal context competition authorities need to think and function a bit like a university hospital: they cannot just “operate patients” (i.e. enforce), but need to invest at least a small part of their time and resources in studying ex post the effects of their “operations” and draw lessons from past successes and failures.

Secondly and most importantly, well planned and targeted ex post evaluations of past enforcement actions can prepare and support wider policy reforms. By way of example the merger remedy study of 2005 [13] served as an important preparatory fact finding exercise for the revision of the Commission's merger remedy guidelines. [14]

Thirdly, ex post evaluation can contribute to a better measurement of the impact and performance of a competition authority and thus strengthen its legitimacy and accountability towards its direct stakeholders (e.g. Parliament, Court of Auditors) as well as towards citizens in general. It can also be used to support the authority's strategic narratives in its advocacy and communication efforts. It can also contribute important facts, data and evidence, which can be relied upon in on-going public policy debates such as the ones on European champions or on possibly rising concentration.

There are however also significant challenges. Firstly, ex post evaluation of competition enforcement is inherently more difficult than the ex post evaluation of the impact of many other public policy measures such as for instance in the fields of road accident prevention or environmental safety.

This is in the first place because in many competition cases the harmful action (e.g. a cartel), the ensuing harm (e.g. higher prices, reduced innovation) and the effects of the competition authority's intervention (e.g. lower prices than in the hypothetical situation without intervention) are “hidden” and not directly or at least not easily observable. “Crime” statistics in antitrust are therefore inherently more difficult to construct and interpret than for example road accident statistics.

Moreover, causalities in competition policy are also more difficult to determine than in other public policy fields, as markets are a bit like “living organisms”: complex, driven by many factors and constantly evolving.

Most importantly, there will always be some tension between the desirability of investing more in ex post evaluation and the reality of what can be achieved given the resources available. At DG Competition, for example, there are limitations on the availability of human resources, which have been exacerbated by the challenges brought about by the pandemic. At other often smaller competition authorities the resource challenge will likely be even more pronounced. Moreover, resource requirements associated with upcoming merger and State aid notifications are difficult to predict. Within this context, it will often be difficult to divert to ex post evaluation human resources from more pressing and often mandatory enforcement and policymaking needs.

Over the past years the team of the Principal Adviser for ex post economic evaluation in DG Competition led by ­Fabienne Ilzkovitz has conducted a number of very interesting evaluations and laid a strong methodological foundation for the ex post evaluation of competition enforcement and policy. In 2020, it published a ground-breaking book [15] discussing methods of ex post evaluation and summarising the various ex post evaluation studies that it had initiated since 2014.

For the coming years, Executive Vice President Vestager has encouraged DG Competition to further step up its evaluation activities. One objective is to increase the number of evaluation studies conducted per year. Another one is to launch at regular intervals larger “flagship” studies covering a relatively large number of decisions. In addition, DG Competition intends to further expand the scope and breadth of its annual customer savings calculations and macroeconomic impact reporting.

When stepping up its evaluation efforts it will however also be necessary to bear in mind that DG Competition's core business of enforcement and policy comes first and that there is significant uncertainty about future workloads in merger and State aid control, for example. Therefore, when deciding on the launch of new evaluation projects DG Competition will have to base its decisions also on the evolving internal resource constraints and the limits to what colleagues can reasonably be expected to contribute.

A further objective will be to ensure that future studies are relevant and impactful besides being as simple and practical as possible. All future evaluations should be directly relevant to the policy objectives of the Commission. They also should lead to constructive and realistic proposals on how to improve Commission enforcement practice. While evaluation projects will continue to be based on state-of-art evaluation techniques, DG Competition cannot afford to invest in projects that are interesting merely from an academic perspective.

Another aim will be also to involve case teams to a greater extent in evaluation exercises, depending on available resources and the nature and objectives of the case decisions to be evaluated.

Moreover, DG Competition wants to deepen and broaden its cooperation with academia and Commission internal partners such as the Joint Research Centre.

Finally, DG Competition intends to invest efforts in building a stronger evaluation culture and invest in evaluation capacity building.

To achieve the above objectives, DG Competition has launched several work streams, which can be grouped around four axes.

DG Competition plans to continue and gradually expand its reporting on the impact of the Commission's enforcement actions by further developing its annual customer savings calculations.

Over the past years, DG COMP has calculated estimates of the direct benefits to consumers from the Commission's merger interventions and cartel prohibitions. [16] Based on conservative assumptions endorsed by the OECD [17], the direct customer savings generated by the Commission's enforcement actions in those enforcement areas have been substantial both in absolute terms and when compared to the benefits reported by other competition authorities. [18]

In 2021, for the first time the calculations will include customer savings generated by non-cartel antitrust enforcement under Articles 101 and 102 of the Treaty on the Functioning of the European Union (TFEU).

DG Competition is also reflecting on how to better estimate the indirect, deterrent effects of its merger and antitrust interventions. Such deterrent effects are from an economic point of view important as they discourage also those companies not directly targeted by the Commission's intervention from engaging in anticompetitive behaviour.

The availability of sound estimates of the importance of such deterrent effects would also have the benefit of strengthening the empirical foundation of the already performed macroeconomic model simulations [19] that are aimed at evaluating the impact of competition policy from a macro perspective.

As regards the calculation of customer savings, DG COMP has also proposed an exchange of best practices within the European Competition Network.

As already set out above, DG COMP intends to launch regularly a “flagship” evaluation study project covering a larger sample of decisions. The idea behind those studies is that a competition authority can learn more about systemic issues in its enforcement practice, when it studies several related cases at the same time rather than focusing on a single intervention. [20]

For example, the EU merger remedy study of 2005 assessed 96 remedies covering 40 merger cases from different sectors. The study enabled DG Competition to identify a number of issues that endangered the effectiveness of the remedies imposed in merger cases. These issues were then addressed in 2008 Commission notice on remedies [21] and resulted in an improved case practice.

A call for tenders for a first flagship evaluation study concerning merger control was launched in June 2021. [22] The study's objective is to evaluate with the benefit of hindsight whether the Commission's ex ante assessments of future market developments, such as entry, expansion or imports, as set out in its merger decisions were correct. The study will cover 15-25 significant and borderline merger decisions adopted between 2012 and 2018, which would allow drawing broader policy conclusions and making recommendations for improving enforcement practice. The findings will hopefully also provide useful contributions to the debate on the assessment of entry and imports from third countries in merger control triggered by the decision in the Siemens / Alstom merger case. [23]

In addition to the larger cross-sectoral flagship studies, DG Competition intends to launch a number of more targeted evaluation studies with a narrower focus in particular on sector specific issues. Not only are such studies less costly, they also may have a more immediate impact on enforcement as the practical lessons to be drawn can possibly be directly translated into an improved enforcement practice in the sector in question.

A good illustration for such a more targeted study is the recently launched study into “killer acquisitions” in the pharma sector which aims to assist the Commission in its recently announced efforts to make increased use of referrals under Article 22 of the Merger Regulation [24] and to support the Commission's antitrust enforcement work.

The study has two main parts. A first, fact-finding part is aimed at finding more evidence on the existence, frequency and typical features of killer acquisitions in the pharmaceutical industry. It would also try to determine under what circumstances killer acquisitions are most likely to occur.

The second part will evaluate, based on a selection of past merger notifications, whether the Commission correctly anticipated the possible damage to innovation competition associated with killer acquisitions. Similarly, it will assess the suitability of the EU antitrust enforcement rules to deal with killer acquisitions that are not structured as concentrations.

The last but not least important axis of activity concerns internal and external capacity building.

First, a light evaluation “network” will be set up within DG Competition bringing together representatives from different parts of the Directorate-General. The network would be a forum to discuss methodological issues and ensure that proposed evaluation projects are impactful, launched and executed rapidly and of good quality. It would also ensure a successful communication around the projects.

A second area for capacity building concerns potential improvements in DG Competition's data gathering toolbox for the purposes of ex post evaluation.

Thirdly, there will be a reflection on whether it is possible to mitigate or partly overcome confidentiality issues, which hinder a greater involvement of outside academics in the ex post evaluation of decisions.

Fourthly, there will be a reflection on how to obtain more and better data on the Commission's enforcement activities possibly through cooperation with external partners [25] and improved data collection internally for example when cases are closed.

Finally, DG Competition intends to strengthen its external network: building on past cooperation it wishes to strengthen its links with the other members of the European Competition Network and exchange best practices on ex post evaluation. It also wants to strengthen links with other Directorates-General within the Commission with experience in market analysis and evaluation, such as the Joint Research Centre, DG Economic and Financial Affairs, DG Internal Market, Industry, Entrepreneurship and SMEs, and EUROSTAT. Moreover, the relationship with the academic world could be strengthened.

As Bill Kovacic rightly noted already in 2006, while ex post evaluation is important for all public institutions, it is particularly important for competition authorities. [26]

Competition enforcement takes place in a highly complex and rapidly changing market environment. It needs to constantly take into account progress made in economic and legal thinking. The economic and societal stakes are high. And as the saying goes “with great powers come great responsibilities”.

Despite the significant challenges discussed above, as competition authorities we owe it to consumers and the companies subject to our enforcement powers that we are making every effort possible to ask ourselves frequently and critically whether we have been doing “the right thing” in past cases, in order to make sure that we do “the right thing” in current and future cases.