30 July 2020

12. Toward a standardized measurement of social impact?

 

Summary: At the heart of evaluation is the question of value. How much value will be attributed to an intervention and its results? Who will decide this? And how? The reason why some people are uncomfortable with the increased use of impact measurement is because they believe that some actions have a great value but that this value is intangible. This means that those outcomes could not be evaluated objectively, quantified, assigned a monetary value or compared to other actions and outcomes. This creates tension between striving toward either standardization or flexibility, as strategies, both of which have advantages and disadvantages.

Can the intangible be measured?

Some desirable outcomes on society are referred to as intangible. The notion of intangibility sometimes emerges when we talk about, for example, democracy, empowerment, social ties, social cohesion, level of awareness and trust (Kanter & Delahaye Paine, 2012, p. 70). By intangible we mean that one cannot “touch” these outcomes and therefore they are difficult if not impossible to capture and observe, even though they are generally considered important by society.

This idea echoes certain proverbial wisdoms, such as “Not everything that counts can be counted, and not everything that can be counted counts,” attributed (probably wrongly) to Einstein, or “What is essential is invisible to the eye,” by Saint-Exupéry in The Little Prince.

In fact, we know of no examples of outcomes that are desirable but absolutely impossible to observe. Intangible therefore does not mean that an outcome is invisible but rather that it is difficult to measure objectively with an external, standardized measuring instrument.


See an example

For example, among the benefits for participants of an intervention implemented by the Centre d’éducation et d’action des femmes de Montréal (CEAF) are, as presented by the Centre de formation populaire (CFP): “improved capacity to focus, reduction of prejudices, greater openness to others, better understanding of systems of oppression and creation of a social network” (Centre de formation populaire, 2017). It is difficult to measure, in the sense of quantifying, whether or not prejudices are internalized, the degree of openness to others, or the level of understanding of systems of oppression. Does this mean that these outcomes do not exist or that an intervention with respect to these variables is not desirable? Not at all. Despite a certain intangibility, they do exist, are important and can nonetheless be expressed, even if only subjectively, by the individuals directly concerned.


Evaluating an intangible outcome

In order to evaluate an outcome deemed to be intangible, we must first properly identify the nature of the problem. The main challenges are:

  1. Identifying the outcome one wishes to evaluate or to measure, in other words, the aspect of reality one is trying to describe.
  2. Finding an indicator or proxy that will reflect this aspect well.
  3. Measuring the change or difference in this aspect.
  4. Establishing a causal link, in other words, demonstrating that a change is due to an activity (to a certain extent).

See an example

  1. Let us take the example of racism against a certain group of people as a barrier to employment. This would be the situation or the aspect that we want to observe, measure, describe and possibly act upon. There is broad consensus that racism has concrete and negative effects on society. However, it is difficult to observe and measure to what extent employers have internalized racism, which accounts for its intangible nature.
  2. We contend that certain indicators could nonetheless serve to assess this situation. For example, we could measure the difference between the number of people of colour being called in for an interview or hired and the same indicators for Caucasians. This would be the indicator that aims to measure an intangible aspect.
  3. If tests had been conducted in this regard in 2010 and repeated today, a change could be observed (e.g., increase, decrease, change in distribution, etc.).
  4. The task then would be to establish a causal link, in other words, to infer that this change was attributable to certain causes and to what extent.


It is important, when addressing this issue, to specify what exactly it is that is intangible. When the aspect to be observed is difficult to be measured objectively with an external and standardized measurement instrument, it is said to be intangible. This means that it is difficult to identify the aspect (step 1), to select indicators that reflect it well (step 2), or to monitor its evolution over time (step 3).

Cases where the causal link between a cause and effect (step 4) is difficult to measure objectively using an external and standardized measuring instrument might also be called intangible. However, such cases are of a somewhat different nature, and is discussed in the section on causality and attribution.

Qualitative methods are generally better suited to evaluating an outcome deemed to be intangible, especially when it involves a change experienced or perceived by an individual. One of the easiest ways of evaluating such an effect is to give the people concerned by the intervention a voice by asking them what they think and feel through questionnaires or interviews. Some methodologies are more rigorous than others in evaluating o-called intangible experienced realities, but none is perfect. It is simply a matter of being aware of the potential biases and subjectivity inherent in any evaluation and agreeing on the level of rigour that is sought.


Learn More

  • For example, the way the questions in an interview or questionnaire are formulated certainly has an influence on the responses that will be generated.
  • There are also a number of biases. For example, the respondent might modify his or her answer or hide part of the truth in order to appear socially compliant—the social desirability bias (Trochim, 2006a).
  • Or, participants’ memory mail fail when asked about a past situation—the recall bias.

To limit these biases, participants may be asked to rank rather than rate an impact (“Which of these impacts are the most significant in qualifying your experience?” instead of “Did you notice the following impacts?”) Alternatively, the same question can be asked in different ways, in order to triangulate and test the robustness of the answers provided. The purpose of this web portal is not to explore each of these methodological aspects but simply to mention their existence. For more information, please refer to the qualitative method manual of your choice:

  • The CSMO-ÉSAC (Binhas, 2007) and Avise (Duclos, 2007) guides discuss the details of constructing a qualitative type of survey in the context of impact measurement (in French).
  • For a general introduction to social science research, see the Research Methods Knowledge Base (Trochim, 2006b).


It is probably when the outcomes to be assessed concern interactions within society (e.g., social cohesion) rather than a particular individual (e.g., sense of self-confidence) that the effects are most intangible, in the sense of being difficult to observe or measure. There are no miracle solutions for this issue and it is up to evaluators to use creativity and rigour to identify the desired effects as best as they can.

For some, the intangibility of an effect is not so much a fundamental obstacle to impact measurement as a methodological challenge to be overcome. Approaches such as cost-benefit analysis (CBA) and social return on investment (SROI) tell us that, in fact, many things that we tend to think of as incalculable can actually be measured, and even assigned a monetary value. For example, to assess the monetary value society places on reducing a given prejudice, a proxy indicator, such as the amount the government invests in awareness campaigns to reduce this prejudice, will be used. Beyond the technical debates about the accuracy of such approximations, some people question the relevance or feasibility of evaluating benefits using a common unit of measurement, such as money.

Comparison: Can and should we monetize everything?

Monetization refers to the act of assigning a monetary value to outcomes that are not traditionally traded on the market. While this has certain advantages, it also raises several criticisms.


Learn More

The willingness to convert different types of effects or benefits into a common unit of measurement (such as a currency) is at the heart of popular methods such as CBA and SROI. However, it is controversial for several reasons. Before addressing the arguments from both sides, let us clarify two things:

  1. This debate should not revolve around any visceral rejection or preference for money. Rather, it is about the desire to convert different kinds of effects into a common unit. This could be a currency, such as the dollar, but also a score or a grade, as used by many financial institutions or certification programs, among them B Corp.
  2. Nor does this debate, as we understand it, imply a preference for “numbers” (quantitative) or “stories” (qualitative).

Maier et al. (2014, pp. 16‒17) explain that this willingness first requires acceptance of the premise that outcomes of a different nature can indeed be compared, that they are commensurable. Indeed, according to them, one might be apt to reject that premise on philosophical grounds, as follows:

To argue against commensuration, we may cite Kant (1994, p. 97), who states that “[…] man as a person, […] is exalted above all price.” Commensuration is a form of exertingunevenly distributedpower by determining the currency in which the value of things is to be expressed. (Maier et al., 2014, p. 16)

Subsequently, they offer an argument in favour of the premise, one which emphasizes the practical aspects instead:

Hopes have been voiced that, by translating often-ignored impacts into the language of money, SROI analysis might put them into the limelight (Edwards et al., 2010; Jardine & Whyte, 2013). In the last decade, possibilities of monetization have extended noticeably. What once seemed unimaginable (e.g., Frumkin, 2003, on monetizing the value of advocacy, orchestra performances and family unity) today can be monetized with economic methods, for example willingness to pay, required compensation, revealed preference techniques, travel cost method or average household spending on certain kinds of goods (Nicholls et al., 2012, pp. 47). (Maier et al., 2014, p. 17)

The feasibility of converting the costs and benefits of an action into a single monetary unit can also be contested, more pragmatically, by pointing to the risks of underestimating the true value of the evaluated projects. As noted by Nicole Alix (2015, p. 114; our translation): “Costs are much easier to assess than benefits, which are often more qualitative than quantitative.” Others oppose monetization on the basis of the risks that these processes create for those who are the subject of the evaluation (the reactivity mentioned in the section on the potential negative impacts of the measure):

Even when standardized, these methods can lead to comparing the social impacts of very heterogeneous organizations. This opens the door to competition in the search for public or private funding, by comparing their performances, and ultimately to distinguishing between organizations that would be supported and those that would not. (Branger et al., 2014, p. 21) (our translation)

If one accepts the premise that it is desirable, albeit difficult, to compare projects of very different natures on the same basis, then one faces several technical challenges. More information on this subject is available at:

In all cases, as noted in the CBA summary sheet, the results of an evaluation where multiple effects are converted and compared using a common unit of measurement should be treated with caution. This is because these studies make many assumptions regarding, for example, the proportion of an outcome that can be attributed to an intervention and the monetary value of the effect thus obtained. Just because there is a number or a score does not mean that the evaluation is not subjective.


The conclusion of this sub-section is that while the attribution of a monetary value to outcomes that traditionally are not traded on the market—monetization—is a potentially useful exercise, the results must be considered with caution. The discerning reader should keep in mind that these are reasonable but not infallible estimates, a range of probabilities rather than a precise figure. It is therefore simply a matter of keeping a critical mind, being aware of the limitations of scores or indicators and not relying on them as absolute truths.


Learn More

This leads authors such as Bhatt and Hebb (2013) to recommend the combination of multiple methods to complement one another in an impact evaluation:

It is important to remember that not all experiences and changes can be captured in monetary terms. Those which cannot be monetized should be discussed in narratives. Narratives are a way of storytelling that can help you to explain the process leading towards the calculation of the ratio and how it was established. It also allows you clarify any assumptions or descriptions of areas which have not been measured or could not be evaluated. (Bhatt & Hebb, 2013, p. 16)

On this subject, the authors of the SROI guide agree that “it is unlikely that the comparison of social return ratios will be relevant, whereas an analysis of the various opinions and decisions expressed during the development of a specific SROI report is much more useful” (ESSEC IIES, 2011, p. 54) (our translation).

Others go further and simply advise against the use of monetization in the context of evaluating the social economy. Instead, they make the following recommendations:

We must promote proximity systems where human judgment can correct the uncertainties resulting from attributing prices to what normally doesn’t have one. We need collaborations between funders and funded to reduce information asymmetries without going through rating and intermediaries that increase the final cost of money. Cooperative banks and solidarity finance have extensive experience (Dupuy & Langendorff, 2014) that is based on such principles, which retain their value. (Alix, 2015, p. 114) (our translation) [1]

The reference to finance in this last quotation leads us to a debate that was particularly topical in the last years (see Social Innovation Strategy and Social Finance). Essentially, social economy organizations, such as TIESS, favour a flexible, negotiated, learning- and improvement-oriented evaluation based primarily on enterprises’ needs. At the same time, other organizations, mainly government funders and organizations wishing to develop the impact investment market, have the legitimate need of analyzing the impact of the organizations they finance through a common and standardized reading grid that allows them to compare, aggregate and convert the social outcomes of their investments.


Tension between standardization and flexibility

The uneasiness surrounding the search for a balance between, on the one hand, a detailed description that takes into account the complexity of the context and, on the other hand, the implementation of practical but potentially reductive common indicators is linked to the different uses that one wishes to make of impact measurement. The following diagram illustrates some of the respective advantages of customized and standardized evaluations:

Social economy enterprises, those who carry out activities to generate positive outcomes on their territory and in their community—in short, the subjects of the evaluation—will generally have a preference for methods that take into account the context and that offer a detailed description not only of results but also of processes, obstacles and successes. They will therefore be more interested in methods and indicators that are adapted and customized to better reflect the relevance of their action.

Donors, whether public (governments) or private (financial or philanthropic), will tend to prefer methods that are standardized and that use the same steps and indicators from one project to another, in order to draw up an overall picture and eventually allocate funds on the basis of this information.

An example of this debate between standardization and flexibility is presented in the Stanford Social Innovation Review. Saul (2014) argues that the entire social sector can be measured through standardized indicators. This is the ambition of the Impact Genome Project. Some researchers, such as Arnold-Fernandez (2014), reply that social transformations are far too complex to be captured by a bank of indicators.


Learn More

Other advantages and disadvantages of standardizing impact measurement are summarized by Flatau et al. (2015, pp. 13‒14) in the following table:

We live in an information age where costs are associated not only with the creation of information but also increasingly with its use. A one-hundred-page report will have the advantage of being able to offer a much more detailed description of a company’s intervention, but it has very little chance of being read. Conversely, a report that assigns a monetary value or a score to the same organization based on a pre-established analysis matrix, despite all the assumptions and shortcuts involved, is much more likely to be considered.

Thus, in an increasingly complex world, managers, in order to keep pace, aspire to a world that could, as Scott explains in this article, be legible for outsiders (Scott, 2010). For Chiapello (2015), this is a process of financialization, a colonization of sectors associated with intervention in the social field through financial thinking:

Financialisation is usually defined as a process of morphological transformation of capitalism, entailing the capture of resources by finance in the broadest sense through expansion of the financial markets, a rise in the number and variety of financial operators and finally the development of a service industry associated with financial activities (audit, consulting, and law firms, rating agencies, etc.). (Chiapello, 2015, p. 15)

When someone decides to give some money to a charity, or when a local authority decides to award a grant to some social actor, there is always some form of valuation beforehand. First of all, the decision must be made to provide support, and then how much. This valuation operation is what is gradually becoming financialised. There is a redefinition of the idea of donations and grants, which become investments that must have returns. These required returns are called ‘‘social returns’’ (but may also be financial). This is a highly specific view of valuation, which requires a connection between the money invested and what the organisation produces, with a view to choosing between organisations based on compared social returns […]

This idea of impact investing should not be confused with the demand for accountability, or the desire to assess the policies conducted, even though some confusion often persists in the debates. We may desire transparency regarding use of the funds or think carefully about the choice of the best policies. But this does not mean that the only way to approach these questions is as an investor who seeks the best return, invests, and divests as opportunities come up. Yet, that is indeed what happens: one of the most entrenched fantasies is the idea that social impact could be reduced to a single indicator that would enable investors to choose quite simply between a range of proposals. Instead of considering only risks and returns, they could simply add a third metric, social impact (Morgan 2012). (Chiapello, 2015, p. 25, emphasis added by TIESS)

Indeed, it is with a view to meeting this demand that a market for standardized impact measurement is developing. IRIS, GRI, SROI, B Corp (covered in the TIESS summary sheets about methods) and several other organizations have the objective to structure and allow the management of an “impact portfolio” (Alix & Baudet, 2015).

Some people are wary of this desire to financialize the universe by designing indicators, seeing therein the threat of the “remote steering of the social sector by non-professionals” (Chiapello, 2013; our translation). Such fears, however, while legitimate, should be put into perspective: The standardization of evaluation does not have to be imposed from above in a logic of control or competition. On the contrary, it could be promoted by the networks of social economy enterprises themselves, who see it as a way of supporting their member enterprises, to compare and improve themselves through benchmarking, or to aggregate their outcomes in order to get a better recognition of their collective impacts. It is also a means of promoting practices deemed positive, as illustrated by the Conseil québécois du loisir (CQL)’s approach to accessibility. In short, it is up to the stakeholders to determine what is best for them.


So, which perspective is preferable? Again, there is no unique answer; the challenge is to find a balance between the total absence or presence of standardization:

Between ad hoc methods and a single format, a third possibility would be to adopt a “common process” that is both flexible enough to allow the company to report on its specificities but also comprehensible to external stakeholders. The aim would be for companies to adopt a common approach while leaving the choice of indicators and data collection techniques to the company. This could also open the door to a valorization of all SSE enterprises, to external stakeholders. (VISES, 2017, p. 31) (our translation)

In Canada, the Common Approach to Impact Measurement Project has taken on the challenge of finding such a balance between standardization and flexibility. The theory behind this project is based on the concepts of construct-based equivalence and bounded flexibility used in financial accounting. For more information, see the article by Ruff (2019) or the official executive summary of the Common Approach to Impact Measurement.

To strike this balance, the evaluation of the social economy will inevitably have to consider the relevance of each indicator and their usefulness for both organizations and their stakeholders (members, users, funders) in an ongoing process of negotiation and deliberation. When following this protocol, it is those approaches and indicators that best meet the objectives of each group of stakeholders that will turn out to be the best ones.


Learn More

In 2014, the search for a balance between flexibility and standardization had also motivated the work of the sub-group on measuring social impact of the European Commission’s Group of Experts on Social Entrepreneurship (GECES):

The sub-group felt that it was impossible to design a rigid set of top-down and a one-size-fits-all indicators that would measure social impact in all situations, for several reasons […] It is for this reason that this report sets out a procedure that should help facilitate the process. (Clifford et al., 2014, p. 7) (our translation)

This procedure is illustrated in the following diagram.

Source: Clifford et al., 2014, p. 8

Since 2015, the alignment of indicators with United Nations Sustainable Development Goals (SDGs) has become a major source of momentum for standardizing the measurement of social impact. The GRI had already taken the lead by launching in 2015, in partnership with the WBCSD and the UN Global Compact, the SDG Compass. More recently, the IRIS indicators were updated in 2019 to better align with the SDGs, and B Lab launched the SDG Action Manager in January 2020.

Source: United Nations

The initial enthusiasm spurred by the idea of integrating all the activities that are desirable for humanity (activities with social impact) within a common grid will probably be followed by some disillusionment about the limits of such an endeavour, but also, potentially, by some progress. This is therefore an important issue to be monitored in the coming years. 


Footnote

[1] original quote: Il faut promouvoir des systèmes de proximité où le jugement humain sait corriger les incertitudes résultant de la fabrication de prix pour ce qui n’en a pas. Des collaborations entre financeurs et financés sont nécessaires pour réduire des asymétries d’information sans passer par le rating et par des intermédiaires qui renchérissent le coût final de l’argent. Les banques coopératives et la finance solidaire ont une longue expérience (Dupuy, Langendorff, 2014), fondée sur de tels principes, qui gardent toute leur valeur. (Alix, 2015, p. 114)


References

Alix, N. (2015). Mesure de l’impact social, mesure du « consentement à investir ». Revue internationale de l’économie sociale : Recma, (335), 111‑116. doi:10.7202/1028537ar

Alix, N. et Baudet, A. (2015). La mesure de l’impact social : facteur de transformation du secteur social en Europe (no 2014/15). Belgique : CIRIEC.

Arnold-Fernandez, E. (2014, 29 mai). The Problem with the Impact Genome Project. Stanford Social Innovation Review. Repéré à https://ssir.org/articles/entry/the_problem_with_the_impact_genome_project 1 sur

Bhatt, B. et Hebb, T. (2013). Measuring Social Value: A Social Metric Primer. Carleton Center for Community Innovation (3CI).

Binhas, L. (2007). Comment mener une étude de A à Z : guide méthodologique à l’intention des regroupements, associations, fédérations, entreprises et organismes du secteur de l’économie sociale et de l’action communautaire. Montréal : Comité sectoriel de main-d’oeuvre économie sociale action communauté.

Bouchard, M. J., Fontan, J.-M., Lachance, É. et Fraisse, L. (2003). L’évaluation de l’économie sociale, quelques enjeux de conceptualisation et de méthodologie.

Branger, V., Gardin, L., Jany-Catrice, F. et Pinaud, S. (2014). Évaluer l’utilité sociale de l’économie sociale et solidaire. Projet Corus-ESS (Connaissance et reconnaissance de l’utilité sociale en ESS).

Centre de formation populaire (CFP). (2017, 12 janvier). Des mots pour exprimer les bienfaits du CÉAF… Un endroit où il fait bon être! ÉvalPop. Repéré le 2 mars 2017 à https://evalpop.com/2017/01/12/des-mots-pour-exprimer-les-bienfaits-du-ceaf-un-endroit-ou-il-fait-bon-etre/

Chiapello, E. (2013, mai). Mesure de l’impact social : pourquoi tant d’intérêt(s) ? Bulletin mensuel de Confrontations Europe.

Chiapello, E. (2015). Financialisation of Valuation. Human Studies, 38(1), 13‑35. doi:10.1007/s10746-014-9337-x

Clifford, J., Hehenberger, L. et Fantini, M. (2014). Approches proposées pour la mesure de l’impact social dans la législation et dans les pratiques de la Commission européenne concernant les FESE et le programme EaSI. Luxembourg : Commission européenne, sous-groupe du GECES sur la mesure d’impact. Repéré à http://dx.publications.europa.eu/10.2767/28927

Duclos, H. (2007). Évaluer l’utilité sociale de son activité  Conduire une démarche d’auto-évaluation. Paris (167 rue du Chevaleret, 75013) : Agence de valorisation des initiatives socio-économiques (AVISE).

ESSEC IIES. (2011). Guide du retour social sur investissement (SROI).

Flatau, P., Zaretzky, K., Adams, S., Horton, A. et Smith, J. (2015). Measuring Outcomes for Impact in the Community Sector in Western Australia (no 1). Sydney, Australia : Centre for Social Impact.

Hequet, C. (2016). L’évaluation monétaire de la nature. IRIS.

Kanter, B. et Delahaye Paine, K. (2012). Measuring the Networked Nonprofit. San Francisco : Jossey-Bass.

Maier, F., Schober, C., Simsa, R. et Millner, R. (2014, septembre). SROI as a Method for Evaluation Research: Understanding Merits and Limitations.

Ruff, K. (2019). Good Food and Good Jobs : How Construct-Based Equivalence and Bounded Flexibility Can Increase Entity-Level Relevance of Social and Environmental Accounting Standards. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.3367614

Saul, J. (2014, 6 février). Cracking the Code on Social Impact. Stanford Social Innovation Review. Repéré à https://ssir.org/articles/entry/cracking_the_code_on_social_impact

Scott, J. C. (2010, septembre 8). The Trouble with the View from Above. Cato Unbound. https://www.cato-unbound.org/2010/09/08/james-c-scott/trouble-view-above

Trochim, W. M. (2006a). Selecting the Survey Method. The Research Methos Knowledge Base. Repéré le 30 avril 2018 à https://socialresearchmethods.net/kb/survsel.php

Trochim, W. M. (2006b). The Research Methods Knowledge Base (2e édition). Repéré à https://www.socialresearchmethods.net/kb/contents.php

VISES. (2017). Orientation stratégique du projet VISES Approche des théories et pratiques. Lille et Louvain-la-Neuve : CRESS et ConcertES.


Want to know more?