30 July 2020

2. Why and for whom to evaluate?

 

Social impact measurement is a special type of assessment that looks at the impacts of an activity (see Definition and main steps).

Evaluation is defined as “a systematic process for estimating the value of an intervention.”


See more details

  • The “process” is an activity which normally involves collecting and then analyzing information. It is “systematic” to the extent that it is based on a method.
  • An intervention is an action taken with the intention of creating an impact on society.
  • For more information, see the Glossary.


To identify the type of approach you need, you must first address a number of topics. These are summarized in the following table.

Why evaluate?

There are several reasons why an organization may want to engage in an evaluation process. Among these are: estimate the relevance of an intervention; support planning; improve an intervention; support accountability; empower participants; value the contribution of staff and volunteers; contribute to the advancement of knowledge; establish credibility; increase its level of recognition; and hope to access funding.


See more details

Regardless of who initiates the evaluation process, it can serve different purposes.

Introductions to evaluation often begin by contrasting summative and formative purposes.

  • A summative evaluation is an evaluation whose purpose is to make a judgment about the relevance of an intervention, and to continue or discontinue it. It is usually carried out at the end of the intervention and, to ensure neutrality, by observers who have not taken part in the action.
  • The evaluation is said to be formative if it aims to assess an intervention during the implementation stage with a perspective of improving operations. It tends to be carried out in close collaboration with those involved in the intervention, and its validity will be more internally focused.

That said, evaluations may be pursuing many other purposes, which then fall somewhere in between the two extremes of this summative-formative spectrum. For example, evaluation can also be a means of:

  • becoming more accountable;
  • empowering participants or valuing the contribution of employees and volunteers;
  • contributing to the advancement of knowledge; and
  • particularly in the case of certain approaches to measuring social impact, getting the results of its action recognized, getting its action legitimized and gaining access to funding.

These various purposes are listed in our decision support tool at the bottom of this page.

What needs or objectives are organizations actually trying to meet?

At the end of 2016, a team from KPMG France conducted a survey among 366 respondents (associations, social enterprises, cooperatives, donors) to find out about their practices in terms of measuring social impact.

Source: KPMG France. (2017). Baromètre de la mesure d’impact social. p. 13.

The donors who responded to this question prioritized the same needs, albeit with slightly different rankings: “Improve our activities” ranked first (85.7%), followed by “Offer greater accountability for our partners” (71.4%) and by “Communicate about our activities” (50%).

This barometer was updated in 2018.

Additional surveys that offer information about the needs and practices of organizations with respect to social impact measurement are:

in Canada

in the United States

in Australia


For whom to evaluate?

The request for an evaluation can be initiated internally (e.g., by managers, board of directors, etc.), externally (e.g., by donors, regulators, etc.) or be motivated by various stakeholders (e.g., members, users, etc.). Consequently, it can also address different audiences.


See more details

Several actors can contribute to initiate an evaluation process:

  • Managers of an organization (whether the board of directors or senior management) may conclude that a more systematic evaluation of their activities would be desirable to help them improve their activities, motivate their employees, clarify and maximize their impact or convince other stakeholders of their relevance.
  • Donors may request such an approach in a more or less restrictive manner, for example, by imposing it as a condition of funding. Funding may or may not depend on the results of the evaluation; but in almost all cases, some form of accountability will be required.
  • Other stakeholders, such as members, employees, users, beneficiaries or consumers of the goods and services provided by the organization, may exert some pressure for accountability. This pressure may be consistent with an internal desire to better communicate the details of the processes, results and impacts associated with the organization’s actions.

Keep in mind that this section is about the recipients of the evaluation, a group of people that is not, usually, involved in conducting the evaluations. The answer to this new question depends on the approach and method you choose.


With a focus on which object?

The evaluation can focus on different aspects: needs, relevance, processes, implementation, effectiveness, efficiency, results, outcomes and impact.

 


 
See more details

Based on the logic model, we can classify the various types of evaluations according to the aspect or timing of the intervention being studied, which we call their object.

The needs assessment aims to determine the needs, understood here as the difference between the current situation and the desired situation.

It is then possible to assess the relevance of the objectives (also called strategic evaluation). The question then becomes one of determining the appropriateness of the link between the objectives of the intervention and the nature of the problem it is intended to address.

The evaluation of the implementation (also known as evaluation of processes) verifies whether the activities were carried out as planned and whether the input and output are as intended. There is a wide variety of evaluations related to implementation. Some are more specific and focus on activities, while others are broader and aim to study the observed impacts in a context-sensitive manner.

The evaluation of effectiveness compares the results obtained with the objectives of an action.

The evaluation of efficiency compares the results to the resources (input) that were required to achieve them.

Outcomes evaluation (also known as performance evaluation) is concerned with both the output and the resulting impacts of the intervention.

The impact evaluation refers to the assessment of all the consequences attributable to an intervention, whether direct, intermediate or ultimate, on or off targets, intended or unintended. In line with the vision we introduced in the “Definition and main steps” section, this evaluation can be understood as synonymous with impact measurement.


 

Which approach?

Finally, the evaluation can also be conducted externally (consultants), internally (staff) or by a combination of both. Several approaches can be mobilized: directive, participatory, practical, emancipatory, evolutionary and many others. The preferred approach will generally depend on your response to the questions above.


See more details

Beyond the questions of purpose and audience, the evaluation can be conducted with a variety of perspectives. A large body of literature exists on the different approaches to evaluation. Hence, this list is not intended to be exhaustive, focusing instead on the concepts with which managers of social economy enterprises are likely to be confronted.

The people conducting the evaluation can be situated along an external-internal axis.

  • An externally led evaluation involves the participation of a person or team independent of those responsible for the intervention.
  • Conversely, an internally driven process is carried out by those in charge and by the agents of the action.

Evaluation approaches can also be organized around a directive-participatory axis. However, while we tend to associate external with directive and internal with participative, these associations are not binding or necessarily applicable. For a better understanding, we specify what we mean by directive, participatory and developmental as follows:

  • In the case of a directive evaluation, the evaluator adopts an expert-like, neutral role, distanced from the object being evaluated, and makes the decisions alone. The role of other actors is limited to that of information source.
  • In contrast, in a participatory evaluation, project stakeholders, from project team members to recipients and donors, have the opportunity to provide feedback on the project and, if appropriate, to influence the development of this or future projects. This second type of approach may be preferred for pragmatic (pushing for the use of results), political (democratic concern for participation) and epistemological reasons, although the ultimate judgement is based on a plurality of points of view. Also, participatory evaluation can be broken down into different categories:
    • Practical, serving a specific purpose such as solving a particular problem, improving a program, or informing a decision.
    • Emancipatory, explicitly serving the purpose of empowerment, with the evaluator acting as facilitator and the participants making the decisions.
    • Negotiated, used in the context of the evaluation of community organizations in Quebec and referring to an evaluation where the organization is recognized as a partner of its funder, the government.
  • Finally, developmental evaluation supports learning in complex and changing contexts by providing real-time feedback on the project.

To find out more about:


  • For more information, see the Glossary.

 

In light of this information, we suggest that, rather than getting directly involved in measuring social impact, you take the time to properly address these issues and to choose an approach that best suits your needs and constraints in terms of purpose, audience, object and, of course, available resources (time, money, human resources). The figure at the top of the page is intended to guide you through this decision.

You may well recognize yourself in more than one of these purposes. In that case, we recommend taking the time to reexamine your needs and means, and to prioritize rather than trying to do everything at once.

In short, make sure you are embarking on an evaluation process for the right reasons. Ideally, it should above all serve you rather than be a constraint.

Want to know more?