Welcome to our fourth session! During this session we will start discussing indicators.

Creating an M&E framework helps us understand what’s most important to us – what change we want to see as a result of our research capacity building activities. If we’re able to measure this change, we can get a good sense of how successful we have been, and what changes we need to make to improve our activity. In order to do this, we need to figure out how to measure what we want to measure. This is where “indicators” come in.

In this session, we will talk about what an indicator is and how to develop and select them for your activity.

Here are our objectives for this session. We’ll return to check in with these at the end of the session.

We can say that an indicator has three basic characteristics. It’s a variable that measures one aspect of an activity.

As you develop your M&E framework, you’ll need at least one indicator per component of the results chain – that’s input, output, outcome, impact.

Now, let’s take a few minutes to unpack these three characteristics.

The value of the indicator varies between a given, reference level, measured at the start of the activity, and another value measured after the activity has had time to lead to results, when the indicator is again measured.

Indicators measure resources, results, or change. So, a good indicator must be measurable. It’s important to keep this in mind as you design your indicators.

It might be easy to think about ways to measure indicators that are inherently numerical, like students that graduate, or M&E Frameworks that are graded, but what does this mean for qualitative indicators? What about student experience? Satisfaction?

There are entire methods of qualitative evaluation, but can we use qualitative data and still have measurable indicators within a results framework?

What do you think?

Qualitative indicators can provide enormous value to M&E processes. There are many ways to develop qualitative indicators to capture this important information and still have comparable and measurable data. We can use scales, rubrics, coding, and other methods to develop these indicators. In some cases, you might also choose to incorporate qualitative methods – like case studies, interviews or focus groups -- into your M&E approach.

Indicators are specific – they should just measure one thing – and should be narrowly defined in order to determine the aspect measured as precisely as possible. A well-rounded M&E plan should have at least one indicator for each component of the results chain. That doesn’t mean that more indicators are necessarily better. It’s key to determine what’s important to you and measure it. More measurement for measurement’s sake can be a waste of time and resources.

Anything can be measured. But not everything you can count matters. And, of course, not everything that matters can be counted.

Indicators are not just anything you can think of to measure. And every measure is not an indicator.

Are there some indicators that we could produce right now about this workshop that are variable, measurable, and deal with a specific aspect of the activity – but that are not very illuminating or important?

Indicators are not objectives or targets, but actual results. For example, 12 students graduating from a course, is not an indicator. Why? What would the indicator be in this case? Why is “12 students” not the indicator?

Indicators don’t use words like “improved,” “increased,” or “gained,”: they don’t show bias.

So why is this important?

Indicators are where data enter our M&E system. They help us to see what we have done and to put it in measurable terms. And they can help us to manage a lot of data and make it understandable and useful. Once this information is placed back into the context of your activities and programmes, it can then be used to check progress and inform future decision making.

So, how do you choose the right indicators?

Well, in some cases you may be given indicators – by a donor, for example. Or you might choose or be asked to use indicators that have been used for other activities at your institution, so that you can easily compare results.

When you can select your own indicators, you can follow five steps to ensure that they are appropriate and effective measures.

First, return to your goals and objectives, and review your results chain. Identify what needs to be measured. Good indicators start with clear results chains. Start with the overall objective or goal and work backwards.

Next, develop a list of all possible indicators you could use. You don’t have to limit yourself at this point - it’s a brainstorming exercise meant to generate as many measurement ideas as possible. Think of internal sources of information for possible measurements, and also look elsewhere for examples of how other programmes have been measured.

When you have a list of possibilities, consider each one.

Is it:

Measurable. Can it be quantified and measured?

Practical. Can you really collect the necessary data for this indicator on a timely basis and at reasonable cost?

Reliable. Do you think it could be measured in the same way, by different people?

Relevant. Does this indicator measure a change that is in some way attributable to your activity?

Useful. Will you be able to use the information you get from it to make changes?

Direct. If you use this indicator, are you measuring what you really intend to measure?

Sensitive. Will changes in the value of this indicator allow you to respond to early warnings of changing conditions?

Capable of being disaggregated. Will you be able to analyse and break down data by gender, age, location, and other important factors?

Once you’ve thoroughly reviewed your options, select the indicators that you will use. Keep in mind that:

You need to balance strong indicators with costs. This balance is important – you want to get the necessary information without overloading your M&E system, making it cumbersome and costly. Limit the number of indicators used to track each component to two or three. When selecting your indicators, keep your audience in mind. What do they need to know?

Finally, once you have chosen your indicators, establish expected or desired results for each one. These results should be realistic and reflect your intention for the performance of the activity. These expected results will help benchmark your M&E process and will give you a point of comparison for your final evaluation.

Sometimes, it’s not possible to measure the actual thing that we want to measure. In these cases, we can choose to use a proxy indicator.

A proxy indicator is an indirect measure that we use to get data that is indicative of the desired result. For any proxy measure, it will be important to reiterate the measurement and its possible limitations, and to account for this limitation as data moves through your M&E system: from collection, to aggregation, and to use.

What are some example of proxy indicators that we might commonly use for research capacity building activities or training programmes? Perhaps a simple one is exam results, which are often used to measure information or knowledge acquired. Is the result of an exam a true measure of knowledge? Of course not, but it is certainly used, at least in some cases, as an acceptable alternative.

As we know, Maria really wants her GCP workshop to improve participants’ competence, so she’s trying to formulate indicators – and determine data sources -- to allow her to do this.

As a group, let’s help Maria complete her thought process.

What do you make of her ideas so far? How would you refine and organise her work to establish the indicator, expected results, and data source?

Lead group discussion to arrive at the three main points.