Let’s get started.

As we discussed in the welcome, over the course of this training we will intersperse discussions of terminology, concepts, and tools with examples from the case study you were introduced to in Block One, and of course sessions in which you will develop your own M&E Frameworks. Our goal is for you to have a nearly completed draft of your M&E Framework by the end of this part of the workshop.

As you’ll see, we’re going to ask for your active participation throughout the sessions. So, we hope it will be an engaging experience for us all.

In this session, we’re going to start with the fundamentals: defining “monitoring” and “evaluation”, discussing why and how monitoring and evaluation can be important for our work in research capacity building and, finally, talking about why it’s essential to establish clear, shared goals and objectives for our research capacity building activities.

As you know, we have some general learning objectives for this workshop, which are for you to be able to:

1. Apply relevant terminology and concepts to describe, discuss, and analyse M&E frameworks and findings; and

2. Generate and implement M&E frameworks for research capacity building activities

We have also defined specific learning objectives for each session during this workshop. Here are our objectives for this first session. We’ll return to check in with these at the end of the session.

We’re going to start out with the basics, by asking you:

“What do the words, “monitoring” and “evaluation” mean to you?”

We probably all have slightly different definitions in mind. These are words that are used in many different contexts, some technical and some every day. At the end of this exercise, we’d like this group to share a common understanding of what monitoring and evaluation mean, and the difference between them.

Here’s how the exercise is organised:

First, each of you will write down what “monitoring” means to you. You have 3 minutes. [Either distribute papers or ask students to complete this exercise on their computers or other devices.]

Next, you will find a partner and exchange what you have written. Discuss the points you agree on, what you disagree on, and what questions you have based on these differences. You will have 7 minutes for this exchange.

Finally, we will come back to the group and each pair will present their definition. As a group, we will develop a shared definition. We’ll give ourselves about 15 minutes for this conversation. When we finish up our definition for “monitoring”, we’ll repeat this exercise for “evaluation”.

[As you develop the group conversation, be sure that you provide sufficient input to help bring the group to a consensus definition consistent with the definition used in this workshop. Pay special attention to potential overlap with “evaluation.”

Repeat the same exercise but this time focusing on the meaning of evaluation. Pay special attention to potential overlap with “monitoring.”

Write key points and contributions during the discussion on a whiteboard / blackboard or flipchart. As you develop a consensus definition, you can write it here as well.]

Great, now that we have this understanding of how we are defining “monitoring and evaluation” in this workshop, let´s get into some more details.

We’re going to start out with the basics, by asking you:

“What do the words, “monitoring” and “evaluation” mean to you?”

We probably all have slightly different definitions in mind. These are words that are used in many different contexts, some technical and some every day. At the end of this exercise, we’d like this group to share a common understanding of what monitoring and evaluation mean, and the difference between them.

Here’s how the exercise is organised:

First, each of you will write down what “monitoring” means to you. You have 3 minutes. [Either distribute papers or ask students to complete this exercise on their computers or other devices.]

Next, you will find a partner and exchange what you have written. Discuss the points you agree on, what you disagree on, and what questions you have based on these differences. You will have 7 minutes for this exchange.

Finally, we will come back to the group and each pair will present their definition. As a group, we will develop a shared definition. We’ll give ourselves about 15 minutes for this conversation. When we finish up our definition for “monitoring”, we’ll repeat this exercise for “evaluation”.

[As you develop the group conversation, be sure that you provide sufficient input to help bring the group to a consensus definition consistent with the definition used in this workshop. Pay special attention to potential overlap with “evaluation.”

Repeat the same exercise but this time focusing on the meaning of evaluation. Pay special attention to potential overlap with “monitoring.”

Write key points and contributions during the discussion on a whiteboard / blackboard or flipchart. As you develop a consensus definition, you can write it here as well.]

Great, now that we have this understanding of how we are defining “monitoring and evaluation” in this workshop, let´s get into some more details.

Monitoring is sometimes referred to as “process evaluation.” It’s an ongoing, continuous process of data collection. Monitoring involves capturing what we are doing and tracking changes in our performance over time.

Evaluation is the use of social research methods to systematically investigate how well our activities have met expected objectives. It requires a special methodological design and, in some cases, might require a control or comparison group.

Let’s take another look at monitoring, from a different perspective.

Monitoring is routine. It tell us what we are doing by tracking inputs and outputs.

Depending on the kind of activity or programme being monitored, monitoring can address questions like: Are activities being carried out as planned? Are the activities accessible? Is the quality of the activities adequate? Is the target population being reached?

Let’s use a simple example based on this activity – a training on monitoring and evaluation. For the purposes of this exercise, let’s say that it’s an activity that is developed over three days. In order to track the quality of this workshop, we will ask you to complete an assessment form at the end of each session or day. This data will help us to track the level of student satisfaction with our activity over the course of the workshop.

Let’s take a look:

The Y-axis represents student satisfaction. The X-axis is time.

Each day we record information about student satisfaction. We could see it develop in different ways. It might start low and improve, start high and stay high (we hope!), start low and stay low, or perhaps worst of all, start high and plummet. In any case, we will capture this data, which we will also use when we evaluate this activity.

Monitoring is routine tracking of information about an activity and its intended inputs, outputs, and outcomes. It measures progress toward achieving objectives; often involves counting what we are doing.

Here’s another quick example:

Let’s say we had a budget of $1,000 USD to develop the materials for this workshop, which took us 4 weeks to produce. Monitoring can help us understand when things are going as planned in terms of budget, and when we might be in trouble. If we collect data on expenditures on a regular basis, we’ll know when (and hopefully well before) we are in trouble.

Evaluation is an episodic assessment of achievement. Did your activity do what it set out to do?

For example, let’s think about the example of Maria and her GCP workshop, which we saw in Block 1. If you remember, Maria has established, “increased staff capacity for implementing GCP in the Phase III vaccine trial,” as her desired outcome for the workshop. She uses pre- and post-training evaluations -- through work observation -- of the level of adherence to GCP among staff members to measure this outcome.

Of course, it’s important to remember that the GCP workshop is not the only factor influencing staff capacity. We’ll get into the issue of attribution later in this session, but Maria can make useful observations about the outcomes of her activity through her post-training evaluation. This will certainly help her assess if she has met her desired and expected outcomes.

Evaluation addresses questions like: What outcomes are observed? Does the activity make a difference? To what extent is the activity responsible for the observed changes? The latter question is directed at measuring the activity’s impact. To measure impact, an evaluation is typically conducted at the start of an activity or programme and again at the end, rather than at repeated intervals while the activity is being implemented.

Since you are all here with us, we assume that you think that M&E is important, or you at least think it might be important. So, let’s have a quick round to see why. Why do you think that monitoring and evaluation are valuable processes and activities?

[Develop a group conversation around the potential benefits of implementing M&E. Write key points and contributions during the discussion on a whiteboard / blackboard or flipchart. As you develop categories of benefits you can write them down as well.]

Here are five big categories of benefits that we have identified:

Effectiveness measures the degree to which results and objectives have been achieved. An effective programme or activity is one that achieves its predetermined targets and objectives.

Efficiency measures how well inputs – resources -- were used in the creation of outputs. An efficient programme or activity is one that achieves its objectives (outcomes) with the minimum expenditures of resources (inputs).

M&E is information use, and we use M&E to share information with people, both inside and outside of our teams, departments, and institutions. The information we generate through M&E can help keep stakeholders engaged.

The data we gather through M&E can help us to objectively compare what we wanted to happen with what really happened.

So, M&E is another tool to help us learn from what we do – our successes and our challenges. In this sense, M&E should be a way of doing things – a mindset – rather than a one-off activity. And it’s not something that just “comes at the end.” Ideally, an M&E framework will be well-integrated with and developed alongside the activity itself.

So, what do we need to begin integrating M&E into our activities?

Well, in order to create an effective M&E plan, we need to know our goals and objectives. This process helps us build our activity and understand if it is successful. We need to be able to answer, very clearly, “why are we doing this?” and “what are we doing this for”? This will help us understand what we consider to be success, and how to best measure that success, and it allows us to share goals with others. It’s almost always necessary to involve team of people in our work and it’s always best if they understand, and hopefully share, our goals and objectives.