Welcome to our last session! During this session we will start discussing data quality. This is a key issue. If you can’t trust your data, you can’t trust anything about your M&E results. We’ll talk about five criteria we can use to evaluate the quality of our data, and briefly discuss legal and ethical considerations, of which we all need to be aware.

When we approach data quality, we will always have to balance it with costs. Realistically, we’re not going to be able to eliminate all threats to quality, but we can identify them and manage them well. In some cases, the best we can do will be to explicitly acknowledge that there is a threat that we can’t mitigate. That’s still important to do. In other cases, we’ll be able to take concrete steps to protect quality.

Data quality can be managed by systematically addressing five principal threats.

But before we discuss how to manage quality, let’s figure out what “quality” really means.

Can I borrow someone’s mobile phone? Actually, I need 4 volunteers to show me
their phones. Thanks!

Ok, let’s examine these 4 phones. They are all just a bit different. Which one do you
think is the ‘best’ mobile phone?

[Ask for several opinions] I like this one because it’s the biggest! Ok, WHY do you
this phone or that phone is the ‘best’?

[Ask for several reasons why one mobile phone may be ‘best’ or ‘worse’ than the other]

So you see, we all have slightly or greatly different ideas of what constitutes “quality”.

When it comes to data quality, we can’t all have different definitions. It’s important that we share an idea of what quality means in this context so that we can analyse, interpret, and share results of our M&E process in a way that generates useable information.

So, let’s focus on how we should define “quality,” not for smartphone, but for data. Here’s how the exercise will work:

• Now, we will divide up into groups of about 3-4 people [change as necessary according to the group size, space, etc.]

• Each group should produce a definition of “quality” as it applies to data. Be prepared to argue for your definition! You have 10 minutes to produce your argument, then we will come back together. Each group will present their definition and then we’ll see if we can produce a consensus.

Use your blackboard, whiteboard or flip chart to capture the main points raised by each group, then guide the group discussion towards a consensus. In the next slides, you will present the definition that we use here in the workshop, which you can compare with the definitions created by the students.

So, here’s how we look at data quality in this workshop. Let’s see how it matches up with what we just came up with in the exercise.

We use five criteria to identify and measure data quality: validity, reliability, timeliness, precision, and integrity. We’ll look at each one in detail.

Validity means we are measuring we have intended to measure or capture. We can’t assume that this is the case. It’s important to go back to the purpose of an indicator to be clear on what we are measuring and why.

Threats to validity include:

Definitional issues. Remember, any word in an indicator could be understood differently by different people, or in different contexts. Even superficially simple words, like “service,” “people,” “person,” or “number of” might not be clear enough.

Proxy measures: We’ve already mentioned proxy measures in our indicator session. Are we sure that the proxy we’ve selected is really getting at the indicator we want to measure?

Inclusions / Exclusions.

Data sources.

In the upper right-hand corner we see an indicator that Maria included in her M&E Framework. What is a threat to validity in the wording of this indicator?

What about the word “trained”? How is this defined? By people who passed a test? By people who attended all the days of the workshop? By people who attended one day? What would you do with this indicator to try to mitigate some of that possible confusion?

What do we mean when we say that something is “reliable”?

Reliable measures are consistent. They measure the same thing, the same way, every time.

There are three factors to consider when we look at threats to reliability: people, places, and time.

These three factor can affect our ability to do things consistently.

Collection methodologies, when not standardized or understood in the same way across collection areas, can affect data reliability. People use different collection methods or use the same collection methods, differently.

Collection instruments can also lead to inconsistent measures.

You can have data quality assurance measures in place, but if data collection personnel are not fully trained, a wide range of threats to the reliability of your data may still exist. Training personnel does not simply mean handing over an instruction sheet or providing an overview of the data collection tool. Thorough training is key.

Look at the upper right-hand part of the page. Maria’s workshop is held over two days, and full attendance is required in order to pass and receive the certificate. On the first day of the workshop, Maria passed this around the classroom at 9am, when the first session started. She then collected it, took it back to her office, and transferred the names to an Excel spreadsheet. On the second day, her colleague taught the workshop. Instead of passing the sheet around at 9am, she herself marked the sheet at 2pm, when the group returned from lunch break. Then, she delivered the sheet to Maria. What are some potential issues with the collection instrument she is using to measure attendance?

Data are collected for use. So, it only makes sense that we need to collect data within a timeframe in which it is useful. If data can’t be collected, reported or used while it is still relevant and useful, then it’s probably not worth the effort or resources to collect it at all.

Threats to timeliness include: Collection frequencies, which can be too frequent or not frequent enough in relation to the data, reporting, or decision making.

If we don’t leave enough time to collect data, we might be tempted to rush, leaving us open to error, but if we leave too much time, we might not capture the information we need.

Let’s think about Maria’s case. With the clinical trial set to start in two months, and the Foundation asking for an evaluation report, what is the timeframe you think she has to work with for collecting post-training work observations? How would you organize it?

When we use data, we know that there will probably be some errors made at some point. This is something that we try to avoid, but it often occurs despite our best intentions. However, is the margin of error in the data less than the expected change the activity was intended to effect? If not, what are we measuring? An introduction of bias into the data likewise affects its accuracy.

Threats to precision can include: Source error: which is when the person or tool doing the measuring introduces bias into the measurement. Instrumentation errors can also occur when a measurement or data capture tool or method is designed in a way that introduces error. Manipulation, as simple as translating numbers into averages or percentages can introduce error. Transcription is a classic way to introduce error.

What if Maria makes a mistake when she moves her sign-in sheet from a paper version to an Excel version? What could she do to avoid that?