Data can deceive. Get to know your data better to avoid this

The Idea: Data-informed decision-making is very valuable, but the users must also ‘know their data’. This means that users should not just jump to drawing conclusions for data they have access to, but also know what specifically the data is measuring, and how the data was collected. Without this crucial aspect of data literacy, the user is susceptible to deception by data. Encouraging questions and conversations during the data analysis process can support the development of this ‘know your data’ aspect of data literacy.

The deceptive data that almost made a nurse commit a fatal mistake

A nurse, Mary, notices that a baby in the NICU is starting to turn blue. Her colleague, also a nurse, looks at the heart monitor - it shows a beating heart so she deduces that it is a respiratory issue and quickly prepares to support the baby's lungs. But Mary takes one look at him and says ‘It’s his heart!’. She knows it is pneumopericardium - where the sac around the heart gets filled with air and restricts the heart from beating. In fact she has seen this happen before to another baby - and that newborn did not survive because of it. The other nurse says that a heart issue is impossible - he is connected to the heart monitor, which shows the heart is beating just fine. But Mary knows the signs too well, and as soon as the doctor walks in, the nurse hands her the syringe, “It’s pneumopericardium, quick, punctures the air sac’. The doctor quickly assesses the nurse's diagnosis, agrees with her and acts. They are correct and the baby’s life is saved.

What happened with the heart monitor? Why was it still showing a heart beat even though the baby’s heart was not beating?

The heart monitor measures the electrical activity that regulates the beating of the heart muscle, not the actual heart beat i.e. the physical contraction of the cardiac muscles. So the heart nerves were firing electrical impulses just fine - telling the heart to beat at the required rate. But the muscles could not contract because its movement was obstructed by the inflated sac around the heart, and it was not actually 'beating'.

What has this got to do with using data to inform decisions?

It perfectly illustrates the potential errors that can occur when you do not ‘KNOW YOUR DATA’ well enough before using it to inform decisions.

If the nurse simply made a decision based on the ‘data’ from the heart monitor was indicating without thinking critically, the results could have been fatal.

(I have adapted a Gary Klein story referenced in the book ‘Made to Stick’ by the Heath Brothers. It has been used to illustrate the power of stories, intuition and much more, but we will use it to illustrate the importance of understanding the limitations of data!)

Know your Data - what does this mean?

In a previous blog post, I had written about 3 mistakes people make when using data to inform decision-making. #3 was ‘Not doubting the data enough’. I will expand on that point in this blog post. 

We have to make many important decisions at work, and it is good practice to use data to inform them.

To do this we need data literacy which involves

1. ‘Knowing your data - what it is and how it is collected’ + 
2. ‘Using it to inform work ‘- analyzing and interpreting it in context +
3. ‘Communicating it to your team’ - sharing information in a meaningful way with other stakeholders

‘Knowing your data’  is knowing what the data can tell you about your work + knowing what specifically the data measures and knowing how the data is measured

In the medical field, we evaluate the health of a person using data from a wide-range of sources like the ECG heart monitors in the story or various others like blood tests, imaging technology like x-ray machines, sonography machines, fMRI machines and so much more.

For the medical example in our story:

Knowing the ‘Heart Rate monitor’ can help me understand how the heart is functioning 

+

Knowing the ‘Heart Rate monitor’ measures the heartbeat/ heart rate
Knowing the ‘Heart Rate monitor’ uses the electrical signal generated by the heart as an indicator of a beating heart.

Knowing what exactly this data measures and how it is measured

In our medical story, knowing how the data is collected - detecting an electrical signal, not muscular movement, was critical. Without this knowledge, the data has the potential to mislead.

In education, knowing what the data measures and how it is collected is also critical. Without this, organisations are at risk of being misled by data.

We evaluate the education / learning of children using achievement on standardised tests, teacher-led assessments, classroom observations of students and teaching practices, stakeholder (parent, student, teachers) feedback etc.

Here are some examples of of what it might mean to ‘Know your data’

In my experience, I have seen situations where the data from standardized tests have not been the best predictors of students ability to implement a project successfully. Or the data from classroom observations has been complicated by the fact that all the evaluators (observers) had a big difference in their own interpretation of the classroom observation rubrics. Or a parent survey data that is impacted tremendously by the 'recency effect' or ‘availability bias’ - where their ratings have been obviously swayed by a positive (enjoyable performing arts showcase)  or negative (announcement of increase in school fee)  experience from the month before.

Every data point has important information about what is measured, and equally important information about WHAT IS NOT MEASURED.

2 things you can do to encourage ‘Knowing your data’:

Encourage conversations between stakeholders during the process.

Very often the teams designing the data system (Leaders, data analytics team), the consumers of the data (teachers, students) and the communicators of data-related information (Leaders, communications teams), are different. And often the process that is set up is an asynchronous one, where each team ‘hands over’ information to the next team, usually because everyone has too much work on their plates and it is not feasible. But to build data literacy, conversations between these stakeholders is essential. Hearing each teams perspective builds a more comprehensive understanding for everyone. 

Encourage data end-users to ask Questions while analyzing data.

I have noticed that many teams are eager to dive into the data to simply draw conclusions. But it is equally important to train teams to ask meaningful questions about what specifically the data is measuring and how the data was collected. What tools were used for this? Who was involved in collecting this? How was it collected? How was the average value calculated? There are literally hundreds of details that are relevant to interpretation. This is an essential part of building data literacy in the team. Point 1 above - encouraging conversations between stakeholders will support this process.

 

I am a big advocate for using data to improve learning in schools. To ensure improvement, it is important to a data culture that values 'knowing your data'!