How To Measure What Matters In Schools


Key Takeaways

  1. Don’t always trust the metrics. Sometimes metrics are not a valid proxy for what you actually want to achieve.
  2. Metrics can be distorted. Metrics are fallible, prone to gaming and corruption – don’t take them as gospel
  3. Use metrics wisely. Be smart about what and how you measure things in schools
  4. Don’t forget the humans. It’s the relationship between the teacher and student that matters most

Want research-informed answers like this direct to your inbox?
Sign up to our mailing list.

How To Measure What Matters Most In Schools

“What gets measured gets managed.” This quote is often attributed to Peter Drucker – the management guru and is used by many in the corporate and educational worlds to suggest that a focus on key metrics is central to success. Accountability is important. In Australia, the Federal and State Governments subsidise both public and private sector schools to the tune of billions of dollars and it is important to know that that tax dollars are being spent wisely to deliver a return on this investment in student outcomes. During the last decade in Australia, this drive towards accountability has increased the use of metrics in schools. Examples of this include the introduction of NAPLAN testing, the ‘my school’ website and the greater focus on international testing such as PISA and TIMSS.

This metrics push may not be entirely good. American Professor Jerry Z. Muller calls into question an overemphasis on tracking quantitative measures. His book ‘The Tyranny of Metrics’ explores numerous case studies from healthcare, education, military and business contexts to argue that metrics are not the promised ‘silver bullet’ to improve outcomes. Indeed, Muller provides a myriad of examples where a metrics culture has not just failed to improve outcomes, but in some cases has actually had a detrimental impact.

Let’s unpack three things that can help us measure the things that matter in schools.

1. Don’t always trust the metrics

One of the examples explored by Muller is Secretary of Defence McNamara’s fixation on key metrics such as productions, bombs dropped, and enemy combatants killed to evaluate the USA’s performance during the Vietnam War. 

From The Tyranny of Metrics:

“These statistics were frequently exaggerated in order to boost the commanding officers’ chance at promotion. And the stream of seemingly objective but actually fallacious information led policymakers and politicians to mistake improvement in the measured performance for real progress.”

2. Metrics can be distorted

Muller highlights a range of similar cases from other fields of endeavour that have a common thread – the importance placed on the metric. Muller argues that are all examples of Campbell’s Law (named after the American social psychologist Donald T Campbell). The law follows as such:

From The Tyranny of Metrics:

The “more any quantitative social indicators is used for social decision making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor” 

Specifically, in the educational field, Muller explores a number of mainly US examples to highlight how schools are also subject to the pitfalls of metrics. These cases illustrate how educational standardised testing can oversimplify the objective of schools when they only focus on the most simplistic aspects of this goal. In addition to this, Muller also depicts how schools can manipulate data through omission or distortion as well as some outright cheating linked to school funding and performance pay. 

In terms of standardised testing, Muller’s work is echoed by former Finnish educational leader and now Professor of Educational Policy at UNSW Pasi Sahlberg. Sahlberg suggests that high stake standardised testing is a symptom of the Global Education Reform Movement (GERM) and that the because of the social and political emphasis placed on testing, the publication of standardised literacy and numeracy test undermines education quality.

3. Use metrics wisely

So, whether you are a classroom teacher or school leader, you probably can’t avoid metrics. In an Australian context, although there are calls to reform standardised measures such as NAPLAN or Year 12 examinations; it is however fairly certain that a version of these tests are with the teaching profession for the foreseeable future. So, how does a school move beyond standardised testing to choose a wider range of metrics that support improvements in teaching and learning? Muller provides 10 points on ‘when and how to use metrics’ with some added consideration for school contexts.

1. What kind of information are you thinking of measuring? Is this information valid and reliable?

If you are measuring simple categorical data such as attendance data, then this can be easy to track and is easy to verify the validity and reliability of the data. However, if the people are involved in the process of the objective you are measuring (most things in a school) then the measurement becomes less reliable. If rewards and punishment are involved it is more likely that the validity of the measure is reduced. However, if staff are ‘on board’ with the measures in an established growth-focused school culture then the validity of the measure is likely to be enhanced. So, before you collect data consider if qualitative, quantitative or mixed data is going to better inform your judgement. Also consider, what human error and/or manipulation may occur if the data has high stakes status.

2. How useful is the information?

Does it need to be measured? If not, don’t measure it. If you already know it, could staff time be used to work on something else?  Consider the costs of gathering the data (see point 4).

3. How useful are more metrics?

Overuse of multiple metrics can cloud out or create ‘noise’ in what you are trying to measure.

As Muller contends “the fact that metrics are helpful doesn’t mean that more metrics are more helpful.”

4. What are to cost of not relying upon standardised measurement?

From The Tyranny of Metrics:

“Are there other sources of information about performance, based on the judgements and experience of… [students] or parents of students? In a school setting, for example, the degree to which a parent’s request for a particular teacher for their children is probably a useful indictor that [that] teacher is doing something right, whether or not the results show up on standardised tests.”

5. To what purposes will the measurement be put, or to it another way, to whom will the information be made transparent?

Whom will this information be viewed and used by? If data is being used with a growth mindset by teachers reflecting on practice and individually or collaboratively reflecting on practice then there is an increased likelihood of ‘buy-in’. If the purpose of this metric is performance management or considered useless or harmful by the teacher/s then there is an increased likelihood of ‘gaming’ the metric. Make sure the purpose of your metrics is growth focused.

6. What are the costs of acquiring metrics?

Metrics are never free, to collect metrics it either cost financial resources, staff labour or both and even more hours to analysis, distribute and discuss. There is an opportunity cost to developing any metric. Choose your metrics wisely, do not let metrics be a distraction from the real work.

7. Ask why the people at the top of the organisation are demanding performance metrics?

Again, is it growth focused or is it compliance focused?

8. How and by whom are the measures of performance developed?

Some metrics such as standardised testing and performance and development measures are pushed onto schools. This is an unavoidable reality. With other in-school metrics being developed, there is a chance to develop them with collaboratively with staff to increase the likelihood of buy-in and ultimately validity and reliability.

9. Remember that even the best measures are subject to corruption or goal diversion?

High stake environments lead to increased risk of a metric not achieving its desired outcome. Reflect on what unintended outcomes this metric might have.

10. Remember that sometimes, recognising the limits of the possible is the beginning of wisdom.

Sometimes, teacher or school-leader judgement is why you are a professional and have spent years of your life developing this capacity. Have faith in the judgement of yourself and your team and wider school community.  

4. Don’t forget the humans

Finally, no matter how well a school or teacher develops metrics to measure student progress, it is important to remember that this information is relationship and contextually dependant. As Dylan William suggested in both of his interviews with Craig Barton. In the end, “it all comes down to the relationship between the teacher and the student”. On a school-wide level, the relationships between staff and schools leader mirror these classroom relations. Indeed, schools can collect the ‘right’ data and have no or limit success in improving student results, if there is not a positive growth-focused culture present.

Key Takeaways

  1. Don’t always trust the metrics. Sometimes metrics are not a valid proxy for what you actually want to achieve.
  2. Metrics can be distorted. Metrics are fallible, prone to gaming and corruption – don’t take them as gospel
  3. Use metrics wisely. Be smart about what and how you measure things in schools
  4. Don’t forget the humans. It’s the relationship between the teacher and student that matters most

Want research-informed answers like this direct to your inbox?
Sign up to our mailing list.