Key Takeaways
- Use five levels of data to systematically evaluate professional learning. Make sure you don’t skip a level.
- All professional development should also be systematically planned. The five-level model can also be used to plan professional development.
How to Know If Professional Development Works.
Better teachers produce better student results. Sounds simple, doesn’t it? So, we just need to improve the teachers. However, we know this can be much harder than it seems. Many of the ‘common sense’ approaches to improving teachers, schools, and student outcomes, simply don’t work. Thankfully, there are evidence-based ‘safe bets’ to improve student outcomes, teacher quality, and our schools. Quality professional development, is one thing that is likely is likely to improve student outcomes.
Let’s face it though, there is quality professional development, and then there is stuff that’s just being run to fill in time. Is the professional learning you’re delivering something that results in better practice or just an exercise in compliance, clock watching or big thinking with little improvement? To really know, you need to evaluate professional development effectively, and this takes more than just knowing that the English teachers “don’t want the cheap biscuits next week.”
Into this space steps, Thomas Guskey, Senior Research Scholar at University of Louisville and Professor Emeritus at the University of Kentucky. Guskey knows his stuff – he’s worked as a teacher, administrator and academic, written more than 25 books and published hundreds of articles and book chapters. Guskey developed a systematic framework for appraising professional develop, the basis of which is adapted from best practice from business and industry.
Let’s have a look at what this model can tell us about developing and evaluating professional learning.
Use five levels to systematically evaluate professional learning.
So, professional development is important, but it’s not easy to know you’re doing it well. This is where Guskey’s framework is so helpful; it provides a systematic way to evaluate professional learning, starting with simple indicators and moving through to more complex data.
From Gauge Impact with 5 levels of Data
“The five levels in this model are hierarchically arranged, from simple to more complex. With each succeeding level, the process of gathering evaluation data requires more time and resources. And because each level builds on those that come before, success at one level is usually necessary for success at higher levels”
So, let’s jump in and consider each level:
Level 1 – Participant Reactions
The first step in understanding the quality of professional development is to gather data on participants’ reaction to the learning provided. This is the most common form of evaluation of and is generally simple information to both collect and analyse.
Guskey suggests asking participants a mixture of different questions, from three broad areas.
- Presenter/s: Was the leader knowledgeable, credible, and helpful? Did they presenter speak clearly?
- Content: Did the content and material make sense? Were the activities relevant to the purpose?
- Context: Was the room/ the right temperature? Were resources conducive to learning?
Guskey acknowledges that some will critique this level as superficial, particularly that of context questions. However, think Maslow’s hierarchy of needs – uncomfortable people are not going to learn anything.
From Gauge Impact with 5 levels of data:
“Some educators refer to these measures of participants’ reactions as “happiness quotients,” insisting that they reveal only the entertainment value of an experience or activity, not its quality or worth. But measuring participants’ initial satisfaction provides data that can help improve the design and facilitation of professional learning in valid ways. In addition, positive reactions from participants are usually a necessary prerequisite to higher-level evaluation results.”
This level of data is certainly simple. It does not guarantee an improvement in teacher practise or student outcomes. If you’re still sceptical on this part, it’s is worth reflecting on this: which professional development is more likely to be successfully implemented and improves student outcomes, a rigorous yet poorly received initiative or a rigorous and well-received initiative? People’s initial reactions matter.
Level 2 – Participant’s Learning
Ultimately, you want professional learning to be more than feel-good moments. You want there to be teacher learning and practice change to improve student outcomes. Therefore, you need to measure this learning. This can present a challenge to some school cultures and individual staff as there can be some anxiety if professional learning is linked to internal performance and development processes. But it’s an important step to build into your evaluation because it tells you what learning has been achieved by teachers.
From Gauge Impact with 5 levels of Data
“Depending on the goals of the professional learning program or activity, this can involve anything from a pencil-and-paper assessment (Can participants describe the critical attributes of effective questioning techniques and give examples of how these might be applied in common classroom situations?) to a simulation or full-scale skill demonstration (Presented with a variety of classroom conflicts, can participants diagnose each situation, then prescribe and carry out a fair and workable solution?).”
One possible strategy is pre and post-testing. Low stakes pre-testing allows the facilitator of the professional development to check prior knowledge and adjust the learning program accordingly. Post-testing could be anything from a quiz, personal reflections, to lesson observation around a given topic. You need to be careful to ensure this remains low-stakes if you want to have a positive and supportive school learning culture.
Level 3 – Organisational Support and Change
Even if there is the most amazing professional development being delivered to an enthusiastic and dedicated staff; you need to consider whether the cultural and organisational norms of your school might undermine this learning translating to practice change.
From Gauge Impact with 5 levels of Data
“Procedures for gathering data at Level 3 differ depending on the goals of the professional learning. They may involve analyzing school records, examining the minutes from follow-up meetings, and administering questionnaires that tap issues related to the organization’s advocacy, support, accommodation, facilitation, and recognition of change efforts.”
Level 4 – Participant’s use of new knowledge and skills
Level four asks the degree to which the intended changes in practice are being enacted. Staff have completed a series of Professional Development sessions, but are they attempting to integrate this learning into their classroom routines? If so, is it being done so effectively?
From Gauge Impact with 5 levels of Data
“Depending on the goals of the professional learning, these data may involve questionnaires or structured interviews with participants and their school leaders. Evaluators might consider oral or written personal reflections or examinations of participants’ journals or portfolios. “
There are two staff practices to consider at level four that can jeopardise the influence of the professional development being conducted.
1. Speed camera behaviour
Just like a speeding car that only slows down for a police camera, some staff can perform the new skills or use the new knowledge when and only when they are being watched.
2. Lack of habit change
Changing habits is hard. Just asking someone who is quitting smoking. As Dylan Wiliam has suggested a teacher of twenty years has asked over a million questions, some new knowledge about questioning techniques is not going to necessarily change these habits overnight, even if they want to. If they do change, it doesn’t mean they won’t revert to their old practice in time, so consider how a longer period of data collection can help you determine if the change is being sustained.
Level 5 – Student Learning Outcomes
The easy solution to this level of Guskey’s model is to check the standardised test score right? If they go up the professional development was a winner, if they go down scrap it and pretend it never happened?
Learning is much messier than this. Student learning is the result of a multi-directional web of causality, not a linear chain of causality – it’s not always straightforward what has actually produced changes in student learning data. Once you add in the correlation versus causation problem, it’s really hard to track how professional learning impacts on student outcomes.
Further, even if you have some ‘winning’ learning data for a cohort of students, what would it mean if there was a very large rise in results for some students, another set of students who made limited progress and a small but significant number of student who went drastically backwards?
There is a lot of thought that needs to go into the selection of metrics used to track level 5. It’s also important to remember that metrics can be misused as often as they are not
One way to increase the validity and reliability of this process is to use multiple measures.
From Gauge Impact with 5 levels of Data
“When providing acceptable data for judging the effects of professional learning, evaluators should always include multiple sources of evidence. In addition, evaluators must carefully match these sources of data to the needs and perceptions of different stakeholder groups”
And, you want to make sure that you use measures that staff have faith in. Often staff see more credibility in measures that are closer to their “nexus of control”
From Gauge Impact with 5 levels of Data
“Teachers put more trust in results from their own assessments of student learning — classroom assessments, common formative assessments, and portfolios of student work. They turn to these sources of evidence for feedback to determine if the new strategies or practices they are implementing make a difference. “
So, it’s not a straightforward ‘look at the test scores’ thing. Instead, mixed measures with some normed reference standardised metrics as well as some classroom developed metrics may increase the support and participation from staff at level five. Student perception surveys can also be helpful here to help you triangulate the impact on student learning.
All professional development should be systematically planned.
The beauty of Guskey’s model is that is can be used both proactively and reactively. That is to both plan and evaluate professional development. To use it proactively to help you plan professional development, Guskey has three suggestions:
1. Each of the five evaluation levels is important
Neglect any level are your peril. Each level must be present for the most effective professional development. Data should be regularly collected and analysed for each level.
2. Tracking effectiveness at one level tells little about impact at the next level.
Just because the food is great doesn’t mean the staff are applying their new knowledge and skills. Conversely, you might need to sharpen your biscuit selection even though staff are kicking goals. Remember to analyse each level independently as well as the five as the whole.
3. Perhaps most important is this: When planning professional learning to impact student learning, the order of these levels must be reversed.
When planning professional development, Guskey suggests backwards planning.
From Gauge Impact with 5 levels of Data
“In backward planning, educators first decide what student learning outcomes they want to achieve and what data best reflect those outcomes (Level 5). Next, they must determine, on the basis of pertinent research, what instructional practices and policies will most effectively and efficiently produce those outcomes (Level 4).”
And so on. This can allow the evaluation to function as an ongoing feedback loop rather than a post-mortem. This allows professional development to respond to the feedback and learning needs of staff.
In addition to this, the emphasis placed on starting with student outcomes (level 5) essential also places students at the forefront of staff professional development, this is essential if you want to be a “student-centred leader”.
If you want more like this, check out, Phil Naylor’s podcast interview with Guskey.
Key Takeaways
- Use five levels of data to systematically evaluate professional learning. Make sure you don’t skip a level.
- All professional development should also be systematically planned. The five-level model can also be used to plan professional development.
Thinking About Teaching 