This study is set in the context of an independent girls’ school which has been participative over a number of years in evidence based practice. As a knowledge building school it has developed commissions of enquiry investigating assessment and reporting practices across its range of faculties. The question of what constitutes fair and valid assessment in the context of a standards framework is clearly one which deserves investigation. Consequently, the Science Department chose to examine how learners in Year 8 perceived outcomes and how they might be assessed. In part, this was driven by a desire to inform both learners’ and teachers’ understanding of standards based assessment. Having a greater awareness of what is valid, and why it is perceived to be so, might be a useful attribute for both teachers and students.
As an action learning project it was essential that there be practical outcomes that could be immediately employed in the classroom. The paper focuses both upon the strategies which were used to ensure that students were fully participative in the processes of designing assessment tasks and also those associated with evaluating their responses to them. One author of this paper was engaged in the school as a teacher researcher; while the other has a long-standing association with the school as researcher in residence.
The study is a particularly timely one in that
it was enacted in a context, in New South Wales, of the introduction
of a standards framework for stage 6 learning outcomes and where a
new science syllabus for both junior and senior students has been
introduced. As, yet, there is little substantive research that is so
thoroughly contextualised that informs our understanding of fair and
valid assessment of outcomes, thus the study will contribute to
professional knowledge in the field of practice. It has a further
relevance to the notion of changing agendas in teacher education in
that it demonstrates the power of enhancing teacher professional
learning on-site in a sustained and embedded fashion.
It has been argued that one of the most effective modes of teacher professional learning is where it occurs in the context of overall school development leading to the improvement of student learning outcomes (McRae, Ainsworth, Groves, Rowland & Zbar, 2001). The study which will be reported here is one which has taken place in a context where teacher inquiry and evidence based practice have become part of the lived life of the school. Not only that, the context is also one where the school itself is a member of a loosely formed alliance which regards the sharing of professional knowledge as a key responsibility of its members.
Independent Girls School has a five year history of developing evidence based practice. A number of papers and journal articles have outlined the work of the school and its orientation to school based research (among them: Groundwater-Smith & Hunter, 2000; Goodwin & Groundwater-Smith, 2000; Groundwater-Smith, 2000; Groundwater-Smith & Mockler, 2001; Groundwater-Smith, 2001a; Groundwater-Smith, 2001b).
The school is a member of the Coalition of Knowledge Building Schools, formed in 2001. Teachers from a small number of schools, three from the government sector and three independent schools, met together and discussed the possible formation of a coalition which acknowledged the work that they were undertaking in relation to school and teacher development. They saw themselves contributing to the ongoing improvement of the work of their schools through the systematic and public collection and discussion of evidence regarding teaching and learning within the lived life of the school. They had a view that evidence was best considered in a forensic rather than adversarial environment; that is to say that it should be constructed and examined in ways which illuminate understanding rather than as a means of proving a particular case.
The participants in the discussion saw themselves having as their purposes:
The embryo coalition believed that by embedding inquiry practices into the daily work of the schools it would be possible to evolve an authentic workplace learning culture. They recognised that professional learning is not an exclusively individualistic enterprise but that learning and growth can take place at the organisational, or corporate level. What is of particular note is that the coalition did not form in response to external initiatives such as a funded program or university partnership, but because the schools themselves had an expressed desire to work in a particular way. Having said that it was also critical that the Centre for Practitioner Research, situated at the University of Sydney was able to, and indeed desirous of, supporting the coalition and providing it with some sort of institutional base.
In considering the necessary factors for the knowledge creating school Hargreaves believes it is likely to be one in which the following conditions, inter-alia, prevail:
Clearly, central to the development of the knowledge building school is the notion of evidence and its relation to practice. Before discussing the detail of this particular project within Independent Girls’ School it is worth considering the notions associated with evidence based practice within knowledge building schools.
How then do we best understand the notion of evidence based practice and the ways in which it contributes to the knowledge building school and practitioner inquiry?
As we have already indicated, we can think of the purposes for gathering evidence in two ways. The first of these is to use the evidence in adverserial settings where it is utilised to prove a case. Those seeking for that elusive, indeed we would argue impossible goal ‘best practice’ would wish to prove that one method is unarguably better than another. Thus, in medicine, using randomised control trials, there are those that seek for the ‘best treatment’ irrespective of the multitude of variables within any medical condition. Similarly education has been beset by the ‘best practice’ holy grail; as if it is possible to identify one best way, for example, to teach reading, or for leaders to organise teacher professional development.
The second purpose in gathering evidence is to conceive of it within a discourse of forensic science, where the investigator is seeking above all else to understand a particular phenomenon, for example how students understand assessment works in favour of their learning. Knowledge building schools clearly wish to achieve a deep understanding of that which happens within them: teaching and learning; managing human and material resources; communication and participation; and so on. Of course, this does not mean that the schools should not concern themselves with the quality of evidence, but rather the purposes to which that evidence is to be put. Norris & Robinson (2001) quite properly point out that a distinction should be made between weak and strong evidence.
Evidence gathered by practitioner inquirers in knowledge building schools needs to stand a number of tests:
Since these issues have been discussed elsewhere (Groundwater-Smith & Hunter, 2000) we shall proceed now to the specific project with which this paper concerns itself.
As a result of the many studies undertaken at Independent Girls School it was determined that a major thrust for 2001 would be a school wide investigation of assessment and reporting practices, conducted by the Heads of Department as Commissions of Enquiry under the guidance of the Director of Curriculum.
The Commissions of Enquiry should be seen as more than just a tool. In 2000 the School decided to change its mode of whole school professional development. Prior to that time, the first day of each term was designated a pupil free day and staff were in-serviced on a topic often by an outside expert. The new method was called Commissions of Enquiry and in 2000 they involved groups of teachers working on topics of interest in an action research mode. The school day was shortened every Wednesday by 25 minutes. Staff were expected to work back every second Wednesday from 3pm to 4:30pm to participate on their Commission. Facilitators were trained in the SCOPE method of action research and the Commissions ran for three terms.
Early efforts were disappointing. Some groups produced work that led to improvements in students learning while others did not. Some groups were too large to be manageable. Many of the facilitators did not have official leadership roles within the school and there was a lack of support for them in their role within a structure which was in transition from hieararchical to flatter management.
However, in 2001 it was decided that the method was worth using again, but that groups would be organised along departmental lines. The Head of Department would be responsible for conducting the enquiry in their discipline area. The Director of Curriculum would support the Heads of Department in monitoring and driving the process.
Assessment and reporting were set as the topic for the Commissions in 2001. This was particularly timely, given the changes to standards based assessment within the New Higher School Certificate. A number of questions, which were to be answered by each department:
Recent changes to the New South Wales Higher School Certificate had moved assessment in year 12 from a norm-referenced procedure to a standards-reference framework. Teachers had had to change their assessment procedures to fit this new system. The mandatory nature of these changes had introduced those staff, taking senior classes to the terminology and practices of a more student centred system of assessment. This would prove useful in achieving change at other year levels.
Term 1 was used as a preparation period with the Heads of Department. Two of the regular Curriculum Committee meetings were used to workshop the introductory questions. Materials used in these workshops could be used by Heads of Department with their departments when the Commissions commenced.
The three terms of the Commissions were to have three phases:- reflection, action and evaluation (see Leitch & Day, 2001). The reflection phase was aimed at achieving a common language of assessment and a common understanding of what were good assessment practices.
A number of checkpoints were placed into the process. These proved very useful for a variety of reasons. They provided opportunities for discussion among Heads of Department leading to cross fertilisation of ideas and positive reinforcement from colleagues. They allowed monitoring of the process by the Director of Curriculum. They provided milestones at which the Heads of Department had to accept responsibility for what they had achieved.
Checkpoints included reports back to the Curriculum Committee, a five minute talk to a whole staff meeting at the end of term 2, a one page progress report to the Director of Curriculum in term 3 and a whole school poster session towards the end of term 4.
The final poster session was particularly important in generating substantial professional dialogue between staff members. Departments took a fairly liberal view of “poster”. There were traditional posters, Power point presentations, performances by the Drama Department, displays of student work and practical demonstrations of technology use in assessment.
All departments reported that they had gained a shared understanding of the issues involved in transforming assessment practices. There were differences in how departments transferred that understanding into changes in their practice.
It is then, within the framework of the Commissions of Inquiry that the project “What students think is fair and valid assessment in Science” was conceived.
Sample and size
The study group was a year 8 top-level streamed science class of 26 learners at a large Independent girls school. It was originally planned to have a non-streamed class included in the study but the demands of a busy school prevented this. The study took place towards the end of year 8, which for these learners ended with the completion of term 3 in the calendar year. At various stages the learners were divided into groups for the purposes of the study. These groups are described below.
In order to arrive at an understanding of what the learners believed to be fair and valid assessment of outcomes in science, a four-step procedure was employed. The first step was designed to determine the learners’ understanding of outcomes and what they might mean. The second entailed a teacher-researcher led discussion of the concepts ‘fair’ and ‘valid’ with the learners. The third required them to design assessment tasks for a nominated outcome taken from a unit of work and the last step was the selection by all learners of what they thought to be the most ‘fair’ and ‘valid’ task. The first three steps undertaken by the learners were designed to assist in the final decision being an informed (if not necessarily unbiased) one. It was further hoped their understanding of outcomes based assessment would be developed and this would benefit them and their teachers as the former move through stages 5 and 6 in the NSW science curricula.
An understanding of outcomes
For this first step the learners were randomly divided into 5 groups of 5 to 6 learners. Each group was instructed to explain, in “plain English”, an outcome from a report they received earlier in the year. The report outcomes were:
The learners were asked to:
By having the learners explain a report outcome in ‘plain English’ it was intended their understanding of outcomes would be enhanced. A clearer understanding of outcomes was also part of a long-term goal of having more independent and responsible learners. It was further intended that if the learners were to construct an assessment instrument then a more informed understanding of outcomes would assist them to do this. The learners’ capacity to judge the adequacy of an assessment task should also have benefitted from a better understanding of outcomes.
‘Fair’ and ‘valid’ as concepts
Having developed a better understanding of what an outcome might mean and prior to the learners devising an assessment task of their own, a discussion between the class and the teacher-researcher about what ‘fair’ and ‘valid’ might mean took place. This was an open discussion initiated by the teacher-researcher that continued for approximately twenty minutes. A discussion of this type was thought to be necessary given the learners’ lack of explicit exposure to these concepts within the context of assessment in science. The notions of ‘fair’ and ‘valid’ have a range of meanings in contexts other than assessment and a more focused set of understandings was deemed to be essential to effective task design.
Designing an appropriate assessment task
Having developed a better understanding of outcomes and come to some consensus about the nature of what might be called ‘fair’ and ‘valid’, the next step was to have the learners design their own assessment tasks. This step was intended to produce three different tasks for each of two outcomes. The class was randomly divided into two groups and then each half was subsequently divided into three. There were two outcomes (one for each half) and these were:
Each of the three groups in each half was randomly given a task to design for their allocated outcome and was given two fifty-minute lessons to complete the task. The tasks were:
Where possible and/or relevant the learners were instructed to word process their tasks. In addition they were instructed to:
The intention of this component was to have the learners engaged as actively and thoughtfully as possible in the construction of a task that fairly and validly assessed a specific outcome. It was also thought that engagement of this proximity to outcomes would be of long term benefit to learners who were to be assessed using a standards approach.
The best task
Once the three groups from each half had completed their task and after some time elapsed (approximately two weeks) all six working groups came together with each taking it in turn to put their case for the ‘fairness’ and ‘validity’ of their task. The two halves, according to outcome (Groups 1 and 2) also examined and discussed each task and wrote down their comments on overhead sheets. This was followed by a whole class discussion on what they thought was the most ‘fair’ and ‘valid’ task overall, with reference to the prepared overhead sheets.
Report outcomes
There was a range of understandings evident in the ‘plain English’ versions produced by the learners ranging from very clear and ‘plain’, to reiteration of the outcome. Two groups indicated that some sort of ‘standard’ or ‘more developed’ ‘level’ (along an unstated continuum) should be reached if “understanding” or ‘recognition’ were to be evident. All groups but one, included a ‘checklist’ of items that served to indicate a learner had achieved an outcome. In various ways this suggests the learners are aware of outcomes having component parts that together contribute to achievement of the outcome. Although not stated by any of the groups, such an inclusion (of checklists) might also indicate recognition of levels of achievement.
Only one group took key words from the outcome and explained what these meant. This related to the outcome, ‘plans and performs a first hand investigation’ and included explanations of “plans”, “performs”, “investigation” and “first hand” with the last one explained as:
“you do it yourself (not pay your brother to do it)”.
This group seemed to get closest to what might be thought of as a ‘plain English’ version and was evident of a deeper understanding of the outcome. While the remaining groups (bar the one mentioned above, which effectively repeated the outcome verbatim) exhibited varying degrees of understanding, much of that relied on indicating how achievement of the outcome would be indicated rather than a ‘plain English’ version of what it meant. However, it should be noted that such an approach had much the same effect as a ‘plain English’ version, which suggests there is more than one way to present such a version (and more than one way to understand an outcome).
A discussion of ‘fair’ and ‘valid’
The learners’ ideas of what is fair and valid showed a range of understanding that included terms such as ‘equitable’, ‘being treated equally’ and “assessing what it says it is”. Most girls contributed to the discussion and after this was completed it seemed to the teacher-researcher that the learners’ understanding of ‘fair’ and ‘valid’ was sufficient for them to proceed on to the next step of designing their tasks.
The tasks and the choice of which is best
The learners produced two tests, two practical tests and one assignment with the remaining assignment not being completed owing to a range of events including: the last day of term 3; the beginning of year 9; illness (teacher-researcher and some learners) and the necessity to complete the last step over several lunchtimes (owing to the progression into year 9 and the splitting of the original class). Three of these (the two tests and the assignment) are included as attachments to this paper.
The discussion between Group 1 and Group 2 and the whole class group
produced the following results -
Group 1
Respiratory test (reasons for choosing):
Reasons for not choosing others:
“It is hard to judge which is the best out of all because they are all different.”
This last comment is one that typifies the quandary facing these learners. They find it difficult to compare one task with another and tended to rely on more simplistic notions of why one is better than another (“easier to mark”, “we wrote the respiratory test”). There are several issues that emerge from comments such as these. That the learners are unable to easily choose because of differences in the tasks suggests that learners should be given a range of tasks for assessment. It might even be worthwhile having a choice of tasks for the same outcome and allowing learners to choose which one they prefer to complete. The almost throwaway line about the respiratory test might have deeper implications that it suggests. If learners prefer a task they wrote, why not provide them (either individually or as a whole group) with the opportunity to do this (and why not use it as an assessment task for internal purposes?).
Group 2
We prefer the “assessment” because:
We don’t like the “other two” (these were the two tests) because:
“The prac (sic) tests are unfair because:
The achievement-oriented nature of this class might explain why they think prac (sic) tests are unfair. This was a group of learners who, whenever possible, liked to have a good idea about what they were to be assessed on (this is also reflected in their comment about the two tests) so that they could study hard for it and do as well as possible. Their comment about the two tests is a legitimate complaint about written tests in that a test might not cover every aspect and therefore disadvantages learners who know some areas better than others. This might be a reality of external testing and hence an argument for training learners to cope with such things, but for internal testing there should be ways in which the learners feel that they are being tested in ways that are ‘fair’ and ‘valid’. This was after all, one of the aims of the project for the Science Department.
The suggestion that a take home assignment provided an excellent opportunity to do their best and which assessed the learners in ways that were ‘fair’ and ‘valid’ also needs to be taken into account when designing assessment tasks for learners. In effect there are three main themes to emerge from this study. They are:
Although the practical test wasn’t chosen &endash; in discussing why it wasn’t, the learners stated that assessment was not about studying “it was about learning”. Although several learners only stated this, it indicates a deeper awareness of outcomes-based learning and standards-driven assessment and a move towards the learning goals the authors and the Science Department intend for these learners and their colleagues.
It may be concluded that the learners:
The implications for practice arising from this study are that:
The improvement of the professional learning of teachers has been the
goal of many, including the practitioners themselves. Education
professionals are facing the dauntingly complex task of managing
teaching and learning in an ever changing world. Commenting upon
international innovations in education, Tedesco (1997)
writes:
... teachers can no longer continue to be overlooked in the process of educational transformation. For example, changes in the structure of education systems, in the administration and management of institutions, in the context of curricula, in teaching/learning methods, in educational materials and equipment, will not achieve the desired effect if they are not accompanied by an integrated policy intended to strengthen the role of teachers. (p.1)
The most notable means for strengthening teachers’ capability is through their workplace learning. Teachers are well aware that the skills and insights they brought to their work as novices, at the beginning of their career, are not sufficient for them to solve the myriad problems and challenges which they will encounter as their careers progress. Career-long professional learning is a concern for every education professional, be she or he a classroom teacher, school administrator or senior officer working in the role of leader and manager in a particular directorate or agency. The Knowledge Building School clearly is a context that recognises and affirms career-long professional learning.
While it is essential that education professionals take responsibility for the improvement of their professional knowledge and the ways in which they may exercise it to the benefit of the consequential stakeholders in schooling (that is to say the students), it is also important to recognise that there are corporate and political imperatives which will also influence their training and development. As Tedesco indicated the organisation and management of schools and of institutions (such as the Independent School whose work has been shared here) are undergoing continuous change as well. Not only that, but they are subject to ongoing policy change, such as the reform of the NSW Higher School Certificate and the shift to standards based assessment protocols.
Studies suggest that local school management has the exciting possibility of leading teachers into sharing their practical knowledge, taking greater risks, and experimenting with different pedagogical and organisational arrangements (Lieberman, Darling-Hammond & Zukerman, 1991; Bryk, Easton, Kerbow, Rollow & Sebring, 1993; Ladwig, Currie and Chadbourne, 1994; and Busher & Saran, 1995).
A promising synthesis arises from the work of Nixon & Ranson (1997) who propose that teacher agency can be strengthened by a form of ‘agreement’ which acknowledges both internal and external relations. Their argument rests upon a notion that “the local is everywhere, even at the centre” (p. 198). This paper takes the position that educational professionals, can and should be, active agents in their own professional learning, while at the same time forging agreements with other cultural forces, including alliances such as the Coalition of Knowledge Building Schools, professional associations and parent groups, with the shared purpose of adding value to their students’ lives in ways which are morally sound and socially just.
Walker (1995) writes of self determination in teaching which comes from “acting and reflecting on one’s understanding of one’s own work” (p.102). He argues that a necessary condition for teachers to act wisely and prudently is that they well understand their work and the context in which it is enacted. He, similarly, sees the power of self determination comes from agreement, agreement with self and agreement with others. This paper is particularly salient because it also extends to agreement with students on those things which matter to them in relation to the assessment of their learning.
In the following school term (Term 4) the teacher-researcher had, as part of his teaching load, the top year 9 Science class that contained approximately half of the class who took part in this study. As part of an internal test, the learners were given the opportunity to design their own test, which they undertook with great enthusiasm. They were divided into 7 groups (one for each component of the unit studied) and were instructed to complete 5 (or more) multiple choice items for their section, after receiving some training in multiple choice item construction. They also had the opportunity to share their items (without answers) with the other groups, which could be taken home for study purposes. This activity produced 35 items divided into seven separate sections (one for each component of the unit). The learners were only required to complete 25 items, which included two sections of 5 each and then any 3 items from the remaining five sections. They appeared to enjoy the opportunity to take part in the testing process, to do well (which is a feature of this group) and were appreciative of being involved in the assessment process (and yes, most of them did very well).
Bennett, N. (1993). Knowledge Bases for Learning to Teach. In N. Bennett & C. Carre (Eds.) Learning to Teach. London: Routledge.
Bryk, A., Easton, J., Kerbow, D., Rollow, S. & Sebring, P. (1993). A View from the Schools: The State of Reform iin Chicago. Chicago: University of Chicago Press.
Busher, H. & Saran, R. (Eds.) (1995). Managing Teachers as Professionals in Schools. London: Kogan Page.
Goodwin, N. & Groundwater-Smith, S. (2000) Beyond the Individual: Corporate Professional Learning in Schools. Paper Presented to the 3rd International Practitioner Research Conference University of Innsbruck, 26th - 28th September, 2000
Groundwater-Smith, S. & Hunter, J. (2000) Whole School Inquiry: Evidence Based Practice. In Journal of In-Service Education, 26 (3) pp. 583 &endash; 600
Groundwater-Smith, S. (2000) Evidence Based Practice - Towards Whole School Improvement. Paper Presented to the Annual Conference Australian Association for Research in Education Sydney, 4th - 7th December, 2000
Groundwater-Smith, S. & Mockler, N. (2001). The Knowledge Building School: From the Outside In, From the Inside Out. Paper presented to the 3rd CERG Conference, University of Technology, Sydney, Kuring-gai Campus, 16th &endash; 17th February.
Groundwater-Smith, S. (2001a). Building Research Capability to Inform Practice.Paper Presented to ISATT 2001 Conference Faro: Portugal , September, 2001.
Groundwater-Smith, S. (2001b).Supporting and Sustaining the Knowledge Building School. Paper presented to the European Educational Research Association Annual Conference Lille, France, September 5th &endash; 8th, 2001
Handy, C. (1994). The Age of Paradox. Columbia, MA: Harvard Business Press.
Hargreaves, D. (1999). The Knowledge Creating School. British Journal of Education Studies, 47 pp. 122 &endash; 144.
Ladwig, J., Currie, J. & Chadbourne, R. (1994). Towards Rethinking Australian Schools: A Synthesis of the Reported Practices of the National Schools Project. Ryde, NSW: National Schools Network.
Lieberman, A., Darling-Hammond, L. & Zukerman, D. (1991). Early Lessons in Restructuring. New York: National Centre for Restructuring Education, Schools and Teaching. Teachers College, Columbia University.
Leitch, R. & Day, C. (2001). Reflective Processes in Action: mapping personal and professional contexts for learning and change. In Journal of Inservice Education, 27 (2) pp. 237 &endash; 261.
McRae, D., Ainsworth, G., Groves, R., Rowland, M. & Zbar, V. (2001). PD 2000 Australia: A National Mapping of School Teacher Professional Development. Canberra: Department of Education, Training and Youth Affairs.
Nixon, J. & Ranson, S. (1997). Professionalism Within the ‘New’ Management of Education. In Discourse, 18 (2) pp. 197 - 214.
Norris, N. & Robinson, J. Generalisation: the linchpin of evidence based practice? In Educational Action Research, 9 (2) pp. 303 &endash; 310.
Raab, N. (1997). Becoming an Expert in Not Knowing: Reframing teacher as consultant. In Management Learning. 28 (1) pp. 161 &endash; 175.
Walker, J. (1995). Self Determination in Teaching and Learning. In Curriculum Inquiry, 25 (1) pp. 101-109.
All correspondence regarding this paper to:
Dr Gerry Corrigan
Lecturer in Science and Technology Education
School of Education
The University of New England
NSW 2351
Phone 02 6773 5072
Email gcorriga@pobox.une.edu.au