Tag Archives: Exam Results

SOLO Research: Conclusions

What have I learnt from my research into the SOLO taxonomy?

Researching and Note Taking

At a personal level, this research has been very useful. I have found out that I really enjoy research and reading academic articles and texts. In particular, writing the literature review was an interesting, challenging and enjoyable part – far more than I had expected it to be. It was like a giant jigsaw which needed to be put together before it would make sense.

notesI tried out a variety of note taking methods for the review, the most effective one turned out to be writing key quotations onto post its which I then sorted into linked areas on large pieces of paper with lines and comments added to show the relationship. This helped organise each part of the review into paragraphs and made the links clearer to see. It was while I was doing this that I realised that this was also a SOLO task – I was moving my knowledge of the literature from the multistructural to the relational level and beyond. Thinking about the process in this way was quite useful as it mitigated some of the frustration I felt at having to go off on tangents in order to understand the bigger picture – it was simply that I didn’t have the knowledge at the multistructural level.

Twitter, the Internet and The Khan Academy

As someone who completed their undergraduate degree in the early 1990s, studying at Masters level was a very different beast. Beyond the level of complexity that obviously exists in the step up, the key difference I noticed was the availability of resources. Now, don’t get me wrong, I love an academic library – the smell of the books, the chance finds in the stacks, the quiet you only get in the obscure corner of the Old English section on floor 10 – but, as a distance learner, the internet has been invaluable to me.

Twitter has been a fantastic source of ideas, suggestions for academic reports and texts as well as a source of data. Without this community of educators, I think my study would have been a sad shadow, and I would have been a very lonely researcher.

One of the challenges I faced was gaining an understanding of descriptive and comparative statistics. As far as I can remember (and it was a very long time ago, so I may be wrong), this was not covered in much detail in my GCSE Maths course. Although I have used maths on a day to day basis in work and as a teacher, this was something I needed to brush up on – that is where The Khan Academy came in. One weekend watching their statistics videos and trying out a few problems, and I had a good understanding of what I needed.

Is the SOLO Taxonomy Effective?

Based on my limited research, it does appear that the SOLO taxonomy can be a useful tool in a teacher’s arsenal. The use of rubrics to identify the knowledge (both declarative and functioning) and stages of learning were particularly useful for making this explicit both for me and the students. The emphasis on looping back through the multistructural-relational-extended abstract levels in order to develop a more detailed and sophisticated understanding helped scaffold the most able and encouraged them to view learning as open-ended.

Knowledge is vital – without relevant knowledge, students cannot progress through the SOLO levels. Direct instruction, whether it is through teacher talk, rubrics or any other direct method, help to provide the  knowledge needed by the student. The rubric can keep this instruction at the forefront while students complete independent tasks – the two do not have to be mutually exclusive.

A key benefit of the SOLO taxonomy is creating a common language for discussion of knowledge and feedback – used by the teacher and in self and peer assessment it can help to ensure the quality and focus of feedback.

Of the SOLO techniques I trialled with my classes, I felt that the use of rubrics, hexagons and SOLO stations were the most useful. The weaker students found the hexagons helpful to pull together their knowledge of a text and bridge the gap between knowing the text and being able to write a clear paragraph about it. SOLO Stations allowed for differentiation, student choice and teacher guidance while giving me the time to work individually with students. The HOT maps were rather hit or miss depending on which type was used – the Part/Whole Analysis was a useful structure for discussing and revising a text in detail.

Given the recent reports from Sir Michael Wilshaw, regarding the brightest students in schools failing to achieve the highest grades, it is certainly interesting that in this small scale study Level 5 students and males taught using SOLO methods did considerably better than their non-SOLO counterparts. Ev ex 2Although it is impossible to know whether SOLO was the key factor in this difference, it suggests that this may be a possibility and would warrant further investigation.Ev ex 4

 

Taken as a whole, based on my personal observations, surveys of teachers and students, a lesson observation and exam data analysis, it appears that the SOLO taxonomy may be effective. As with any teaching technique, it is not a panacea – however, it is certainly worth trying.

SOLO Research Project – Findings Part 2

Exam Data

In addition to the other data collection methods, I chose to analyse the modular examination results for a group of Year 10 students, one group (n=29) taught using SOLO methods, and a larger group (n=82) who were not.

To try to ensure that, as much as possible, the groups were comparable, I chose students who had started school with either a level 4 or 5 from their Key Stage 2 English tests. The students were all from one of the two parallel top sets. Students who did not have a Key Stage 2 level were excluded from the analysis.

All students in the groups selected took GCSE English Literature exam module A663 (Prose from Different Cultures) in the summer of Year 10 and studied the same text – Of Mice and Men. The results were analysed using descriptive statistics to gain an overview and identify areas where the data warranted a closer look. In areas which appeared to show a difference, a chi-square test was applied to test significance; a significance threshold level of p<0.5 was set to ensure that any significance was meaningful.

Although every attempt was made to make the analysis as unbiased as possible, for example choosing to focus on a module which was externally marked, it is important to remember that looking:

At results before and after a new intervention is rolled out…can be very misleading, as other factors may have changed at the same time. (Goldacre, 2013:9)

 

In addition, as the group selection was not randomized, or carried out over a longer period of time, the results may not be replicable, although I feel that they may provide some indications for areas which would be worth investigating further.

Analysis

The non-SOLO group shows normal distribution with a modal grade of a B. The SOLO group also shows a relatively normal distribution, which is less steep than the non-SOLO group. Ev ex 1 The tail at the higher end of the SOLO group does not drop off to the extreme of the non-SOLO group. The modal grade for the SOLO group is one grade higher, an A grade.

Comparing the results of the combined level 4 and 5 students between the SOLO and Non-SOLO groups, using a chi-square test, suggests that the probability of this distribution happening by chance is 4%; a significant result. These results suggest that, based on this limited study, SOLO may have a positive impact on exam achievement.

To identify whether this impact can be pinpointed, the data was explored in subsets according to level and gender.

Exploring the subset data, it is apparent that, in this sample, the SOLO group females did not achieve significantly different grades to the non-SOLO group. 

When comparing the difference between the level 4 students, the difference is significant. Ev ex 3

However, it is when comparing the achievement of students starting school on a level 5  and male groups that a very highly significant difference is evident.

Ev ex 2

Comparing the results of the students entering school with a level 5, using a chi-square test, suggests that the probability of this distribution happening by chance is 0%; a very highly significant result.Ev ex 4

As suggested by the graph there is a significant difference between the achievements of the two male groups. Looking at this distribution, using a chi-square test suggests that this distribution (p=0.0000) is very highly significant.

Although there are limitations to this particular aspect of the study, the results suggest that SOLO techniques may have a measurable impact on student exam results. Therefore, it would certainly be worth further, structured research.

The Language of Education

Over the past year or so, there has been a subtle (and at times, not so subtle) drift in the language used in education. There have been references to attainment, progress and achievement for some time – and, although measuring these can be fraught with difficulty, this is no bad thing. All teachers, I’m sure, want their students to make progress and to improve.

However, the change recently has been more insidious, more negative in its tone. We have heard of ‘the race to the bottom’, ‘cheating’, ‘dumbing down’. The implication is clear, education is in a mess and the Government need to ride in and take swift ruthless action before it is too late.

The Powers That Be

A brief look at some of the loudest voices illustrates my point. Ofqual‘s press release regarding the problems with the Summer 2012 GCSE results included the following:

Glenys Stacey said: “It is clearly hard for teachers to maintain their own integrity when they believe that there is a widespread loss of integrity elsewhere. No teacher should be forced to choose between their principles on the one hand and their students, school and career on the other.”

A barbed little comment – is it ‘clear’? Are teachers being ‘forced to choose’? Although conciliatory in tone, the underlying implication is that lots of teachers are cheating – but where is the evidence? Which teachers have had to choose to cheat or risk their career? If this is true, then it is a scandal…if. The implication is enough to fuel sensationalist headlines, even without evidence.

Wordle: Ofqual

In his ‘Good to Great’ speech, Michael Wilshaw said:

“we need radical improvements to the education system”

Is this really the case in most schools? Or is it a man, who has a passion for education, overstating the case – a rhetorical device?

If the system was so bad where did all these “brightest and best graduates” come from? That in itself is an unpleasant myth – writing off pretty much everyone already in the profession at the moment – a quick glance at twitter will show that new teachers don’t have the monopoly on innovation, motivation and passion for the job.

Wordle: OfSted

The current Education Minister (current, as we have had 8 different ministers in the past 10 years – this in itself could explain some of the problems in education, as each one wants to make their mark), Michael Gove, in his speech to Brighton College said:

“And because we recognise that Governments must take sides in debates – we must be for aspiration, ambition, hard work and excellence – for success based on merit and a celebration of those who do succeed.”

Truly a comment worthy of Orwell’s Squealer – disagree with us and you want the opposite. The image of the rabid, stike-ready, trade unionist teacher, who cares only for their pension and doing as little work as possible, is lurking in the background. No grey areas, no acknowledgement that we may want the same but disagree with the methods, especially the methods espoused by a man with no teaching experience, a love of the independent sector and little experience of the English state system.

The Truth?

The real situation is probably somewhere between the extremes. Some schools, some individual teachers, some students may well cheat; some may look for the easiest route to tick the boxes on the performance tables. Realistically, most schools will push the boundaries as far as they can, while still staying within the rules.

What is the motivation for this? We have to look at the way schools are judged/ If we don’t want a system where each school focuses on league tables, then we need to remove them – or change the way the tables work. Schools are not created equal, so judgements made on the numbers of top grades will only reveal what we expect to see – selective schools and those with a more affluent intake doing better than those in deprived areas. Teaching alone is not sufficient to change this pattern. Looking at value-added impact gives a more balanced view, but again is fraught with problems – not least who this information is for and how understandable it is.

There are no easy solutions here, but the bottom line is that schools will do their best to meet the standards set for them. It is churlish for ministers to criticize schools for trying to meet standards that they, or their predecessors, have set for them.

Schools

Beyond the Government and the press hounds slavering for a juicy headline, the language used within schools also seems to be taking a disturbing turn. Now, these are my personal bug-bears (right up there with the usual culprits of BS bingo). I can live with the shift from ‘Teaching and Learning’ to ‘Learning and Teaching’ – obviously little learning took place until this semantic change! The BLP ‘learning muscles’ set my teeth on edge, however, the two phrases that I find the most poisonous are ‘customer service’ and any reference to ‘getting them their C grade’.

Don’t get me wrong, I don’t think that poor quality teaching should be accepted, I have issue with the language itself. Both phrases have the suggestion of something being ‘done’ to a ‘customer’ – a customer who doesn’t have to put in any effort themselves – rather like an expensive massage.

The danger is that all language contains messages – and sometimes the message received is not the one intended. Phrases like ‘getting them their C grade’ suggests the student is entitled to the grade, there is no suggestion of work, of effort, of mastery – and, sadly, this is a message that is received loud and clear by some of our young people. They believe the hype and headlines, and their chances are negatively affected (admittedly, largely through their own lack of work). We are failing our ‘customers’ if we let them think this, but that is the message of this type of language, as well as those easy to pass exam equivalents.

So, what is my point? Really, that language is powerful. It is easy to accidentally, flippantly or deliberately create a damaging impression. This was where education in the UK seemed to be at the end of 2012 – hopefully 2013 can be more positive.

Lies, Damned Lies and GCSE English Results

The exam results period is never short of controversy, each year there seems to be a new issue with marking, supposed ‘soft’ subjects, the BTEC v. GCSE debate…however, this year is different and not in a good way.

You would have had to have been in a cave for the past week to avoid all the news stories about the drop in GCSE English results, or, in my case, in Egypt. There have been other articles written about this, an excellent one by @RealGeoffBarton for example, many focusing on the AQA qualification. I am writing this post partly to get my own head around the situation from the perspective of someone teaching the OCR qualification, but also to cut through some of the media misunderstanding of the situation, and – let’s be brutally honest here – to prepare myself for the oncoming storm from SMT and parents.

So, there has been a drop in the number of students achieving a C grade GCSE  or higher in English – so much we know. A quick trawl of the Internet will show that this drop varies from school to school from a few percent to shocking figures of 16-20%. Following Gove’s repeated grandstanding (minus hard evidence I hasten to add) about ‘falling standards’ and claims from some parts of the press about how easy the qualifications were to pass (anyone heard the myth that if you write your name correctly you get a grade?), it was not surprising that there was going to be some fall out and that grades were likely to take a hit. However, as Orwell’s ‘Animal Farm’ suggests, all students and exams are equal, but some are more equal than others.

The press have reported in varying levels of accuracy and froth, the Daily Mail for example reported: “claims that pupils who took the exam in January found it easier to gain C grades than those who sat it in the summer”. Others reported exam boards explaining that the difference was due to the new syllabus. I hope state the case as I see it and explain, hopefully in layman’s terms, why this deflation of grades is unfair.

The New Syllabus

This summer marked the first cohort going through the new GCSE syllabus. The syllabus was introduced in September 2010 and included several changes to the previous exams – the introduction of ‘controlled assessment’, a type of coursework being completed under exam conditions, being the most notable one. Yes, you would expect a few teething problems as students, examiners and teachers get to grips with the changes, but these should be fairly minor as the core of English remains the same – reading, writing, speaking and listening.

The mark schemes for the new exam ‘controlled assessments’ changed, due to the insistence of the now-defunct QCDA (Qualifications and Curriculum Development Agency), descriptions of students’ skills being matched to bands rather than grades. This was a more complex change (something many of us are used to at A-Level), but the description of C grade skills for teachers experienced in marking C grade work, and guidance via exemplars from the board, meant that, although the boundaries were a little fuzzy in places, ultimately the skills and quality needed for a C grade were pretty much the same as under the old syllabus.

Obviously, it would be unfair if, because of an accident of birth, you needed to achieve a much higher range of skills in order to get the same C grade as previous years, wouldn’t it? Surely that is the point of the grading, and exams of this type, if you get a C you have X range of skills, so colleges and employers can compare applicants fairly? If this is not the case, why do we bother with the exams at all?

The Harder Summer Exam

Much of the press reporting focuses on the assertion that the summer exam was much harder than the January one. This may well be the case. I would not be surprised if the mark schemes were more stringent and that, where a student was previously given the benefit of the doubt, this was no longer the case. This does need to be investigated and, if it turns out to be the case that the exam was much harder than previous sessions, adjustments should be made to ensure that there is consistency and fairness.

As each exam series has a new paper, you expect there to be a little movement in the marks needed to achieve particular grades – we as teachers expect this, a really tricky paper will generally have a lower grade threshold than an easier one to ensure parity. We expect the grade boundary for a paper to shift by a few marks – that is fair. What seems unfair is a sudden shift of grades by 10 marks or more, suggesting that it is harder to achieve a C grade than before – as I said above, if a summer 2012 C grade is not the same as a Jan 2012, summer 2011 or 2008 grade, then the whole point of assessing students via GCSE exams is flawed and unfair.

 The Controlled Assessment

This is the element marked in school and samples sent to the exam board for moderation. Typically, 60% of the GCSE is made up of internal assessment and 40% by the exam. The exam boards set the tasks, we teach the skills and content and the students complete the task under exam conditions. We can’t mark drafts or give feedback on the piece until it has been completed. The teacher then marks the assessment piece using the mark scheme provided by the exam board, this is split into bands and marks, not grades. We send these marks to the exam board, as well as an estimated grade (based on our professional judgement and previous experience). So far, so good.

I teach the OCR course for GCSE English Language, I have been to the board training sessions, we have moderated the work as a department, we have sent off our sample and (post-results) received confirmation that there has been “no adjustment” needed, that is that we have applied the exam board mark scheme accurately – matching work to bands and the relevant marks. So no problem there…well, yes! The controlled assessment is a huge, and, what I think is, a key part of the unfairness of this summer’s exam results.

This is where it gets a bit technical. Each exam series, the boards produce a list of grade boundaries for the marks awarded for each module, these ‘raw’ marks are then converted to UMS (Uniform Mark Scale) – this allows for adjustments to the boundaries, for example the differences in exam papers I mentioned above. While the public exam boundary may change within the same series, the controlled assessment boundaries should not (although they may change slightly from year to year) as the tasks are the same, can be completed at any point over the two years and are marked using the same mark scheme. The only difference with the controlled assessment is that the marks could be submitted to the exam board in either January or May (depending on which elements of the course were being counted towards the 40% terminal rule).

Following me so far? Good.

The controlled assessment tasks and mark schemes have not changed over the two-year course, so there is no variation in content that we might see in the external exam. Our moderator reports (and, I expect, many other schools’) state that there is ‘No Adjustment’ to either CA unit, so they agree our marks and our application of the mark scheme. As the qualification is criterion referenced, the grade equivalent for the mark awarded for the CA units (certainly within a single cohort) should not change – if they do, it suggests that C grades from different years and sessions are not actually the same which is obviously unfair and makes the whole exam system a farce.

Ah, I hear some of you say, the boundaries have been changed to avoid ‘dumbing down’, to increase ‘challenge’, to make the exams ‘harder’. Ok, so, if that is the case then surely we will see a similar increase in the boundaries for all grades?

No! The changes to the marks are not equitable, they hit the C/D and lower grades rather than the B-A*. Across the two English Language CA units (A651 and A652) the difference in marks (for OCR) to achieve a particular grade are as follows:

  • A* – 1 mark less than January
  • A – the same as January
  • B – 4 marks more
  • C – 8 marks more
  • D – 8 marks more
  • E – 9 marks more
  • F – 9 marks more
  • G – 9 marks more

I suggest that this is a political move, as if it were about rigour and ensuring challenge then surely all grades should have been affected? It seems to suggest that those in selective or high achieving schools (hmm, the children of many of our politicians perhaps?) are less likely to be affected. Perhaps the powers that be don’t wish to upset their privileged friends? Those students who are most in need of the C grade for college, apprenticeships or jobs, who need a good education to improve their chances in life seem to be the target of this change. It smacks of social engineering at its worst. This is unfair.

The second issue is the change within the same exam series depending on when the CA marks were submitted – if we had submitted the controlled assessments in January  the same pieces of work, by the same students, marked by us and given the same raw mark which was agreed by the exam board, were worth up to 9 marks less if we submitted in the summer rather than January. This unfairly penalizes  students, as in other schools, those who had the same or slightly lower raw score would have been awarded a higher mark if they were entered in January. The scaling for UMS makes this even bigger, so some students are 10 or 11 UMS worse off.

 What Should Be Done?

Firstly, the summer exam for all English exams should be reviewed to check that those who sat it in the summer were not unfairly penalised due to political pressure. Basically, would a response in the summer exam have achieved a higher grade in January? If so, they should be amended.

Secondly, the grade boundaries from January for the controlled assessment grades should be applied to the summer entry – ensuring that all students have been treated equally.

Finally, an urgent review of the whole situation with clear recommendations in plenty of time to avoid this situation next year. I am not advocating ‘giving’ students grades they don’t deserve, but be fair. If the exams are too easy – make them harder for all. Tell us what each grade is and give us examples to illustrate it – that will make it clear for everyone. Otherwise we are in the bizarre situation, to use an Olympic analogy, of a high jump final where no one knows how high you need to jump to win or even qualify.

I will be following Ofqual’s investigation and the outcome very closely.

—————————————————————————————————————-

Ofqual’s less that fab report here.