Student Feedback dos and don'ts

Which of the following do you think has the biggest impact on 'student evaluation of teaching' (SET) feedback?

1. How hard the course is 
2. The grade the student gets 
3. The teacher’s gender 
4. The teacher’s personality 
5. How ‘hot’ the teacher is 

Ready...SET...

There is a ton of research into SETs starting over 80 years ago (Clayson 2009) and including, as of 1990 over 2000 articles (Feldman in Felder 1992). The literature includes several meta-analyses and even one meta-analysis of meta-analyses (Wright & Jenkins-Guarnieri 2012). In short, it is a well-researched field. Since university professors' careers can depend on these evaluations, perhaps this isn't surprising. Despite the large body of research (or perhaps because of it?) the science is not settled (Spooren et al 2013). 

There are however some general observations which can be made, reasonably confidently, about the effect of certain variables on SETs. So according to the literature* which factors have the biggest effect on student feedback? What follows is my hand list of dos and don'ts to improve your student feedback.

Do be likeable!

One of the variables which correlates highly with positive student feedback is personality and there is a substantial relationship between a teacher’s personality and the feedback they will be given (Feldman 1986, Cardy and Dobbins 1986, Williams and Ceci 1997). Foote et al suggest that “[instructors] who score highly on evaluations may do so not because they teach well, but simply because they get along well with students” (2003:17). One researcher writes that personality is such a strong predictor of SET results that "the SET instrument could be replaced with a personality inventory with little loss of predictive validity” (Clayson online). 

There also seems to be something of a Halo effect at work with SETs. Basically, one positive attribute (Good looks) may cause people believe other positive things about a person (they are trustworthy, for instance). This is the reason handsome criminals get shorter prison sentences for the same crime than less attractive ones. This means that student opinions of personality might colour other variables and subsequently ‘likeable’ teachers may be judged positively in areas unrelated to ‘likeability’, such as teaching ability or professionalism. 

Does attraction affect scores?

This is problematic because it means the feedback you get will be tainted by the students general opinion of you. The picture on the left shows some feedback I recently received. Clearly the student had a high opinion of my teaching. Ho-hum.

The last column asks how useful the virtual self-access centre (VASC) was, the student has written 'very useful'. Now, being the teacher of the course, I can say with some confidence that I said not a word about the VSAC nor did any part of the course use the VSAC. Studies seem to corroborate this phenomenon showing that students are more than happy, to report false information to either reward of punish teachers (Clayson & Haley 2011).It should be noted that the Halo effect also works in reverse, so whatever happens, don't be disliked! 

Do be hot! 

Company promotes bribery

There is evidence that teachers who are perceived to be physically attractive tend to score more highly than their plainer colleagues. Riniolo et al (2006) found a 0.8 advantage on a 5 point scale for ‘hot’ teachers. After analysing the ratemyprofessor.com website, where teachers can be given a ‘hot’ rating, Felton et al (2004) found that ‘sexy’ teachers generally rated more highly than ‘non-sexy’ teachers. The authors note:

If these findings reflect the thinking of American college students when they complete in-class student opinion surveys, then universities need to rethink the validity of student opinion surveys as a measure of teaching effectiveness (91).

Do be expressive!

Despite various methodological flaws, the landmark ‘Dr. Fox’ studies (Naftulin et al. 1973), created interest in the question of the validity of SETs and what exactly it is that students are assessing when they complete feedback. In this study (see the actual study in the video below) an actor lectured a group of medical students with a largely meaningless talk that he had learnt the previous day. The student were told the speaker, Myron Fox was an expert in 'game theory'. 

The actor’s expressiveness and charm was seemingly enough for him to receive positive feedback from three separate audiences. Later researchers showed that even the meaningless talk was unnecessary. Ambady & Rosenthal's (1993) “thin slice” study asked students to evaluate teachers based on a silent 15 second clip of them teaching. The authors found a remarkable similarity between the term-end evaluations and those made after watching the short clips. 15 silent seconds was enough time to give an 'accurate' evaluation of the teacher. 

Do be a man!

Russell is annoying, his class is boring 

Researchers tend to agree that gender plays a minor role in overall evaluation. That is, one gender is not consistently rated lower than the other. In fact, “when significant differences were found, they generally favoured the female teacher” (Feldman in Pounder 2007). So what does 'be a man' mean? Well, despite this seeming equality, different genders may be rated on the basis of stereotyped views of gender (Laube et al 2007). For example, the most highly scoring men were described as ‘funny’ whereas the lowest scoring men were ‘boring’ in contrast the highest scoring women were ‘caring’ whereas the lowest scoring were either 'too smart' or 'not smart enough' or were simply a ‘bitch’ (Sprague & Massoni 2005).

There is also the question of whether a male teacher has to work as hard to get a top SET score as a female teacher. Women may suffer from the ‘Ginger Rogers effect’. That is "Ginger Rogers, one-half of the famous dance-team of 1930s movies, had to do everything Fred Astaire did, only she had to do it backwards and in high heels" (Sprague & Massoni 2005:791).  

Do grade generously!

There is a reasonably strong correlation between the grade, expected or real, and the type of feedback a teacher gets. This correlation can be summarised thus, “to put it succinctly, university teachers can buy ratings with grades” (Hocutt in Pounder 2007:185). 

The highest rated prof on RateMyProfessor.com

Clayson (online) notes that in his research 50% of students asked, admitted purposefully either lowering or inflating feedback grades as retribution or reward, and adds that whether or not grades actually affect scores is perhaps less important than whether faculty believe this to be the case as the belief is potentially enough to alter the way grades are given. Pounder backs this up noting “many university teachers believe that lenient grading produces higher SET scores and they tend to act on this belief” (Pounder 2007:185). However, It should be noted though that this is something of a controversial area with a large number of studies finding no relation between SET score and grades. (see Aleamoni 1999)

And if this isn't enough...

Here are a few more killer tips taken from the literature (Pounder 2007)

Do

• bribe students with food 
• let students leave early 
• praise the class on its ability before doing SETs 
• do the SETs when the weak students are absent 
• do a ‘fun activity’ before the SETs 
• stay in the room 
• teach small classes 

Don't 

• teach maths or science (Aleamoni 1999)
• worry about experience, it makes no difference (Marsh, 2007)
• teach early in the week (Koh and Tan, 1997)
• worry about doing research (Feldman, 1987)

Not convinced yet? 

Here's a satisfied customer's testimony. From a remarkable paper published under the pen name name "A Great Teacher". This teacher, faced with the prospect of losing his job over poor SETs decided to throw out his morals and aim for good ratings. He stopped being such a 'tough' teacher and 'sucked up' to the students instead, making the course easy and trying to build rapport with his students:

What were the results of my experiment? The consequences for learning were not good. Students did less well than expected even on deliberately easy quizzes. Their final exam papers proved to be among the worst I had seen in years. Most students displayed only a superficial knowledge of the material. It was clear that some had concluded that with a kinder, gentler me, one didn’t need to work as hard. Although the pedagogical consequences were poor, the results for me were great! My [SET] scores went through the roof (2010:495-6)

And so, armed with this information, you too can become an well-loved teacher. Alternatively, you can treat student feedback with the caution it probably deserves.   

* Seldin (2010) suggests “one can find empirical support for any common allegation pertaining to student ratings” (in Hughes and Pate 2013:50). It's also worth noting that all of this research was carried out (like much research) on American University students. THere has been very little research on carried on in this are on FL students.

boooooo! hurrah!

Penn and Teller's show 'Bullshit' was a favourite of mine. Every week they debunked commonly held beliefs from 12-step-programs to cryptozoology. In one particular episode they asked people to sign a petition to ban Dihydrogen monoxide -a substance found in 'pesticides, baby food and the water supply'.

Hundreds of people signed up to demand the government ban H2O, more commonly known as water. So why would someone want to ban water? Probably because it was presented to them as a scary sounding chemical and 'chemical' is for many people a 'boo' word. 

'Boo' words, and their opposite 'Hurrah' words come from an old theory called Emotivism which holds that "ethical sentences do not express propositions but emotional attitudes". I'm not too concerned about the philosophical theory but I rather like the notion of boo and hurrah words. Put simply boo words are things that are just accepted as bad, and hurrah words, the opposite. When we hear 'Chemical' we mentally relegate it to the pantomime villain category and boo accordingly. 

So what are boo and hurrah words in education? Swan noted that:

the applied linguistic equivalents of democracy and motherhood - include 'learner-centred', 'meaning based', 'holistic', 'discourse', 'discovery', 'process', 'interaction', 'negotiation' and 'strategy'. On the other side of the communicative fence, concepts related to 'bad' pedagogic attitudes felt to be discredited and undesirable include 'teacher-dominated', 'form-based', 'discrete', 'sentence-level', 'transmission model', 'product', memorization', 'repetition', and 'drill'. (2009:167). 

I would probably add 'testing' and 'textbooks' to this list. These words are often placed in 'boo' or 'hurrah' boxes and there they linger with little examination. And it's not just ELT, as a comment on the now defunct 'Web of Substance' blog wryly notes:

I am disappointed in you as well Harry. You should know by now that, in polite education society you label your OWN ideas as "authentic", "innovative", "Child-centred" and "21st Centruy" so that when anyone disagrees they are, essentially, arguing for a counterfeit, old-fashioned, child-hating, Victorian education. 

We often take our views 'off-the-peg', after all, none of us really have the time to go and read up on every single subject which may concern us. What, for instance, is the link between wanting relaxed gun laws and thinking climate change is a hoax? Seemingly nothing, and yet (American) people with one of these views will often have the other. Have these people really reasoned out the pros and cons of each side, or have they adopted the views of the 'tribe' they most identify with? 

What this boils down to is ideology. Once we choose an ideology to follow, be it socialism, Islamism or environmentalism, we reshape reality to fit that frame. A petition to ban a chemical? Sure, where do I sign!

Is this a problem? As long as our chosen ideology is sound, the views that follow will also be sound, won't they? Perhaps. But I'm uncomfortable, for two reasons. 

Firstly, our views are often unexamined. I can't speak for other teachers, but I often find a lot of the TEFL discourse confusing because I can never sure the terms people are using mean the same thing to them as they do to me.  

Take for instance the discussion on PowerPoint on the Minimal Pair podcast. One of the presenters said something about trying to avoid using PowerPoint because they're so 'teacher centric'. The assumption in this statement is that 'teacher centric' (whatever that means) is bad and should be avoided. I kept thinking, 'are they teacher centric and if they are is that a problem?' 

Secondly, we've seen this go wrong before. Learning styles rode an ideological wave to success. It is an appealing notion to imagine that every learner has their own special abilities and if we just teach them in the right way, tapping into their unique 'intelligence' they will flourish. It's certainly more appealing than the notion that some people are just smarter than others and will do better than them no matter what we do. Learning styles is attractive, ideologically, but unfortunately its not true. 

Alan Waters, who passed away recently, wrote several articles examining ideology in applied linguistics noting that "a good deal of its discourse promotes or proscribes language teaching ideas on the basis of ideological belief rather than pedagogical value." A view supported by 40 years of learning styles promotion. Dana Ferris, who is perhaps the leading scholar in written error correction notes that, on largely ideological grounds "composition theorists have for decades ignored, minimized, or even openly disparaged any issues related to error treatment in writing courses." (2011:61) And Hyland suggests that although process approaches to writing may be appealing "there is little hard evidence that they actually lead to significantly better writing in L2 contexts." (2003:17-8)

These examples make me wonder, what teaching practices we are currently being ignoring because they don't fit our ideology. And likewise, what teaching practices are popular because they appeal to our world view? Is a teacher-centric lesson bad because it limits learning, makes students unhappy and is boring, or is it because it's 'authoritarian' and 'traditional' while we are modern, democratic, freedom loving sorts? Is there a difference between claiming you teach in a 'a learner-centric, communicative way using only authentic materials' and say claiming that you only eat 'organic, gluten free, locally sourced, food?'

Walters wrote several papers on this theme, taking quite an extreme position at times. He claimed, for instance that the EFL world engages in a kind of Orewelian 'newspeak' where unacceptable views are supressed  and only, "approved’ ways of thinking, such as in the use of the term ‘authentic’" are acceptable. (2015) He argued that getting rid of textbooks or advocating learner autonomy or ELF are not just pedagogical choices, but markers of right thinking people. 

And perhaps he has a point. Are textbooks disliked more because they present materials in pedagogically unsound ways or because they are written by large companies who make lots of money? Arguably it's a bit of both. So how do we stop ideology slipping into our teaching? I think it's important to carefully scrutinise our beliefs. The first step would be making sure we have a clear and accurate definition of what it is we're talking about. Take autonomy for instance, most teachers would consider it a good thing but as Mike Swan noted at a recent talk, while autonomy can certainly be good, the logical end point of autonomy, is no teacher. 

Next, we need to examine our biases, -what would we like to be true. I correct my students mistakes in class. Therefore I hope that that helps them learn. If I found out it didn't help them, -even hindered them, I'm likely to feel pretty bad about that. Therefore, I have a vested interest in trying to find data that back that view up. I'll also fight harder against, and examine closer articles which contradict that view.

Lastly, we should ask ourselves what our beliefs about teaching are based on. Do you teach the way you do because it's the way you were taught to teach, or because it's how everyone else teaches? What reason do you have to believe the things you do and more importantly, what would it take to change your mind. If the answer to the former is 'I just know' or 'common sense' and the answer to the latter 'nothing' then what you are describing is dogma. 

A chemical like H2O may save your life or, like H2O2 it might be poisonous. Chemicals themselves are not inherently bad, and H2O2 is excellent for dying hair while water may drown you.  

Review of ELT podcasts

2014 was a great year for EFL podcasts with several sprouting up like veritable fungi. I'm a huge fan of podcasts and think they can be a great way of learning while doing other stuff. So what TEFL podcasts are there and more importantly are they any good? 


1. TEFLology 

TEFLology is 45 mins, fortnightly podcast. The three guys who host it are, I think, lecturers in Japanese universities, which perhaps gives the podcast more of a slant towards applied linguistics, over TEFL topics. The very early episodes were quite unpolished, and there are still moment where the conversation just seems to fade out into  'yeah...mmm....right' kind of moments but they seem to be getting better at editing these out. The Podcasts is usually divided into a 'TEFL pioneers' section, TEFL news and a more general discussion of some ELT topic like DuoLingo, linguistic imperialism or TPR. Overall The podcast is well-researched and well worth a listen. In fact the level of research they seem to put into the episodes does make me fear they will burn themselves out. The podcast has recently had an impressive list of guests such as Nina spada, Widdowson and even an 'explicit' interview with Rod Ellis. It's also worth listening to for the 'home-made' jingle at the start. 

2. The Minimal Pair Podcast

This podcast is almost the complete opposite to TEFLology. It's ESL focused rather than EFL and is hosted by two Americans,  Jean Dempsey and Stephanie Axe who I think are adjunct professors (?) at a US university. They have had a number of interesting episodes on things like 'What's the last P in parsnip' and  recap of goings on at the TESOL conference. I find this podcast interesting because I feel I get very little exposure to US TEFL culture and ideas. Obviously ELT is big over there too and I know their system is somewhat different to the UK, but I'm not entirely sure how. That said, in a number of episodes they have talked at length about catering for student learning styles and then were quite positive about prescriptive grammar, -my two pet hates.  Consequently I wrote a rather negative review of them. Since that there hasn't been another episode. I hope the two events are not related. They have reassured me they will be back in the New Year, so here's hoping. 

3. KKCL podcast 

I thought kKCL was a pretty good podcast, with fairly high production values and a nice style. Their fifth episode was on the topic of learning styles. Guest Marjorie Rosenberg, discussed her new book with host Phil Keegan. I thought this particular episode was a good illustration of the problems with learning styles and so I wrote about it here. Unfortunately the podcast seems to have stopped after this. I hope the two events were not related. The curse of EBEFL? I hope not. Will 2015 see a reappearance of KKCL? Only time will tell.  

4. Masters of TESOL 

This podcast is the brand spanking new kid on the block. With only 3 episodes so far it may not seem worth reviewing but host Andrew Bailey has already managed to bag interviews with Scott thornbury and Ahmar Mahboob.And if that weren't enough he also got a guest anecdote from none other than the Master of TESOL himself, Mike Griffin.I f you've heard the 'freakanomics' podcast, you may feel this has a similar vibe.  This podcast is new so it's hard to say how it'll turn out but it's compact and slick and I've got this on my 'one to watch' list. It certainly has a lot of potential. 

5. ELTchat  

Last but not least is ELTchat, the companion to the twitter #ELTchat. I have to include this because James Taylor would kill me if I left it out. This is a great podcast which includes well known, tweeters and bloggers like Vicky Loras, Tony Gurr and Marisa Constantinides. However so far it has only had about 12 episodes over four years and has only had one episodes in the last year (2014) which makes me wonder if perhaps it isn't in need of a bit of love and attention? James? 

Hopes for 2015

I hope some of the podcasts mentioned here are produced a bit more regularly. It'd also be great to see a podcast offering actual advice for teachers about jobs, something like "guide to teaching in..." and each week the country would be different. It'd also be nice if podcasts included more NNS as hosts and if we saw more women hosts as well.

Did I get anything wrong here? Anything I need to add? Did I miss out any podcasts you think are great? Let me know in the comments. 

Part 2 here. 

Review of ELT podcasts part 2

In my previous review of podcasts I wrote "2014 was a great year for EFL podcasts with several sprouting up like veritable fungi". Well not only had I missed some, but also more soon sprouted up like...more fungi?

1. Lives of teachers

When I first wrote about podcasts Darren Elliott commented that I'd left his podcast out. I had! I was shocked to discover a TEFL podcast that had existed since 2010 and which started with an interview of Paul Nation as it's first episode! Elliott has interviewed EFL luminaries like Mike Swan, Scott Thornbury and Jennifer Jenkins. The interviews are great and Darren is an excellent host. My only criticism of this podcast (apart from its irregularity) is the fact that the sound quality is poor at times. It has improved recently but early episodes, particularly at the start, were very quiet. 

2. The TEFL show  

this show started in July and hosts Marek Kiczkowiak and Robert McCaul have already managed to pump out 16 episodes. They've covered a wide variety of topics such as 'Chinese v Western education systems' and 'product v process approaches to teaching writing'. It's quite 'loose' in style and of the 'two dudes talking' school of podcasting (Marek tells me he doesn't worry much about editing). At times the sound quality isn't great (the 'live from the language show' episode sounded like it was recorded in a submarine) but I'd still say it's well worth a listen. I'm a little biased however since they invented me on to one of their recent episodes and let me ramble on for about half and hour. It'll be interesting to see what happens with this podcast. 

3. The TEFL Commute

The 'commute', hosted by Shaun Wilden and Lindsay Clandfield's (and James Taylor at times*) is a rare beast. A TEFL podcast that isn't about teaching. Instead they deal with peripheral issues such as 'photocopiers' and 'translation'. My favourite episode so far was their examination of the movie 'dead poets' society' from a teaching perspective. I really enjoyed that one. 

I would say that this podcast has far and away the best production values of these podcasts. It has clear sections, good art, good editing and (usually) great sound quality. They generally avoid teaching but do say in their blurb that it "might crop up" A recent interview with Scott Thornbury which touched on 'example sentences' got me wondering if this podcast would be even better if it did actually deal with teaching issues. 

4. SAGE language and linguistics (language testing bytes)

Glenn Fulcher started language 'bytes podcast' in 2010 and has so far produced around 20 episodes, so it's a pretty infrequent. The episodes are also very short with 26 minutes being the longest and 8 the shortest. What it lacks in quantity it makes up for in quality. Glenn is a leading expert in language testing and has guests like Alan Davies and Stephen box discussing issues like 'aviation English testing' and 'rather bias in speaking assessment'. The podcast has been combined with one of Sage's other podcasts so the language testing is interspersed with 'child language teaching' which seems like a rather odd combination to me. 

 
5. EdTechConcerns

Another podcast I really enjoyed was EdTechConcerns. It was also hosted by Shaun Wilden and Lindsay Clandfield (with Philip Kerr) and ran for 7 episodes. It focused on the use of tech in education and the potential problems associated with that. It was packed with interesting interviews and was a high quality production. I'm not sure that you can listen to it now as it doesn't seem to be available. Was it a perhaps a trial run for the TEFL commute? 

   
So there's been a huge expansion in ELT podcasts but a few seemed to have died off. The minimal pair which I talked about last time and KKCL podcast both now seem defunct. I still think there is room for more so here are a few ideas:

1. A TEFL podcast that focuses on actually getting jobs in various countries. So each episode would be about a certain country/sector including an interview with someone there.

2. Similar to the above but getting a local teacher from different countries to talk about the particular language issues that students they teach have.

3. An Applied linguistics podcast. There's a lot of good stuff in TEFLology and and language testing bytes but it would perhaps be good to have a podcast about more academic issues with more in-depth discussion -but not too complex as to turn off listeners.

4. Academic reading circle. A podcast that discusses important/interesting ELT articles. One per episode. Even better if they could interview the authors.

5.A TEFL podcast with a female host.*

 Here's looking forward to a 2016 of great podcasting! 

*As Shaun Wilden notes in the comments, the TEFL commute does in fact have a female host  Ceri Jones. So apologies Ceri!