What’s so SMART about targets? Arguments Against

[A]ny claim that a particular method or resource is ‘good’ or ‘best practice’ needs to be met with the following questions: Who says so? On what evidence? Using what criteria? ‘Best’ for whom? Under what conditions? With what type of students?” (Coffield & Edward, 2009)

Individualisation in regular classes must fail and does fail. (Hattie, 1999)

Perhaps unsurprisingly, SMART targets are something born out of the business world, along with SWOT analysis and the habit of education people coming up with a nifty acronym to hang their theory on (this includes educational software designers – I find it hard to believe they came up with Modular Object Oriented Dynamic Learning Environment first, but rather that they just thought “Moodle” sounded cool).

It has since moved into the generic post-16 educational context. It is taken as read that an ILP with SMART targets will improve learning, although scant information is offered as to how this will happen, and indeed evidence. Here’s one quote from an article on functional skills which more or less paraphrases every other publication on this:

“For learners to succeed in functional skills, a ‘personalised’ approach to learning is needed. This allows the development of skills appropriate to the individual learner. This approach to delivering functional skills is vital in engaging learners and developing their competences in ways that they can identify as relevant and motivating.”

Would it be foolish or inaccurate to equate personalised with individualised? Seems reasonable to me, in which case let me refer you to John Hattie’s well-known work (reference below, but explained very well by Geoff Petty here: http://www.geoffpetty.com/research.html) which listed innovations by effect size, i.e. how much they improved student performance, and individualisation comes way way down in the bottom 10 (an effect size of 0.14 against a benchmark of 0.4, rating barely above “Finances/Money” and “Behavioural Objectives”.) To add a little perspective there an effect size of 1 is the same as a 100% improvement.

Let’s look at the other side of our profession – the side that comes from language and language learning (as opposed to the side that comes from adult/general education). There exists a large body of literature, research and practical help internationally which we can access, loosely collected under the label of applied linguistics.

The concept of targets assumes by default that the learning process and the sequence in which things are learned is a linear, predictable process. For example, in order to learn to make an omelette you need to learn how to crack an egg, then whisk the eggs, choose and prepare any omelette fillings, operate a cooker to turn on the heat and then select an appropriate pan, and so on. These are clear and one leads to another in a fairly logical sequence (like you can’t make an omelette without…).

Over the years there have been a large number of studies in applied linguistics relating to the sequence in which particular items are acquired (see Ellis, 1997 & Lightbown & Spada 2006 for further references). The stages through which a learner goes are fairly predictable and clear, but at several times during that process they might produce a structure which resembles the target language, but in fact represents an aspect of their interlanguage. For example: By the end of this term, I will be able to write 5 past tense sentences about the weekend using irregular verbs. Fairly SMART and useful, for a consolidating E2 learner. But let’s dig a little deeper.

As a rule, and this is not a fixed rule, but can vary on an individual basis, learners generally produce the past simple irregular verbs in something like the following sequence:

  1. They produce a present tense form or the base form but with a past time marker. “I go yesterday”
  2. They learn the irregular past form as an item of vocabulary and produce the apparently accurate “I went yesterday”
  3. They then learn the rule “add –ed” and start to produce “I goed”
  4. They may then overgeneralise this rule and produce forms like “I wented”
  5. Finally the rule sorts itself out and the learner can distinguish between irregular verb forms and regular verb forms. “I went.”

(Ellis, 1997 is the source for this, but the same sequence applies to first language learning as well – ask a handy parent). If you are interested in exploring the irregular verb weird brain thing in a bit more detail then I recommend Stephen Pinker’s Words and Rules which is truly fascinating.

A cursory glance at this tells us quickly that at two points the learner produces “accurate” past tense verb forms. The learner may well produce the correct form, say at point 2, but is not, by any measure anywhere close to learning the correct past simple tense. However, they have evidenced it. They may even stay at this point for some time, producing “correct” sentences, according to the rule of OFSTED, several times. It is specific, measurable and has been achieved. But they have not necessarily learned it.

I admit, that was a bit easy, a rather soft target, if you’ll excuse the expression. However, the majority of grammatical structures are not learned in an easily measurable way. There is a sequence, for example, in how we acquire the negative. Most learners progress through these sequences in a fairly predictable manner, although at different speeds. It would be a valid learning goal to put something like “use not before the verb to indicate the negative” (e.g. “I not go”). This represents a real achievement in a beginnner learner’s learning (as would “I goed”) and would indeed be SMART in a real sense of the term. However, I suspect that a skills for life observer would be highly critical of that target, and that the same learner should be working towards do not. Which would be true, but unrealistic for a very low level learner. Not all learners go through all stages, but they do exist, and have been drawn from analyses of learners’ speech and writing as it progresses. This isn’t just ivory tower academics, but real studies of real performance.

One common argument for SMART targets is that of increasing learner motivation – they know what they need to learn and what they need to improve, and achieving the targets marks a real stage for them, thereby increasing motivation. This is a proper and realistic observation, and this is where, perhaps, target setting, if done well and properly might be able to make something of a claim. Apart from age and aptitude, motivation is probably the biggest factor influencing language acquisition (see Lightbown and Spada 1999 for the most accessible summary of factors affecting langugage acquisition). People don’t like to talk about age and aptitude (“an ear for languages” a knack, if you like) as the idea suggests that some people just won’t succeed. Bad news, I’m afraid, some people just won’t be successful language learners and there is not really a great deal we can do about either their age or their aptitude for languages.

Motivation, is something we can have a profound impact on – making our lessons interesting, topical, relevant, etc. Setting and meeting goals may lead to one type of motivation (an element of integrative motivation) out of many different types of motivation. These would include extrinsic motivation (the why a learner is learning), instrumental motivation (to achieve a required end, for example citizenship) and also instructional motivation – that the lessons are in and of themselves basically interesting. A lesson can be interesting and not tied to a learner’s individual learning goals – the very first thing to go once you start down that path is meaningful and useful student interactions as learners close in on themselves, something the ESOL Effective Practice Project (http://www.nrdc.org.uk/publications_details.asp?ID=89) highlighted as a key factor in improving ESOL learners’ performance. Not to mention the limitations that workshop style lessons place on meaningful output and feedback, which most models (e.g. Gass & Selinker, Krashen) highlight as crucial in the effective acquisition of language. Parcelling off a lesson into individual needs is unlikely to maximise opportunities for genuine learner interactions.

Even if we do take that motivation can really arise from these, and sufficient motivation to succeed, meaningful and real target setting requires real knowledge on the part of the learner of the stages they need to progress through – like they can watch someone make an omelette and identify the stages fairly easily. In vocational (and indeed academic) subjects there are clear and identifiable stages that may require skill to do, but can at least be seen by an external non-expert who can then make a guess at how it works. In short SMART targets work in these contexts because the intervening stages are clear to the student, and can be explained and shared in a real and worthwhile manner.

But the intervening stages are not clear for an ESOL learner. Even the stages mentioned above can vary a little from learner to learner, sections can be skipped. To be honest, they’re not that clear to the teacher, and the research into 2LA has yet to come up wth a neat unified theory that can lay this out.

Then there are beginner and Entry level 1 students. How can they meaningfully engage with the process of target setting, gaining ownership of the targets when they don’t share a first language with the tutor? Unless you are lucky enough to share a first language wioth the student then you might be able to get somewhere, as long as you can explain the terms and concepts in both languages and the learner can identify the stages as mentioned above. I find this unlikely. Level 1 and 2 students have the same problems, learners quoted by Cooke & Simpson (2009) had real trouble describing their needs.

This is usually the section where people who support ILPs (usually generically trained, and subsequently specialised ESOL teachers) would point out something along the lines of “but it’s not just about grammar”. Of course it’s not. Second language learning is about marrying up the interlocking systems of grammar, phonology, lexis with the skills of reading, writing, speaking and listening.

There is nothing at all wrong with action research and reflection leading to a “hey this worked for me, why not give it a go and we’ll see what you think” type of joint practice development. In fact I embrace it – I’m willing to give anything a bash to see how it goes. But by the same measure, under this approach I have the freedom to either accept or reject the same idea, or to do it with one group because it suits them, but not with others. But SMART target setting is not something over which we have this freedom. We can’t critically analyse it, try it out and decide for ourselves whether it works because the inspection teams and senior managers were all conditioned into thinking “target setting good” by the external requirements placed upon them. “If you do SMART targets,” goes the reasoning, “you must be meeting individual needs in the class and that is good practice.” However, as soon as you start asking the powers that be for evidence and proof (as the same people like to do of you) that this has value, you get a variation on the theme of the blank stare that says “Chuh! Don’t you know? it’s good practice.” I have yet to meet someone who can refer me to a proper, robust, rigorous study that bears out the idea that setting ESOL learners these targets will improve their English language skills over a given period.

It’s quite ironic, given this context of tutors and colleges having to supply robust and rigorous evidence to the powers that be (OFSTED, BIS, SFA, etc.) , that so much, including the particular practice of SMART target setting for ESOL learners, has, as far as I can tell, and I have looked, honestly hoping I’m wrong, no evidence whatsoever to support it. In 2004 an article by Meryl Wilkins & Helen Sunderland in Reflect Issue 1 (http://www.nrdc.org.uk/content.asp?CategoryID=539&ArticleID=462) asked for precisely this same evidence.

Since then, nothing has been forthcoming.

***

Coffield, Frank and Edward, Sheila(2009) ‘Rolling out ‘good’, ‘best’ and ‘excellent’ practice. What next Perfect practice?’, British Educational Research Journal, 35: 3, 371 — 390

Cooke, M. & Simpson, J. ESOL: A Critical Guide. Oxford: OUP

John Hattie, Inaugural Lecture, University of Auckland, 1999 http://www.education.auckland.ac.nz/uoa/home/about/staff/j.hattie/hattie-papers-download/influences/ (accessed 20th November 2010)

Lightbown, P. & Spada, N. 2006. How Languages are Learned. Oxford: OUP

Ellis, R. 1997. Second Language Acquisition (Oxford Introduction to Language Study ELT). Oxford: OUP.

Advertisements

9 comments

  1. Wow. Interesting. Aside from the point about being given a choice to use it and no proof etc….. Do you think if the targets were used bearing all your other points in mind, such as assessing correctly, considering their personal motivation, apapting for lower levels, they would then be really useful? Have you in fact just written a set of guidelines for what to consider when using targets and how to use them effectively? Or is your main argument here about the fact that there’s no choice involved when using them?

  2. A really good point. If and this is a mighty big if, the conditions were met that
    a) they were done entirely in language the learner could understand,
    b) the learners could have full understanding of what linguistic competence at their level is (i.e. they know what it is they don’t know),
    c) the assessment was thorough, detailed and informative, and…
    d) language learning really was achieved in sequences which could realistically be assessed by setting & achieving SMART targets

    then there might be room for the concept that they aid motivation and thus aid language learning.

    Except…
    Point a is unlikely for a lot of ESOL teachers & learners, point b assumes that you can go into a group of E1 learners, teach them to understand about the language they need to learn, before teaching them the language, although I do think our organisation has done pretty well for point C.

    And point d is never going to be true. The sequences of acquisition idea varies (oh the irony) from learner to learner, so the only way that the sequence of acquisition could be used to develop SMART targets would be to make massive generalisations that all learners definitely go through every stage. They don’t – some jump stages, go back, go forwards three stages and so on. Some do get it straight away, many don’t.

    And yes, there is a bigger point here about the imposition of apparently arbitrary and absolute pronouncements like “setting SMART targets is good practice” without even the slightest nod to any kind of evidence. Start to scratch the surface of this and you end up with a kind of “well, that’s all very well but OFSTED say.” And in our current climate of hyper-audit and funding cuts, OFSTED & the SFA have developed almost omnipotent levels of control over what we do. I do think this goes beyond target setting and is beginning to go into issues around how and why definitions of good practice develop in colleges, and that, as they say, is another blog post for another day…

  3. This gave me a wry smile.

    I was searching for some hard, empirical evidence that quantitavely demonstrates a correlation between the implementation of SMART Targets for groups of learners, and the success rates of those learners – for example, “SMART” students tracked and compared versus a “Non-SMART” control group.

    After a frutiless search I stumbled upon this article which makes points I very much agree with, again questioning this gaping hole where the evidence should be! Alas, I feel search is proving increasingly futile, and even asking an OFSTED inspector for pointers towards such literature was met with a non-committal fob off.

    My first question is if you have any tips to aid student engagement with said targets? I am currently finding it a heck of a challenge. At one instant I could have a student animated and expressive, wanting me to know he struggles with punctuation, only to see that swiftly taken away and replaced with confusion as I take his meaningful feedback and turn it into an auditable piece of rhetoric in his learning plan.

    Finally, in your honest opinion, do you foresee SMART targets going out of fashion in education any time soon? Being relatively new to FE I’m not sure how long precisely they’ve been in education for, yet I’m baffled (and worried) by the fact they haven’t been laughed out of the house. I get the feeling at times I must be missing something, but after hunting for research and gleaning opinions from a variety of colleagues I’m not so sure.

    1. Thanks for the reply. I’ve been meaning to write something more useful and practical on target setting for a while. But in essence I think it would make a world of difference if we turned the M into Meaningful. So watch this space.

      I think that we are pretty much stuck with them for the foreseeable future. It goes back to my theory on “good practice” in the top down culture of FE, that something identified as good practice by a good, enthusiastic new teacher becomes strongly worded advice by the time they reach management level, becoming dogma by the time they are OFSTED inspectors, consultants and policy bods. In the meantime at the level of the simple teacher they have either incorporated it or rejected it , and either way moved on. But it’s too late because said approaches are now “best practice” and there’s little scope for discussion and debate, let alone rejecting or accepting something because it will (or won’t) work for your learners.

      I would like to finish saying that no, I don’t agree with them, and that they do run counter to most of the things i have learned about teaching and learning generally and especially to how we learn language, but on the other hand, they might work for some learners sometimes, so there is a case to be made for having a go first. Hell, I even wrote the proforma for an ILP once.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s