Program Evaluation in Language Education

1.1 Introduction
Evaluation has many meanings in language programs. It is part of the
novice teacher’s checklist to guide the development of initial lesson plans and
teaching practice, a process of determining learning achievements or student
satisfaction, and a dimension of the analysis of data in a formal evaluation
or research study. It refers to judgements about students by teachers and by
external assessors; the performance of teachers by their students, program
managers and institutions; and programs, departments and institutions by
internal assessors, external monitors and inspectors. Evaluation is about the
relationships between different program components, the procedures and
epistemologies developed by the people involved in programs, and the
processes and outcomes which are used to show the value of a program –
accountability – and enhance this value – development.
This chapter provides an overview of this territory. It identifies themes
and notions examined in their historical context in Part 1, in the case studies
in Part 2, and in the ways forward for language program evaluation in Part 3.
In this chapter we outline three characteristics of language program evaluation
as a field of study (section 1.2). Then we set out five challenges for
evaluation – themes which together constitute a framework for developing
the theory and practice in relation to aspects of language programs informed
by Applied Linguistics on the one hand, and by the fields of education and
management on the other.
1.2 Three features of evaluation
The study and practice of evaluation has developed in diverse ways over recent
decades. These developments are driven by issues from within evaluation
and aspects of the wider socio-political context. Three features of evaluation
theory and practice illustrate the complexity of these developments and the
difficulties inherent in the task of mapping achievements and directions.

First, there is the question of definition; evaluation is a form of enquiry,
ranging from research to systematic approaches to decision-making. Our
account of the history of evaluation over the past several decades in
chapter 2 illustrates a progression from reliance on stripped-down statistical
representations of a program to inclusive, multi-perspective approaches.
The common thread – the making of judgements in a shared context –
gives a problematically wide-ranging basket of activities. Thus, in the context
of an innovative language program, evaluation might include periodic
reviews of the budget, staff appraisal and decisions relating to professional
development, iterated classroom observation for professional development
of teachers or for quality assurance purposes, narratives of experience from
participants, as well as a one-off study to inform on the success of the
innovation.
Second, there are two perspectives on evaluation research. It is viewed, on
the one hand, as a type of study which has both research functions – rolling
back the frontiers of knowledge – and evaluation functions – providing
information for judgements or decision-making; and, on the other, as research
into the processes of evaluation. The former perspective has been significant
in language program evaluations, as evidenced by edited collections such
as those by Alderson and Beretta (1992) Rea-Dickins and Lwaitama (1995)
and Rea-Dickins and Germaine (1998). In the latter perspective evaluation
research can be seen as analogous to the research which has for decades
underpinned the validity and reliability of language testing processes. (For
a recent account of the processes and issues here, see Weir 2004 in this
series.) There is no doubt that evaluation processes require such epistemological
and methodological underpinning – Lowe (1995) and Dornyei
(2003), for example, examine in detail the issues involved in questionnaire
design and completion; while Alderson and Beretta (1992) and Saville and
Hawkey (2004) note that validation procedures in test design have tended
to be much more extensive than in the design of other evaluation instruments.
Without an understanding of the data types which are most appropriate for
the different uses of evaluation, there may be a tendency to inefficient
scrutiny of all practices, documents and perspectives, with constant doubts
regarding the extent to which they actually evidence the success or otherwise
of the program in question.
Third, many accounts of evaluation do not reach the public domain. For
a range of reasons, some proper, others less so, evaluation processes and
findings remain either insufficiently documented or unpublished. One outcome
of this feature of evaluation is the difficulty of mapping theory and
practice when some of the terrain is obscured from view. Evaluations of social
programs are for the most part funded from the public purse. In addition,
they involve aspects of people’s lives in a way that profiles legal and ethical
issues. Thus, there are contending forces for transparency and confidentiality,
which means that the issue of publishing and not publishing evaluations
is difficult. The case studies and discussion in Part 2 illustrate in particular
evaluation contexts both potential conflicts in, and principled resolutions
to, managing accountability and anonymity in evaluation practice. In Part 3
we revisit these issues in the context of guidelines for practice in, and research
into, evaluation processes.
Together, these features present difficulties, but also opportunities. In this
book we bring together perspectives from published evaluations, unpublished,
but researched evaluations, and from the wider discourses of evaluation
in particular fields. The case studies in Part 2 address issues of evaluation
purpose and design, the role of evaluation in program decision-making and
policy development; the roles of stakeholders in evaluations, and of evaluation
in the lives of stakeholders, evaluation and learning in language programs, and
evaluation as a procedure for quality management in programs, departments
and institutions. In each case study, we pay particular attention to the construct
of evaluation – what the data represent and how they correspond to the stated
objectives on the one hand, and the wider purposes of the program on the
other. In Part 3 we explore options for future development in terms of research
into evaluation policy positions within programs, frameworks and guidelines
for practice and methodological orientations. In addition, we examine research
possibilities into the cross-cutting issues of stakeholder evaluation, ethicality
and fairness, and ‘learning to do’ evaluations. This broad-based perspective
on language program evaluation as the examination of situated language
programs can complement, on the one hand, the more theoretical orientations
to understanding language learning in instructed settings in Applied Linguistics
and, on the other, the local development of teaching skills, learning materials
and other program components in schools, universities and ministries of
education world-wide.
To develop this analysis of the potential of evaluation, we set out five
challenges. These are reflections of the features outlined above in two
ways: first, they have proved enduring issues in the development and practice
of evaluation in recent decades (we explore the issues here more fully
in the following chapters and in Part 2); and second, they represent areas
for evaluation theory and for evaluators in different program settings to
engage with.
1.3 Five challenges for evaluation
There are five challenges which we see as characterising the theoretical
orientation and practice of evaluation. The challenge in each case is to understand
and communicate the issues involved in the following dimensions of
evaluation:
1. The purpose of evaluation in its social and political context.
2. The informants who people programs and evaluations
3. The criteria which generate evaluation frameworks, instruments and
ultimately judgements.
4. The data which validate these approaches and instruments, and complete
the construction of judgements.
5. The use of evaluation findings in managing social programmes.
Evaluation purpose: The challenge of evidence-based public policy
development
The ideas which shape public and social policy in this period of late or
postmodernity represent a shift away from ideology-driven programs derived
from philosophical positions and grand theories. New perspectives on the
social aspects of our human nature, such as evolutionary psychology, activity
theory and game theory, give a view of individual and social behaviour
which is infinitely complex. This complexity combines with reassessments
of the success or appropriateness of the social projects of high modernity to
generate a need to move beyond debates focused on nature/nurture, social/
individual and public/private in defining and developing the role of the
state, and of public programs in the lives of citizens. These debates have
been characterised in Western democracies over the last decade by new
syntheses of Right and Left in public sector programs relating to health
care, social welfare and education, such as the Third Way (Giddens 1998).
In education in particular, the task has shifted from universal provision to
effectiveness for particular groups in particular settings (in England and
Wales this educational debate has been characterised by the issue of ‘bogstandard
comprehensives’ – should there be one national approach to the
structure, resourcing and curriculum of secondary schools, or should there
be a diversity of approaches as determined by local stakeholders and factors?).
In Applied Linguistics, understanding second language acquisition and the
teaching strategies which best facilitate this are engaging with diverse
social and personal factors rather than focusing on universals of cognition:
Cook (2000), Block (2003), Lantolf (2000) and Kramsch (2002), for example,
explore those social and cultural dimensions of language learning which
generate new perspectives on the roles of context and identity in language
learning.
The focus on what works in such policy development implies a strong role
for evaluation. Patton (1997: 192–4) lists 58 types of evaluation, all of which
involve understanding the impact of these programs on the problems to be
resolved or the situation to be improved. The unifying theme in these different
purposes is their shared platform of evidence, and the ways in which it can
serve to inform on programs and policies. The verbs – appraise, assess, audit,
examine, monitor, review, etc. – all suggest judgements based on empirical
scrutiny of the program in operation.