Learner observation tasks as a learning tool for pre-service teachers

2.2. 1.                        Classifications of errors in the process of observation

There is always the possibility of error in the observation process. Fassnacht (1982:43) reviews Campbell’s (1958) classifications of errors in representing data in psychological and social studies. Some of these errors frequently occur when making judgements and primarily concern language behaviour:

a)                  error of central tendency

b)                 error of leniency or generosity

c)                 primacy or recency effect

d)                 halo effect

e)                  logical error

A first error occurs in using a rating scale. Hollingworth (1910) called the effect ‘central tendency’ in a series of judgements about objectivity of quantifiable stimuli, when the large stimuli are underestimated and the small ones overestimated.

An error of leniency or generosity could arise in making favourable verbal judgements using personality scales. Fassnacht (1982:40) clarifies that in the personality scales a number of questions relating to one particular personality trait are drawn together and the answers to these questions are given in the form of ‘yes’, ‘no’, ‘sometimes’, ‘often’ which might not reflect objective reality.

A third error occurs as a result of the order in which perceptual events happen. The problem is that in behaviour testing the first impression could have a distorting effect on later data collection and thus lead to errors. Bailey (1990:218) admits that in diary keeping, events that are embarrassing or painful when they occur ‘often lose their sting after weeks of reflection’.

A fourth error, halo effect, is described by Mandl (1971) when the evaluator ‘has the tendency when judging a personality trait to be influenced by a general impression or a salient characteristic’.

Logical errors or error of theory reveals due to the theoretical assumptions of the observer. It is now widely accepted that observation is always ‘theory-laden’ (Phillips 1993:62). He continues that observations can not be ‘pure’, free from the influence of background theories or hypotheses or personal hopes and desires. Ratcliffe (1983:148) supports this assumption in that ‘most research methodologists are now aware that all data are theory-, method-, and measurement-dependent’. As Bailey (1990:226) suggests in conducting 'pure research' it is better to avoid reading the research literature in the field, to keep from biasing the results.

2.2.2.  The problem of ‘observable’ items

The item ‘observable’ in the definition given by Seliger and Shohamy (1989:118) mentioned above emphasizes the problem of what items to be treated as observable in classroom setting. Thus, Smith and Geoffrey (1968) make valid assertions criticising systematic observation systems:

The way the teacher poses his problems, the kind of goals and sub-goals he is trying to reach, the alternatives he weighs … are aspects of teaching which are frequently lost to the behavioural oriented empirical who focuses on what the teacher does to the exclusion of how he thinks about teaching. Smith and Geoffrey (1968:96)

McIntyre and Macleod (1986:14) generalize the problem of observable items and limitation of data obtained through systematic observation claiming that there is ‘no direct evidence on the actions of participants which are not overt’. The detailed criticism of systematic observation is given in Chapter 2.6.2.

2.2.3.  Data recording problems

The problem of accurate recording

Data collection, description procedures face problems of the accuracy and explicitness of records. ‘The crucial problem is to be able to render interpretable the process of events and behaviour as it occurs naturally’ (McKernan 1996:60).

Hutt and Hutt (1970:34) emphasise the difficulty of accurate description of the behaviour. They emphasize the problem with the vocabulary choice in that there are many thousands of words which describe motor and language behaviour but ‘unfortunately, the words are injunctive concepts, learned by usage rather than by definition’ (Hutt and Hutt 1970:34). Other than that, it is frequently found that some definitions are over encompassing in that they cover patterns of behaviour for which ordinary language has two or more terms. Lofland and Lofland (1995:93) recommend employing behaviouristic and concrete vocabulary rather than abstract adjectives and adverbs, which are based on paraphrase and general recall.

The problem of objective recording

Another problem with the written commentary to be discussed is the problem of objectivity. All researchers agree that the data are often subjective, reflect personal impressions, inferential and interpretative. Events may not be viewed the same way by different observers. ‘It is common to find that witnesses to an accident give differing accounts of what happened’ (Lofland 1995:127).

Eisner (1993:49) defines objectivity as being ‘fair, open to all sides of the argument’. He considers that to reduce subjectivity the observer must achieve correspondence not only in what s/he perceives or understands but how she or he represents it. Schaffer (1982:75) continuous the problem of vocabulary choice saying that there are some aspects of reality which can be described fairly objectively and those which can only be described subjectively, and ‘it is difficult to know where the borderline between objectivity and subjectivity lies’. Scheurich (1997:161) doubts in ‘the very existence of gross material reality’. He claims that research mainly addresses interpretation of meaning or constructions of ‘reality’.

To sum the problems with data recording I can suggest that an observer may describe and interpret an event in subjective way due to personal bias, theoretical assumptions, s/he can experience difficulty in the choice of an object/behaviour to observe and words to record an event in accurate and explicit way.

2.2.4.  The choice of an approach to observation

An observer faces the dilemma in choosing systematic or ethnographic approaches. The main problem of ethnographical approach lies in its very nature – it is so broad that it demands a highly trained observer to do a competent and reliable observation. ‘An untrained observer may be overwhelmed by the complexity of what goes on and not be able to focus on important events in the classroom’ (Day 1990:44). Pre-specified coding systems in systematic observation are exclusively concerned with ‘what can be categorized or measured’ (Simon and Boyer 1974). Thus they may distort or ignore the qualitative features which they claim to investigate. At the same time limiting the attention of the observer can help improve reliability.

2.3.          Reliability and Validity

2.3.1     Types of reliability

Reliability and validity are the most important criteria for assuring the quality of the data collection procedures. The criterion of reliability provides information on whether the data collection procedure is ‘consistent and accurate’ (Seliger and Shohamy 1989:185). The researchers suspect that observers may unintentionally impose their own biases and impressions on the observed situation. Seliger and Shohamy (1985:185) claim that for different types of data collection procedures different types of reliability are relevant. Thus they determine for the ethnographic approach the following types:

a)                  inter-rater reliability (to examine to which different observers agree on the data collected from the observation);

b)                 test-retes reliability (to check stability of data collection over time);

c)                 regrounding (to repeat the data collection and compare both results);

d)                 parallel form (to examine to which extent two versions of the same data collection procedure are really collecting the same data)

To assure reliability different methodologists suggest involving at least two observers to carry a ‘sequential analysis’ (Becker 1970:79), or to achieve ‘inter-observer agreement’ (Croll 1986:150). The idea of the former procedure is to carry out the analysis concurrently with data collection in the sense that ‘one may ‘step back’ from the data, so as to reflect on their possible meaning’ (Fielding 2001:158). Thus further subsequent data gathering will direct the observer either to abandon or pursue the original hypothesis. In the later procedure two observers look at the same events from different locations to categorise these events and compare the outcomes. Using systematic schemes with pre-specified categories they refine, or ‘index’ (Fielding 2001:159) the definitions and categories of observation by ‘applying in a consistent manner the procedures for data selection, collection, grouping, inclusion, exclusion etc.’ (Simpson and Tuson 1995:65).

2.3.2 Types and evidences of validity

Just as there are different types of reliability, Seliger and Shohamy (1989: 102) suggest that there are different types of validity which provide ‘evidence’ for validity. Thus, their typology of ‘evidences’ of validity comprises

a)                  evidence on content validity which demonstrates appropriateness of data collection against the content to be measured;

b)                 criterion validity which provides an indication as to whether the instrument can be measured against some other criterion and compared with the previous results (concurrent validity), and whether the procedure is capable of foretelling certain behaviour (predictive validity);

c)                 construct validity which examines whether the data collection procedure is a good representation of and consistent with current theories underlying the variable being measured.

Chaudron (1988:24) gives another term to the content validity and suggests ‘treatment validity’ which relates to the process component of process-product study and demonstrates that the treatment was in fact implemented and that it was identifiable different from whatever it was being compared with.

For the results of the second language research Seliger and Shohamy (1989:104) identify internal and external validity. They propose that a study has internal validity if the outcomes of the observational data can be directly and unambiguously attributed to the treatment that is applied to the observed group, and that the interpretation of these data is not dependent on the subjective judgement of an individual researcher. Internal validity in this sense relates to three areas: ‘representativeness, retrievability, and confirmability of the data’ (Seliger and Shohamy 1989:104). External validity involves the extent to which the findings of a study can be generalized and applied to another situation and the categories of the study are treated as basic, applied, and practical.

To achieve evidences of validity items or questions of an instrument must be analyzed in the process of data collection. A researcher or observer should obtain information on whether the items are of ‘low-inference’ or ‘high-inference’ (Long 1980), too easy or too difficult, and whether the items are phrased and easily understood by the respondents. All these aspects are recommended to examine in the pilot phase of the research that is likely to be proved by evidences from a variety of sources, such as additional questionnaire data from pupils or teachers, interviews, surveys. Another way of examining the validity of observation is to ask colleagues to study the categories and to define the purpose of the observation. Simpson and Tuson (1995:65) treat this method as a useful check on face validity. Thus to achieve reliable and valid observation an evaluator should take into account the spatial location of an observer, engage more than one observer, involve ‘low-inference’ categories that do not require complex interpretation and check agreement of key aspects against independent studies.

2.4.          Items of observation

2.4.1 The importance of items

In so far the language classroom observation ‘does not simply mean watching classes’ (Wallace 1991:123). An observer may record either very narrowly defined data such as a specific speech act, or more general kinds of language learning activity such as turn-taking, group work.

Any scientific research or observation is characterised by terms as ‘structured’, ‘organised’, ‘methodical’, and ‘systematic’. To follow these characteristics any data collection obtains a structure or format, and guided by some questions or variables. Croll (1986:55) notifies a variable as a basic unit that represents the process by which a concept of interest is turned into a set of working definitions whereby the results of observation or some other data collecting process can be categorized and measured.

2.4.2 Items of observation in the language classroom

For classroom observation as a learning tool Richards (1998:143) proposes three perspectives on a lesson for pre-service training to develop a deeper understanding of how and why teachers teach the way they do and the different ways teachers approach their lessons. They are:

1)                 Teacher-centered focus: the teacher is primary focus; factors include the teacher’s role, classroom management skills, questioning skills, presence, voice quality, manner, and quality instructions.

2)                 Curriculum-centered focus: the lesson as an instructional unit is the primary focus; factors include lesson goals, opening, structuring, task types, flow, and development and pacing.

3)                 Learner-centered focus: the learners are the primary focus; factors include the extent to which the lesson engaged them, participation patterns, and extent of language use.

Wallace (1998:68) substitutes the focus on the curriculum with the focus on the context in which the teacher teaches: the classroom layout, the teaching aids available and how they are used.

Low-inference and high-inference categoreis

The presentation of items involves constructing sets of categories into which occurrences must be coded unambiguously. In this respect Long (1980:3) introduces low-inference and high-inference measures. Low-inference categories include things that can be counted or coded without the observer having to infer their meaning from observable behaviour. Such categories according to Allwright and Bailey (2000:73) involve the number of times the student raises her/his hands, or the frequency with which the teacher uses the student’s name. High-frequency items demand that the observer make a judgement that goes beyond what is immediately observed. The samples of this type of categories cover factors like learner’s attention, or the social climate. I can conclude that observation data should cover categories of observable behaviour that does not require much interpretation.

2.5.          Typology of observation

Typology of classroom observation instruments is worked out by Wallace (1991:66) and he presents the following oppositions:

1.                  system-based, ethnographic or ad-hoc

2.                  global or specific

3.                  evaluative, formative or research-related

4.                  teacher-focused, learner-focused or neutral in focus

5.                  quantitative or qualitative

He admits that some of the oppositions are not clear-cut and overlap. For example, observation techniques which are primarily evaluative may be employed for formative purposes, ethnographic approach is treated as global and qualitative. System based approach can focus on teacher’s activity and learners’ activities. System-based (systematic), ethnographic and ad-hoc approaches encompass other characteristics of the classification provided. Thus, I outline the features of the first opposition.

2.5.1 System-based approach

By system-based observation Wallace (1991:67) means the observation that is based on a system of fixed and pre-specified categories. They are global in nature, i.e. ‘they are intended to give general coverage of the most salient aspects of the classroom process’ (Wallace 1991:110). Any system contains a finite array of categories. The endeavour of all system-based observation instruments is the analysis of teacher-class interaction. The two most influential systems are devised by Bellack (1966:267) and by Flanders (1970:314). Wallace (1991:112) has identified the characteristic features of the first system as:

1)                 the data are measured from a transcript, i.e. the data have to be first recorded and then transcribed;

2)                 the central place of labelled units of discourse are structure, solicit, response, reaction.

In the ‘Flanders tradition’ there is a form of documented recall where tallies are made every three minutes under one range of categories. In chapter 2.6 the analysis of a range of interaction schemes, their advantages and disadvantages are presented with more details. They are widely used by researchers as they are ready-made, well known and ‘it does not to be trialled and validated’ (Wallace 1991:111).

2.5.2. Ethnographic approach

The observation techniques share many of qualities of ethnographic practices. Ethnography is a detailed sociological observation of people which immerses the researcher in an intense period of observation ‘which guides and informs all subsequent data gathering’. (Radnor 2002:49)

Ethnographical approach is originally developed from the methodologies of field anthropologists and sociologists concerned with studying human behaviour within the context in which that behaviour would naturally occur. Methodologically, ‘anthropological’ classroom studies are based on participant observation, during which the observer immerse him/herself in the ‘new culture’. Initial data gathered by the ethnographer are open-ended and relatively unstructured that ‘allows and encourages the development of new categories’ (Delamont and Hamilton 1976:13). An ethnographer uses a holistic framework. S/he makes no attempt to manipulate, control or eliminate variables. At the same time s/he reduces the breadth of research problems systematically to give more concentrated attention to the emerging salient issues.

The great strength of the ethnographic research is that it gets away from the simplistic behavioural emphasis of the pre-specified codes. (Delamont and Hamilton 1976:37).

The main purpose of the ethnographic approach is the search for meaning and is based on the description of the studied phenomenon. However, Lutz (1986:112) warns that not everyone who can write a paragraph describing an encounter between a teacher and a student is an ethnographer, and he points out that an observer should be trained in ethnographic methods, particularly participant-observer field methods.

2.5.3 Ad-hoc approach

The term ‘ad-hoc’ is used to describe something that has been devised for a particular purpose, ‘with no claims to generality’ (Wallace 1991:113). The ad-hoc approach relates to structured approaches but the categories derive from a particular problem or research topic. That is why this system is more popular with practising teachers. What is more this approach is flexible and eclectic, and involves both quantitative and qualitative data where each seems appropriate. Wallace (1991:113) assumes that each different area of concern will yield a different system of analysis. Ad-hoc approach is considered to be the most appropriate in teacher-training education as it is basically guided discovery approach that drive student-teachers to focus and reflect on an important area of language teaching, and provide a meta-language with which to discuss. The instrument of ad-hoc approach is known as observation tasks (Wajnryb 1992) and is described in Chapter 2.6.2.

2.6.          Methods and techniques of observation

2.6.1 Classification of data collection techniques

Seliger and Shohamy (1989:158) present classification of data collection procedures according to the degree of explicitness. On one end of the scale they set broad and general techniques which do not focus on a particular type of data and are considered to be of a low degree, while at the other end they tend to put procedures which are more explicit and structured and thus reveal high degree of explicitness. Collecting data by procedures of a low degree of explicitness is done by means of open and informal description, which tends to be done simultaneously with its occurrence. Typical procedures of this kind are field notes, records, diaries, journals, lesson reports, personal logs, life history accounts, informal interviews with the subjects of observation. Collecting data by means of procedures of a high degree of explicitness involves the use of formal and structured types of data collection procedures. Examples of such procedures are interaction schemes, checklists, observation schedules, observation tasks, formal interviews, surveys, structured questionnaires, case studies, rating numerical scales. Different procedures imply different techniques for data collection. Data obtained from more structured observations are presented in the form of checks, tallies, frequencies, and ratings, while data obtained from the informal observations are presented in the form of narration, field-notes, or transcripts.

Страницы: 1, 2, 3, 4, 5, 6, 7