These studies demonstrate how a problem can be identified and clarified. They do not attempt to track the problems back to their root causes, which would be outside the scope of this project. They do, however, show examples of possible solutions to the problems, derived from the information elicited about the nature of the problem.
The emphasis in this report is on showing how different techniques can be used to tackle different aspects of the problem, so that anyone interested in applying these techniques in their own work can do so. The techniques are well established in other disciplines, and good tutorial texts are available.

The underlying foundation of these studies is that choice of elicitation method for gathering data is an important issue: different methods are able to elicit different types of information, and, conversely, will also differ as regards the types of information which they fail to elicit. This success or failure is not random, and is predictable in relation to the types of information involved. No single technique is able to elicit all types of information. Interviews and questionnaires will systematically miss a range of types of information which can be easily elicited using other techniques, as demonstrated in these studies.

This issue has far-reaching implications for work in this area. Fortunately, the shortcomings of questionnaires and interviews can be quite easily overcome by using other techniques, as described below. This offers the prospect of a wide range of new insights into issues involved in retention. We hope that this study will be of use to other researchers in this area.

1.2: Finding out why students drop out

It is easy to ask students why they drop out from a course. However, the extent to which students’ stated reasons correspond to reality is quite another question. Practitioners and researchers in this area appear well aware of this problem, but appear unsure how best to tackle it. The apparent problem is students’ unwillingness to give completely truthful answers; however, the real problem is a subtly different one, namely students’ inability to give complete and correct answers, for reasons described below. Although the difference may be subtle, it is also extremely important, and has far-reaching implications for any attempt to investigate this area.

In the domain of software engineering, for example, it became clear that the problem was not users’ unwillingness to give relevant information about their requirements, but users’ inability to give it, even when their lives would depend on the software working properly (for instance, in the case of flight control systems for “fly by wire” aircraft). This led to considerable research into the elicitation problem, and to the realisation that any attempt to elicit information from human beings had to take into account the constraints imposed by a range of factors involving cognition, communication, and organisational behaviour.

This section describes a possible solution.

This possible solution is based on a framework recently developed for software engineers faced by similar problems when trying to obtain a complete and correct set of requirements for a new software system (Maiden & Rugg, 1996). The original framework was subsequently extended for more general use (Rugg & McGeorge, 1999), and is described below.

Any attempt to investigate the real reasons why students drop out needs to take these factors into account. An outline of these factors, based on Maiden & Rugg (1996) and Rugg & McGeorge (1999) is as follows.

A key issue is that most types of memory and skill cannot be reliably or validly accessed by interviews or questionnaires, for reasons which will become apparent.

The lay model of memory is that it is unitary – there is only one type of human memory. This, however, is not the case. The type of memory described as “explicit” memory in the list below is what most people think of when they refer to “memory”. It is the sort of memory we use to answer questions such as “What is the capital of France?” or “Where were you born?” This form of memory, though imperfect, is very long-lasting, and has enormous capacity.

There are, however, other types of memory, with very different characteristics. One of these is short term memory (STM), which has a very limited capacity of about seven items (Miller 1956), and a duration of only a few seconds. It is this sort of memory which we use for purposes such as remembering a phone number for a few seconds until we can write it down. Since STM only stores items for a few seconds, any attempt to find out what is in STM at a given moment has to be conducted at that time – a few minutes later that information will be irretrievably gone. This is an important issue in software design for safety-critical systems, where it is essential to know what information the user needs for any given task, and to make sure that the information is readily available on-screen. Interviews and questionnaires have no hope of accessing STM: the only method which offers a means of doing this is to ask the person to think aloud while doing the task (“on-line self-report”).

This is an issue which has significant implications for studying the very earliest stages of the retention process. Prospective students read sources of information such as prospectuses and Web pages, and use these to choose a course. If an information source misleads them into applying for an inappropriate course, then there is a potential retention problem right from the outset. Any attempt to identify the factors which a prospective student notices when reading a prospectus or Web page has to take STM into account, and an example of this is given in the first study described in this report.

Although STM cannot be accessed via interviews, it can at least be accessed via one technique. There are, however, types of memory and skill which cannot be accessed directly via any technique. A classic example of a skill of this sort is touch typing: touch typists can operate at high speed and high precision, but this does not involve some of the skills which one might expect. If a skilled touch typist is asked which key is to the right of “g” on the keyboard, the typical response is for the typist to imagine themselves typing, and then, in effect, to “read off” the name of the relevant key from that mental image. Similarly, skilled drivers will typically be unaware of the precise sequence of actions involved when they change gear.

1.3: Frameworks for guiding choice of elicitation technique

Although there are many elicitation methods, there are few systematic frameworks to guide choice of an appropriate technique. One quite widely used approach is triangulation, in which two or more techniques are used to cross-check each other. The problem with this approach is that it does not provide solid guidelines for deciding which techniques to use for the triangulation.

A more structured approach is a framework developed by Maiden & Rugg and subsequently expanded by Rugg & McGeorge, which describes the main types of memory and communication, groups them into categories, and provides guidance on the choice of appropriate elicitation technique. A brief description is given here, with particular reference to implications for studying student retention. The four main categories are:
· future systems knowledge
· explicit knowledge
· semi-tacit knowledge
· tacit knowledge

These are described below.

Future systems knowledge

This term originates in computer system development, but is equally applicable to retention. If a stakeholder is asked what they would like from a new system, their response is not based directly on memories of past events, but is based instead on their model of their own future behaviour and wants. Even in cases where there are no issues of negotiation between conflicting stakeholder groups, there are solid grounds for caution about people’s statements about their future requirements or behaviour. A considerable body of research in several disciplines has shown there is a very poor fit between people’s statements about their future behaviour and what actually happens – for instance, the substantial body of work in the “heuristics and biases” tradition deriving from Kahneman, Slovic & Tversky’s seminal work (Kahneman, Slovic & Tversky, 1982).

For retention studies, the implication is that any statements from students about how they might behave if the academic system were changed should be treated with caution. However, because the main factors involved have been identified, it may be possible to make allowances for known biases when predicting student responses, and to produce a more accurate prediction as a result.

Explicit knowledge

This is “common or garden” knowledge, as described above: available to introspection, and accessible via any elicitation technique. It should, however, be treated with caution all the same. Memory is not like a photograph which starts off as a clear, accurate record and then becomes more faded and scratched over time. Instead, memory is an active process, both at the encoding and at the retrieval stages, and can be surprisingly easy to distort, even to the extent of implanting memories of events which never happened (Loftus & Palmer, 1974). It also tends to process information at the encoding stage so that the memories fit into expected templates; this processing can extend to reversing the memory of the order in which events actually happened (Bartlett, 1932).

Semi-tacit knowledge

This is knowledge which is accessible via some elicitation methods, but not all. It can be subdivided into a number of types.

Short Term Memory (STM) has been described above. It is a short-term, low-capacity form of memory, only accessible via on-line self-report. It is important in information processing – knowing which information is being processed, and which information is being ignored, is important for anyone producing information sources such as Web pages and prospectuses.

Recall and recognition access the same information, but recall (active memory) is usually considerably less powerful than recognition (passive memory). This can lead to knowledge being missed by techniques dependent on recall, such as interviews. Presenting material to prompt recognition can help with this, but at the risk of cueing the respondent in the direction of the material presented.

Taken For Granted (TFG) Knowledge refers to one of Grice’s maxims of communication: you do not say anything explicitly if you take it for granted that the other person already knows that information (Grice, 1975). For instance, you do not say “My aunt, who is a woman” because you can take it for granted that the other person knows that aunts are women. This is an extremely useful maxim, since it keeps communications down to a manageable length. However, it leads to problems when the speaker’s assumptions about what can be taken for granted are wrong.

A classic example for students coming from Further Education to Higher Education involves bibliographic references. Lecturers will advise the students to make proper use of bibliographic references when writing assignments, dissertations, etc; however, the lecturers frequently take it for granted that the students will know what is meant by “good” references, and not bother to spell this out. This typically leads to students feeling lost, or, arguably worse, misunderstanding what is meant (e.g. viewing textbooks as better references than journal articles because the textbooks are standard canonical sources of information).

Taken For Granted Knowledge will by definition often be missed by interviews and questionnaires, but is usually detected fairly easily by observation and by laddering (Hinkle, 1975, Rugg & McGeorge, 1995), which breaks down terms systematically in a way which usually leads to specific and unambiguous explanations.

The second set of studies in this project demonstrate this approach, using laddering to uncover Taken For Granted Knowledge involved in perceptions of what makes for a good dissertation.

Not Worth Mentioning (NWM) Knowledge is similar in some ways to TFG knowledge (Forsyth, 2000). The difference is that TFG knowledge may be viewed as extremely important, whereas NWM knowledge is by definition viewed as not very important. Much “craft skill” knowledge falls into this category – “tricks of the trade” which are used by experts, but which are typically viewed as not being sufficiently important to be worth teaching explicitly in formal classes, or mentioning in textbooks, but which may be essential skills.

Front and back versions derive from Goffman’s model of life as akin to the theatre, with a “front stage” version for public consumption, and a “backstage” version which is not shown to outsiders (Goffman, 1959). This is a significant filtering factor in issues such as students’ decisions about what to report or to omit when giving their reasons for withdrawing from a course.

One way of accessing back versions is via projective approaches – asking respondents to say what response they think someone else might give in the relevant situation. This approach has been widely used in market research, where it is considered to give more accurate predictions about customer behaviour than asking customers about their own opinions.

This approach has been used in the second set of studies reported here, to investigate perceptions of students’ reasons for withdrawing.

Tacit knowledge is knowledge which is not accessible via introspection – it is effectively “black box” knowledge. Much of experts’ skilled behaviour comes into this category. The area is the subject of considerable ongoing research (and disagreement) in psychology, but the general points are fairly well agreed. It should be noted that the term “tacit knowledge” is used in some domains to include any knowledge which is not stated explicitly. It is used here in a stricter sense, to refer only to knowledge which cannot be accessed via introspection.

There are two main categories into which tacit knowledge can be divided.

Implicit learning(e.g. Seger, 1994) is learning which occurs without ever passing through conscious awareness. This typically involves learning by seeing large numbers of examples. There is debate about the degree of skilled performance which results, and there is some evidence that experts may believe they have learned things which are not in fact present in the data.

Compiled skills are skills which were once non-tacit, but which have become tacit as a result of practice (Anderson, 1990). These are usually, but not necessarily, motor skills. Compiled skills are normally much faster and more efficient than the non-compiled version of the same skill – the classic example is expert versus novice drivers.

The language used by many academics to describe marking criteria strongly suggests that some form of tacit knowledge is involved in parts of the task: for instance: “It’s hard to put into words, but after a while you can recognise a good dissertation just by glancing at the front page.”

1.4: Conclusion

Eliciting valid replies to the question: “Why do students drop out?” is a real problem, and one with significant implications. Any attempt to improve retention rates which is based on unreliable answers to this question is likely to encounter problems, to waste resources and to cause further difficulties for all involved in further and higher education.

The framework described above offers a systematic, theoretically grounded way of approaching this question, based on best practice from other disciplines. The case studies described below illustrate how this framework can be used in practice.