People who need people
Introduction
Thank you for coming to this lecture "People who need people" I am delighted to have been invited by the Vice-Chancellor to talk on this subject because, since my appointment at City University, people have asked me no end of questions about what psychometrics is, what it does, whether it works, and is it ethical. I appreciate the opportunity to address some of these issues in public.
As you know, psychometrics has become an increasingly controversial subject in recent years, not just in human resources but in society generally. I would like to begin by viewing psychometrics as it might appear from the point of view of the man or woman in the street. What are their impressions and concerns likely to be? Well, there are several. First of all, their knowledge of psychometrics is likely to have come from several sources, some personal, some critical, and some esoteric. They will have read articles in the press and will probably have completed questionnaires in magazines. They will have seen quiz shows on TV. But, perhaps more importantly, their impressions will be based on their own experiences of being assessed. They will have sat examinations at school and possibly at University, and will very likely have completed psychometric tests when applying for jobs. All of these experiences will almost certainly have affected their lives in important ways. Hence, the proverbial man in the street is unlikely to be neutral about the use of psychometric tests.
On the basis of all this, how is he likely to react when invited to take a psychometric test as part of a recruitment programme? Well, he will almost certainly be apprehensive. If it is an ability test that he is about to take, he will worry about not doing well. First he will think back to previous tests that he has done, in particular his GCSE and A Level examinations taken at school. He will begin to consider how he should prepare for the test. Many sources are often available for such preparation although not all are helpful. I once read a book, an Almanac written in India , intended to provide useful advice to job applicants. To the question "What do I need to do to do well on an ability test?" the reply was "Ability tests are not difficult, all you need is a quick mind and a good intelligence". I think you would agree that this is not particularly helpful.
For the candidate asked to take a personality test, the concerns will be different. He may think "Does this mean that they will be able to read my mind, to know what I'm thinking?" He will think about aspects of his character that might make him unsuitable for the job. And he will wonder about how much they could actually find out. And, of course, his attention will turn to ways in which he might be able to cover some things up. He will ask himself what he should do to present himself in the best light, and what he should say or not say.
Once the whole assessment process has been completed, whether he succeeds or fails in his job application, he will be left with many questions unanswered. Did he fail because he was too honest? Can psychometric testing truly allow for his individuality?
Did the assessment fail to take account of the fact that English was his second language, or that he had a disability such as dyslexia? Did he succeed just through luck or because he cheated? And how did his score compare with that of other candidates? And these concerns are not just his concerns, but also those of the selection panel and, ultimately, of the psychometricians who design and maintain the tests and questionnaires themselves. What answers can we provide?
I shall endeavour to address some of these questions by first addressing the question "What is psychometrics?" I shall then go on to consider common misconceptions about psychometrics, followed by an examination of current debates that involve psychometric issues. I shall conclude by addressing the key question of this lecture "Does psychometrics have a part to play in staff selection?"
The four principles of psychometrics
So what exactly is psychometrics? Psychometrics has been defined as "The science of psychological assessment". The four major principles of psychometrics are reliability, validity, freedom from bias, and standardisation.
Reliability
Reliability is the extent to which an assessment is free from error. There are several different types of reliability. Inter-rater reliability examines the level of agreement between two raters on the same assessment. This might be an observation of behaviour in assessment centre, or marks given for an in-tray exercise where the candidate was asked to draft a letter to a client. It would be foolish to make use of an assessment if there was evidence that raters were unable to agree on the quality of the candidates' performance. Another type of reliability is test-retest reliability. Here the same test is given to the same group of candidates on more than one occasion, and the consistency of scores across these occasions is examined. If there is little consistency then this tells us that we can have little confidence in the results. There is simply no point in using a test that gives a completely different score every time we use it - no different really from weighing yourself every morning and getting a different reading on the scales when you know that it is extremely unlikely that you have lost or gained weight. As a psychometric test undergoes development its reliability is monitored closely and the test adjusted to keep this value to a maximum.
Validity
Validity is the extent to which a test or questionnaire assesses what it purports to assess. There are four main types of validity. These are face validity, content validity, predictive validity and construct validity. Face validity exists if the candidate and all involved see the assessment as appropriate. For example, a personality test, designed for clinical use that contained the question "Do you hear voices" would lack face validity in most recruitment situations. The candidate may feel that that this is an inappropriate question to ask of someone who is applying for a job. A student once described face validity as "giving a test to a group of people and looking at the expressions on their faces". Not a bad definition really.
With content validity we match the test specification, the blueprint that underlies the development of the test, to the job description for the post for which the person is applying. If the two do not match then the use of the test would in be inappropriate for the job in question. Predictive validity concerns the availability of evidence that the test will be effective in predicting how well a person will perform if appointed. This is the prime objective for most psychometric tests used in staff selection and promotion.
Finally, construct validity assesses the extent to which the assessment relates to its construct. Constructs are ideas like 'emotional intelligence' or 'IQ'. Simply describing a test as a test of emotional intelligence is not enough. We need to be able to present arguments and convince others that 'emotional intelligence' is a real entity and say exactly what we mean by it. We also need to show how the items we have chosen to test emotional intelligence are actually measuring this construct and not something else.
An assessment can be reliable but not valid. Handwriting analysis can be quite reliable in that two handwriting experts trained in the same system may have a high level of agreement about what is meant by a particular aspect of handwriting. That is, they may both believe that if handwriting leans backwards this means that the writer has a yearning to return to things as they were in the past. But this interpretation may be wrong. It could well be that there are other reasons why peoples' writing leans backwards. Maybe it has something to do with whether a person is right or left handed? In this case we would say that while handwriting analysis can be reliable in that it produces consistent reports from people trained in its methods, this does not necessarily mean that it has any validity whatsoever for the purpose of assessing personality.
Equivalence
I would like to turn now to the third psychometric principle, fequivalence or reedom from bias. Test bias arises if the test is unfair to certain groups, that is, if one consequence of using the test is a disproportionate representation of people of a particular background, or of men or women, in the workforce. When this happens the test is described as having 'adverse impact'. There are several ways in which a psychometric test might be biased. It may be that some of the items contain bias. An item may contain very colloquial words or phases the meaning of which may elude speakers of English as a second language. Irony and sentences such as "He goes round with a chip on his shoulder' should be avoided as they can cause confusion.
Standardisation
Finally, standardization refers to the availability of appropriate comparison groups, so that the score of an individual can be compared with the average for the population - this is called 'norm referencing' - or interpreted alongside those of other people in a similar job; called criterion referencing. Before we can make sensible use of any test we need to know what a particular score means. If someone is told that they have achieved a score of 75 on a test they might initially be quite pleased - until they are told that everyone else who applied for the job scored over four hundred and fifty. Also, whatever anyone else may have scored on a test, we as practitioners need to know the sort of performance we can reasonably expect from a person with this particular score.
IQ testing
I would now like to take a moment to address the controversial issue of the construct validity of the intelligence test, more commonly known as the IQ test. The first laboratory for psychometrics was set up by Sir Francis Galton at the Grand Exhibition of 1883 held in South Kensington. Sir Francis was the first person to attempt to measure intelligence psychometrically. He offered to test the faculties of people who were visiting the exhibition, and made a charge of three pence per person for this service. But what is intelligence? Edwin Boring in 1923 gave a particularly unconvincing argument for the construct validity of intelligence when he defined it as "That which is measured by intelligence tests".
In fact, psychologists use the term 'intelligence' very sparingly, if at all. It is a very old word carrying with it rather too much conceptual baggage. Intelligent or 'gifted' individuals were seen by the Ancient Greeks as having received 'a gift from the gods". Since the time of Darwin it is more likely to be seen as a gift from nature, but a gift nonetheless. And if you have not received this gift, the worse for you - you can't fight god and you shouldn't fight Mother Nature. Such a belief undermines the evidence that people can change, can learn, and can increase their ability to do any number of things.
Another reason that psychologists and psychometricians are so cautious about the use of intelligence testing is its murky role in the eugenics movement of the last century. Before 1940, IQ tests were routinely used not just to assess special educational needs but also to provide evidence for sterilization programmes for those who were believed to be 'subnormal'. This abuse of the science is something that most of us hoped had been buried by World War II. I feel deeply uneasy when I watch TV programmes such as 'Test the Nation" and sincerely believe that intelligence testing is not something to be trifled with so readily. The multicultural society we are developing in London has been a great achievement, and anything that reinforces prejudices that some groups are better or worse than others 'by nature' is highly damaging.
Questions for recruiters
Anyone involved in staff selection, whether using psychometric tests or not, should be able to satisfy themselves that their assessment procedures have complied with the four psychometric principles. Also, they should be in a position to provide a justification of their choice of assessment procedures to others if called upon to do so. They need to know that their assessment system is reliable in that it can be depended upon to produce a consistent result. They also need to know that it is valid in that the person selected is likely to be the best person for the job in comparison to all the other candidates. They will also need to be sure that their selection procedure is free from bias. Finally, they need to be sure that the standards they have set are the correct ones.
Common misconceptions about psychometrics
I am now going to move on to consider three common misconceptions about psychometrics that underlie many of the questions that I and other psychometricians are asked by our clients, students and the interested public. These are the belief that the four psychometric principles only apply to psychometric testing, the belief that it is easy to cheat in a personality test, and the belief that psychometric tests fail to take account of a person's true individuality.
The misconception that psychometric principles only apply to psychometric tests
The first common misconception is that the four psychometric principles of reliability, validity, standardisation and freedom from bias may be right for psychometric tests and questionnaires, but that they do not apply to other forms of assessment such as school examinations or the more open-ended type of interview. This is not true. They apply to any assessment. In terms of reliability it would be counter-productive to reject a psychometric test on the grounds that it was not reliable, if instead selection was by an interview that proved to be even less reliable. Or, in relation to the psychometric principle of freedom from bias, it would be inappropriate to reject a psychometric test on the grounds that it discriminated against a particular ethnic group, if selection were to be based instead on a procedure that was even more biased.
An obvious alternative to a psychometric test of ability might be the use of GCSE grades. But this will also have its problems. For example, given that, on average, young Black Londoners, particularly boys, are less successful in obtaining good GCSE results than kids from members of other ethnic groups in London , the use of GCSE grades will have considerable adverse impact on them as a group. In the sense in which the term 'bias' is used in psychometrics, GCSE results are very biased indeed!
With respect to the psychometric principal of standardisation, it has recently been suggested that psychometric tests might be a more effective technique for selecting students for University than the 'A' level system. This has arisen, perhaps, because the 'A' level system itself has not paid sufficient attention to the need for standardisation. Universities have always had pre-existing ideas of how students with particular 'A' Level grades might perform as learners. If entry standards change, for example if students with 3 'A's in science at A level are less able to cope with mathematical concepts than were students with these grades in years earlier, then more tuition may be required, or some existing courses may have to be reduced in complexity or abandoned altogether. There will also be knock-on effects once these students graduate. Employers may begin to suspect that today's graduates are not as effective as were previous graduate recruits. Finally, when it comes to the psychometric principle of validity, there are many cases in employee selection where psychometric tests have been rejected on the grounds of suspected poor validity, only to be replaced by alternatives such as a particular type of interview, without any objective consideration of whether these alternatives are themselves valid. The need to demonstrate validity should be just as important for an interview as it is for a psychometric test.
I have talked so far only about the application of psychometric principles to recruitment situations. However, there are many other forms of assessment, such as the driving test, licensing qualifications for doctors, for accountants and for other professionals, psychiatric diagnoses, and the assessment of special educational needs in children, to which they equally apply. What I would like to emphasise here is that, whatever you may think of psychometric tests themselves, the psychometric principles are inescapable in any assessment process. Far more training is needed in the application of these principles in all areas of assessment. And applicants with psychometric experience are increasingly being sought by examination boards, management and recruitment consultancies and executive search organizations, as well as by human resources departments.
The misconception that it is easy to cheat on personality tests
The second misconception I would like to address is the widespread belief that it is easy to get a 'good' result on a personality test by faking answers to questions. Now, of course, there can be no doubt that it is possible to lie when answering a question in a questionnaire. But this applies when answering a question in an application form, or in an interview. In all these cases, the recruiter needs to be aware of this possibility and to have well developed strategies for dealing with it. Hence, when a test is constructed, precautions will normally be taken and safeguards built into the test itself.
Many questionnaires contain a lie detector. There will be, scattered among all the other items, a series of items such that people will only tend to answer them in a particular way when they are lying. An example of such an item might be "It always pays to tell the truth (Agree or Disagree)". Clearly the statement cannot be true, certainly not 'always' as the question specifically states. Most people will reluctantly disagree with the statement. Some may agree because at first glance it seems to them to be true, perhaps because this is what they were taught at Primary School. But the majority who agree will do so because they think that 'Agree' is the desired answer, even though they do not seriously believe it to be true. Of course, any one item like this on its own could not predict lying. But if the same pattern was observed for a whole series of lie-detector items then we would probably feel rather hesitant about accepting the candidate's answers to the real questions. This is only one of a number of strategies to detect lying that are used by psychometricians when constructing a test. Hence, as with an interview, people may lie, but by carefully choosing which questions to ask, by ordering the questions in a particular way, and by setting internal alarms matched to certain patterns of answers, liars should know that their subterfuge may well be detected.
So failure to detect dishonest responding is not the most important issue when considering the relative merits of personality questionnaires and interviews as assessment techniques. There are more important ones. An advantage of the personality questionnaire is that it puts exactly the same questions to large numbers of people in different situations. In this way we build up a detailed knowledge of how people generally respond to particular questionnaire items and combinations of items. This provides a background against which an individual person's responses can be interpreted. But the questionnaire does not have the ability of the interviewer to react to the applicant, to tailor the questions to the facts as they emerge. Well - we are working on it. In the meantime the personality questionnaire can be a valuable addition to the selection process. Anything we can do to supplement the interview is to be welcomed for, as most of you will know, in many situations the interview is not particularly reliable, or valid, or standardized or free from bias.
The misconception that psychometric tests fail to take account of a person's individuality
A third common misconception is the widespread belief that the people who design and work with psychometric tests are only interested in numbers and fail to understand the human side of assessment and evaluation. This belief is also shared by some social scientists, who see psychometrics as purely quantitative, while interviews and other 'soft' techniques of assessment are believed to be more qualitative. I believe that rather too much is made of the distinction between the qualitative and the quantitative approaches. However unstructured an interview, it must always result in a quantitative ranking of candidates in terms of perceived suitability for the post, otherwise no appointment could be made. Similarly, psychometric tests, whatever mathematical techniques are used in their construction, must have a qualitative aspect. They must take account of the real-world situation in which people live.
This was bought home to me in my first research job where I was employed on a Home Office Grant to validate a questionnaire designed to identify people with anti-social personality disorder. My Professor, a distinguished psychologist of the day, sent me to Wandsworth Prison to administer the questionnaire to a selection of inmates who had been designated as recidivists, prisoners who returned to prison again and again, usually for petty crimes. He was delighted when his questionnaire was found to discriminate very highly between this group and a control group of non-prisoners. I found on further analysis that almost all of the discrimination could be accounted for by two items: These were "There are people who have it in for me." And "I would get on much better in the world if people did not get in my way". Need I say more?
For academic reasons some people may want to draw a sharp line between qualitative and quantitative methods, but in practice this is usually unhelpful. A prudent human resource professional will ignore neither aspect of the assessment process. Of course, psychometrics as a discipline has always been back-footed in the quantitative / qualitative debate by its very name 'psycho-metrics'. The 'metric' part refers to measurement, surely the epitome of quantitative analysis. And many psychometricians have taken pleasure in quoting a phrase from William Thompson, also known as Lord Kelvin, who was the arch-measurer of the 19 th Century and was responsible for the Kelvin scale of temperature. He said "Everything that exists must exist in some quantity and can therefore be measured".
Today, however, psychometrics is no longer 'the science of psychological measurement'. It is now 'the science of psychological assessment' and there is a subtle but important difference between the two. Often, but not always, we can obtain someone's measure by summing up their positive attributes, subtracting their negative attributes and examining this total, rather like the scales of justice. But the whole can be more than the sum of the parts, as perhaps can best be demonstrated by what in psychometrics is referred to as the cloning problem. Suppose we are able to identify the perfect personality profile for 'a good salesman'. When we use this profile in staff selection we will be able to 'clone' this ideal profile and select a group of good salesman. But will this be a good team? Probably not, because teams need diversity. The simple addition of attributes fails to recognize that there are many different types of good salesman. There are many different profiles that fit the bill. In this way the 'quantitative' psychometric approach appears to fall down when compared with the more 'qualitative' interview. But does psychometrics have to be about simply totalling up scores? Not any more. In the past, tests and questionnaires were necessarily limited to paper and pencil, but today questions can be presented on computer screen, and the session can be interactive, just as in an interview. The technology for doing this is complex and state-or-the-art, but a great deal of progress has already been made. Computer-assisted testing programs are in widespread use across the United States for their minimum competency testing programme in Primary Schools. And there are other interesting developments.
A questionnaire can be treated like an expert system. This is a term used in artificial intelligence research to describe the type of computer program that has largely replaced the role of the traditional bank manager in making decisions on who should or should not be allowed a loan. Such programs can also make quite a good job of medical diagnosis on health websites. Most of you probably know of them in the form of the 'wizards' (for example that annoying paper-clip) that give advice when you use a computer.
More advanced artificial intelligence systems are likely to find increased utility in biometrics and psychometrics alike. Neural networks can learn to improve their performance on the basis of their successes and failures at hitting targets Once a target had been set, such as only selecting staff who would stay in a job for at least 6 months, a neural network could learn to predict this outcome from the patterning of answers given in personality tests at selection. With the advent of the internet this presents both problems and promise. The problems mostly involve human rights issues, the right to privacy in particular, but after some initial stumbling with the implementation of the provisions of the Data Protection Act, most organizations are taking seriously and coming to grips with the issues involved. The hugh growth in spam, not just emails but faxes and now telephone calls as well, does somehow focus the mind!
The speed at which the information revolution is moving is now beginning to break down barriers between adjacent disciplines, both in academia and in practice. Is the 'lie detector' a biometric or a psychometric device? Is a psychometric test of achievement a psychometric or an educational test? Is the use of keyboard behaviour to detect fraud psychometrics or is it actuarial science? And, if we link assessments of psychological constructs such as fear and greed to the behaviour of the stock market, is this psychometrics or econometrics? Maybe in all these cases, it is both. It's not just politics and the media that have been thrown into turmoil by the communications revolution. We are moving into a new era in which psychometrics and its fellow sciences will inevitably play an increasing role.
The promises are many and varied, so I will focus on just one, the potential for improved prediction. Applied psychometrics is, in essence, a predictive technology, an actuarial science. We use selection tests to predict how candidates would perform if given the job, and we select those who best fit this prediction. All actuarial sciences are currently undergoing revolutionary changes, whether it be the prediction of investor behaviour, life expectancy, insurance or credit risk. Actuarial prediction is a necessary part of all these functions, just as it is for human resources. But the enormous advances made have not been without their problems. New Artificial Intelligence systems probably had a role in several of the stock market crashes of the late 1990s; the use of postcodes is raising a number of new issues; as is genetic testing for insurance purposes. Hence we cannot expect the use of predictive technology in human resources to escape controversy. But neither can we bury our heads in the sand. The future is after all inescapable. Many of the mistakes occurred not because people took unnecessary risks, but because they failed to foresee the consequences of the new technology. Inaction also holds risks. Most personnel directors can think of occasions where a failure to predict the outcome of a particular appointment has proved disastrous and anything we can do to improve our predictions must be a major priority.
Application of psychometrics to current debates
I would now like to suggest various ways in which a psychometric perspective can help us better appreciate some current issues. The three issues I would like to talk about are University entrance, the recruitment of minorities, and the development of on-line testing.
University entrance
Recently there has been increasing debate about standards of entry into university, a debate that culminated last year in what became known as 'the A Level fiasco'. The question 'have standards fallen?' is ongoing, and apparently unresolved. But in many ways the type of answer you receive depends on what is meant by 'standards'. As well as a general increase or decline in standards it is possible for the academic standard of the students to go down and the standard of teaching to go up, and vice versa. The parties are often arguing at cross purposes. But this still leaves real concerns about academic standards. How can psychometric principles help here?
Let's start by agreeing that the 'A level' examination has certainly changed. While previously, for many academic subjects, it might have been the case that 'all that was needed to succeed was a quick mind and a sharp intelligence' as the almanac says, this is becoming increasingly less so. The student is now required to complete large amounts of course work and is expected, much more than in the past, to take part in activities in order to obtain credits for continuous assessment. All of these changes have particularly benefited students of average ability - who are by far the majority. But what about those who are above or below average in ability? Have they actually been disadvantaged by the success of their more average fellows? In some ways, yes. The few university applicants who in the past obtained three straight AAAs at 'A level', a pattern of scores that almost guaranteed them a good university place, have now been joined by countless others who would previously have done less well. The effect is that Universities find it increasingly difficult to identify potential high-flyers.
While the elite students and universities may receive little sympathy for this dilemma, there is also cause for concern at the other end of the ability scale. Until recently it was the norm not to have 'A levels'. This was also true, but to a lesser extent, for GCSEs, at least as far as what counted for a 'good' result, was concerned. As more and more pupils and students achieve the qualifications they desire, not to have done so becomes an increasing stigma, a situation that has not been helped by the tendency of some schools to be more likely to exclude these children in order to meet targets. The increasing marginalizing of under-achievers seems to be a natural result of a policy designed to increase access. And for the others? Well, one of the main aims in life at that age is to make something of oneself, to be somebody. But when looking for work after graduation many find that the Gilbert and Sullivan dictum "If everybody's somebody then no-one's anybody" is more than just a music hall joke. While the target of increased access to University has largely been achieved, this has not been without cost. And while applying 'the art of the possible' must always have its costs as well as its benefits, surely not all of these outcomes were intended? Many appear to be better explained by Matthew Parrish's 'law of unintended consequences'. An application of psychometric principles at an earlier stage would have allowed a more informed debate to have taken place, and might even have suggested some less chaotic alternatives.
Recruitment of minorities
A second current issue I would like to address is the difficulty that many employers face in recruiting staff that represent the social mix in their locality, an important issue given transport difficulties and housing costs in the inner city. As I've said before, under-achievement at GCSE by a large number of minority pupils in London is an example of a biased assessment system, that is, the use of GCSE results alone for recruitment would result in adverse impact. There are two principal ways in which adverse impact can be addressed, either by positive discrimination or by affirmative action. Positive discrimination demands that applicants from each group be treated differently, either with different entry requirements or by using a quota system. Affirmative action refers to an action programme designed to remedy the sources of inequality.
The new university entry policy urged on the universities by the Government is an example of a positive discrimination system. Students might, for example, be selected from poor inner city schools with only three grade Bs at 'A level', but if they had attended a private school, three grade A's might be required. There are many examples worldwide of successful positive discrimination systems, but they are controversial. Recently, the University of Michigan 's positive discrimination programme in favour of poorer students went all the way to the US Supreme Court and still rattles on through Senate and Congress. But while such schemes may have short or even medium-term benefits, on their own they do not address the root causes of inequality in societies.
In the university entry scheme the use of positive discrimination will increase access for disadvantaged groups but will not address the root causes of underachievement in some inner city schools.
An affirmative action program would target this and other causes of under-recruitment, such as different ambitions or expectations from the system. In many cases of adverse impact there is a tendency to blame the assessment procedures used, but this is often just a case of blaming the messenger. What is often needed is a scientific analysis of all the alternatives and their role in the adverse impact. A psychometric approach enables data to be collected and analysed, trends to be monitored, improvements to be designed and evaluated, and ultimately allows policy decisions to be well-informed.
On-line testing
The third issue I want to talk about demonstrates the need to be prepared for the unexpected. The internet, in particular, continually throws up surprises. It is probably already obvious to all here that on-line internet assessment is here to stay and will continue to evolve. A recent survey has shown that 70% of corporations use tests to measure human performance and nine out of ten are going to be using more tests in 2004. Already, many of these tests are delivered over the web. In the meantime on-line sales, on-line banking, and on-line booking are now starting to rise exponentially.
It is important to remember that, before the breakthrough in internet testing, there were genuine concerns about the impact of this new technology, and many of these still remain. It is interesting to review some of these with the benefit of hindsight. Much of the initial concern was about different levels of competence in computer use. Clearly someone who is not familiar with a keyboard is unlikely to shine on a computer administered test. But today 'a computer in every classroom' has become as important a social campaign as was 'literacy for all' exactly 100 years ago when Alfred Binet first introduced ability tests into the Primary Schools of France. While there are clear dangers of adverse impact against older candidates, not being able to use a computer, in itself, disqualifies people from many jobs. The need remains, however, for more to be done to support life-long learning in computer use among the over 30s.
While more research is needed to demonstrate that on-line testing is as good as or better than paper and pencil testing, for many practical applications this need has simply being overtaken by events. When an organization is recruiting for posts that demand computer proficiency, and if the necessary tests are available on computer, it often makes no sense to use paper. Indeed as fewer and fewer people are proficient at handwriting or managing without a spelling checker and a wizard, a new concern is emerging. Do paper and pencil test adversely impact computer users?
At City University we are partnering with the major US on-line testing company Performance Assessment Network, or PAN, to provide a Proctored High Stakes Testing Centre. Pan supplies Procter & Gamble's worldwide internet based recruitment programme as well as the system used by the Canadian Government for its Civil Service. The new On-line Testing Unit at City University offers companies, consultants and individual psychologists a unique resource that combines the benefits of internet delivery with a standardised environment staffed by expert test administrators.
The unit, based in The Psychometrics Centre, is the first of its kind to be set up in the United Kingdom . It will be used by occupational consultants, HR departments in organisations, recruitment companies, and clinical and educational psychologists to undertake large scale recruitment and development testing; to administer batteries of tests for high level coaching and counselling, and for the assessment of children and adults for educational, rehabilitation and legal purposes. And we are delighted that the launch of this new service will coincide with our move to the University's new Social Science building, later this year.
Does psychometrics have a place in staff selection?
I would now like to come back to the main question of this lecture "Does psychometrics have a place in staff selection?" I am sure you will not be surprised to hear that my answer is "yes" .. But you might perhaps like some more details.
Let me give you a practical example. Psychometric tests can be classified into two broad categories: those that assess ability and those that assess personality, and these can be used separately or together depending on the particular position to be filled. Let's say that you want to recruit a senior manager in an insurance company. This person must be able to interpret and implement policy, to understand the implications of policy for financial planning, be able to manage staff and demonstrate a high level of personal integrity.
The following four tests would give you a standardised assessment of each of these characteristics to produce an overall profile. These tests go under the names of the Watson-Glaser Critical Thinking Appraisal, RANRA, Orpheus and Giotto. The Watson Glaser Critical Thinking Appraisal is designed to assess high level verbal reasoning. Its questions are derived from five key areas of critical thinking - inference, assumptions, deduction, interpretation and evaluation. The Watson-Glaser produces an overall score on a scale that assesses a person's ability to apply the skills necessary for effective critical reasoning.
The second test that might be used to select the manager is RANRA. This is a companion to the Watson Glaser and is designed to assess numerical reasoning. RANRA produces an overall score of a person's ability to apply their numerical skills in a critical manner.
The third test that might be used is the work-based personality questionnaire 'Orpheus'. The five major scales of Orpheus measure: (i) Fellowship, that is the degree of extraversion or introversion; (ii) Authority, that is the capacity to make tough decisions; (iii) Emotion, the ability to work under stress; (iv) Conformity, with its inverse of openness to experience; and (v) Detail that assesses the extent to which the candidate can focus down on minor details when necessary. Orpheus contains 190 questions, and an answer is required for each question on a four point scale ranging from 'strongly agree' to 'strongly disagree'. Scores on these five traits can be presented as a profile. Orpheus also produces a narrative report that gives a detailed account of the candidate's expected functioning with respect to these five areas. This is two pages long and so I will not attempt to present it on screen.
The last test is Giotto. This is a test of personal integrity that produces a score on each of seven subscales that assess carefulness, orientation to work, slowness to anger, fair-mindedness, loyalty to organisational values, trustworthiness, and the ability to cope with change. So, in combination, these four tests would give you an all-round picture of each candidate in terms of the four key aspects of the job. These four tests are just a small sample of the many hundreds of recognised tests that are available.
Human Resourse Professionals need to be qualified in order to administer psychometric tests. Both psychologists and non-psychologists can obtain these qualifications. A Level A certificate is required for ability testing, and a Level B certificate for personality testing. The British Psychological Society is the isssuing body for these certificates.
I will end by summarising ten ways that psychometrics can help you:
- It can provide an efficient, standardised assessment of an individual's strengths and capabilities. Moreover, if psychometric tests or questionnaires have been administered prior to an interview, they can provide information concerning each candidate's likely strengths and possible weaknesses, and these can be pursued as part of the interview process.
- Many psychometric tests come accompanied by an extensive research based literature that can provide justification for their use in a wide variety of situations and settings.
- Psychometrics can be applied to a recruitment activity of any scale, ranging from a single post to a multi-centre multi-national recruitment campaign carried out world-wide.
- Numbers are no object. We can test a single individual, or we can test millions, as with the minimum competency tests for children used in the US.
- Psychometrics provides a set of principles - the psychometric principles of reliability, validity, standardisation and freedom from bias - that form the basis for the objective examination of the merits and demerits of a wide range of different assessment systems.
- The psychometric principles provide a framework within which the effectiveness of policies such as an organisation's equal opportunities policy can be monitored.
- Psychometrics can be a specialist career path within human resources that is exciting, rewarding and much in demand.
- Knowledge of psychometrics enables human resource executives to increase their influence in the boardroom by being at the forefront of new developments in IT and the use of the web.
- The possession of psychometric expertise provides an opportunity for human resources to influence management decisions through the presentation of scientific evidence that can demonstrate links between recruitment policy and company success.
- Because the area is so fast moving, psychometrics offers those organisations involved a competitive advantage over their less reactive counterparts.
Our job at the Psychometrics Centre is to help you in these activities as best we can. We offer advanced training in psychometrics that is suitable for numerate and highly motivated human resourse professionals, as well as for graduates with a good science degree who wish to follow a career in psychometrics.
© John Rust, 26th February, 2004. All rights reserved.