Major breakthrough in ipsative test scoring methodology

The 2011 Best Doctoral Dissertations Prize from the Psychometric Society was awarded to Dr Anna Brown of The University of Cambridge Psychometrics Centre. The thesis, entitled “How Item Response Theory can solve problems of ipsative data”, was carried out at the University of Barcelona under the supervision of Professor Alberto Maydeu-Olivares. The prize was awarded at the 17th International Meeting of the Psychometric Society, held in Hong Kong in July 2011.

Anna and Alberto’s approach is mind-boggling, and resolves a longstanding and sometimes acrimonious debate in the field of ipsative testing. A psychometric test is ipsative if respondents are required to compare two or more options and pick the one that is most preferred. Ipsative tests are popular as, by forcing respondents to choose between equally desirable options, they are more difficult to fake.  For example, in the integrity test ‘Giotto’, published by Pearson Assessment, respondents decide whether the most appropriate description of them is either ‘Resourceful’ or 'Cheerful’.  The ipsativity lies in the fact that by choosing ‘Resourceful’ they are also saying something about the degree to which ‘Cheerful’ applies to them, and vice versa, and hence the single item contributes to scores on two separate scales; Originality and Mood. A problem is solved, but another one created as the assumption of statistical independence of the two scales, on which so much psychometric rigor depends, is broken. Ipsative scores distort scale relationships and reliability estimates, and make interpretation of scores problematic.

Anna’s research demonstrates how Item Response Theory (IRT) modelling may be applied to overcome these problems. A multidimensional IRT model for forced-choice questionnaires is introduced, suitable for use with any forced-choice instrument composed of items fitting the dominance response model, with any number of measured traits, and any block sizes (i.e. pairs, triplets, quads etc.). While personality is the main focus of the research, the method can equally be applied to the assessment of motivation, interests, beliefs and attitudes. 

The model is grounded in Thurstone's framework for comparative data. Thurstonian IRT is based on normal ogive models with structured factor loadings, structured uniquenesses, and structured local dependencies. These can be straightforwardly estimated using the Structural Equation Modelling (SEM) software Mplus. Simulation studies show how the latent traits are recovered from the comparative binary data under different conditions. The Thurstonian IRT model is tested with real participants in both research and occupational assessment settings.  It is concluded that when the recommended design guidelines are met, scores estimated from forced-choice questionnaires with the proposed methodology reproduce the latent traits well. 

The Psychometric Society is a non-profit professional organisation devoted to the advancement of quantitative measurement practices in psychology, education and the social sciences. It publishes the journal Psychometrika which contains articles on the development of quantitative models of psychological phenomena, as well as statistical methods and mathematical techniques for evaluating psychological and educational data.