A psychometric analysis from the Item Response Theory: step-by-step modelling of a Loneliness Scale
DOI:
https://doi.org/10.22235/cp.v14i1.2179Keywords:
item response theory, pedagogical, loneliness, graduated response model, advanced psychometricsAbstract
The Item Response Theory (IRT) is a set of psychometric models used in the development, assessment, improvement, and scoring of evaluating scales. This pedagogical article provides an initial overview of how to conduct, as well as interpret and present the results from, the application of IRT models suitable for ordered polytomous response options. The data used as an example for IRT modelling corresponds to the administration of the Buenos Aires Loneliness Scale (BALS), a new instrument for the assessment of loneliness self-perception. This data also corresponds to a non-probabilistic and incidental sample of 509 participants residing in the Buenos Aires Metropolitan Area (53 % women). The objective of this article is to present an overview of the general steps and components needed to perform an IRT analysis, as a way to increase access to this powerful psychometric technique.
Downloads
Download data is not yet available.
References
Abal, F. (2013). Comparación de modelos politómicos y dicotómicos de la Teoría de la Respuesta al Ítem aplicados a un test de Comportamiento Típico. Tesis de Doctorado, Facultad de Psicología de la Universidad de Buenos Aires.
Abal, F. J. P., Auné, S. E., Lozzia, G. S., & Attorresi, H. F. (2017). Funcionamiento de la categoría central en ítems de Confianza para la Matemática. Evaluar, 17(2), 18-31.
Aiken, L. R. (1980). Content validity and reliability of single items or questionnaires. Educational and Psychological Measurement, 40, 955-959. https://doi.org/10.1177/001316448004000419
Aiken, L. R. (1985). Three coefficients for analyzing the reliability and validity of ratings. Educational and Psychological Measurement, 45, 131-142. https://doi.org/10.1177/0013164485451012
Asún, R. & Zuñiga, C. (2008). Ventaja de los modelos politómicos de teoría de respuesta al ítem en la medición de actitudes sociales. El análisis de un caso. Psykhe, 17(2), 103-115.
Attorresi, H. F., Lozzia, G. S., Abal, F. J. P., Galibert, M. S., & Aguerri, M. E. (2009). Teoría de Respuesta al Ítem. Conceptos básicos y aplicaciones para la medición de constructos psicológicos. Revista Argentina de Clínica Psicológica, 18(2), 179-188.
Auné, S., Abal, F., & Attorresi, H. (2017a). Propiedades psicométricas de una prueba de conducta empática. Revista Iberoamericana de Diagnóstico y Evaluación Psicológica, 3(45), 47-56. https://doi.org/10.21865/RIDEP45.3.04
Auné, S., Abal, F., & Attorresi, H. (2017b). Versión argentina de la Escala de Felicidad de Lima. Diversitas, 13(2), 201-214.
Auné, S., Abal, F., & Attorresi, H. (2019). Construction and psychometric properties of the Loneliness Scale in adults. International Journal of Psychological Research, 12(2), 82-90. http://dx.doi.org/10.21500/20112084.425782
Auné, S., Abal, F., & Attorresi, H. (2020). Modeling of the UCLA Loneliness Scale According to the Multidimensional Item Response Theory. Current Psychology, 1-8. https://doi.org/10.1007/s12144-020-00646-y
Baker, F. B., & Kim, S. H. (2017). The Basics of Item Response Theory Using R. New York, NY: Springer.
Bock, R. D. (1997). The nominal categories model. In W. van der Linden & R.K. Hambleton (Eds.), Handbook of Modern Item response Theory (pp. 33-50). N.Y.: Springer.
Cai, L. (2012). flexMIRT: Flexible multilevel item factor analysis and test scoring [Computer software]. Seattle, WA: Vector Psychometric Group, LLC.
Cai, L., Thissen, D., & du Toit, S. (2011). IRTPRO user's guide. Lincolnwood, IL: Scientific Software International.
Cappelleri, J. C., Lundy, J. J., & Hays, R. D. (2014). Overview of classical test theory and item response theory for the quantitative assessment of items in developing patient-reported outcomes measures. Clinical Therapeutics, 36(5), 648-662. https://doi.org/10.1016/j.clinthera.2014.04.006
Chen, W., & Thissen, D. (1997). Local dependence indices for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22, 265-289.
De Ayala, R. J. (2009). The theory and practice of item response theory. New York, NY: Guilford.
Edelen. M. O., & Reeve, B. B. (2007). Applying item response theory (IRT) modeling to questionnaire development, evaluation, and refinement. Qual Life Res., 16(1), 5-18. http://dx.doi.org/10.1007/s11136-007-9198-0
Ferrando, P. J., & Loranzo-Seva, U. (2017a). Assessing the quality and appropriateness of factor solutions and factor score estimates in exploratory item factor analysis. Educ. Psychol. Measur., 1-19. https://doi.org/10.1177/0013164417719308
Ferrando, P. J., & Lorenzo-Seva, U. (2017b). Program FACTOR at 10: Origins, development and future directions. Psicothema, 29, 236-240.
Haberman, S. J. (1978). Analysis of qualitative data: Vol. 1: Introductory topics. New York, NY: Academic Press.
Hair, J. F., Anderson, R.E., Tatham, R. L. & Black, W. C. (1999). Análisis Multivariante. Madrid, España: Prentice Hall Iberia.
Langer, M. (2008). A reexamination of Lord’s Wald test for differential item functioning using item response theory and modern error estimation (Unpublished doctoral dissertation). University of North Carolina, Chapel Hill.
Lozano, L. M., García-Cueto, E. & Muñiz, J. (2008). Effect of the Number of Response Categories on the Reliability and Validity of Rating Scales. Methodology, 4(2), 73-79. https://doi.org/10.1027/1614-2241.4.2.73
Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47(2), 149-174.
Masters, G. N. (2016). Partial Credit Model. En W. J. van der Linden (Ed.). Handbook of Item Response Theory, Volume 1: Models (pp. 109-126). Boca Raton: Chapman & Hall/CRC.
Maydeu Olivares, A., & Joe, H. (2005). Limited and full information estimation and testing in 2n contingency tables: A unified framework. Journal of the American Statistical Association, 100, 1009-1020. http://dx.doi.org/10.1198/016214504000002069
Maydeu Olivares, A., & Joe, H. (2006). Limited information goodness-of-fit testing in multidimensional contingency tables. Psychometrika, 71, 713-732. http://dx.doi.org/10.1007/s11336-005-1295-9
Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm, Applied Psychological Measurement, 16, 159-176. https://doi.org/10.1002/j.2333-8504.1992.tb01436.x
Orlando, M., & Thissen, D. (2000). Likelihood-based item fit indices for dichotomous item response theory models. Applied Psychological Measurement, 24, 50-64. https://doi.org/10.1177/01466216000241003
Orlando, M., & Thissen, D. (2003). Further investigation of the performance of S-χ2: An item fit index for use with dichotomous item response theory models. Applied Psychological Measurement, 27, 289-298. https://doi.org/10.1177/0146621603027004004
Paek, I., & Cole, K. (2019). Using R for Item Response Theory Model Applications. New York, NY: Routledge.
Sacchi, C. & Richaud de Minzi, M. C. (1997). La Escala Revisada de Soledad de UCLA: Una adaptación argentina. Rev. Argent. Clín. Psicol, 6(1), 43-53.
Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika monograph supplement, 17(4), 2. http://dx.doi.org/10.1002/j.2333-8504.1968.tb00153.x
Samejima, F. (2016). Graded response models. In Wim J. van der Linden (Ed.), Handbook of Item Response Theory, Volume One (pp. 123-136). Chapman and Hall/CRC.
Timmerman, M. E., & Lorenzo-Seva, U. (2011). Dimensionality assessment of ordered polytomous items with parallel analysis. Psychological Methods, 16, 209-220. http://dx.doi.org/10.1037/a0023353
Toland, M. (2013). Practical guide to conducting an item response theory analysis. The Journal of Early Adolescence, 34(1), 120-151. https://doi.org/10.1177/0272431613511332
Woods, C. (2009). Empirical selection of anchors for tests of differential item functioning. Applied Psychological Measurement, 33(1), 42-57. https://doi.org/10.1177/0146621607314044
Abal, F. J. P., Auné, S. E., Lozzia, G. S., & Attorresi, H. F. (2017). Funcionamiento de la categoría central en ítems de Confianza para la Matemática. Evaluar, 17(2), 18-31.
Aiken, L. R. (1980). Content validity and reliability of single items or questionnaires. Educational and Psychological Measurement, 40, 955-959. https://doi.org/10.1177/001316448004000419
Aiken, L. R. (1985). Three coefficients for analyzing the reliability and validity of ratings. Educational and Psychological Measurement, 45, 131-142. https://doi.org/10.1177/0013164485451012
Asún, R. & Zuñiga, C. (2008). Ventaja de los modelos politómicos de teoría de respuesta al ítem en la medición de actitudes sociales. El análisis de un caso. Psykhe, 17(2), 103-115.
Attorresi, H. F., Lozzia, G. S., Abal, F. J. P., Galibert, M. S., & Aguerri, M. E. (2009). Teoría de Respuesta al Ítem. Conceptos básicos y aplicaciones para la medición de constructos psicológicos. Revista Argentina de Clínica Psicológica, 18(2), 179-188.
Auné, S., Abal, F., & Attorresi, H. (2017a). Propiedades psicométricas de una prueba de conducta empática. Revista Iberoamericana de Diagnóstico y Evaluación Psicológica, 3(45), 47-56. https://doi.org/10.21865/RIDEP45.3.04
Auné, S., Abal, F., & Attorresi, H. (2017b). Versión argentina de la Escala de Felicidad de Lima. Diversitas, 13(2), 201-214.
Auné, S., Abal, F., & Attorresi, H. (2019). Construction and psychometric properties of the Loneliness Scale in adults. International Journal of Psychological Research, 12(2), 82-90. http://dx.doi.org/10.21500/20112084.425782
Auné, S., Abal, F., & Attorresi, H. (2020). Modeling of the UCLA Loneliness Scale According to the Multidimensional Item Response Theory. Current Psychology, 1-8. https://doi.org/10.1007/s12144-020-00646-y
Baker, F. B., & Kim, S. H. (2017). The Basics of Item Response Theory Using R. New York, NY: Springer.
Bock, R. D. (1997). The nominal categories model. In W. van der Linden & R.K. Hambleton (Eds.), Handbook of Modern Item response Theory (pp. 33-50). N.Y.: Springer.
Cai, L. (2012). flexMIRT: Flexible multilevel item factor analysis and test scoring [Computer software]. Seattle, WA: Vector Psychometric Group, LLC.
Cai, L., Thissen, D., & du Toit, S. (2011). IRTPRO user's guide. Lincolnwood, IL: Scientific Software International.
Cappelleri, J. C., Lundy, J. J., & Hays, R. D. (2014). Overview of classical test theory and item response theory for the quantitative assessment of items in developing patient-reported outcomes measures. Clinical Therapeutics, 36(5), 648-662. https://doi.org/10.1016/j.clinthera.2014.04.006
Chen, W., & Thissen, D. (1997). Local dependence indices for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22, 265-289.
De Ayala, R. J. (2009). The theory and practice of item response theory. New York, NY: Guilford.
Edelen. M. O., & Reeve, B. B. (2007). Applying item response theory (IRT) modeling to questionnaire development, evaluation, and refinement. Qual Life Res., 16(1), 5-18. http://dx.doi.org/10.1007/s11136-007-9198-0
Ferrando, P. J., & Loranzo-Seva, U. (2017a). Assessing the quality and appropriateness of factor solutions and factor score estimates in exploratory item factor analysis. Educ. Psychol. Measur., 1-19. https://doi.org/10.1177/0013164417719308
Ferrando, P. J., & Lorenzo-Seva, U. (2017b). Program FACTOR at 10: Origins, development and future directions. Psicothema, 29, 236-240.
Haberman, S. J. (1978). Analysis of qualitative data: Vol. 1: Introductory topics. New York, NY: Academic Press.
Hair, J. F., Anderson, R.E., Tatham, R. L. & Black, W. C. (1999). Análisis Multivariante. Madrid, España: Prentice Hall Iberia.
Langer, M. (2008). A reexamination of Lord’s Wald test for differential item functioning using item response theory and modern error estimation (Unpublished doctoral dissertation). University of North Carolina, Chapel Hill.
Lozano, L. M., García-Cueto, E. & Muñiz, J. (2008). Effect of the Number of Response Categories on the Reliability and Validity of Rating Scales. Methodology, 4(2), 73-79. https://doi.org/10.1027/1614-2241.4.2.73
Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47(2), 149-174.
Masters, G. N. (2016). Partial Credit Model. En W. J. van der Linden (Ed.). Handbook of Item Response Theory, Volume 1: Models (pp. 109-126). Boca Raton: Chapman & Hall/CRC.
Maydeu Olivares, A., & Joe, H. (2005). Limited and full information estimation and testing in 2n contingency tables: A unified framework. Journal of the American Statistical Association, 100, 1009-1020. http://dx.doi.org/10.1198/016214504000002069
Maydeu Olivares, A., & Joe, H. (2006). Limited information goodness-of-fit testing in multidimensional contingency tables. Psychometrika, 71, 713-732. http://dx.doi.org/10.1007/s11336-005-1295-9
Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm, Applied Psychological Measurement, 16, 159-176. https://doi.org/10.1002/j.2333-8504.1992.tb01436.x
Orlando, M., & Thissen, D. (2000). Likelihood-based item fit indices for dichotomous item response theory models. Applied Psychological Measurement, 24, 50-64. https://doi.org/10.1177/01466216000241003
Orlando, M., & Thissen, D. (2003). Further investigation of the performance of S-χ2: An item fit index for use with dichotomous item response theory models. Applied Psychological Measurement, 27, 289-298. https://doi.org/10.1177/0146621603027004004
Paek, I., & Cole, K. (2019). Using R for Item Response Theory Model Applications. New York, NY: Routledge.
Sacchi, C. & Richaud de Minzi, M. C. (1997). La Escala Revisada de Soledad de UCLA: Una adaptación argentina. Rev. Argent. Clín. Psicol, 6(1), 43-53.
Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika monograph supplement, 17(4), 2. http://dx.doi.org/10.1002/j.2333-8504.1968.tb00153.x
Samejima, F. (2016). Graded response models. In Wim J. van der Linden (Ed.), Handbook of Item Response Theory, Volume One (pp. 123-136). Chapman and Hall/CRC.
Timmerman, M. E., & Lorenzo-Seva, U. (2011). Dimensionality assessment of ordered polytomous items with parallel analysis. Psychological Methods, 16, 209-220. http://dx.doi.org/10.1037/a0023353
Toland, M. (2013). Practical guide to conducting an item response theory analysis. The Journal of Early Adolescence, 34(1), 120-151. https://doi.org/10.1177/0272431613511332
Woods, C. (2009). Empirical selection of anchors for tests of differential item functioning. Applied Psychological Measurement, 33(1), 42-57. https://doi.org/10.1177/0146621607314044
Downloads
Published
2020-06-01
How to Cite
Auné , S. E. ., Abal , F. J. P. ., & Attorresi , H. F. . (2020). A psychometric analysis from the Item Response Theory: step-by-step modelling of a Loneliness Scale. Ciencias Psicológicas, 14(1), e-2179. https://doi.org/10.22235/cp.v14i1.2179
Issue
Section
ORIGINAL ARTICLES
License
The reproduction rights of the article are granted under the Creative Commons Attribution 4.0 International License (CC BY 4.0), open access system.
