Integrating SDT and IRT Models for Mixed-Format Exams

Every time you take a standardized test — an IQ assessment, a college entrance exam, a professional certification — the questions have been calibrated using sophisticated statistical models that most test-takers never learn about. Item Response Theory (IRT) is the mathematical framework behind virtually all modern psychological and educational testing, and understanding its basics illuminates why tests work the way they do. Read more →

Most psychometrics textbooks teach the classical "correction for attenuation" — Spearman's century-old technique for estimating what the correlation between two psychological constructs would be if the tests measuring them were perfectly reliable. The technique is simple: divide the observed correlation by the square root of the product of the two reliabilities. The technique is also limited: it adjusts the relationship between two scales, but assumes the reliability values plugged into the denominator are themselves accurate. A 2022 paper by Jari Metsämuuronen in Applied Psychological Measurement argues that this assumption is broken in practice. Reliability estimates produced by Cronbach's alpha and similar formulas are themselves attenuated by the same mechanical errors that attenuate correlations — and in some datasets, alpha may be deflated by 0.40–0.60 units of reliability. Metsämuuronen's contribution is a class of deflation-corrected reliability estimators that apply the classical attenuation logic inside the reliability formula rather than only to correlations between scales. Read more →

The General Knowledge (GK) subtest of the Jouve Cerebrals Crystallized Educational Scale (JCCES) measures factual breadth — the accumulated stock of information about the world that crystallized intelligence theory treats as a core component of acquired cognitive ability. A subtest of this kind has to satisfy two structural requirements: items should span a meaningful range of difficulty, and they should order along a single underlying continuum of factual breadth rather than tapping multiple unrelated dimensions. This study examined the item structure of the JCCES GK subtest using multidimensional scaling (MDS) on response data from 588 respondents, and recovered the empirical signature of a well-ordered unidimensional construct: a horseshoe-shaped scaling pattern that is the canonical evidence of a Guttman simplex — items ordered cleanly along a single difficulty continuum. Read more →

Cronbach's coefficient alpha is the most-reported reliability statistic in psychology and educational measurement. It is also one of the most-misunderstood. The classical formula assumes that test items measure a single construct with equal factor loadings (tau-equivalence), uncorrelated errors, and continuously distributed scores. Real psychological measurement rarely meets all three assumptions: most scales use Likert responses (discrete), have items with unequal contributions to the construct (congeneric), and produce score distributions that depart from normality. The natural question is how badly alpha breaks under these violations and which alternatives perform better. A 2023 simulation study by Xiao and Hau in Educational and Psychological Measurement provides a systematic answer, with implications for the routine reliability reporting that fills psychometric methods sections. Read more →

Mixed-format exams, commonly used in large-scale assessments, present a challenge for researchers seeking to model responses across different item types. Historically, multiple-choice items have been analyzed using frameworks like signal detection theory, while open-ended items are typically modeled using item response theory. DeCarlo’s work builds on these approaches, introducing a method to unify them through the probability of knowing, a concept that bridges both models.

Unified Framework: The article demonstrates how the SDT choice model and IRT sequential logit model can be integrated into a single framework. This approach captures latent states such as "know" and "don’t know" to analyze responses across item types. Psychological Processes: By modeling both item types simultaneously, the approach highlights differences