PsychoLogic

Evidence-based psychology, psychometrics, and cognitive science research

Dimensional Structure of JCCES Math Items

By , Psychometrician

Published · Reviewed

1,824 words · 8 min read · 6 references cited

Mathematical problem-solving is not a single cognitive ability — it draws on numerical reasoning, algorithmic computation, applied analytic thinking, spatial visualization, and quantitative pattern recognition, with different problems weighting these components in different proportions. A test built to measure mathematical problem-solving therefore covers a functional space rather than a single dimension, and the practical question for test development is how many distinct dimensions are needed to represent the structure of real test items adequately. This study used Alternating Least Squares Scaling (ALSCAL; Young, Takane, & Lewyckyj, 1978) — a flexible nonmetric multidimensional-scaling procedure — to investigate the dimensional structure of the 38 Mathematical Problems items from the Jouve Cerebrals Crystallized Educational Scale (JCCES) on a sample of 588 respondents.

Background: dimensions of mathematical problem-solving

The cognitive psychology literature recognizes that mathematical problem-solving is not unidimensional. Geary (1994) framed mathematical development as the interplay of biologically primary number sense, secondary culturally-acquired computation, and higher-order strategy use; the three contribute distinguishably to performance on real problems. Swanson and Beebe-Frankenberger (2004) tied mathematical problem-solving to working-memory subsystems, finding that visuospatial sketchpad and central executive demands are differentially loaded across problems. Hecht and Vagi (2010), looking specifically at fraction skills, identified separable contributions from conceptual understanding, procedural knowledge, and strategic flexibility.

The implication for assessment design is that any reasonably broad mathematical-reasoning test will tap multiple cognitive dimensions, and the question is how many. A unidimensional test would be measuring only one of these dimensions; a hyper-multidimensional test would be redundant. The empirical answer depends on the specific test and the sample, and the appropriate analysis is dimensionality estimation from the data.

The JCCES Mathematical Problems (MP) subtest contains 38 items selected to span numerical, algorithmic, and applied reasoning content. The test is administered as part of the broader JCCES crystallized intelligence battery, which also includes Verbal Analogies and General Knowledge subtests. The MP subtest’s dimensional structure has practical consequences for how its items are interpreted, scored, and used in composite calculations.

Method

Five hundred and eighty-eight respondents completed the JCCES MP subtest under standardized administration. The 38-item correlation matrix was submitted to ALSCAL with dimensional solutions ranging from two through five dimensions. Convergence criteria were set at improvement < 0.001 in successive stress values; Kruskal's Stress Formula 1 (Kruskal, 1964) was the goodness-of-fit measure, with RSQ (proportion of variance in disparities accounted for by the solution) as the complementary index.

The decision criterion for selecting the preferred dimensionality was: the smallest dimensionality that achieves RSQ > 0.80 and stress within the 0.10 to 0.20 range. Higher dimensionalities provide better fit by construction but risk over-fitting; the criteria balance fit against parsimony.

Results

The four candidate dimensionalities produced the following fit statistics:

00.250.50.7512-D3-D4-D5-DStressRSQDimensionalityFit statistic
Figure 1. Fit improves with dimensionality; the 4-dimensional solution is the first to clear RSQ > 0.80 with acceptable stress (Kruskal, 1964).
  • 2-dimensional: stress = 0.305, RSQ = 0.647. Failed the criteria — stress too high, RSQ too low.
  • 3-dimensional: stress = 0.210, RSQ = 0.739. Stress at the upper limit, RSQ below threshold. Failed.
  • 4-dimensional: stress = 0.155, RSQ = 0.815. Both criteria satisfied. Preferred solution.
  • 5-dimensional: stress = 0.127, RSQ = 0.851. Better fit but at the cost of an additional dimension.

The 4-dimensional solution captures 81.5% of the variance in inter-item disparities and operates at a stress level (0.155) that ALSCAL conventions characterize as “fair to good” (Kruskal, 1964). The 5-dimensional solution improves both metrics modestly — stress drops by 0.028, RSQ rises by 3.6 percentage points — but the additional dimension does not yield a substantively distinct cluster. Under the parsimony principle, the 4-dimensional solution is preferred.

The four dimensions correspond to distinct cognitive substrates that the MP items collectively tap. A formal dimensional labeling requires substantive analysis of which items load on which dimensions, which is beyond the scope of the present analysis but is the natural follow-up: examining item content within each dimension to identify the cognitive demand each represents.

Interpreting the four-dimensional structure

The 4-D solution implies that the JCCES MP subtest measures four separable but related cognitive dimensions, consistent with the prior literature on mathematical problem-solving (Geary, 1994; Swanson & Beebe-Frankenberger, 2004; Hecht & Vagi, 2010). Plausible candidate dimensions, based on the substantive content of the MP items:

Numerical reasoningAlgorithmic / procedural fluencySpatial / visualizationWord-problem analytic reasoning
Figure 2. The preferred 4-dimensional solution corresponds to four separable cognitive substrates the JCCES math items tap.
  • Numerical reasoning: items requiring direct manipulation of numerical quantities, mental arithmetic, and quantitative comparison. The most basic mathematical demand.
  • Algorithmic / procedural fluency: items requiring application of standard mathematical procedures (long-division algorithms, equation-solving routines, formula application).
  • Spatial / visualization: items requiring mental visualization of geometric or spatial relationships, distance, area, or directional reasoning.
  • Word-problem analytic reasoning: items presented in narrative form requiring extraction of mathematical structure from natural-language description, then application of appropriate mathematical operations.

The four-dimensional structure is consistent with the multi-component frameworks proposed by Geary (1994) and Swanson and Beebe-Frankenberger (2004), and with the more recent fraction-skill decomposition of Hecht and Vagi (2010). The MP items, in aggregate, tap the same general cognitive substrates that the math-cognition literature has identified, suggesting good content validity for the broad construct of mathematical problem-solving rather than a single narrow ability.

Implications for JCCES scoring and math-test design

The dimensional finding has direct implications for how MP items are scored and reported. If the subtest is unidimensional (one dominant factor), a single composite score adequately summarizes a respondent’s performance. If the subtest is multidimensional with four separable dimensions, a single composite can hide meaningful between-respondent differences in which dimensions are strong or weak. A respondent strong in numerical reasoning but weak in word-problem analysis would receive an average composite score that is less informative than dimension-specific scores.

For composite-score interpretation, the JCCES MP composite is best understood as a broad mathematical-reasoning index rather than a measure of a single ability. The factor analysis of JCCES + GAMA on a different sample showed that the JCCES MP subtest itself loads on a crystallized factor distinct from nonverbal-fluid measures, but the within-MP dimensional structure is finer-grained than that across-battery analysis revealed.

For test development, the 4-dimensional structure suggests that future revisions of the MP subtest could deliberately balance items across the four dimensions, ensuring that the composite represents a defined mix rather than depending on which item types happen to dominate the item pool. Item-development practice that targets each dimension explicitly tends to produce more interpretable composites than item-development that selects for difficulty alone.

Methodological caveats

The 588-participant sample, while substantially larger than typical Cogn-IQ research samples, was a convenience sample drawn from online media. Generalizability to the full population of mathematical-problem-solving respondents — children, professional populations, non-Western cultural contexts — is not established and should be addressed in replication studies.

ALSCAL is an exploratory technique. It identifies a defensible dimensional solution but does not formally test it against alternatives. A confirmatory analysis — through structural equation modeling with hypothesized dimensional structure or item response theory with multidimensional models — would strengthen the inference from “this is the most parsimonious dimensional solution that fits the data” to “this is the structure that the data support more than alternatives”. Multidimensional IRT methods in particular are well-suited to the next analytical step.

The criteria thresholds for stress and RSQ are conventions; reasonable analysts would set them differently and might prefer the 5-dimensional solution under stricter thresholds. The 4-D vs 5-D choice is a parsimony judgment, not a definitive empirical verdict.

Connection to the broader Cogn-IQ research program

This analysis sits within a broader program of psychometric work on the JCCES. The JCCES + GAMA MDS analysis showed that JCCES subtests cluster on a crystallized side distinct from GAMA’s nonverbal cluster. The JCCES + GAMA factor analysis recovered the same two-factor structure under different statistical machinery. The present analysis goes one level deeper: within the MP subtest itself, four dimensions account for the within-subtest structure. The picture across studies is that the JCCES is a hierarchically structured battery — broad crystallized intelligence at the top, distinguishable subtest-level constructs at the next level, and finer item-level dimensions within each subtest at the bottom.

Frequently Asked Questions

What is ALSCAL and how does it differ from other MDS methods?

Alternating Least Squares Scaling (Young, Takane, & Lewyckyj, 1978) is a nonmetric multidimensional-scaling algorithm that allows ordinal-level proximity data and uses an alternating least squares optimization to minimize stress. It is more flexible than classical metric MDS, which assumes interval-level proximities, and is implementable in standard statistical software including SPSS and R packages.

Why prefer the 4-dimensional solution over 5-dimensional?

The 5-D solution improves stress by 0.028 and RSQ by 3.6 percentage points — modest gains for an additional dimension. The parsimony principle favors the simpler model when fit improvements are marginal. The 4-D solution satisfies the predefined criteria (stress 0.80) and is more interpretable.

What does Kruskal’s stress measure?

Kruskal’s Stress Formula 1 (Kruskal, 1964) measures the discrepancy between the observed proximities and the distances reproduced by the MDS solution, normalized by the squared distances. Lower stress indicates better fit. Conventional benchmarks: stress 0.20 poor.

How does the dimensional structure inform test scoring?

If the test is multidimensional with four separable dimensions, a single composite score collapses meaningful information. Reporting dimension-specific scores alongside the composite gives a more granular picture of a respondent’s mathematical-reasoning profile and can identify specific strengths and weaknesses that the composite obscures.

What is the next analytical step beyond ALSCAL for this data?

Confirmatory analysis with multidimensional item response theory (MIRT) or structural equation modeling, using the 4-dimensional structure as a hypothesized model. The exploratory ALSCAL result is suggestive; a confirmatory test against the data would establish whether the structure replicates and whether it fits better than alternative dimensional models.

References

  • Geary, D. C. (1994). Children’s mathematical development: Research and practical applications. American Psychological Association. https://doi.org/10.1037/10163-000
  • Hecht, S. A., & Vagi, K. J. (2010). Sources of group and individual differences in emerging fraction skills. Journal of Educational Psychology, 102(4), 843–859. https://doi.org/10.1037/a0019824
  • Kruskal, J. B. (1964). Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis. Psychometrika, 29(1), 1–27. https://doi.org/10.1007/BF02289565
  • Kruskal, J. B., & Wish, M. (1978). Multidimensional scaling. Sage. https://doi.org/10.4135/9781412985130
  • Swanson, H. L., & Beebe-Frankenberger, M. (2004). The relationship between working memory and mathematical problem solving in children at risk and not at risk for serious math difficulties. Journal of Educational Psychology, 96(3), 471–491. https://doi.org/10.1037/0022-0663.96.3.471
  • Young, F. W., Takane, Y., & Lewyckyj, R. (1978). ALSCAL: A nonmetric multidimensional scaling program with several individual-differences options. Behavior Research Methods & Instrumentation, 10(3), 451–453. https://doi.org/10.3758/BF03205177

Related questions

What is item response theory?

Every time you take a standardized test — an IQ assessment, a college entrance exam, a professional certification — the questions have been calibrated using sophisticated statistical models that most test-takers never learn about. Item Response Theory (IRT) is the mathematical framework behind virtually all modern psychological and educational testing, and understanding its basics illuminates why tests work the way they do. Read more →

What is attenuation-corrected reliability?

Most psychometrics textbooks teach the classical "correction for attenuation" — Spearman's century-old technique for estimating what the correlation between two psychological constructs would be if the tests measuring them were perfectly reliable. The technique is simple: divide the observed correlation by the square root of the product of the two reliabilities. The technique is also limited: it adjusts the relationship between two scales, but assumes the reliability values plugged into the denominator are themselves accurate. A 2022 paper by Jari Metsämuuronen in Applied Psychological Measurement argues that this assumption is broken in practice. Reliability estimates produced by Cronbach's alpha and similar formulas are themselves attenuated by the same mechanical errors that attenuate correlations — and in some datasets, alpha may be deflated by 0.40–0.60 units of reliability. Metsämuuronen's contribution is a class of deflation-corrected reliability estimators that apply the classical attenuation logic inside the reliability formula rather than only to correlations between scales. Read more →

How are JCCES General Knowledge items structured?

The General Knowledge (GK) subtest of the Jouve Cerebrals Crystallized Educational Scale (JCCES) measures factual breadth — the accumulated stock of information about the world that crystallized intelligence theory treats as a core component of acquired cognitive ability. A subtest of this kind has to satisfy two structural requirements: items should span a meaningful range of difficulty, and they should order along a single underlying continuum of factual breadth rather than tapping multiple unrelated dimensions. This study examined the item structure of the JCCES GK subtest using multidimensional scaling (MDS) on response data from 588 respondents, and recovered the empirical signature of a well-ordered unidimensional construct: a horseshoe-shaped scaling pattern that is the canonical evidence of a Guttman simplex — items ordered cleanly along a single difficulty continuum. Read more →

How well does coefficient alpha perform with non-normal data?

Cronbach's coefficient alpha is the most-reported reliability statistic in psychology and educational measurement. It is also one of the most-misunderstood. The classical formula assumes that test items measure a single construct with equal factor loadings (tau-equivalence), uncorrelated errors, and continuously distributed scores. Real psychological measurement rarely meets all three assumptions: most scales use Likert responses (discrete), have items with unequal contributions to the construct (congeneric), and produce score distributions that depart from normality. The natural question is how badly alpha breaks under these violations and which alternatives perform better. A 2023 simulation study by Xiao and Hau in Educational and Psychological Measurement provides a systematic answer, with implications for the routine reliability reporting that fills psychometric methods sections. Read more →

Why is mathematical problem-solving treated as multidimensional?

The cognitive psychology literature recognizes that mathematical problem-solving is not unidimensional. Geary (1994) framed mathematical development as the interplay of biologically primary number sense, secondary culturally-acquired computation, and higher-order strategy use; the three contribute distinguishably to performance on real problems. Swanson and Beebe-Frankenberger (2004) tied mathematical problem-solving to working-memory subsystems, finding that visuospatial sketchpad and central executive demands are differentially loaded across problems. Hecht and Vagi (2010), looking specifically at fraction skills, identified separable contributions from conceptual understanding, procedural knowledge, and strategic flexibility.

How were the JCCES math items analyzed?

Five hundred and eighty-eight respondents completed the JCCES MP subtest under standardized administration. The 38-item correlation matrix was submitted to ALSCAL with dimensional solutions ranging from two through five dimensions. Convergence criteria were set at improvement < 0.001 in successive stress values; Kruskal's Stress Formula 1 (Kruskal, 1964) was the goodness-of-fit measure, with RSQ (proportion of variance in disparities accounted for by the solution) as the complementary index.

Related reading