Revisiting the Effect of Varying the Number of Response Alternatives in Clinical Assessment: Evidence From Measuring ADHD Symptoms

Revisiting the Effect of Varying the Number of Response Alternatives in Clinical Assessment: Evidence From Measuring ADHD Symptoms

Shi D, Siceloff ER, Castellanos RE, Bridges RM, Jiang Z, Flory K, Benson K.

Assessment. 2020 Sep 11:1073191120952885.

doi: 10.1177/1073191120952885.

Commentary* by Dr. Margaret Weiss: We measure ADHD symptoms on a 4 point scale, but interpret the results in a binary (present or absent) way. This suggests that our categorical approach to interpretation of dimensional data may lead to a considerable loss of important information.

ABSTRACT

This study illustrated the effect of varying the number of response alternatives in clinical assessment using a within-participant, repeated-measures approach. Participants reported the presence of current attention-deficit/hyperactivity disorder symptoms using both a binary and a polytomous (4-point) rating scale across two counterbalanced administrations of the Current Symptoms Scale (CSS).

Psychometric properties of the CSS were examined using (a) self-reported binary, (b) self-reported 4-point ratings obtained from each administration of the CSS, and (c) artificially dichotomized responses derived from observed 4-point ratings. Under the same ordinal factor analysis model, results indicated that the number of response alternatives affected item parameter estimates, standard errors, goodness of fit indices, individuals’ test scores, and reliability of the test scores.

With fewer response alternatives, the precision of the measurement decreased, and the power of using the goodness-of-fit indices to detect model misfit decreased.

These findings add to recent research advocating for the inclusion of a large number of response alternatives in the development of clinical assessments and further suggest that researchers should be cautious about reducing the number of response categories in data analysis.

* Abstracts are selected for their clinical relevance by Dr. Margaret Weiss, Director of Clinical Research, Child Psychiatry, Cambridge Health Alliance, Harvard University. Her commentary reflects her own opinion.  It is not approved or necessarily representative of the CADDRA board.