This glossary collects the key concepts, methods and people in this lesson. It can be used as a reference while working through the material or as a review before assessments. Typing in the search box filters the entries.

Lesson 7 · Section 1

From Constructs to Scores

A score is useful only when the path from the idea being measured to the number in the dataset is clear.

Running case

A reviewer asks about the loneliness scale

The authors add the six De Jong Gierveld items into a single score. Do the items measure one construct or two? The emotional subscale has three items. Is it reliable enough to analyze on its own?Reviewer 2, on Amira's manuscript

Each section of the lesson adds one part of the reply, and Section 4 drafts the revised Measures paragraph.

Vocabulary

From construct to instrument

Term	Meaning	In the running case
Construct (latent variable)	The idea being measured	Loneliness
Domain	A distinct part of the construct	Emotional and social loneliness
Item (indicator)	One question and its answer	"I often feel rejected"
Scale and subscale	Items combined into a score	Total score 0 to 6; subscales 0 to 3
Instrument	The whole tool	The CSCS questionnaire

Scales and indices

Which way do the arrows run?

Scale (reflective)

The construct causes the answers, so the items are expected to correlate.

An example is the De Jong Gierveld Loneliness Scale.

Index (formative)

The components define the construct, so they need not correlate.

Examples are an area deprivation index and the Steptoe social isolation index.

Internal consistency applies to scales and has no meaning for indices.

Recoding

Text answers become numbers, and three items are reversed

Scoring rules

The published rule turns answers into a score

• Each item scores 1 for "More or less" or the lonely answer.
• The emotional and social subscales each run from 0 to 3.
• The total score runs from 0 (not lonely) to 6 (most lonely).
• Amira's scores match the supplied score for all 3,415 people.

Classical test theory

Observed score = true score + error

The classical test theory model

\[ \color{#0B7B6B}{X} = \color{#6D28D9}{T} + \color{#C2410C}{E} \qquad \text{reliability} = \frac{\color{#6D28D9}{\sigma^2_T}}{\color{#0B7B6B}{\sigma^2_X}} = \frac{\sigma^2_T}{\sigma^2_T + \sigma^2_E} \]

X observed score T true score E random error

Reliability runs from 0 (all error) to 1 (no error).

Attenuation

Measurement error weakens correlations

Spearman's attenuation formula

\[ r_{\text{observed}} = r_{\text{true}} \times \sqrt{\text{rel}_X \times \text{rel}_Y} \]

Carry forward

What to take into the next section

• A score is linked to its construct through domains, items, an instrument and a scoring rule.
• The items of a scale are expected to correlate, whereas the components of an index need not.
• Reliability is the true-score share of observed variance, and low reliability weakens correlations.

Introduction and Overview

Many of the variables in health research describe things that cannot be observed directly. Loneliness, social support, depressive symptoms and health literacy are measured by asking people several questions and combining their answers into a score. The quality of every later analysis depends on that score. Lesson 2, Section 3 computed Cronbach's alpha, corrected item-total correlations and alpha-if-item-deleted, ran a parallel analysis and one-factor and two-factor exploratory factor analyses, and built prorated scores for depression and anxiety items that had no reverse-worded items. This lesson revisits those steps on the six De Jong Gierveld items, three of which are reverse-worded. This lesson takes the analyst's view of the same task: how scales are built, how their reliability and validity are judged, and how factor analysis and item response theory test whether the items behave as the scale's authors intended. This first section sets out the vocabulary of measurement, separates scales from indices, prepares and scores the loneliness items used throughout the lesson, and introduces classical test theory.

Box 7.1 introduces the running case of the lesson, a manuscript on loneliness whose reviewer has raised two questions about how loneliness was measured.

Answering either question requires precise terms for the parts of a measure, from the construct that a study sets out to measure to the score that enters the analysis. The first part of this section defines those terms.

The Vocabulary of Measurement

Measurement terms are often used loosely, and a reviewer will notice when a manuscript calls an index a scale or treats a single item as a construct. The terms below are used with these meanings throughout the course (Streiner, Norman & Cairney, 2015; DeVellis, 2017).

Figure 7.1 shows how the terms connect for the De Jong Gierveld Loneliness Scale, from the construct at the top, through its two domains and six items, to the instrument and scoring rule at the bottom. The six flip cards that follow the figure define each term and give an example of its use.

Two of these terms, scale and index, describe measures that both combine several items into one number. They differ in how the items relate to the construct, and the next part explains why that difference decides which statistics can be used to judge a measure.

Scales and Indices: Reflective and Formative Measurement

A scale and an index both combine several items into one number, yet they rest on different models of how the items relate to the construct (Bollen & Lennox, 1991; Diamantopoulos & Winklhofer, 2001). In reflective measurement, the construct causes the answers. A person who is lonely is more likely to report a sense of emptiness and more likely to report feeling rejected, so the answers to the items rise and fall together, and any one item could be replaced by a similar item without changing what is measured. In formative measurement, the components define the construct. A person is socially isolated because they are unmarried, rarely see family or friends and belong to no groups; isolation does not cause those circumstances. The components of an index can be unrelated to each other, and removing one changes the meaning of the index.

Figure 7.2 draws the two models side by side. The three tabs that follow it describe one reflective scale and two formative indices, and Box 7.2 explains why the distinction matters when a measure is analyzed.

A reflective scale and a formative index are therefore judged by different evidence. Either kind of measure must first be scored, and the next part turns to the levels of measurement and the scoring rules on which scoring depends.

Levels of Measurement, Scoring Rules and Cut-points

Before the answers to several items are combined, the analyst needs to know what kind of values the answers are and how the authors of the scale intended them to be scored. Box 7.3 reviews the four levels of measurement and explains why each De Jong Gierveld item is treated as ordinal and a sum of items as approximately interval. The published scoring rule of the scale and the use of cut-points are described after the box.

A published scale comes with a scoring rule, and the rule should be followed exactly so that scores can be compared with other studies. The De Jong Gierveld rule is dichotomous: each item scores 1 if the answer is "More or less" or the lonely answer, and 0 otherwise (De Jong Gierveld & Van Tilburg, 2006). The three emotional items are added to give the emotional subscale (0 to 3), the three social items give the social subscale (0 to 3), and the two subscales are added to give the total (0 to 6). Some scales also publish a cut-point, a score at or above which a person is classified as having the condition. The CSCS file supplies a yes or no version of the total in which scores of 2 to 6 are classified as lonely. A cut-point converts a score into a category and discards information, so it should be taken from the scale's documentation, stated in the manuscript, and used only when the research question needs a category.

The published rule gives a point for an answer of "More or less" or for the lonely answer on each item, so the lonely answer must be identified for every item. For three of the six De Jong Gierveld items the lonely answer is "No", and these items must be recoded before they are combined, as the next part explains.

Reverse-Worded Items

Many scales mix negatively and positively worded items so that a respondent who agrees with every statement, a habit called acquiescence, does not receive an extreme score. The De Jong Gierveld scale does this. For the three emotional items ("I experience a general sense of emptiness", "I miss having people around me" and "I often feel rejected"), "Yes" is the lonely answer. For the three social items ("There are plenty of people I can rely on when I have problems", "There are many people I can trust completely" and "There are enough people I feel close to"), "No" is the lonely answer. Before the items can be combined, the positively worded items must be reverse-coded so that a higher number means a lonelier answer on every item.

Equation 7.1 gives the general rule for reversing the codes of an item, whatever the range of those codes.

Leaving a reverse-worded item unreversed is one of the most common errors in survey analysis. The item then counts in the wrong direction, the total score mixes loneliness with its opposite, and every statistic computed from it is distorted. Section 2 shows in R how much the reliability of the scale falls when this step is skipped. A cross-tabulation of the original and reversed codes, as in Activity 7.1, is a quick check that the reversal worked.

The two R activities that close this part prepare the data used throughout the lesson. Activity 7.1 loads the 2021 wave of the CSCS, converts the text answers of the six items to numbers, applies Equation 7.1 to the three social items and checks each step with a cross-tabulation. Activity 7.2 applies the published scoring rule to form the two subscale scores and the total score, checks the total against the score supplied with the CSCS file, and draws the bar chart of total scores shown in Figure 7.3.

github_url <- "https://raw.githubusercontent.com/jorgeandr3s/heal/main/cscs/public_data/CSCS2025_full_cleaned_deidentified_data_and_metadata.RData"
load(url(github_url))     # loads a data frame called data
dim(data)                 # rows (survey responses) and columns

# Keep the 2021 wave, which gives one row per person
data <- data[data$SURVEY_collection_year == 2021, ]
dim(data)

# A small data frame with short names for the six items
dj <- data.frame(
  emptiness = data$LONELY_dejong_emotional_social_loneliness_scale_emptiness,
  miss      = data$LONELY_dejong_emotional_social_loneliness_scale_miss,
  rejected  = data$LONELY_dejong_emotional_social_loneliness_scale_rejected,
  rely      = data$LONELY_dejong_emotional_social_loneliness_scale_rely,
  trust     = data$LONELY_dejong_emotional_social_loneliness_scale_trust,
  close     = data$LONELY_dejong_emotional_social_loneliness_scale_close)

# The answers are stored as text categories
table(dj$emptiness, useNA = "ifany")
table(dj$rely, useNA = "ifany")

# Text to numbers: No = 1, More or less = 2, Yes = 3
# Anything else ("Presented but no response" or NA) stays NA
dj$emptiness_n <- NA
dj$emptiness_n[dj$emptiness == "No"] <- 1
dj$emptiness_n[dj$emptiness == "More or less"] <- 2
dj$emptiness_n[dj$emptiness == "Yes"] <- 3

dj$miss_n <- NA
dj$miss_n[dj$miss == "No"] <- 1
dj$miss_n[dj$miss == "More or less"] <- 2
dj$miss_n[dj$miss == "Yes"] <- 3

dj$rejected_n <- NA
dj$rejected_n[dj$rejected == "No"] <- 1
dj$rejected_n[dj$rejected == "More or less"] <- 2
dj$rejected_n[dj$rejected == "Yes"] <- 3

dj$rely_n <- NA
dj$rely_n[dj$rely == "No"] <- 1
dj$rely_n[dj$rely == "More or less"] <- 2
dj$rely_n[dj$rely == "Yes"] <- 3

dj$trust_n <- NA
dj$trust_n[dj$trust == "No"] <- 1
dj$trust_n[dj$trust == "More or less"] <- 2
dj$trust_n[dj$trust == "Yes"] <- 3

dj$close_n <- NA
dj$close_n[dj$close == "No"] <- 1
dj$close_n[dj$close == "More or less"] <- 2
dj$close_n[dj$close == "Yes"] <- 3

# Check: each answer should map to one number
table(dj$emptiness, dj$emptiness_n, useNA = "ifany")

# Reverse-code the three positively worded items: for these
# items "No" (1) is the lonely answer. Subtracting from 4 flips
# the scale: 1 becomes 3, 2 stays 2, and 3 becomes 1.
dj$rely_r  <- 4 - dj$rely_n
dj$trust_r <- 4 - dj$trust_n
dj$close_r <- 4 - dj$close_n
table(dj$rely_n, dj$rely_r)      # check: 1 -> 3, 2 -> 2, 3 -> 1

# One data frame of six items, higher = lonelier
items <- data.frame(emptiness = dj$emptiness_n,
                    miss      = dj$miss_n,
                    rejected  = dj$rejected_n,
                    rely      = dj$rely_r,
                    trust     = dj$trust_r,
                    close     = dj$close_r)
items <- na.omit(items)   # people who answered all six items
nrow(items)

R Activity 7.2: score the scale with its published rule

Files for this activity Data: the CSCS 2021 wave, loaded from GitHub in the codeno download needed Answer key (R script)revealed after you save your responses

This activity continues from the one above and uses the dj data frame created there. It applies the published dichotomous scoring rule, forms the two subscale scores and the total score, and checks the total against the score supplied with the CSCS file.

# The published scoring rule: an item scores 1 for "More or
# less" or the lonely answer (a code of 2 or 3), and 0 otherwise.
dj$emptiness_01 <- as.numeric(dj$emptiness_n >= 2)
dj$miss_01      <- as.numeric(dj$miss_n >= 2)
dj$rejected_01  <- as.numeric(dj$rejected_n >= 2)
dj$rely_01      <- as.numeric(dj$rely_r >= 2)
dj$trust_01     <- as.numeric(dj$trust_r >= 2)
dj$close_01     <- as.numeric(dj$close_r >= 2)

# Add the items: two subscales (0 to 3) and a total (0 to 6)
dj$emotional_score <- dj$emptiness_01 + dj$miss_01 + dj$rejected_01
dj$social_score    <- dj$rely_01 + dj$trust_01 + dj$close_01
dj$total_score     <- dj$emotional_score + dj$social_score
table(dj$total_score, useNA = "ifany")

Console output

0 1 2 3 4 5 6 <NA> 61 274 346 510 665 873 686 630

A comparison such as dj$emptiness_n >= 2 gives TRUE or FALSE, and as.numeric() turns TRUE into 1 and FALSE into 0. The table shows the total score for the 3,415 people with complete answers; the 630 people with any missing item have no total. Scores pile up toward the top of the range: 873 people score 5 and 686 score 6.

# Check against the score supplied with the CSCS data
table(dj$total_score == data$LONELY_dejong_emotional_social_loneliness_scale_score,
      useNA = "ifany")

# Plot the total score
barplot(table(dj$total_score),
        main = "De Jong Gierveld Loneliness Scale scores",
        xlab = "Total score (0 = not lonely, 6 = most lonely)",
        ylab = "Number of participants", col = "grey80")

Console output

TRUE <NA> 3415 630

All 3,415 totals equal the score supplied with the data, and the 630 NA values are the same people in both versions. Agreement with a supplied score is a useful check whenever a data team has already scored a scale.

R Reflect on what you just ran

Use the questions below to interpret the output you produced. Look at your console output and plots before answering.

1. Under the published rule, which answers score 1 on the item "There are enough people I feel close to", and which code on dj$close_r do they correspond to?

Model answerThe item is positively worded, so the lonely answer is "No". Under the published rule, "No" and "More or less" both score 1, and "Yes" scores 0. After reverse-coding, "No" has close_r = 3 and "More or less" has close_r = 2, which is why the code uses dj$close_r >= 2.

2. The CSCS file classifies total scores of 2 to 6 as lonely. Using the table of total scores, how many of the 3,415 people would be classified as lonely, and what information does the classification discard?

Model answerThe people with scores of 0 or 1 number 61 + 274 = 335, so 3,415 − 335 = 3,080 people (about 90%) would be classified as lonely. The classification treats a person who scores 2 the same as a person who scores 6, so it discards the differences in degree of loneliness that the full score records. It should be used only when the research question needs a category and the cut-point comes from the scale's documentation.

3. Why is it useful to compare your total score with LONELY_dejong_emotional_social_loneliness_scale_score before analyzing it?

Model answerThe comparison checks every step of the preparation at once: the text-to-number recoding, the reversal of the three social items and the scoring rule. A single mistake, such as forgetting to reverse an item or using the wrong threshold, would make some totals differ from the supplied score. Here all 3,415 match, so the scored variable can be used with confidence and described in the Measures paragraph as scored according to the published rule.

Save responses

Saved.

The six items are now coded in the same direction and scored by the published rule, and the totals match the score supplied with the data. Figure 7.3 shows that the total scores are skewed toward the lonely end of the scale. Even a correctly prepared score contains some error, and the next part introduces the model that classical test theory uses to describe it.

Classical Test Theory

Classical test theory, set out formally by Lord and Novick (1968), is the simplest model of what an observed score contains. It states that each person's observed score X is the sum of a true score T and a random error E. The true score is the average score the person would obtain over many hypothetical repetitions of the measurement under the same conditions. The error is everything else: a question misread, a passing mood, a guess between two answers. The model assumes that errors average zero, that they are unrelated to the true score, and that the errors of different measurements are unrelated to each other.

Equation 7.2 states the model and the definition of reliability that follows from it.

Reliability is therefore a property of scores in a population, and it depends on how much people in that population differ. The same scale can have higher reliability in a population with a wide range of loneliness than in a population in which almost everyone is equally lonely, because the true-score variance is larger in the first. For this reason a manuscript reports the reliability observed in its own sample as well as the value from the scale's original publication.

Measurement error weakens associations

Random measurement error has a predictable effect on correlations. Spearman (1904) showed that when two variables are measured with error, the correlation between the observed scores is smaller than the correlation between the true scores, a result called attenuation.

Equation 7.3 gives Spearman's formula for the size of this effect and applies it to one set of values.

The corresponding result for a regression slope is simpler. When the exposure is measured with random error, the observed slope equals the true slope multiplied by the reliability of the exposure, a factor called the regression dilution ratio. With an exposure reliability of 0.60, a true slope of 2.0 is observed as about 1.2. Random error in the outcome leaves the slope unbiased and widens its confidence interval. For optional reading, HSCI 230 Lesson 7, Section 1 and HSCI 230 Lesson 9, Section 3 discuss attenuation and regression dilution.

The simulation in Interactive 7.1 generates 250 people whose true loneliness and true depressive symptoms have a chosen correlation, then adds random error to each measure. Lowering the reliability of either measure spreads the cloud of points and pulls the observed correlation toward zero. The panel also reports the correlation expected from Equation 7.3, so that the simulated and expected values can be compared.

The same weakening applies to the associations in Amira's manuscript. Box 7.4 returns to the running case and shows why the reliability of the emotional subscale matters for her results.

This section has defined the parts of a measure, separated reflective scales from formative indices, prepared and scored the six De Jong Gierveld items, and introduced the classical test theory model, in which reliability is the proportion of observed-score variance that is true-score variance. Because true scores are never observed, reliability has to be estimated from data, which is the task of Section 2. The knowledge check and reflection that follow review the material of this section.

Introduction and Overview

Section 1 defined reliability as the proportion of the variance in observed scores that is true-score variance. This definition cannot be applied directly, because true scores are never observed. Reliability is therefore estimated from the consistency of a measurement: across two occasions, across two raters, or across the items of a scale. This section describes these kinds of reliability, explains why a single survey supports only one of them, and then teaches the most widely reported reliability statistic, Cronbach's alpha, together with its assumptions, its limits and its main alternative, McDonald's omega. It answers the second question in the reviewer's comment: whether the emotional subscale of the De Jong Gierveld scale is reliable enough to analyze on its own.

The section begins with the three kinds of reliability and the study design that each one requires, because the design of the CSCS limits which of them Amira can estimate.

Three Kinds of Reliability

Every reliability statistic measures consistency, and the kinds of reliability differ in what the consistency is across. The choice is driven by the main source of error in the measurement: the occasion on which it is made, the person who makes it, or the particular items used.

Table 7.1 sets out the three kinds of reliability, the question each one answers, the design it needs and the statistic usually reported.

Table 7.1. Three kinds of reliability, with the design and the usual statistic for each.

Test-retest reliability for a continuous score is usually summarized with an intraclass correlation coefficient, which measures agreement as well as correlation. A Pearson correlation between two occasions can be high even if everyone scored five points higher the second time, whereas the intraclass correlation penalizes such a shift. Worked Example 7.1 shows the difference. It applies Equation 7.4, the intraclass correlation for absolute agreement, to hypothetical scores for six people measured on two occasions, and then repeats the calculation after a uniform shift in the second set of scores.

Inter-rater reliability for categories, the second row of Table 7.1, is usually summarized with Cohen's kappa. Box 7.5 gives its formula as Equation 7.5 and works through two examples, the second of which shows how a rare category lowers kappa.

Box 7.6 applies Table 7.1 to the CSCS and identifies the kinds of reliability that Amira can estimate from her data.

Internal consistency is therefore the only kind of reliability that Amira can estimate from the CSCS. The next part introduces the statistic most often used to estimate it, Cronbach's alpha.

Cronbach's Alpha

Cronbach's alpha is the most widely reported estimate of internal consistency, and the course has used it once already. Box 7.7 recalls that analysis from Lesson 2 and poses a retrieval question, which the standardized form of the formula in this part answers.

Cronbach (1951) introduced coefficient alpha as a lower bound to the reliability of a sum of items, and it remains the statistic that reviewers most often expect to see. Alpha compares the variances of the individual items with the variance of the total score. When the items move together, people who score high on one item tend to score high on the others, so the total varies much more than the items do separately and alpha approaches 1. When the items are unrelated, the variance of the total is close to the sum of the item variances and alpha approaches 0.

Equation 7.6 gives alpha in its raw form, computed from the variances of the items and of the total score, and in its standardized form, computed from the average correlation between pairs of items.

The standardized formula shows that alpha depends on two things: the number of items and the strength of their correlations. With an average correlation of 0.25, three items give an alpha of 0.50, but twelve items with the same average correlation would give 3 / 3.75 = 0.80. Short subscales therefore tend to have lower alphas than long scales, even when their items are of equal quality, which is part of the reason the three-item emotional subscale has a low value.

Several conventions are used to judge alpha. Nunnally (1978) suggested that about 0.70 suffices in the early stages of research, that about 0.80 is adequate for basic research, and that 0.90 or higher is needed when scores are used to make decisions about individuals, such as a clinical cut-point; the widely used minimum of 0.70 comes from the first of these. Very high values, above about 0.90 to 0.95, can indicate that some items are close paraphrases of each other, which adds length without adding information (Streiner, 2003). These conventions are guides for judgement, and a manuscript should report the value and its confidence interval and let readers judge it in context.

Activity 7.3 applies these ideas to the De Jong Gierveld items in R, and the rest of this section draws on its results.

# install.packages("psych")   # run once, if not yet installed
library(psych)    # alpha() and omega()
emotional_items <- items[, c("emptiness", "miss", "rejected")]
social_items    <- items[, c("rely", "trust", "close")]
alpha(social_items)

round(alpha(emotional_items)$total, 2)               # overall alpha only
round(alpha(emotional_items)$alpha.drop[, 1:2], 2)   # alpha if each item is dropped
round(alpha(emotional_items)$item.stats[, "r.drop", drop = FALSE], 2)   # item-total

# Alpha for all six items, correctly reverse-coded
round(alpha(items)$total[, 1:2], 2)

# What happens if we forget to reverse-code?
not_reversed <- data.frame(emptiness = dj$emptiness_n,
                           miss      = dj$miss_n,
                           rejected  = dj$rejected_n,
                           rely      = dj$rely_n,
                           trust     = dj$trust_n,
                           close     = dj$close_n)
round(alpha(na.omit(not_reversed))$total[, 1:2], 2)

# McDonald's omega from a one-factor model of each subscale
f_social <- fa(social_items, nfactors = 1)   # one-factor model
lambda <- f_social$loadings[, 1]             # the three loadings
round(lambda, 2)
sum(lambda)^2 / (sum(lambda)^2 + sum(f_social$uniquenesses))   # omega

f_emo <- fa(emotional_items, nfactors = 1)
lambda <- f_emo$loadings[, 1]
round(lambda, 2)
sum(lambda)^2 / (sum(lambda)^2 + sum(f_emo$uniquenesses))

# An index: the Steptoe social isolation index (CSCS supplies 0/1 codes,
# where 1 = isolated on that component)
iso <- data.frame(unmarried = data$LONELY_steptoe_isolation_index_unmarried_num,
                  clubs     = data$LONELY_steptoe_isolation_index_clubs_num,
                  friends   = data$LONELY_steptoe_isolation_index_friends_num,
                  children  = data$LONELY_steptoe_isolation_index_kids_num,
                  family    = data$LONELY_steptoe_isolation_index_other_fam_num)
iso <- na.omit(iso)
round(cor(iso), 2)
round(alpha(iso)$total[, 1:2], 2)   # shown only to explain why it is not reported

Activity 7.3 gives an alpha of 0.75 for the social subscale and 0.51 for the emotional subscale, shows the six-item alpha falling from 0.59 to 0.42 when the reversal is skipped, and gives an alpha of 0.34 for an index whose components are not expected to correlate. Reading such values correctly requires knowing what alpha assumes, which the next part examines.

What Alpha Assumes and What It Cannot Show

Alpha is easy to compute and is often over-interpreted (Sijtsma, 2009; Dunn, Baguley & Brunsden, 2014). The accordion below sets out the five misconceptions that this lesson addresses, each with the evidence from the CSCS loneliness items.

The first item in the accordion also contains Box 7.8, which revisits a claim about alpha made in Lesson 2 and asks for it to be rewritten.

Alpha also assumes that every item measures the construct equally well, a condition called tau-equivalence. Formally, each item is assumed to have the same loading on the common factor. When some items are stronger indicators than others, alpha underestimates the reliability of the total score. The assumption is plausible for the social items, whose loadings are similar, and clearly false for the emotional items, where one item is much weaker than the other two.

When the items differ in quality, a reliability coefficient that allows each item its own loading gives a better estimate of the reliability of the total score. The next part introduces such a coefficient, McDonald's omega.

McDonald's Omega

McDonald's omega (McDonald, 1999) estimates the reliability of a sum score from a factor model, which allows each item to have its own loading. For a scale with a single factor, omega is the share of the variance of the total score that is explained by the factor.

Equation 7.7 gives omega for a one-factor model and applies it to the loadings of the three social items.

When the loadings are equal, omega equals alpha, and when they differ, omega is the better estimate. The section's R activity computes omega for both subscales from a one-factor model fitted with fa(), which Section 4 explains in detail. The social loadings are similar (0.73, 0.73 and 0.67), and alpha and omega are both 0.75. The emotional loadings are very unequal (0.83, 0.20 and 0.58), so omega (0.57) is higher than alpha (0.51). Both values are low. Omega is now recommended alongside or in place of alpha in many fields, and reporting both lets readers compare a study with earlier work that reported only alpha.

Box 7.9 returns to the running case and applies these results to the weak emotional item identified in Activity 7.3.

The card below links to a narrated R walkthrough of the same reliability analysis.

This section has shown that a single survey supports only internal consistency, that the social subscale of the De Jong Gierveld scale has adequate internal consistency (alpha and omega of 0.75), and that the emotional subscale has low internal consistency (alpha of 0.51 and omega of 0.57). It has also shown that alpha must be read with its assumptions in mind. A reliable score can still measure the wrong construct, so Section 3 turns to validity. The knowledge check and reflection that follow review the material of this section.

Lesson 7 · Section 3

Validity

Do the scores mean what we claim they mean, for this use and in this population?

Reliability and validity

Consistent is different from correct

Tight and centred

The measure is reliable and valid.

Tight and off target

The measure is reliable and measures the wrong thing.

Scattered

The measure is unreliable, which also limits its validity.

Reliability is necessary for validity, and it is never enough to establish it.

Validity as an argument

Valid for what, and for whom?

Validity is an integrated evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores or other modes of assessment.Messick (1989, p. 13)

The argument names the intended use and population, then gathers evidence for each claim.

Kinds of evidence

Four families of validity evidence

Content and face

The items cover the construct, as judged by experts and by respondents.

Criterion

Scores agree with a gold standard now (concurrent) or predict a later outcome (predictive).

Construct

Scores relate to other variables as theory predicts (convergent, discriminant and known groups).

Structural

The items group into the domains the theory describes (Section 4).

Convergent and discriminant evidence

Correlations in 3,053 CSCS participants

Measure	Total	Social subscale	Emotional subscale
UCLA Loneliness Scale (3 items)	0.45	0.32	0.39
Perceived social support (MSPSS)	−0.49	−0.48	−0.26
Anxiety symptoms (GAD-2)	0.33	0.21	0.32

Social loneliness tracks social support more closely, and emotional loneliness tracks anxiety and the UCLA scale more closely.

Known groups

Scores fall as the number of close friends rises

COSMIN taxonomy

Three domains of measurement properties

Reliability

This domain covers internal consistency, test-retest and inter-rater reliability, and measurement error.

Validity

This domain covers content (and face), criterion and construct validity, including structural validity, hypothesis testing and cross-cultural validity.

Responsiveness

This domain covers the ability of scores to detect change over time.

Interpretability is listed as an important characteristic beside the three domains.

Population matters

Does the scale work the same way in every group?

• Measurement invariance asks whether the items relate to the construct in the same way across groups.
• Without it, a difference in mean scores may arise from how the items are read in each group.
• It is tested with the confirmatory factor analysis introduced in Section 4.

Carry forward

What to take into the next section

• Validity is an argument for a use of scores in a population, built from several kinds of evidence.
• The COSMIN taxonomy organizes the evidence into reliability, validity and responsiveness.
• CSCS correlations and known groups support the scale and suggest two distinct subscales.

Introduction and Overview

A reliable score is consistent, and consistency does not show that the score measures the intended construct. Validity concerns the meaning of scores: whether a loneliness score reflects loneliness or instead reflects low mood, social desirability or something else that the items happen to share. HSCI 230 Lesson 7 introduced construct validity and measurement error from the perspective of a reader appraising a study. HSCI 230 used construct validity for the broad question of whether an instrument captures its construct, which this lesson calls validity. This lesson reserves construct validity for evidence that scores relate to other measures as theory predicts (convergent, discriminant and known-groups evidence), and the COSMIN taxonomy later in the section groups structural and cross-cultural validity under construct validity. This section takes the analyst's perspective. It presents the kinds of validity evidence and the COSMIN taxonomy, frames validity as an argument for a particular use of scores in a particular population, and gathers construct validity evidence for the De Jong Gierveld scale from the CSCS data.

The section begins by setting reliability and validity side by side, and the parts that follow describe how evidence of validity is framed, classified and gathered.

Reliability and Validity

Reliability concerns random error, the scatter of scores around a person's true score. Validity concerns systematic error, whether the true score itself corresponds to the intended construct. The two are linked in one direction only. A score that is mostly random error cannot correlate strongly with anything, so low reliability limits validity. A highly reliable score can still measure the wrong construct, as a bathroom scale that always reads three kilograms heavy is consistent and wrong.

Figure 7.4 illustrates the relation with three dartboards, in which the spread of the darts stands for reliability and their position relative to the bullseye stands for validity.

A consistent score can therefore still miss its target, and validity has to be supported by evidence of its own. The next part describes how current measurement theory frames that evidence.

Validity as an Argument

Early textbooks listed "types of validity" as if a scale either had or lacked each one. Current measurement theory treats validity as a single, overall judgement about how well the evidence supports a particular interpretation and use of scores (Messick, 1989, 1995; AERA, APA & NCME, 2014). Kane (2013) describes validation as the construction of an argument in two steps. The first step states the interpretation and use: for example, "scores on the De Jong Gierveld scale rank adults in an online Canadian survey by their degree of loneliness, and differences in mean scores between groups reflect differences in loneliness". The second step gathers evidence for each claim that this interpretation depends on, including that the items cover loneliness, that the answers form the intended domains, that the scores relate to other variables as expected, and that the items work the same way in the groups being compared.

Two consequences follow. Support for a scale is always relative to a stated use in a stated population, so a scale cannot be called "valid" in general. Evidence that the De Jong Gierveld scale works well in the populations in which it was developed supports, without settling, its use with Canadian adults of all ages in an online survey. Validation is also cumulative: each study that uses a scale and reports how its scores behave adds to the argument.

Validity is therefore judged for a stated use in a stated population, and the next part describes the kinds of evidence from which the argument is assembled.

Kinds of Validity Evidence

Validity evidence is usually grouped into several kinds, each of which supports a different claim in the argument. The six flip cards below define content, face, criterion, convergent and discriminant, known-groups and structural evidence, and the paragraph after them explains which kinds carry most weight for a construct such as loneliness.

Criterion validity is the most direct evidence when a trusted gold standard exists, as it does for many diagnostic tests. For psychological constructs such as loneliness there is no gold standard, so the argument relies mainly on content, construct and structural evidence (Cronbach & Meehl, 1955). Construct validity is tested by writing down, before looking at the data, the correlations and group differences that theory predicts, and then checking them. COSMIN calls this hypothesis testing. Stating the hypotheses in advance guards against reading support into whatever pattern appears.

The next part presents the COSMIN taxonomy, which places these kinds of validity evidence alongside reliability and responsiveness.

The COSMIN Taxonomy

The COSMIN initiative (Consensus-based Standards for the selection of health Measurement Instruments) developed, through an international Delphi study, a taxonomy and definitions for the measurement properties of health instruments (Mokkink et al., 2010). It is widely used in systematic reviews of measurement instruments, and it is a useful checklist for the Measures paragraph of a manuscript.

Table 7.2 lists the three COSMIN domains with their measurement properties, the question that each property answers, and the evidence for the De Jong Gierveld scale that this lesson provides.

Table 7.2. The COSMIN domains and measurement properties, with the evidence for the De Jong Gierveld scale in this lesson.

COSMIN also describes interpretability (for example, what size of difference in scores is meaningful) as an important characteristic of an instrument, alongside the three domains.

Interpreting change: the minimal important difference

Responsiveness asks whether scores change when the construct changes, and interpretability asks what a change of a given size means. The minimal important difference (MID) is the smallest change in scores that people would regard as important, or that would justify a change in care (Jaeschke, Singer & Guyatt, 1989). It is estimated in two ways. An anchor-based estimate links score changes to an external judgement, the anchor: for example, participants report at follow-up whether their loneliness is "a little better", and the MID is the mean change in score among those who chose that answer. A distribution-based estimate uses only the spread and the reliability of the scores. Two common versions are half a standard deviation of the baseline scores (Norman, Sloan & Wyrwich, 2003) and one standard error of measurement (Wyrwich, Tierney & Wolinsky, 1999).

Equation 7.8 defines the standard error of measurement used in the second distribution-based estimate.

Distribution-based values describe how large a change is compared with the spread of scores and the error in them, and they say nothing directly about importance, so anchor-based estimates are preferred when they exist and distribution-based values serve as a check. As an illustration, consider a loneliness scale scored 0 to 6 whose baseline scores have a standard deviation of 1.6 and a reliability of 0.75 (hypothetical values). Half a standard deviation is 0.8 points, and the SEM is 1.6 × √0.25 = 0.8 points; the two coincide whenever the reliability is 0.75. If participants who reported feeling "a little better" improved by 1.0 point on average, the anchor-based MID would be 1.0. A program that lowers the mean score by 0.4 points then produces a change below every estimate of the MID, even if a large trial finds it statistically significant.

Table 7.2 also shows that, beyond internal consistency, the CSCS data can add two kinds of validity evidence for the De Jong Gierveld scale: hypothesis testing through correlations and known groups, which the next part computes, and structural validity through factor analysis, which Section 4 tests.

Construct Validity Evidence from the CSCS

Before computing anything, Amira writes down what the theory of the scale predicts. Loneliness is usually defined as the distress that follows from a gap between the relationships a person has and those they want (Perlman & Peplau, 1981), and Weiss (1973) distinguished its emotional and social forms. The total score should therefore correlate positively with another loneliness measure, the three-item UCLA Loneliness Scale (Hughes et al., 2004), and negatively with perceived social support, measured in the CSCS with the Multidimensional Scale of Perceived Social Support (Zimet et al., 1988). It should correlate less strongly with anxiety symptoms (measured with the two-item GAD-2), a related but distinct construct. The social subscale, which concerns the wider network, should relate more strongly to social support and to the number of close friends than the emotional subscale does.

Activity 7.4 tests these predictions in the CSCS data, with a correlation matrix for the convergent and discriminant hypotheses and a comparison of subscale means by number of close friends for the known-groups hypothesis.

# Other measures in the CSCS file (already scored by the survey team)
val <- data.frame(total     = dj$total_score,
                  social    = dj$social_score,
                  emotional = dj$emotional_score,
                  ucla      = data$LONELY_ucla_loneliness_scale_score,
                  support   = data$PSYCH_zimet_multidimensional_social_support_scale_score,
                  anxiety   = data$WELLNESS_gad_score)
val <- na.omit(val)     # people with every score
nrow(val)
round(cor(val), 2)      # correlations between every pair of measures

# Known groups: people with no close friends should score higher
friends <- data$CONNECTION_social_num_close_friends_grouped
friends[friends == "Presented but no response"] <- NA
friends <- factor(friends, levels = c("None", "1–2", "3–4", "5 or more"))
round(tapply(dj$social_score, friends, mean, na.rm = TRUE), 2)
round(tapply(dj$emotional_score, friends, mean, na.rm = TRUE), 2)

The results support most of these predictions. The total score correlates 0.45 with the UCLA scale and −0.49 with perceived social support, and only 0.33 with anxiety, so the convergent correlations exceed the discriminant one. The gap is modest, which is common for constructs that share an emotional component. The two subscales show the distinct patterns the theory predicts: the social subscale relates most strongly to social support (−0.48, compared with −0.26 for the emotional subscale), whereas the emotional subscale relates more strongly to the UCLA scale and to anxiety. The known-groups comparison shows a steady gradient by number of close friends, much steeper for social loneliness (2.47 to 1.13) than for emotional loneliness (2.45 to 1.74). The two subscales correlate only 0.21 with each other.

Figure 7.5 plots the known-groups comparison for the two subscales. Box 7.10 then sets out two reasons for caution in reading these validity correlations.

Figure 7.5 compares mean scores between groups, and any such comparison rests on an assumption that the next part examines.

Measurement Invariance

A comparison of mean scores between groups, such as older and younger adults, assumes that the items relate to the construct in the same way in each group. If younger adults interpret "I miss having people around me" differently from older adults, a difference in mean scores could arise from the item's wording. Measurement invariance testing, introduced in HSCI 230 Lesson 7, checks this assumption by fitting the confirmatory factor model of Section 4 in each group and testing whether the loadings and item intercepts are equal. A full treatment is beyond this lesson. In a manuscript that compares groups, it is good practice to cite any published invariance evidence and to note comparisons for which it is missing.

With the kinds of evidence, the COSMIN headings and the CSCS results in place, the last part of this section shows how they are assembled into a validity argument for a Measures paragraph.

Assembling the Validity Evidence for a CSCS Measure

A Measures paragraph for a CSCS measure needs a short validity argument. The task in Box 7.11 sets out how to assemble one. It draws on the measure's published sources and on the analysis of the CSCS data.

This section has treated validity as an argument for a particular use of scores in a particular population, organized the evidence for it with the COSMIN taxonomy, and gathered construct validity evidence for the De Jong Gierveld scale from the CSCS. Structural validity, the evidence that the reviewer's first question calls for, requires a latent variable model, and Section 4 introduces the models used to test it. The knowledge check and reflection that follow review the material of this section.

Lesson 7 · Section 4

Latent Variable Models: Factor Analysis and Item Response Theory

Do the six items measure one construct or two, and how well does each item measure it?

The factor model

Each item is caused partly by a factor

One item in a factor model

\[ \color{#0B7B6B}{\text{item}_i} = \color{#6D28D9}{\lambda_i}\,\color{#065C50}{F} + \color{#C2410C}{u_i} \]

λ_i loading F latent factor u_i unique part of the item

Emotional factor	Loading	Social factor	Loading
Emptiness	0.82	Rely on	0.75
Miss people	0.20	Trust	0.73
Rejected	0.57	Feel close	0.65

The two factors correlate 0.25 (standardized estimates from the confirmatory model).

Exploratory and confirmatory

Explore in one half, confirm in the other

Exploratory (EFA)

Every item may load on every factor, and the data suggest the number of factors.

This step uses half 1 (1,708 people).

Confirmatory (CFA)

Each item loads only on its assigned factor, and the model is tested against the data.

This step uses half 2 (1,707 people).

Exploratory factor analysis

Parallel analysis suggests two factors

• The overall KMO is 0.68, which is adequate.
• Parallel analysis keeps two factors.
• Oblimin rotation allows the factors to correlate (r = 0.29).
• The miss item loads below 0.3 on both factors.

Confirmatory factor analysis

The model is written in lavaan syntax

model_2f <- '
  emotional =~ emptiness + miss + rejected
  social    =~ rely + trust + close
'
cfa_2f <- cfa(model_2f, data = cfa_half)
summary(cfa_2f, fit.measures = TRUE, standardized = TRUE)

• The operator =~ is read as "is measured by".
• The first loading of each factor is fixed at 1 to set its scale.
• The Std.all column holds the standardized loadings.

Model fit

Two factors fit far better than one

Index	Guide for good fit	Two factors	One factor
CFI	0.95 or higher	0.949	0.739
TLI	0.95 or higher	0.905	0.565
RMSEA	0.06 or lower (0.08 acceptable)	0.082	0.177
SRMR	0.08 or lower	0.060	0.110

The chi-square difference test gives 386.95 on 1 degree of freedom (p < 0.001), and the AIC is 20,825 for two factors against 21,210 for one.

Running case

One construct or two?

• The two-factor model fits far better than the one-factor model in a separate half of the sample.
• The fit of the two-factor model is acceptable, with CFI = 0.949 and RMSEA = 0.082.
• The miss item loads weakly (0.20), which accounts for part of the misfit.
• The emotional items are all negatively worded and the social items all positively worded, so wording direction may contribute to the separation.

Item response theory

Each item has its own curve

Two-parameter logistic (2PL) item characteristic curve

\[ P(\text{lonely answer} \mid \color{#0B7B6B}{\theta}) = \frac{1}{1 + e^{-\color{#6D28D9}{a}(\color{#0B7B6B}{\theta} - \color{#C2410C}{b})}} \]

θ latent trait a discrimination (slope) b difficulty (location)

The Rasch model fixes a to be equal for all items, the 2PL lets it vary, and the graded response model handles ordered answers.

Curves from the CSCS

Where on the trait do the items measure well?

Carry forward

Three ways to look at the same items

Classical test theory

It gives one reliability for the whole score in one sample.

Factor analysis

It tests whether the items form the domains the theory describes.

Item response theory

It describes each item along the trait and shows where measurement is precise.

The reflection, the knowledge check and the final assessment follow the reading.

Introduction and Overview

Sections 2 and 3 produced two hints that the six De Jong Gierveld items do not form a single construct: the six-item alpha (0.59) is lower than the alpha of the social subscale alone (0.75), and the two subscales correlate only 0.21 and relate differently to other measures. A latent variable model can test this directly. This section teaches exploratory factor analysis (EFA), which lets the data suggest a structure, and confirmatory factor analysis (CFA), which tests a structure stated in advance. It then introduces item response theory (IRT) at a conceptual level, using curves estimated from the loneliness items, and compares it with classical test theory. The section ends with the Measures paragraph of Amira's revised manuscript. Lesson 8 extends the confirmatory model of this section into structural equation models.

The first part presents the common factor model on which both exploratory and confirmatory factor analysis rest.

The Common Factor Model

A factor model treats the answer to each item as the sum of two parts: a part caused by a latent factor that the items share, and a unique part that belongs to the item alone (Spearman, 1904; Thurstone, 1947). The loading λ describes how strongly the item depends on the factor. When the items and the factor are standardized, the loading is the correlation between the item and the factor, and its square is the share of the item's variance that the factor explains, called the communality. The rest is the unique variance, which includes measurement error and anything specific to the item's wording.

Equation 7.9 writes the model for one standardized item.

Figure 7.6 draws the two-factor model of the De Jong Gierveld scale, with the standardized loadings and the factor correlation from the confirmatory model fitted later in this section.

Factor analysis and the reliability statistics of Section 2 rest on the same reflective model, so they answer related questions. Alpha and omega summarize how well a set of items measures whatever they share, assuming there is one thing they share. Factor analysis asks how many things they share and which items measure each one.

A factor model can be used to find out how many factors a set of items share or to test a structure stated in advance. The next part describes both uses and the order in which Amira applies them.

Exploratory and Confirmatory Factor Analysis

Exploratory factor analysis lets every item load on every factor and estimates the loadings from the data. It is used when the structure is unknown, for example when a new scale is being developed, and it suggests how many factors there are and which items belong together. Confirmatory factor analysis tests a structure stated in advance: each item loads only on the factor the theory assigns it to, and all other loadings are fixed at zero. A confirmatory model can be rejected by the data, which makes it the stronger test of a published scale's structure.

Exploring and confirming in the same data would let the confirmatory model reward whatever the exploration happened to find. Amira therefore splits the 3,415 participants at random into two halves of 1,708 and 1,707, explores in the first and confirms in the second. The split uses set.seed(2021) so that it is the same every time the code is run, and it matches the split in the Factor Analysis and Scale Scoring walkthrough.

Exploratory factor analysis in four steps

An exploratory factor analysis proceeds in four steps, set out in the accordion below: checking that the items are suitable, choosing the number of factors, rotating the solution and reading the loading matrix. Step 2 includes the parallel analysis plot for the exploratory half as Figure 7.7, and Step 3 includes Box 7.12, which revisits the rotation used in Lesson 2.

Step 1. Check that the items are suitable for factor analysis

Factor analysis needs items that correlate. The Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy compares the correlations between items with their partial correlations. Kaiser (1974) labelled values of 0.90 or above "marvelous", values in the 0.80s "meritorious", the 0.70s "middling", the 0.60s "mediocre", the 0.50s "miserable" and values below 0.50 "unacceptable", and about 0.6 is often treated as a practical minimum. Bartlett's test of sphericity tests whether the correlation matrix differs from one in which all correlations are zero; with samples of this size it is almost always significant, so it is a minimal check. For the exploratory half, the overall KMO is 0.68 and Bartlett's test gives a chi-square of 1,912 on 15 degrees of freedom (p < 0.001).

Step 2. Choose the number of factors with parallel analysis

Each factor has an eigenvalue that describes how much of the items' variance it explains. The old rule of keeping every factor with an eigenvalue above 1 (the Kaiser criterion) often keeps too many factors. Parallel analysis (Horn, 1965) compares each eigenvalue from the data with the eigenvalues obtained from random data of the same size, and keeps the factors whose eigenvalues exceed the random ones. For the loneliness items, two eigenvalues lie above the random line, and fa.parallel() reports that "the number of factors = 2".

Step 3. Rotate the solution

An unrotated solution is hard to read, because most items load on the first factor. Rotation turns the factors so that each item loads strongly on as few factors as possible. An orthogonal rotation, such as varimax, keeps the factors uncorrelated. An oblique rotation, such as oblimin, allows them to correlate. Constructs in health research are usually related, and emotional and social loneliness are expected to be, so an oblique rotation is the usual choice. If the factors turn out to be uncorrelated, an oblique rotation gives nearly the same answer as an orthogonal one.

Box 7.12: Revisiting Lesson 2

The two-factor analysis of the depression and anxiety items in Lesson 2, Section 3 used a varimax rotation, which forces the factors to be uncorrelated. Depression and anxiety are expected to correlate, so an oblique rotation such as oblimin is the better choice for those items, as it is for emotional and social loneliness. Lesson 2 also defined a factor loading as the correlation between an item and a factor. That holds only in a one-factor or an orthogonal solution. In an oblique solution, the reported (pattern) loading is the item's weight on one factor with the other factors held constant, and it differs from the item-factor correlation whenever the factors correlate.

Question. What does an oblique solution report that a varimax solution cannot?

Show answer

AnswerThe correlation between the factors. A varimax solution fixes it at zero, whereas an oblique solution estimates it, as in the correlation of 0.29 between the emotional and social factors in this section. When that correlation is not zero, the oblique solution also keeps the pattern loadings separate from the item-factor correlations.

Step 4. Read the loading matrix

Each row of the loading matrix is an item and each column is a factor. Loadings below about 0.3 are often hidden to make the pattern clearer. A clean structure has each item loading strongly on one factor and weakly on the others. An item that loads weakly on every factor measures little of what the others share, and an item that loads strongly on two factors (a cross-loading) is ambiguous. In the loneliness data, the three social items load 0.71, 0.74 and 0.68 on the first factor, emptiness and rejection load 0.86 and 0.56 on the second, and the item about missing people loads below 0.3 on both. The two factors correlate 0.29.

Activity 7.5 carries out the four steps in R on the exploratory half of the sample. Its output supplies the values quoted in the accordion, and its call to fa.parallel() draws the plot in Figure 7.7.

# install.packages(c("psych", "GPArotation"))   # run once
library(psych)        # KMO(), cortest.bartlett(), fa.parallel(), fa()
library(GPArotation)  # the rotation methods that fa() uses
round(cor(items), 2)  # correlations between every pair of items

# Explore in one half, test in the other half
set.seed(2021)    # makes the random split the same every time
efa_rows <- sample(nrow(items), size = round(nrow(items) / 2))
efa_half <- items[efa_rows, ]     # half 1: exploratory
cfa_half <- items[-efa_rows, ]    # half 2: confirmatory
nrow(efa_half)
nrow(cfa_half)

KMO(efa_half)    # overall MSA above 0.6 is usually adequate
cortest.bartlett(cor(efa_half), n = nrow(efa_half))

fa.parallel(efa_half, fa = "fa")   # how many factors?

efa_2 <- fa(efa_half, nfactors = 2, rotate = "oblimin")
print(efa_2$loadings, cutoff = 0.3)  # hide loadings below 0.3
round(efa_2$Phi, 2)                  # correlation between the factors

The exploratory half points to two correlated factors, with the three social items on one, emptiness and rejection on the other, and the item about missing people loading weakly on both. The next part tests this structure in the confirmatory half of the sample.

Confirmatory Factor Analysis in lavaan

The lavaan package (Rosseel, 2012) fits confirmatory factor models and the structural equation models of Lesson 8. A model is written as text, with one line per factor. The operator =~ is read as "is measured by", so social =~ rely + trust + close says that the social factor is measured by those three items. A latent factor has no natural scale, so lavaan fixes the first loading of each factor at 1 by default, and the other loadings are estimated relative to it. The standardized loadings (Std.all in the output) are on a common scale and are the values usually reported.

Judging model fit

A confirmatory model implies a pattern of correlations among the items, and fit statistics measure how closely the implied pattern matches the observed one. The chi-square test asks whether the misfit could be due to chance. With large samples it rejects almost every model, including models with trivial misfit, so it is reported but rarely decisive. Four indices are usually reported alongside it. The guides for good fit below follow Hu and Bentler (1999), who proposed values close to 0.95 or higher for the CFI and TLI, close to 0.06 or lower for the RMSEA and close to 0.08 or lower for the SRMR. The description of an RMSEA up to 0.08 as acceptable follows Browne and Cudeck (1993), who called such values a reasonable error of approximation. They are rules of thumb, and a model near a guide is better described as borderline than as passing or failing.

Table 7.3 lists the four indices, what each one measures and its guide for good fit, together with the values for the two-factor and one-factor models of the loneliness items that Activity 7.6 produces.

Table 7.3. Four fit indices for a confirmatory factor model, with the values for the two-factor and one-factor models of the De Jong Gierveld items.

Two models fitted to the same data can be compared directly. When one model is a special case of the other (the one-factor model is the two-factor model with the factor correlation fixed at 1), a chi-square difference test compares them, and the AIC and BIC can also be compared, with lower values preferred. These comparisons are made with anova() in Activity 7.6.

# install.packages("lavaan")   # run once
library(lavaan)    # cfa() for confirmatory factor analysis
# "=~" means "is measured by"
model_2f <- '
  emotional =~ emptiness + miss + rejected
  social    =~ rely + trust + close
'
cfa_2f <- cfa(model_2f, data = cfa_half)
summary(cfa_2f, fit.measures = TRUE, standardized = TRUE)

fitMeasures(cfa_2f, c("cfi", "tli", "rmsea", "srmr"))

model_1f <- '
  loneliness =~ emptiness + miss + rejected + rely + trust + close
'
cfa_1f <- cfa(model_1f, data = cfa_half)
fitMeasures(cfa_1f, c("cfi", "tli", "rmsea", "srmr"))
anova(cfa_1f, cfa_2f)    # chi-square difference test

Box 7.13 returns to the running case and uses the results of the two factor analyses to answer the reviewer's first question.

The card below links again to the narrated walkthrough of Section 2, which also demonstrates both factor analyses on these items.

Factor analysis has answered the reviewer's question about the structure of the scale. Item response theory, the subject of the next part, examines the same items one at a time and shows where along the trait each item measures well.

Item Response Theory

Classical test theory and factor analysis describe the scale as a whole. Item response theory (IRT) models how each item behaves along the latent trait, usually written θ (theta) and scaled so that 0 is the average and most people fall between −3 and 3 (Embretson & Reise, 2000). Each item is described by an item characteristic curve, the probability of a given answer at each level of the trait. Item response theory is presented here conceptually; the optional Worked code 7.1 at the end of this part shows how the curves are estimated, and it is not assessed through code.

Difficulty and discrimination

For an item scored 0 or 1, the two-parameter logistic (2PL) model describes the curve with two numbers. The difficulty b is the trait level at which the probability of the keyed answer (here, the lonely answer) is 0.5. An item with a high b is endorsed only by people high on the trait. The discrimination a is the steepness of the curve at that point. An item with a high a separates people just below b from people just above it sharply, and an item with a low a separates them weakly. The Rasch model (Rasch, 1960), also called the one-parameter model, gives every item the same discrimination, which makes the total score a sufficient summary of a person's trait level. The two-parameter model (Birnbaum, 1968) lets discrimination vary between items. The graded response model (Samejima, 1969) extends the 2PL to items with ordered categories, such as "No", "More or less" and "Yes", by estimating one discrimination per item and one threshold for each boundary between adjacent categories.

Equation 7.10 gives the 2PL item characteristic curve and the information that an item provides at each level of the trait.

The explorer in Interactive 7.2 draws the curve for any combination of a and b, together with the item's information. Raising a makes the curve steeper and the information peak taller and narrower; changing b slides both along the trait.

Curves for the social loneliness items

Figures 7.8 to 7.10 come from the CSCS data. Figure 7.8 shows 2PL curves for the three social items scored 0 or 1 by the published rule. All three curves are steep (a between 1.90 and 2.32) and cross 0.5 close together (b between −0.45 and −0.19). The three items therefore behave almost interchangeably: each one separates people just below the middle of the trait from people just above it, and none of them distinguishes well among people who are very connected or very isolated.

A graded response model keeps all three answer categories. It gives each item a discrimination and two thresholds: b1, the trait level at which a person becomes more likely than not to give "More or less" or the lonely answer, and b2, the level at which the lonely answer itself becomes more likely than not. For the three social items, b1 lies between −0.43 and −0.21 and b2 between 0.99 and 1.31. The test information curve adds the information from the three items. It is high between a θ of about −1 and 2, with two peaks that correspond to the two sets of thresholds, and it falls toward zero below about −2 and above about 3. Because the standard error of a person's estimated trait is 1 / √(information), a peak information of about 4 gives a standard error of about 0.5, whereas information near zero gives a very large standard error.

The two tabs that follow show the graded response model results: Figure 7.9 is the test information curve described above, and Figure 7.10 shows the category curves for each item.

Test information Category curves

Item response theory compared with classical test theory

Item response theory and classical test theory describe the same items in different ways and suit different tasks. Table 7.4 compares them on six features, from the unit of analysis to the demands that each places on the analyst.

Table 7.4. Classical test theory and item response theory compared.

The graded response model above was fitted to the three social items only, because IRT models of this kind assume that the items measure one trait (unidimensionality), and the factor analysis has shown that the six items measure two. For a manuscript that uses an established scale such as the De Jong Gierveld scale, classical reliability and a confirmatory factor analysis are usually sufficient. Item response theory becomes important when a scale is being built, shortened or compared across groups, and the PROMIS item banks used in health research are a prominent example of its use (Cella et al., 2010).

Worked code 7.1 shows how the graded response model for the three social items was fitted with the mirt package and how Figure 7.9 and Figure 7.10 were drawn.

# OPTIONAL: install.packages("mirt")   # run once
library(mirt)
# A graded response model for the three social loneliness items
grm <- mirt(social_items, model = 1, itemtype = "graded", verbose = FALSE)
round(coef(grm, IRTpars = TRUE, simplify = TRUE)$items, 2)

plot(grm, type = "trace")    # category curves for each item
plot(grm, type = "info")     # test information curve

Item response theory has described each social item along a continuous trait. The next part introduces latent class analysis, a latent variable model of a different kind.

Latent Class Analysis

Factor analysis and item response theory assume that the latent variable is continuous: people differ by degree along a trait such as loneliness. Latent class analysis is the latent variable model for a categorical latent variable. It assumes that the sample is a mixture of a small number of hidden groups, called latent classes, and that each class has its own probability of each answer to each item (Lazarsfeld & Henry, 1968; Collins & Lanza, 2010). Within a class, the answers to different items are assumed to be unrelated (local independence), so the classes account for all of the association between the items. The model estimates the size of each class and, for every person, a set of class-membership probabilities: the probability of belonging to each class given that person's answers. People are usually placed in their most likely class, and the membership probabilities show how certain each placement is.

The number of classes is chosen by fitting models with one, two, three or more classes and comparing them. The BIC balances fit against the number of parameters, and lower values are preferred, although in large samples it often keeps falling as classes are added. Entropy summarizes how clearly people are assigned to classes, on a scale from 0 to 1, with values near 1 indicating well-separated classes. The size of each class and whether the classes make substantive sense also guide the choice. Factor analysis groups items that measure the same trait, whereas latent class analysis groups people who answer in similar ways.

The card below links to an optional narrated walkthrough that fits latent class models to CSCS data in R.

The last part of the section returns from latent variable models to Amira's manuscript and shows how the results of this lesson are reported.

Reporting Measurement in a Manuscript

The Measures paragraph of a manuscript tells readers enough about each scale to judge the scores. It names the scale and its source, describes the items and response options with an example, states the scoring rule (including reverse-coding and any cut-point), reports the reliability observed in the study sample, and summarizes any structural or other validity evidence produced in the study. It also describes how missing items were handled. Claims such as "a validated scale" are replaced by a statement of what was validated, for what use and in which population.

Worked Example 7.2 applies this structure to Amira's manuscript, drawing on the scoring of Section 1, the reliability estimates of Section 2 and the factor analyses of this section. Box 7.14 then describes how the same structure applies to other scales.

Worked Example 7.2 completes Amira's reply to the reviewer. It reports that the six items measure two related constructs and that the social subscale has adequate internal consistency and the emotional subscale low internal consistency, and it states how the analysis takes account of both findings. The knowledge check and reflection that follow review the material of this section, and the Key Takeaways and final assessment of the lesson bring the four sections together.

Bringing It All Together

This lesson took the analyst's view of the multi-item scales that most analyses of the CSCS use. Section 1 traced the chain from a construct, through its domains and items, to an instrument and a scoring rule. It separated reflective scales, whose items are caused by the construct and are expected to correlate, from formative indices, whose components define the construct. It prepared and scored the six De Jong Gierveld items in R, including the reversal of the three positively worded items, and introduced classical test theory, in which reliability is the true-score share of observed variance and measurement error weakens correlations.

Section 2 estimated reliability. A single survey supports only internal consistency, and in the CSCS the social subscale had an alpha and an omega of 0.75 while the emotional subscale had an alpha of 0.51 and an omega of 0.57, with one weak item. Alpha rises with the number of items, assumes equal loadings, cannot show that a scale is unidimensional and has no meaning for an index, and it fell from 0.59 to 0.42 when the reversal step was skipped. Section 3 treated validity as an argument for a stated use in a stated population, organized the evidence with the COSMIN taxonomy, and found convergent, discriminant and known-groups evidence that supports the scale and distinguishes its two subscales. Section 4 used exploratory factor analysis in one half of the sample and confirmatory factor analysis in the other to show that two related factors (CFI 0.949, RMSEA 0.082) fit far better than one (CFI 0.739, RMSEA 0.177), and it introduced item response theory, in which each item has a curve and precision varies along the trait.

Together these steps answered both of the reviewer's questions and produced a Measures paragraph that reports the scale's source, scoring, reliability in the study sample and structural evidence. The final assessment asks for these steps to be applied to new scales. The next lesson, Mediation, Moderation and Path Analysis, extends the confirmatory factor model of Section 4 into path models and structural equation models.

The final assessment covers all four sections. All 15 questions must be answered correctly (100%), and the final reflection completed, to finish the lesson.

Final Knowledge Assessment