library(tidyverse)
library(tidystata)
gss24 <- readRDS("gss24.rds")19 Group and reshape
How far is a respondent’s schooling above or below the sample average for their region? How can we compare several questionnaire items when each occupies a separate column? These questions require two different changes to a table. Grouping changes which observations enter a calculation. Reshaping changes what a row represents.
By the end of this chapter, you will be able to:
- Add regional comparisons while controlling whether grouping persists.
- Compare overall and regional age positions without changing respondents.
- Reshape questionnaire items and verify a complete roundtrip.
- Prepare ordered school-to-work histories and distinguish duration from transitions.
We use the column operations from Chapter 12 and factors from Chapter 15. The earlier custom summary example is optional; the grouping lesson below explains the summary syntax it needs.
The main examples use the prepared GSS extract. Prepare it following Appendix A, then put gss24.rds in your project folder. A final application uses school-to-work histories because the GSS extract has no repeated monthly observations:
19.1 Add a regional comparison
Each row in gss24 represents a respondent. We will keep those rows and add a regional comparison to each one. Select respondent ID, region, schooling, and family income. The appendix already prepared region as a factor with readable categories.
respondents <- gss24 |>
select(region, id, educ, realinc)
respondents# A tibble: 3,986 × 4
region id educ realinc
<fct> <int> <dbl> <dbl>
1 northeast 1 16 43560
2 northeast 2 16 24502.
3 northeast 3 14 43560
4 northeast 4 12 58080
5 northeast 5 13 24502.
6 northeast 6 14 2722.
7 northeast 7 13 117565.
8 northeast 8 14 3267
9 northeast 9 14 117565.
10 northeast 10 12 NA
# ℹ 3,976 more rows
educ records completed years of schooling. Its missing answers remain in the table. The regional mean uses respondents with recorded schooling and gives each equal weight; it is an unweighted description of this survey sample. na.rm = TRUE excludes missing schooling only from the mean calculation. A respondent whose schooling is missing keeps a missing deviation.
Inside mutate(), mean(educ, na.rm = TRUE) calculates a mean and the name to its left creates a column. Adding .by = region tells mutate() to calculate separately within each region. Start with that mean alone:
respondent_means <- respondents |>
mutate(region_mean_educ = mean(educ, na.rm = TRUE), .by = region)
respondent_means# A tibble: 3,986 × 5
region id educ realinc region_mean_educ
<fct> <int> <dbl> <dbl> <dbl>
1 northeast 1 16 43560 14.4
2 northeast 2 16 24502. 14.4
3 northeast 3 14 43560 14.4
4 northeast 4 12 58080 14.4
5 northeast 5 13 24502. 14.4
6 northeast 6 14 2722. 14.4
7 northeast 7 13 117565. 14.4
8 northeast 8 14 3267 14.4
9 northeast 9 14 117565. 14.4
10 northeast 10 12 NA 14.4
# ℹ 3,976 more rows
Every respondent receives their own region’s mean. The regional value repeats because the rows still represent respondents. Now subtract that column from each person’s schooling:
respondent_comparison <- respondent_means |>
mutate(educ_deviation = educ - region_mean_educ)
respondent_comparison# A tibble: 3,986 × 6
region id educ realinc region_mean_educ educ_deviation
<fct> <int> <dbl> <dbl> <dbl> <dbl>
1 northeast 1 16 43560 14.4 1.56
2 northeast 2 16 24502. 14.4 1.56
3 northeast 3 14 43560 14.4 -0.442
4 northeast 4 12 58080 14.4 -2.44
5 northeast 5 13 24502. 14.4 -1.44
6 northeast 6 14 2722. 14.4 -0.442
7 northeast 7 13 117565. 14.4 -1.44
8 northeast 8 14 3267 14.4 -0.442
9 northeast 9 14 117565. 14.4 -0.442
10 northeast 10 12 NA 14.4 -2.44
# ℹ 3,976 more rows
The original schooling remains in educ. A positive deviation means that a respondent’s schooling is above the mean for their recorded region; a negative deviation means it is below. Both the mean and the deviation are in years. The repeated regional mean is intentional: we need it alongside each respondent, rather than in a separate summary table.
nrow() counts a table’s rows. Compare the input and output:
nrow(respondents)[1] 3986nrow(respondent_comparison)[1] 3986All 3986 respondent rows remain. Grouped mutate() adds columns without reducing the data to one row per region. The .by instruction applies only to this call; respondent_comparison carries no persistent grouping.
19.2 When grouping persists
Sometimes several consecutive operations should use the same groups. group_by(region) returns a table that stores the grouping instruction. It retains all rows and columns:
respondent_groups <- respondents |>
group_by(region)
respondent_groups# A tibble: 3,986 × 4
# Groups: region [4]
region id educ realinc
<fct> <int> <dbl> <dbl>
1 northeast 1 16 43560
2 northeast 2 16 24502.
3 northeast 3 14 43560
4 northeast 4 12 58080
5 northeast 5 13 24502.
6 northeast 6 14 2722.
7 northeast 7 13 117565.
8 northeast 8 14 3267
9 northeast 9 14 117565.
10 northeast 10 12 NA
# ℹ 3,976 more rows
Notice the Groups line above the output. Subsequent operations use those stored regions. Reuse the familiar calculation on this grouped table:
respondent_grouped <- respondent_groups |>
mutate(
region_mean_educ = mean(educ, na.rm = TRUE),
educ_deviation = educ - region_mean_educ
)
respondent_grouped# A tibble: 3,986 × 6
# Groups: region [4]
region id educ realinc region_mean_educ educ_deviation
<fct> <int> <dbl> <dbl> <dbl> <dbl>
1 northeast 1 16 43560 14.4 1.56
2 northeast 2 16 24502. 14.4 1.56
3 northeast 3 14 43560 14.4 -0.442
4 northeast 4 12 58080 14.4 -2.44
5 northeast 5 13 24502. 14.4 -1.44
6 northeast 6 14 2722. 14.4 -0.442
7 northeast 7 13 117565. 14.4 -1.44
8 northeast 8 14 3267 14.4 -0.442
9 northeast 9 14 117565. 14.4 -0.442
10 northeast 10 12 NA 14.4 -2.44
# ℹ 3,976 more rows
ungroup() removes the stored instruction. It retains respondent rows and the calculated columns:
respondent_ungrouped <- respondent_grouped |>
ungroup()
respondent_ungrouped# A tibble: 3,986 × 6
region id educ realinc region_mean_educ educ_deviation
<fct> <int> <dbl> <dbl> <dbl> <dbl>
1 northeast 1 16 43560 14.4 1.56
2 northeast 2 16 24502. 14.4 1.56
3 northeast 3 14 43560 14.4 -0.442
4 northeast 4 12 58080 14.4 -2.44
5 northeast 5 13 24502. 14.4 -1.44
6 northeast 6 14 2722. 14.4 -0.442
7 northeast 7 13 117565. 14.4 -1.44
8 northeast 8 14 3267 14.4 -0.442
9 northeast 9 14 117565. 14.4 -0.442
10 northeast 10 12 NA 14.4 -2.44
# ℹ 3,976 more rows
The familiar identical() check compares the deviation columns, extracted with $:
identical(
respondent_comparison$educ_deviation,
respondent_ungrouped$educ_deviation
)[1] TRUETRUE confirms that the two grouping routes produced the same deviations. Their grouping instructions differ until we remove the persistent groups.
19.2.1 Reduce observations to summaries
summarise() returns calculated summaries rather than retaining respondent rows. The earlier custom-summary example was optional, so we introduce the operation here. Start from the ungrouped table. In mean_educ = mean(educ, na.rm = TRUE), the left side names the result and the right side computes it:
respondent_ungrouped |>
summarise(mean_educ = mean(educ, na.rm = TRUE))# A tibble: 1 × 1
mean_educ
<dbl>
1 14.2
This result has one row: the mean completed years of schooling across all respondents with recorded schooling. It gives each such respondent equal weight. It does not replace respondent_ungrouped.
To request one summary per region for this operation alone, add .by = region:
respondent_ungrouped |>
summarise(mean_educ = mean(educ, na.rm = TRUE), .by = region)# A tibble: 4 × 2
region mean_educ
<fct> <dbl>
1 northeast 14.4
2 west 14.5
3 midwest 14.0
4 south 14.2
The result has one row for each observed region. .by is an instruction inside this call, not a new column or a persistent group. Missing region values would form a separate unknown-region group if present.
Stored groups give the same regional scope without writing .by again:
respondent_grouped |>
summarise(mean_educ = mean(educ, na.rm = TRUE))# A tibble: 4 × 2
region mean_educ
<fct> <dbl>
1 northeast 14.4
2 midwest 14.0
3 south 14.2
4 west 14.5
This result still has 4 rows. A researcher expecting one overall mean would therefore need to remove the groups first. Neither grouping route changes anyone’s schooling. The overall respondent mean differs from an equal-weight mean of regional means: regions with more recorded respondents contribute more observations to the respondent mean.
For a single grouped operation, prefer .by because its scope is visible in that call. Use group_by() when you intend subsequent operations to share the groups, and use ungroup() when that intention ends. Do not add .by to an already grouped table.
19.3 Compare positions within groups
sum() adds numbers, as in the word-count check in Section 17.3. It can also count true conditions: in a logical input, TRUE contributes 1 and FALSE contributes 0. This small calculation has two true values:
sum(c(TRUE, FALSE, TRUE))[1] 2The result is 2. For a column, !is.na(age) returns TRUE for a recorded age and FALSE for a missing one. These tests are themselves known for every row, so their sum counts recorded ages without dropping respondents or treating missing ages as age zero.
The age groups in Section 14.5 distinguish fixed intervals from sample quantiles. Grouping introduces another choice: a position in the whole sample or a position within a region. Before using regional quantiles, count recorded ages in each region:
gss24 |>
summarise(recorded_age = sum(!is.na(age)), .by = region)# A tibble: 4 × 2
region recorded_age
<fct> <int>
1 northeast 598
2 west 850
3 midwest 898
4 south 1513
Every region has recorded ages here. An entirely missing or empty group would have no age distribution from which to calculate quartiles; decide how to report that group before applying the calculation. In this extract all respondents also have a recorded region.
First request four overall groups with n = 4. At this step, retain the interval labels so we can see the age ranges:
overall_age_groups <- gss24 |>
select(id, region, age) |>
mutate(overall_quartile = cut_quantile(age, n = 4))
overall_age_groups# A tibble: 3,986 × 4
id region age overall_quartile
<int> <fct> <dbl> <ord>
1 1 northeast 33 [18,34]
2 2 northeast 64 (48,64]
3 3 northeast 69 (64,89]
4 4 northeast 19 [18,34]
5 5 northeast 70 (64,89]
6 6 northeast 53 (48,64]
7 7 northeast 48 (34,48]
8 8 northeast 30 [18,34]
9 9 northeast 60 (48,64]
10 10 northeast 25 [18,34]
# ℹ 3,976 more rows
The returned factor records each respondent’s position using the whole sample’s age boundaries. Missing age remains missing. To compare positions across differently bounded regions, request integer codes instead: labels = FALSE returns codes 1 through 4.
age_positions <- gss24 |>
select(id, region, age) |>
mutate(overall_quartile = cut_quantile(age, n = 4, labels = FALSE))
age_positions# A tibble: 3,986 × 4
id region age overall_quartile
<int> <fct> <dbl> <int>
1 1 northeast 33 1
2 2 northeast 64 3
3 3 northeast 69 4
4 4 northeast 19 1
5 5 northeast 70 4
6 6 northeast 53 3
7 7 northeast 48 2
8 8 northeast 30 1
9 9 northeast 60 3
10 10 northeast 25 1
# ℹ 3,976 more rows
Now repeat the same code-producing calculation separately within each region:
age_positions <- age_positions |>
mutate(
regional_quartile = cut_quantile(age, n = 4, labels = FALSE),
.by = region
)
age_positions# A tibble: 3,986 × 5
id region age overall_quartile regional_quartile
<int> <fct> <dbl> <int> <int>
1 1 northeast 33 1 1
2 2 northeast 64 3 3
3 3 northeast 69 4 4
4 4 northeast 19 1 1
5 5 northeast 70 4 4
6 6 northeast 53 3 2
7 7 northeast 48 2 2
8 8 northeast 30 1 1
9 9 northeast 60 3 3
10 10 northeast 25 1 1
# ℹ 3,976 more rows
Comparing two unknown quartile codes gives NA, so adding those comparisons without a missingness rule would return an unknown sum. In sum(..., na.rm = TRUE), the option excludes unknown contributions from this calculation. It does not remove the corresponding respondents from age_positions or assign them a quartile:
age_positions |>
summarise(
recorded_age = sum(!is.na(age)),
changed_group = sum(overall_quartile != regional_quartile, na.rm = TRUE),
.by = region
)# A tibble: 4 × 3
region recorded_age changed_group
<fct> <int> <int>
1 northeast 598 80
2 west 850 65
3 midwest 898 14
4 south 1513 112
Each row reports recorded ages and known changed assignments within one region. The comparison concerns relative age positions among recorded ages, rather than movement between regions.
The table retains all 3986 respondents and their original ages. Unknown ages receive unknown quartile codes. In total, 271 respondents change groups. The code 1 means the youngest quarter of the distribution used by that call, so its age limits can differ between regions. Inspect the boundaries explicitly. quantile() calculates distribution positions; .25, .5, and .75 request the three boundaries dividing four groups:
age_positions |>
summarise(
q1 = quantile(age, .25, na.rm = TRUE),
q2 = quantile(age, .5, na.rm = TRUE),
q3 = quantile(age, .75, na.rm = TRUE)
)# A tibble: 1 × 3
q1 q2 q3
<dbl> <dbl> <dbl>
1 34 48 64
age_positions |>
summarise(
q1 = quantile(age, .25, na.rm = TRUE),
q2 = quantile(age, .5, na.rm = TRUE),
q3 = quantile(age, .75, na.rm = TRUE),
.by = region
)# A tibble: 4 × 4
region q1 q2 q3
<fct> <dbl> <dbl> <dbl>
1 northeast 37 53.5 65
2 west 35 46 63
3 midwest 35 48 64
4 south 33 46 63
The overall boundaries are 34, 48, and 64 years. The northeast’s are 37, 53.5, and 65. The value 53.5 is an interpolated distribution boundary, not a newly observed age. Thus two people with the same age can occupy different regional quartiles. Use common fixed age bands when the question requires common age limits across regions.
Repeated ages stay together under this quantile rule, so four groups need not have exactly equal sizes. Exclude unknown ages only from the denominator of the following membership check:
regional_age_sizes <- age_positions |>
filter(!is.na(age)) |>
count(region, regional_quartile) |>
mutate(percent = 100 * n / sum(n), .by = region)
regional_age_sizes# A tibble: 16 × 4
region regional_quartile n percent
<fct> <int> <int> <dbl>
1 northeast 1 154 25.8
2 northeast 2 145 24.2
3 northeast 3 150 25.1
4 northeast 4 149 24.9
5 midwest 1 227 25.3
6 midwest 2 226 25.2
7 midwest 3 228 25.4
8 midwest 4 217 24.2
9 south 1 397 26.2
10 south 2 366 24.2
11 south 3 384 25.4
12 south 4 366 24.2
13 west 1 226 26.6
14 west 2 202 23.8
15 west 3 216 25.4
16 west 4 206 24.2
The smallest group contains 23.8 percent of its region’s recorded ages. None is below 15 percent here. In another sample, inspect small groups and ties before using the categories; reconsider boundaries only for a substantive reason. These unweighted groups describe positions in this sample, not population age quartiles.
19.4 Change what a row represents
The GSS confidence questions ask about major companies, education, and medicine. In the prepared extract, conbus, coneduc, and conmedic are factors with the same response categories. Select them with the respondent identifier:
confidence_wide <- gss24 |>
select(id, conbus, coneduc, conmedic)
confidence_wide# A tibble: 3,986 × 4
id conbus coneduc conmedic
<int> <fct> <fct> <fct>
1 1 a great deal a great deal a great deal
2 2 only some only some a great deal
3 3 only some a great deal a great deal
4 4 NA NA NA
5 5 only some hardly any only some
6 6 hardly any only some only some
7 7 only some only some a great deal
8 8 hardly any hardly any hardly any
9 9 only some only some only some
10 10 NA NA NA
# ℹ 3,976 more rows
This is a wide table: one row represents one respondent, and each item has its own column. id identifies the respondent. A missing answer can reflect an unavailable questionnaire item as well as nonresponse; it does not mean “hardly any” confidence.
Before combining the item columns, inspect their factor levels. As above, $ extracts a named column from a table; levels() lists a factor’s categories in their stored order.
levels(confidence_wide$conbus)[1] "a great deal" "only some" "hardly any" levels(confidence_wide$coneduc)[1] "a great deal" "only some" "hardly any" levels(confidence_wide$conmedic)[1] "a great deal" "only some" "hardly any" All three use “a great deal”, “only some”, and “hardly any” in the same order. We can combine these responses into one factor column while preserving their meanings. Combining columns with different response scales would require a substantive decision before reshaping.
pivot_longer() gathers selected columns into rows. Its cols argument selects the columns to gather; the range conbus:conmedic selects all three adjacent items. Start with its default output names:
confidence_positions <- confidence_wide |>
pivot_longer(cols = conbus:conmedic)
confidence_positions# A tibble: 11,958 × 3
id name value
<int> <chr> <fct>
1 1 conbus a great deal
2 1 coneduc a great deal
3 1 conmedic a great deal
4 2 conbus only some
5 2 coneduc only some
6 2 conmedic a great deal
7 3 conbus only some
8 3 coneduc a great deal
9 3 conmedic a great deal
10 4 conbus NA
# ℹ 11,948 more rows
name contains the former column names and value contains the answers. The unselected id repeats for each item. A row now represents a response position, including a position with no recorded answer. We will choose names that express that meaning. First, names_to = "item" replaces the output name name:
confidence_named <- confidence_wide |>
pivot_longer(cols = conbus:conmedic, names_to = "item")
confidence_named# A tibble: 11,958 × 3
id item value
<int> <chr> <fct>
1 1 conbus a great deal
2 1 coneduc a great deal
3 1 conmedic a great deal
4 2 conbus only some
5 2 coneduc only some
6 2 conmedic a great deal
7 3 conbus only some
8 3 coneduc a great deal
9 3 conmedic a great deal
10 4 conbus NA
# ℹ 11,948 more rows
The answers still occupy value. Add values_to = "response" to name that column as well:
confidence_long <- confidence_wide |>
pivot_longer(
cols = conbus:conmedic,
names_to = "item",
values_to = "response"
)
confidence_long# A tibble: 11,958 × 3
id item response
<int> <chr> <fct>
1 1 conbus a great deal
2 1 coneduc a great deal
3 1 conmedic a great deal
4 2 conbus only some
5 2 coneduc only some
6 2 conmedic a great deal
7 3 conbus only some
8 3 coneduc a great deal
9 3 conmedic a great deal
10 4 conbus NA
# ℹ 11,948 more rows
Missing response positions remain by default. Write values_drop_na = FALSE explicitly when retaining them is central to the task, as it is here:
confidence_long <- confidence_wide |>
pivot_longer(
cols = conbus:conmedic,
names_to = "item",
values_to = "response",
values_drop_na = FALSE
)
nrow(confidence_long)[1] 11958levels(confidence_long$response)[1] "a great deal" "only some" "hardly any" Now one row represents one respondent-item combination. There are 3986 respondents and three items, so the result contains 3 x 3986 = 11958 rows. These are not additional respondents. Even a respondent with no recorded answer to any of the three items contributes three rows. The single response column retains the shared factor levels.
This organization lets us apply the same operation to every item. In the familiar tab() call below, .by = item requests a separate frequency table for each questionnaire item:
confidence_long |>
tab(response, .by = item)# A tibble: 12 × 7
item response n percent cum valid valid_cum
<chr> <fct> <int> <dbl> <dbl> <dbl> <dbl>
1 conbus a great deal 372 9.33 9.33 14.4 14.4
2 conbus only some 1555 39.0 48.3 60.1 74.5
3 conbus hardly any 661 16.6 64.9 25.5 100
4 conbus NA 1398 35.1 100 NA NA
5 coneduc a great deal 513 12.9 12.9 19.6 19.6
6 coneduc only some 1479 37.1 50.0 56.6 76.2
7 coneduc hardly any 622 15.6 65.6 23.8 100
8 coneduc NA 1372 34.4 100 NA NA
9 conmedic a great deal 710 17.8 17.8 27.2 27.2
10 conmedic only some 1408 35.3 53.1 53.9 81.1
11 conmedic hardly any 492 12.3 65.5 18.9 100
12 conmedic NA 1376 34.5 100 NA NA
Within each item, n counts respondents in each response category. percent uses all 3986 respondents as its denominator, including missing answers; valid uses only respondents with a recorded answer to that item. The missing row keeps the coverage difference visible. Among recorded answers, “only some” is the most common response to each item. These are unweighted descriptions of this sample, rather than population estimates.
19.5 Return to one row per respondent
To reverse the reshape, we need to know which values belong in each cell. A key is a column, or combination of columns, that identifies a row uniquely. In the wide table, id should identify one respondent row. In the long table, the combination of id and item should identify one response row.
count() returns one row per observed key and names its row count n. Use filter() to look for invalid keys. In the first check, is.na(id) detects missing identifiers and n != 1 detects identifiers occurring more than once. The | operator means either condition is sufficient. The second check counts combinations of the two columns.
confidence_wide |>
count(id) |>
filter(is.na(id) | n != 1)# A tibble: 0 × 2
# ℹ 2 variables: id <int>, n <int>
confidence_long |>
count(id, item) |>
filter(n != 1)# A tibble: 0 × 3
# ℹ 3 variables: id <int>, item <chr>, n <int>
Both results have zero rows: no respondent identifier is missing or repeated, and no respondent-item pair is repeated. The item names came directly from the three selected column names, so none is missing. Together with the row count, these checks confirm that there is exactly one response position for every respondent and item.
pivot_wider() reverses the direction: it places response positions into columns. The default-name table confidence_positions has the expected name and value columns, so we can start with a call that uses those defaults:
confidence_positions |>
pivot_wider()# A tibble: 3,986 × 4
id conbus coneduc conmedic
<int> <fct> <fct> <fct>
1 1 a great deal a great deal a great deal
2 2 only some only some a great deal
3 3 only some a great deal a great deal
4 4 NA NA NA
5 5 only some hardly any only some
6 6 hardly any only some only some
7 7 only some only some a great deal
8 8 hardly any hardly any hardly any
9 9 only some only some only some
10 10 NA NA NA
# ℹ 3,976 more rows
The result again has one respondent per row and one column per item. We can adapt the defaults one role at a time. confidence_named already renamed name to item, while retaining value. Set names_from = item to tell the reshape where the new column names come from:
confidence_named |>
pivot_wider(names_from = item)# A tibble: 3,986 × 4
id conbus coneduc conmedic
<int> <fct> <fct> <fct>
1 1 a great deal a great deal a great deal
2 2 only some only some a great deal
3 3 only some a great deal a great deal
4 4 NA NA NA
5 5 only some hardly any only some
6 6 hardly any only some only some
7 7 only some only some a great deal
8 8 hardly any hardly any hardly any
9 9 only some only some only some
10 10 NA NA NA
# ℹ 3,976 more rows
This returns the same wide layout. In confidence_long, the answer column is also renamed, so add values_from = response to tell the reshape which values enter those cells:
confidence_long |>
pivot_wider(names_from = item, values_from = response)# A tibble: 3,986 × 4
id conbus coneduc conmedic
<int> <fct> <fct> <fct>
1 1 a great deal a great deal a great deal
2 2 only some only some a great deal
3 3 only some a great deal a great deal
4 4 NA NA NA
5 5 only some hardly any only some
6 6 hardly any only some only some
7 7 only some only some a great deal
8 8 hardly any hardly any hardly any
9 9 only some only some only some
10 10 NA NA NA
# ℹ 3,976 more rows
The remaining id column identifies rows here. Make that role explicit with id_cols = id. With more columns in a long table, explicitly identifying the row key protects against accidentally using a measurement as part of it:
confidence_restored <- confidence_long |>
pivot_wider(
id_cols = id,
names_from = item,
values_from = response
)
confidence_restored# A tibble: 3,986 × 4
id conbus coneduc conmedic
<int> <fct> <fct> <fct>
1 1 a great deal a great deal a great deal
2 2 only some only some a great deal
3 3 only some a great deal a great deal
4 4 NA NA NA
5 5 only some hardly any only some
6 6 hardly any only some only some
7 7 only some only some a great deal
8 8 hardly any hardly any hardly any
9 9 only some only some only some
10 10 NA NA NA
# ℹ 3,976 more rows
identical(confidence_wide, confidence_restored)[1] TRUEThe check returns TRUE: the original respondent order, item columns, answers, missing values, and factor levels have all survived the round trip. This exact agreement is possible because we retained every response position and preserved the ordering used to create the long table.
If more than one response occupied the same respondent-item key, a single output cell could not hold those responses as one ordinary value. With multiple responses for one cell, pivot_wider() can warn and return a list column: each cell holds several values, as in Chapter 17. The command may finish, but its result no longer has one ordinary response per cell. Investigate the respondent-item keys before continuing. Choosing an average merely to make the reshape run would change the observations. Filling missing answers with zero would also change their meaning: no response is not a zero-confidence answer.
Completion of absent region-schooling summary cells is a separate question from reshaping respondent records. We develop that reporting grid in Section 20.5, after learning to join tables.
19.6 Application: school-to-work histories
How much time do young people spend in education, training, employment, or joblessness after compulsory schooling? Answering this question requires observations of the same people over time. The public mvad dataset from McVicar and Anyadike-Danes’s school-to-work study follows 712 young people in Northern Ireland monthly from July 1993 through June 1999. TraMineR’s documentation describes the source and its categories.
Download the book’s captured CSV into your project’s examples folder. Its data note records the source and selected columns. The file keeps every respondent, the original ID and sample weight, and all monthly states. It is already a curated teaching dataset; our task is to prepare its layout and time information for analysis. TraMineR is not needed to run this lesson.
19.6.1 Turn month headings into observations
Reuse the explicit import rules from Section 18.3. col_types supplies the familiar cols() specification: character columns are the default, while the exceptions read ID as an integer and the source weight as a double. This keeps monthly activity labels as text. The collectors control storage; the weight is retained as a source variable, not applied automatically.
school_wide <- read_csv(
"examples/school-to-work.csv",
col_types = cols(
.default = col_character(), id = col_integer(), weight = col_double()
)
)
school_wide# A tibble: 712 × 74
id weight Jul.93 Aug.93 Sep.93 Oct.93 Nov.93 Dec.93 Jan.94 Feb.94 Mar.94
<int> <dbl> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr>
1 1 0.33 training train… emplo… emplo… emplo… emplo… train… train… emplo…
2 2 0.57 jobless… joble… FE FE FE FE FE FE FE
3 3 1.59 jobless… joble… train… train… train… train… train… train… train…
4 4 1.59 training train… train… train… train… train… train… train… train…
5 5 0.57 jobless… joble… FE FE FE FE FE FE FE
6 6 1.59 jobless… joble… joble… train… train… train… train… train… train…
7 7 0.57 jobless… joble… FE FE FE FE FE FE FE
8 8 2.75 employm… emplo… FE FE FE FE FE FE FE
9 9 2 jobless… joble… train… train… train… train… train… train… train…
10 10 3.6 employm… emplo… school school school school school school school
# ℹ 702 more rows
# ℹ 63 more variables: Apr.94 <chr>, May.94 <chr>, Jun.94 <chr>, Jul.94 <chr>,
# Aug.94 <chr>, Sep.94 <chr>, Oct.94 <chr>, Nov.94 <chr>, Dec.94 <chr>,
# Jan.95 <chr>, Feb.95 <chr>, Mar.95 <chr>, Apr.95 <chr>, May.95 <chr>,
# Jun.95 <chr>, Jul.95 <chr>, Aug.95 <chr>, Sep.95 <chr>, Oct.95 <chr>,
# Nov.95 <chr>, Dec.95 <chr>, Jan.96 <chr>, Feb.96 <chr>, Mar.96 <chr>,
# Apr.96 <chr>, May.96 <chr>, Jun.96 <chr>, Jul.96 <chr>, Aug.96 <chr>, …
school_wide |>
count(id) |>
filter(is.na(id) | n != 1)# A tibble: 0 × 2
# ℹ 2 variables: id <int>, n <int>
The key check returns zero rows. Each of the 712 rows identifies one respondent; Jul.93 through Jun.99 identify 72 observation months. The state labels are school, FE (further education), HE (higher education), training, employment, and joblessness. Keep these distinctions: for example, joblessness is the source category and should not be renamed unemployment without checking its definition.
We reuse pivot_longer(), retaining ID and weight while gathering the monthly columns. month_label preserves the original heading, and state contains the recorded activity:
school_long <- school_wide |>
pivot_longer(
cols = -c(id, weight),
names_to = "month_label",
values_to = "state",
values_drop_na = FALSE
)
school_long# A tibble: 51,264 × 4
id weight month_label state
<int> <dbl> <chr> <chr>
1 1 0.33 Jul.93 training
2 1 0.33 Aug.93 training
3 1 0.33 Sep.93 employment
4 1 0.33 Oct.93 employment
5 1 0.33 Nov.93 employment
6 1 0.33 Dec.93 employment
7 1 0.33 Jan.94 training
8 1 0.33 Feb.94 training
9 1 0.33 Mar.94 employment
10 1 0.33 Apr.94 employment
# ℹ 51,254 more rows
nrow(school_long)[1] 51264One row now represents one person in one month. The 51264 rows are 712 x 72 person-months, while the number of people remains 712. All monthly states in this capture are recorded. In another source, a missing state would remain unknown; reshaping would not justify carrying a previous activity forward.
19.6.2 Put each person’s months in order
Reuse the parser and English locale from Section 16.1. Here paste() adds day 01 to the original month heading. The format %d %b.%y reads that day, an abbreviated English month, the literal period, and a two-digit year. The source documents 1993–1999, so a successful parse must also agree with that century and observation window.
school_months <- school_long |>
mutate(month = parse_date(
paste("01", month_label),
format = "%d %b.%y",
locale = locale(date_names = "en")
)) |>
arrange(id, month)
school_months |>
filter(is.na(month))# A tibble: 0 × 5
# ℹ 5 variables: id <int>, weight <dbl>, month_label <chr>, state <chr>,
# month <date>
school_months |>
count(id, month) |>
filter(n != 1)# A tibble: 0 × 3
# ℹ 3 variables: id <int>, month <date>, n <int>
Both checks return zero rows: every month parsed, and every person-month key is unique. Check the date window and every respondent’s monthly coverage as well:
school_months |>
filter(month < as.Date("1993-07-01") | month > as.Date("1999-06-01"))# A tibble: 0 × 5
# ℹ 5 variables: id <int>, weight <dbl>, month_label <chr>, state <chr>,
# month <date>
school_months |>
count(id, name = "months") |>
filter(months != 72)# A tibble: 0 × 2
# ℹ 2 variables: id <int>, months <int>
sum(is.na(school_months$state))[1] 0The first result has zero rows: no parsed month falls outside July 1993 through June 1999. The coverage check returns no exceptions to 72 months per person, and the source has no missing monthly states. There are 72 possible observation months in that inclusive window. Together with the unique person-month keys, these checks cover all 712 histories. No missing parse results alone could have established the correct century. The first day represents an observation month; it does not assert that an activity changed on that day. Sorting the original headings alphabetically would mix months and years, so we sort parsed dates within ID.
19.6.3 Count time and transitions separately
First count months in each state for each person. name = "months" names the count column. A person with no observed months in a particular state has no row for that state in this table.
state_months <- school_months |>
count(id, state, name = "months")
state_totals <- state_months |>
summarise(person_months = sum(months), .by = state) |>
mutate(mean_months = person_months / nrow(school_wide))
state_totals# A tibble: 6 × 3
state person_months mean_months
<chr> <int> <dbl>
1 employment 22937 32.2
2 training 5272 7.40
3 FE 8331 11.7
4 HE 5980 8.40
5 joblessness 4399 6.18
6 school 4345 6.10
state_months |>
summarise(total_months = sum(months), .by = id) |>
filter(total_months != 72)# A tibble: 0 × 2
# ℹ 2 variables: id <int>, total_months <int>
The last check returns zero rows: each person’s totals cover all 72 months. In state_totals, person_months sums observed time across the cohort. Dividing by all 712 respondents gives the average months per person in that state, including zero months for people who never entered it. Averaging only the rows present in state_months would instead describe people who experienced that state. The averages across all six states sum to 72 months.
Time in employment and entry into employment answer different questions. lag() returns the preceding value in the current row order, with NA at the first position. Apply it separately within each ID using .by = id; school_months is already ordered by person and date:
school_previous <- school_months |>
mutate(previous_state = lag(state), .by = id)
school_previous |>
select(id, month, state, previous_state)# A tibble: 51,264 × 4
id month state previous_state
<int> <date> <chr> <chr>
1 1 1993-07-01 training NA
2 1 1993-08-01 training training
3 1 1993-09-01 employment training
4 1 1993-10-01 employment employment
5 1 1993-11-01 employment employment
6 1 1993-12-01 employment employment
7 1 1994-01-01 training employment
8 1 1994-02-01 training training
9 1 1994-03-01 employment training
10 1 1994-04-01 employment employment
# ℹ 51,254 more rows
sum(is.na(school_previous$previous_state))[1] 712The returned column records the preceding observed month for the same person. It has 712 first-month absences, one per respondent. Those are unavailable prior observations, not missing source states. Now compare current and previous states:
school_history <- school_previous |>
mutate(changed = state != previous_state)
observed_transitions <- school_history |>
filter(!is.na(previous_state))
observed_transitions |>
count(id, name = "comparisons") |>
filter(comparisons != 71)# A tibble: 0 × 2
# ℹ 2 variables: id <int>, comparisons <int>
nrow(observed_transitions)[1] 50552work_entries <- observed_transitions |>
filter(previous_state == "joblessness", state == "employment") |>
count(previous_state, state, name = "transitions")
work_entries# A tibble: 1 × 3
previous_state state transitions
<chr> <chr> <int>
1 joblessness employment 182
The first month of each history has no earlier observed state, so it contributes no adjacent-month comparison. There are 50552 comparisons: 71 per person. The result counts 182 observed moves from joblessness in one month to employment in the next. A person can contribute more than one such move. These records cannot show changes occurring entirely between monthly observations or locate transitions on exact days.
All calculations here describe the observed cohort without weights. The source weight remains in the imported table but is not applied. These summaries do not estimate national rates or establish why an activity changed. The useful sequence is to preserve the source, establish the person-time key, order observations, and choose whether the question concerns time or change.
19.7 Review
mutate()can add group means and deviations while retaining the original observations.group_by()stores groups for subsequent operations;ungroup()removes that instruction.summarise()reduces observations to summaries, and stored grouping determines their scope.- Within-group quartiles describe relative positions; inspect their boundaries before comparing regions.
pivot_longer()changes separate item columns into respondent-item rows; retain missing answers when they mark response positions.count()checks identifying keys;pivot_wider()restores item columns when each key identifies one cell. A list-column warning calls for key investigation.parse_date()interprets month labels; orderedlag()comparisons within each person identify adjacent-month transitions.
Reshaping changes the layout of one table. Chapter 20 next considers how rows and columns from separate tables belong together.
19.8 Exercises
Start each exercise from its saved source. Recreate needed objects; no exercise depends on another answer.
- Annotate income and diagnose grouping. Add each respondent’s regional mean
realincand deviation, keeping missing income and all respondents.realincrecords family income in constant 1986 dollars. State the units and describe a missing deviation. Then use persistent regional grouping to annotate schooling and reproduce a summary mistakenly expected to be one overall row. Explain its row count and correct it, retaining respondent rows until the final summary. Explain what an entirely missing group would contribute to a mean. - Compare age positions. From a fresh GSS import, compare overall and regional age quartiles. Report changed assignments, inspect regional boundaries and actual group shares, and explain why regional quartile 1 is not a common age band. Check that neither respondents nor missing ages were lost.
- Reconstruct confidence items. Select ID and the three confidence items from a fresh import. Reshape to
itemandresponse, retaining missing answers. Compare total and item-specific counts before and after selecting recorded answers. Identify the long-table row unit and say how many distinct respondents were added. Explain the denominator error from dividing one item’s category count by all long-table rows. Check both keys, restore the wide table, and verify exact agreement. Explain why replacing missing answers with zero or averaging duplicate responses would defeat preservation. - Distinguish duration and transitions. From the school-to-work CSV, calculate each respondent’s combined months in
school,FE, orHE, including zero-month respondents, and the cohort mean. Then recreate ordered histories and reproduce the error from an ungroupedlag(). Count cross-person comparisons, correct the grouping, and verify first-month absences and adjacent comparisons. State the units, denominators, and limits of monthly observations.
Worked answers
NoteRegional comparisons and grouping scope
Start a fresh session with the packages and saved GSS extract.
library(tidyverse)
library(tidystata)
answer_gss24 <- readRDS("gss24.rds")answer_income <- answer_gss24 |>
select(region, id, realinc) |>
mutate(
region_mean_income = mean(realinc, na.rm = TRUE),
income_deviation = realinc - region_mean_income,
.by = region
)
answer_income# A tibble: 3,986 × 5
region id realinc region_mean_income income_deviation
<fct> <int> <dbl> <dbl> <dbl>
1 northeast 1 43560 36596. 6964.
2 northeast 2 24502. 36596. -12094.
3 northeast 3 43560 36596. 6964.
4 northeast 4 58080 36596. 21484.
5 northeast 5 24502. 36596. -12094.
6 northeast 6 2722. 36596. -33874.
7 northeast 7 117565. 36596. 80968.
8 northeast 8 3267 36596. -33329.
9 northeast 9 117565. 36596. 80968.
10 northeast 10 NA 36596. NA
# ℹ 3,976 more rows
nrow(answer_income)[1] 3986answer_income |>
filter(is.na(realinc))# A tibble: 423 × 5
region id realinc region_mean_income income_deviation
<fct> <int> <dbl> <dbl> <dbl>
1 northeast 10 NA 36596. NA
2 northeast 18 NA 36596. NA
3 northeast 30 NA 36596. NA
4 northeast 31 NA 36596. NA
5 northeast 32 NA 36596. NA
6 northeast 36 NA 36596. NA
7 northeast 56 NA 36596. NA
8 northeast 57 NA 36596. NA
9 northeast 67 NA 36596. NA
10 northeast 69 NA 36596. NA
# ℹ 413 more rows
The mean uses respondents with recorded family income within each region, giving each equal weight. The result still has 3986 respondent rows. The final output identifies respondents with missing income: their regional means are available, but their deviations remain missing because subtracting a known mean from an unknown measurement cannot produce a known difference. Both new columns are in constant 1986 dollars. These are unweighted sample comparisons, not population estimates.
If every income in a region were missing, mean(..., na.rm = TRUE) would return NaN: there would be no measured values to average. No such mean could provide a meaningful regional deviation.
Now diagnose the stored-group problem on schooling. Restore the extract again so this part also has an explicit source.
Load the same packages and saved extract in a fresh session:
library(tidyverse)
library(tidystata)
answer_gss24 <- readRDS("gss24.rds")answer_grouped <- answer_gss24 |>
select(region, id, educ) |>
group_by(region) |>
mutate(region_mean_educ = mean(educ, na.rm = TRUE))
answer_regional_summary <- answer_grouped |>
summarise(mean_educ = mean(educ, na.rm = TRUE))
answer_regional_summary# A tibble: 4 × 2
region mean_educ
<fct> <dbl>
1 northeast 14.4
2 midwest 14.0
3 south 14.2
4 west 14.5
answer_respondent_rows <- answer_grouped |>
ungroup()
nrow(answer_respondent_rows)[1] 3986answer_overall_summary <- answer_respondent_rows |>
summarise(mean_educ = mean(educ, na.rm = TRUE))
answer_overall_summary# A tibble: 1 × 1
mean_educ
<dbl>
1 14.2
The first summary has 4 rows because the regional grouping persists after mutate(). Removing that grouping retains all 3986 respondent observations and their new regional-mean column. The final summary then has 1 row, containing the unweighted mean of recorded schooling in years. We average the original educ measurements, not the repeated regional-mean column.
NoteOverall and regional age positions
Prepare the extract following Appendix A, then restore it:
library(tidyverse)
library(tidystata)
gss24 <- readRDS("gss24.rds")answer_age_positions <- gss24 |>
select(id, region, age) |>
mutate(overall_quartile = cut_quantile(age, n = 4, labels = FALSE)) |>
mutate(
regional_quartile = cut_quantile(age, n = 4, labels = FALSE),
.by = region
)
nrow(answer_age_positions)[1] 3986answer_age_positions |>
summarise(
respondents = sum(!is.na(id)),
missing_age = sum(is.na(age)),
missing_overall = sum(is.na(overall_quartile)),
missing_regional = sum(is.na(regional_quartile)),
changed_group = sum(overall_quartile != regional_quartile, na.rm = TRUE)
)# A tibble: 1 × 5
respondents missing_age missing_overall missing_regional changed_group
<int> <int> <int> <int> <int>
1 3986 127 127 127 271
answer_age_positions |>
summarise(
q1 = quantile(age, .25, na.rm = TRUE),
q2 = quantile(age, .5, na.rm = TRUE),
q3 = quantile(age, .75, na.rm = TRUE),
changed_group = sum(overall_quartile != regional_quartile, na.rm = TRUE),
.by = region
)# A tibble: 4 × 5
region q1 q2 q3 changed_group
<fct> <dbl> <dbl> <dbl> <int>
1 northeast 37 53.5 65 80
2 west 35 46 63 65
3 midwest 35 48 64 14
4 south 33 46 63 112
All 3,986 respondents remain, including the 127 with unknown ages; those same respondents have unknown quartile codes in both calculations. There are 271 changed assignments: 80 in the northeast, 14 in the midwest, 112 in the south, and 65 in the west. The regional boundaries are 37/53.5/65, 35/48/64, 33/46/63, and 35/46/63 years, respectively, compared with overall boundaries of 34/48/64. Regional quartile 1 describes a position relative to each region’s recorded ages. Common fixed intervals would be needed to compare identical age bands. Neither the respondents’ ages nor their regions have changed.
The non-missing ID count verifies respondent retention because the prepared source has a unique, recorded ID for every respondent. Also inspect actual regional quartile shares, excluding missing age only from this check:
answer_age_positions |>
filter(!is.na(age)) |>
count(region, regional_quartile) |>
mutate(percent = 100 * n / sum(n), .by = region)# A tibble: 16 × 4
region regional_quartile n percent
<fct> <int> <int> <dbl>
1 northeast 1 154 25.8
2 northeast 2 145 24.2
3 northeast 3 150 25.1
4 northeast 4 149 24.9
5 midwest 1 227 25.3
6 midwest 2 226 25.2
7 midwest 3 228 25.4
8 midwest 4 217 24.2
9 south 1 397 26.2
10 south 2 366 24.2
11 south 3 384 25.4
12 south 4 366 24.2
13 west 1 226 26.6
14 west 2 202 23.8
15 west 3 216 25.4
16 west 4 206 24.2
Ties keep equal ages together, so quarters need not contain exactly 25 percent. The displayed shares let us inspect small groups rather than assuming that requesting four groups made them balanced. Compare the original missing-age count with both missing-code counts above; they agree, so no unknown age was converted into a known relative position.
NoteConfidence positions and reconstruction
Import the prepared extract in a fresh session:
library(tidyverse)
library(tidystata)
answer_gss24 <- readRDS("gss24.rds")answer_items_wide <- answer_gss24 |>
select(id, conbus, coneduc, conmedic)
answer_items_long <- answer_items_wide |>
pivot_longer(
cols = conbus:conmedic,
names_to = "item",
values_to = "response",
values_drop_na = FALSE
)
answer_items_long# A tibble: 11,958 × 3
id item response
<int> <chr> <fct>
1 1 conbus a great deal
2 1 coneduc a great deal
3 1 conmedic a great deal
4 2 conbus only some
5 2 coneduc only some
6 2 conmedic a great deal
7 3 conbus only some
8 3 coneduc a great deal
9 3 conmedic a great deal
10 4 conbus NA
# ℹ 11,948 more rows
nrow(answer_items_wide)[1] 3986nrow(answer_items_long)[1] 11958answer_items_long |>
count(item)# A tibble: 3 × 2
item n
<chr> <int>
1 conbus 3986
2 coneduc 3986
3 conmedic 3986
The wide table has 3986 respondents. Its three item columns become 11958 respondent-item rows, with 3986 rows per item. No respondents have been added. A missing response marks a retained position with no recorded answer, including items unavailable on a respondent’s questionnaire. Dropping those rows would describe answered positions rather than the complete set of selected respondent-item positions.
Inspect those answered positions explicitly:
answer_items_long |>
filter(!is.na(response)) |>
count(item)# A tibble: 3 × 2
item n
<chr> <int>
1 conbus 2588
2 coneduc 2614
3 conmedic 2610
Each count is the number of respondents with a recorded answer to that particular item. A percentage of all respondents uses the original wide-table row count; a percentage among recorded answers uses that item’s observed count. The total long-table row count pools three item positions per respondent and is neither denominator for a single item’s percentage. These questions ask one confidence response per institution, not a checkbox question where a respondent can select several options for one item.
Return to the source to verify the reverse reshape independently.
Import the prepared extract again so the answer does not depend on any earlier object:
library(tidyverse)
library(tidystata)
answer_gss24 <- readRDS("gss24.rds")answer_original <- answer_gss24 |>
select(id, conbus, coneduc, conmedic)
answer_long <- answer_original |>
pivot_longer(
cols = conbus:conmedic,
names_to = "item",
values_to = "response",
values_drop_na = FALSE
)
answer_original |>
count(id) |>
filter(is.na(id) | n != 1)# A tibble: 0 × 2
# ℹ 2 variables: id <int>, n <int>
answer_long |>
count(id, item) |>
filter(n != 1)# A tibble: 0 × 3
# ℹ 3 variables: id <int>, item <chr>, n <int>
nrow(answer_long)[1] 11958answer_restored <- answer_long |>
pivot_wider(
id_cols = id,
names_from = item,
values_from = response
)
answer_restored# A tibble: 3,986 × 4
id conbus coneduc conmedic
<int> <fct> <fct> <fct>
1 1 a great deal a great deal a great deal
2 2 only some only some a great deal
3 3 only some a great deal a great deal
4 4 NA NA NA
5 5 only some hardly any only some
6 6 hardly any only some only some
7 7 only some only some a great deal
8 8 hardly any hardly any hardly any
9 9 only some only some only some
10 10 NA NA NA
# ℹ 3,976 more rows
identical(answer_original, answer_restored)[1] TRUEThe key checks return zero rows and the long table has 11958 rows. The exact comparison returns TRUE, covering values, missingness, row and column order, and factor levels. A zero is not one of these factor responses and does not mean an unknown answer. Averaging duplicates would combine source records instead of restoring them. Either change would replace preservation with a new data decision, so investigate unexpected keys or missing values before making it.
NoteEducation duration and transition boundaries
Read the CSV again and gather its monthly columns. Keep all monthly records, so people who never appear in education still contribute to the result:
library(tidyverse)
answer_school_wide <- read_csv(
"examples/school-to-work.csv",
col_types = cols(
.default = col_character(), id = col_integer(), weight = col_double()
)
)
answer_education <- answer_school_wide |>
pivot_longer(
cols = -c(id, weight), names_to = "month_label", values_to = "state"
) |>
mutate(in_education = state %in% c("school", "FE", "HE")) |>
summarise(education_months = sum(in_education), .by = id)
answer_education# A tibble: 712 × 2
id education_months
<int> <int>
1 1 0
2 2 70
3 3 34
4 4 0
5 5 70
6 6 0
7 7 30
8 8 22
9 9 0
10 10 10
# ℹ 702 more rows
answer_education |>
summarise(
respondents = sum(!is.na(id)),
mean_months = mean(education_months),
zero_months = sum(education_months == 0)
)# A tibble: 1 × 3
respondents mean_months zero_months
<int> <dbl> <int>
1 712 26.2 216
The logical indicator is TRUE for the three selected states and FALSE for the others. sum() counts true months within each ID. The source has no missing monthly states; with incomplete histories, an unknown state would require an explicit coverage rule before calculating time in education. Every respondent remains represented, including those with zero education months. The mean describes months in the three education states during the 72-month window, averaged over all 712 respondents. It does not measure the number of qualifications or time in education outside that window.
The respondents count uses non-missing IDs: each row in answer_education represents one source respondent. Next rebuild the ordered histories to diagnose comparisons across people.
Start from the CSV, reshape it, and parse and sort the months before comparing states. lag(id) lets us identify comparisons that cross a person’s boundary even if the two recorded states happen to be identical:
library(tidyverse)
answer_history <- read_csv(
"examples/school-to-work.csv",
col_types = cols(
.default = col_character(), id = col_integer(), weight = col_double()
)
) |>
pivot_longer(
cols = -c(id, weight), names_to = "month_label", values_to = "state"
) |>
mutate(month = parse_date(
paste("01", month_label), format = "%d %b.%y",
locale = locale(date_names = "en")
)) |>
arrange(id, month)
answer_wrong <- answer_history |>
mutate(previous_id = lag(id), previous_state = lag(state))
answer_wrong |>
filter(!is.na(previous_id), id != previous_id) |>
count(name = "cross_person_comparisons")# A tibble: 1 × 1
cross_person_comparisons
<int>
1 711
answer_correct <- answer_history |>
mutate(previous_state = lag(state), .by = id)
answer_correct |>
summarise(first_months = sum(is.na(previous_state)))# A tibble: 1 × 1
first_months
<int>
1 712
answer_correct |>
filter(!is.na(previous_state)) |>
count(name = "adjacent_month_comparisons")# A tibble: 1 × 1
adjacent_month_comparisons
<int>
1 50552
The ungrouped calculation crosses 711 boundaries: the first month of every person after the first is paired with the preceding person’s final month. Grouping by ID restarts lag() for each history, producing 712 first-month NA values and 50,552 valid adjacent-month comparisons. These first-month values mean there is no earlier observation within this window. They are not missing answers in the recorded state column. Sorting establishes chronology; grouping establishes whose chronology we are following.