library(tidyverse)
library(tidystata)
gss24 <- readRDS("gss24.rds")21 Nested datasets
How can we keep a separate respondent table for each region while managing those tables together? A list column can contain whole tables as well as the word vectors introduced in Chapter 17. Nesting places several respondent tables inside one outer table. We can inspect them, repeat a calculation on each, and return to the original respondent rows when we need them.
By the end of this chapter, you will be able to:
- Nest respondent records by region and distinguish outer rows from inner rows.
- Extract and inspect a table stored in a list column.
- Apply the same row-count or coverage calculation to each inner table.
- Unnest the records and check that their values, identifiers, and factor levels survive.
Selection, column creation, and list columns are prerequisites. Chapter 19 explains calculations within groups; Chapter 20 explains identifier checks. We teach the mapping operations needed here without requiring the optional Programming chapter.
21.1 Prepare respondent records
We return to the prepared 2024 General Social Survey, or GSS. Follow the one-time preparation recipe in Appendix A and save gss24.rds in your project folder. The file retains respondents with missing answers. Load it with the packages used below:
Our application concerns the availability of schooling information across regions. educ records the respondent’s completed years of schooling; maeduc records the mother’s schooling or the younger parent’s schooling for same-sex parents, following the source definition. Both measurements are in years. Below, parent schooling refers to this specific maeduc measure. We need the respondent identifier and region alongside those two measurements. Select them into a new table, leaving gss24 unchanged, and inspect its first rows and dimensions:
schooling <- gss24 |>
select(id, region, educ, maeduc)
head(schooling)# A tibble: 6 × 4
id region educ maeduc
<int> <fct> <dbl> <dbl>
1 1 northeast 16 14
2 2 northeast 16 NA
3 3 northeast 14 12
4 4 northeast 12 NA
5 5 northeast 13 11
6 6 northeast 14 5
nrow(schooling)[1] 3986ncol(schooling)[1] 4isid(schooling, id)Variable `id` uniquely identifies the rows.
There are 3986 respondent rows and 4 columns. isid() confirms that id uniquely identifies rows and has no missing values. It returns the table invisibly without changing it; a repeated or missing identifier would instead raise an error. Keep these identifiers inside the records we will nest. A row total alone could not reveal one duplicated respondent replacing another.
Before making regional containers, check their membership. count() counts respondent rows in each region, including a missing category if one occurs. sum(is.na(region)) separately counts respondents whose region is unavailable:
schooling |>
count(region)# A tibble: 4 × 2
region n
<fct> <int>
1 northeast 616
2 midwest 923
3 south 1553
4 west 894
sum(is.na(schooling$region))[1] 0The extract has 4 observed regions and 0 missing region values. The regional counts describe this unweighted sample. We have not excluded respondents with missing schooling, because those respondents are part of the coverage question.
21.2 Put tables inside a column
nest() from tidyr takes a table and returns an outer table containing a list column of inner tables. .by = region identifies the column to keep outside: one outer row represents each observed region. The other columns move together into an inner table, stored in a list column named data:
nested_schooling <- schooling |>
nest(.by = region)
nested_schooling# A tibble: 4 × 2
region data
<fct> <list>
1 northeast <tibble [616 × 3]>
2 west <tibble [894 × 3]>
3 midwest <tibble [923 × 3]>
4 south <tibble [1,553 × 3]>
The result has 4 outer rows and two columns. region identifies the container. Each cell in data holds a tibble with respondent rows and the three columns id, educ, and maeduc. The printed <tibble [...]> abbreviation describes a stored table’s dimensions, rather than a text value. Read the row and column counts in this output for each region.
Nesting has changed the organization of the records. It has not replaced respondents with regional averages. All their IDs and schooling values remain inside the stored tables. This differs from a grouped summary, which returns calculated results and does not retain all the underlying respondent records in its output. It also differs from group_by(), which leaves respondent rows on the outside and attaches a grouping instruction.
The region factor keeps its existing levels. We neither recode the regions nor give them a new order. Outer rows follow the groups encountered in the input; use the region labels to identify them rather than assuming that a row position has a permanent geographic meaning. An unused factor level is a possible category, but has no respondent records to supply an inner table here.
If some respondents had an unknown region, nest(.by = region) would retain them together in an outer row with NA as its key. That container would represent unknown region, not a fifth geographic region. This extract has no such records. Missing educ or maeduc values remain in their respondents’ inner rows regardless of grouping.
21.3 Look inside one table
Printing the outer table shows its structure, but not the respondent values inside each cell. pull(data) extracts the whole data column as a list. Assignment gives that list a name we can inspect separately:
region_tables <- nested_schooling |>
pull(data)
region_tables[[1]]
# A tibble: 616 × 3
id educ maeduc
<int> <dbl> <dbl>
1 1 16 14
2 2 16 NA
3 3 14 12
4 4 12 NA
5 5 13 11
6 6 14 5
7 7 13 12
8 8 14 NA
9 9 14 0
10 10 12 NA
# ℹ 606 more rows
[[2]]
# A tibble: 894 × 3
id educ maeduc
<int> <dbl> <dbl>
1 196 14 20
2 197 12 NA
3 198 12 13
4 199 14 16
5 200 9 3
6 201 12 12
7 202 20 16
8 203 15 13
9 204 13 2
10 205 15 12
# ℹ 884 more rows
[[3]]
# A tibble: 923 × 3
id educ maeduc
<int> <dbl> <dbl>
1 329 12 8
2 330 12 12
3 331 12 12
4 332 12 12
5 333 12 12
6 334 18 3
7 335 12 12
8 336 18 8
9 337 15 13
10 338 14 NA
# ℹ 913 more rows
[[4]]
# A tibble: 1,553 × 3
id educ maeduc
<int> <dbl> <dbl>
1 432 16 12
2 433 13 12
3 434 16 12
4 435 14 12
5 436 17 15
6 437 13 5
7 438 17 15
8 439 16 9
9 440 12 12
10 441 16 12
# ℹ 1,543 more rows
The result is a list of 4 tibbles, in the same order as the outer rows. The region labels remain in nested_schooling; pulling only data does not bring that separate column along. Retain the outer table when those labels are needed.
Double square brackets extract one list component itself. In region_tables[[1]], 1 means the first component in the current order. It returns the stored tibble, which we can assign and preview with the familiar head(). The familiar slice(1) selects the first outer row so we can read that component’s region:
first_region <- region_tables[[1]]
head(first_region)# A tibble: 6 × 3
id educ maeduc
<int> <dbl> <dbl>
1 1 16 14
2 2 16 NA
3 3 14 12
4 4 12 NA
5 5 13 11
6 6 14 5
nrow(first_region)[1] 616nested_schooling |>
slice(1) |>
select(region)# A tibble: 1 × 1
region
<fct>
1 northeast
The preview now shows respondent IDs and schooling values, rather than a list-column abbreviation. There are 616 respondents inside this component. The final output identifies its region using the corresponding outer row. region is absent from the inner table because it already labels the container; the same region applies to every respondent stored there.
This extraction returns an object for inspection. It does not remove the component from region_tables or alter the source records. We can therefore return to nested_schooling to work on all regions together.
21.4 Repeat an operation
We want a respondent count beside each region. purrr::map_int() applies a function to every component of a list and collects one integer per component. Supply data as the list and nrow as the function. Write nrow without parentheses because map_int() supplies each inner table to it in turn. The _int suffix requires each call to return one integer, which a table’s row count does.
Inside mutate(), the resulting vector becomes a column aligned with the original outer rows:
region_sizes <- nested_schooling |>
mutate(respondents = purrr::map_int(data, nrow))
region_sizes# A tibble: 4 × 3
region data respondents
<fct> <list> <int>
1 northeast <tibble [616 × 3]> 616
2 west <tibble [894 × 3]> 894
3 midwest <tibble [923 × 3]> 923
4 south <tibble [1,553 × 3]> 1553
sum(region_sizes$respondents)[1] 3986The outer table still has one row per region, but the new column counts the respondents inside each cell. Its sum is 3986, matching the original respondent total. nrow(region_sizes) would instead count the regional containers. Whenever a workflow contains tables within tables, say which level a row count describes.
The same distinction matters for averages. The regions contain different numbers of respondents. Averaging four regional schooling means would give each region equal weight, whereas averaging observed respondent schooling gives each recorded respondent equal weight. Nesting does not choose a statistical weighting rule for us. Here we use it to organize records and check coverage, without estimating population differences.
21.5 Application: schooling coverage
We want the same small coverage report for every region: the number of respondents, how many have recorded schooling, and how many have recorded parent schooling. First count the rows of the already extracted first_region inside a familiar summary. n() supplies the number of rows in the current table or group. Unlike counting recorded schooling, it includes respondents with unknown schooling:
first_region |>
summarise(respondents = n())# A tibble: 1 × 1
respondents
<int>
1 616
This one-row result counts all 616 respondents in that inner table. n() is used inside table operations such as summarise(); nrow(first_region) remains the direct way to count a named table outside those operations. Here both count the same respondents.
Now give the repeated coverage calculation a name. function(x) defines a function with one input called x; braces enclose its instructions. The final expression is the returned result. Here that result is a one-row summary table:
schooling_coverage <- function(x) {
summarise(
x,
respondents = n(),
educ_recorded = sum(!is.na(educ)),
maeduc_recorded = sum(!is.na(maeduc))
)
}
schooling_coverage(first_region)# A tibble: 1 × 3
respondents educ_recorded maeduc_recorded
<int> <int> <int>
1 616 611 525
The function definition stores an operation; the call supplies one actual inner table as x. Test that call before repeating it. The three returned columns count all respondents, recorded respondent schooling, and recorded parent schooling. Each !is.na() condition is true for a recorded measurement, and sum() counts those true values. No respondent is removed and no unknown schooling value becomes zero.
21.5.1 Collect one numeric percentage per region
Sometimes we need just one number from each table. Define a second named function that returns the recorded-schooling percentage rather than a report table. $educ extracts the input table’s schooling column. The numerator counts recorded schooling and nrow(x) counts all respondents in that table:
schooling_percent <- function(x) {
100 * sum(!is.na(x$educ)) / nrow(x)
}
schooling_percent(first_region)[1] 99.18831This test returns one numeric percentage. Its denominator includes respondents with missing schooling. The inner tables here are nonempty observed regions, so their row counts are positive.
purrr::map_dbl() applies the named function to each list component and collects one numeric value per component in a double vector. Each call must return exactly one numeric value; a respondent-length vector or a multi-column report does not meet that contract. As with map_int(), omit parentheses after the function name because the mapper supplies each table:
region_percent <- nested_schooling |>
mutate(educ_percent = purrr::map_dbl(data, schooling_percent))
region_percent |>
select(region, educ_percent)# A tibble: 4 × 2
region educ_percent
<fct> <dbl>
1 northeast 99.2
2 west 99.2
3 midwest 99.6
4 south 98.8
The vector aligns with the outer rows, so each displayed region receives the percentage calculated from its own respondent table. These unweighted percentages describe coverage within this sample. They are not mean years of schooling or population estimates. map_int() fits integer row counts; map_dbl() fits numeric percentages, which need not be whole numbers.
21.5.2 Keep a complete coverage report
The schooling_coverage() result has several columns. To retain both schooling measures and their counts, collect whole report tables rather than a single percentage from each call. purrr::map() applies a function to each list component and returns a list of its results. As before, supply the function’s name without parentheses:
coverage_nested <- nested_schooling |>
mutate(coverage = purrr::map(data, schooling_coverage))
coverage_nested |>
select(region, coverage)# A tibble: 4 × 2
region coverage
<fct> <list>
1 northeast <tibble [1 × 3]>
2 west <tibble [1 × 3]>
3 midwest <tibble [1 × 3]>
4 south <tibble [1 × 3]>
Each coverage component holds a one-row report for the corresponding data component. The original respondent tables remain in coverage_nested$data. The displayed selection shows only the region and report containers, so we can inspect their arrangement before expanding them.
unnest() turns the rows and columns of stored tables back into ordinary table rows and columns. First select region and coverage, then ask unnest(coverage) to expand that list column. Every report has one row, so each region will still occupy one row:
coverage_report <- coverage_nested |>
select(region, coverage) |>
unnest(coverage)
coverage_report# A tibble: 4 × 4
region respondents educ_recorded maeduc_recorded
<fct> <int> <int> <int>
1 northeast 616 611 525
2 west 894 887 779
3 midwest 923 919 814
4 south 1553 1535 1327
We can now compare availability without opening each respondent table. respondents is the denominator for each region’s coverage; educ_recorded and maeduc_recorded count recorded values separately. A respondent can contribute to one count and not the other. These are not counts of respondents with both measurements.
For a compact comparison, turn the recorded counts into percentages of all respondents in that region:
coverage_percent <- coverage_report |>
mutate(
educ_percent = 100 * educ_recorded / respondents,
maeduc_percent = 100 * maeduc_recorded / respondents
) |>
select(region, respondents, educ_percent, maeduc_percent)
coverage_percent# A tibble: 4 × 4
region respondents educ_percent maeduc_percent
<fct> <int> <dbl> <dbl>
1 northeast 616 99.2 85.2
2 west 894 99.2 87.1
3 midwest 923 99.6 88.2
4 south 1553 98.8 85.4
The percentages include respondents with missing schooling in their denominators. Across regions, recorded respondent schooling ranges from 98.8 to 99.6 percent; recorded parent schooling ranges from 85.2 to 88.2 percent. These are unweighted descriptions of this extract’s coverage. Availability can reflect questionnaire coverage and nonresponse; it does not establish that every recorded value is valid or explain why an answer is missing.
21.6 Return to respondent rows
The coverage report is a summary. To recover the respondent records, return to nested_schooling and expand its original data column. The outer region is repeated alongside each respondent from its container. select() then restores our starting column order:
schooling_restored <- nested_schooling |>
unnest(data) |>
select(id, region, educ, maeduc)
head(schooling_restored)# A tibble: 6 × 4
id region educ maeduc
<int> <fct> <dbl> <dbl>
1 1 northeast 16 14
2 2 northeast 16 NA
3 3 northeast 14 12
4 4 northeast 12 NA
5 5 northeast 13 11
6 6 northeast 14 5
nrow(schooling_restored)[1] 3986ncol(schooling_restored)[1] 4isid(schooling_restored, id)Variable `id` uniquely identifies the rows.
We again have 3986 respondents and 4 columns. The ID check confirms unique, non-missing respondent identifiers. Nesting gathers each region’s records together, so expansion need not reproduce an input in which regions were interleaved. Matching dimensions and keys still do not prove that the measurements survived.
For an exact comparison, sort both tables by the unique ID. identical() checks values and attributes, including types and factor levels. We also inspect the region levels explicitly to connect that check to our categorical grouping key:
identical(
arrange(schooling, id),
arrange(schooling_restored, id)
)[1] TRUEidentical(levels(schooling$region), levels(schooling_restored$region))[1] TRUEBoth checks return TRUE. The first comparison includes every respondent’s ID, region, and both schooling values, including missing values. The second confirms that the same region levels remain in the same order. Sorting for comparison makes respondent order explicit; it does not repair altered values or missing respondents.
Our workflow has moved between two levels: regional containers on the outside and respondents on the inside. We kept the original records while adding separate reports, then verified the return to respondent rows. Use an ordinary grouped summary when only a regional report is needed. A nested table is useful when keeping and inspecting the underlying tables is part of the task.
21.7 Review
nest()stores respondent tables in a list column, with grouping keys on the outside.pull()extracts a list column; double square brackets extract one stored component.map_int()requires one integer per component;map_dbl()requires one numeric value per component.n()counts every row in the current summary table or group, including rows with missing measurements.function()defines a reusable operation;map()collects its results in a list.unnest()expands stored tables; the contents determine the resulting row unit.- Check dimensions, identifiers, values, and factor levels after a roundtrip, and state the respondent denominator of coverage results.
The next chapter, Chapter 22, returns to ordinary observation tables to build figures. Keep track of the row unit when carrying tables and summaries into statistical analysis.
21.8 Exercises
Each exercise starts from the package and file-loading setup above. Recreate any objects you need; none depends on another exercise’s answer.
- Count both levels. Select
id,region,educ, andmaeduc, then nest by region. Predict whatmap_int()returns before adding respondent counts. Compare the outer row count with the sum of inner row counts. What does each count represent? - Inspect one component. Recreate the nested table. Predict the object returned by pulling
dataand by extracting its first component, then perform both operations. Show its first respondents and identify its region from the outer table. Why isregionabsent from the component, and why should you not assume that component 1 always names the same region? - Report coverage and diagnose output shape. Recreate the nested table and define a numeric schooling-percentage function and
schooling_coverage()locally. Test each on one inner table before mapping. Predict the double-vector result frommap_dbl(), the list frommap(), and the expanded report fromunnest(). Retain both measures and calculate recorded parent-schooling percentages. Diagnose why mapping a function that returns a recorded-answer flag for every respondent cannot produce one numeric value per region, and repair it. State each denominator. Would nesting discard missing schooling, and what would an unknown region mean? - Check a roundtrip. Recreate the selected and nested tables. Predict the row unit and row count after expanding
data, then unnest the original records and restore the selected column order. Check dimensions, unique IDs, exact equality after sorting by ID, and region levels. Explain why a correct row count is insufficient by itself.
Worked answers
NoteCount both levels
Load the extract independently, retain the four requested columns, and count the respondents inside each regional container:
library(tidyverse)
library(tidystata)
gss24 <- readRDS("gss24.rds")answer1_schooling <- gss24 |>
select(id, region, educ, maeduc)
answer1_nested <- answer1_schooling |>
nest(.by = region) |>
mutate(respondents = purrr::map_int(data, nrow))
answer1_nested# A tibble: 4 × 3
region data respondents
<fct> <list> <int>
1 northeast <tibble [616 × 3]> 616
2 west <tibble [894 × 3]> 894
3 midwest <tibble [923 × 3]> 923
4 south <tibble [1,553 × 3]> 1553
nrow(answer1_nested)[1] 4sum(answer1_nested$respondents)[1] 3986sum(answer1_nested$respondents) == nrow(answer1_schooling)[1] TRUEmap_int() returns an integer vector with one respondent count per stored table; assignment aligns it with the regions. The outer count is 4 regional containers. The inner counts sum to 3986 respondents, matching the selected input, so the equality check returns TRUE. Four containers do not imply four respondents or equally sized regional samples.
NoteInspect one component
Rebuild the nested table before extracting its first component. Use the corresponding outer row to identify the region:
library(tidyverse)
library(tidystata)
gss24 <- readRDS("gss24.rds")answer2_nested <- gss24 |>
select(id, region, educ, maeduc) |>
nest(.by = region)
answer2_tables <- answer2_nested |>
pull(data)
answer2_first <- answer2_tables[[1]]
head(answer2_first)# A tibble: 6 × 3
id educ maeduc
<int> <dbl> <dbl>
1 1 16 14
2 2 16 NA
3 3 14 12
4 4 12 NA
5 5 13 11
6 6 14 5
answer2_nested |>
slice(1) |>
select(region)# A tibble: 1 × 1
region
<fct>
1 northeast
Pulling returns a list; double brackets return the tibble stored in one component. The inner rows are respondents; the outer row identifies their common region. region remains outside because it is the nesting key. Component 1 is a position in this particular list, whose order follows encountered groups. A differently ordered input could place another region first. Neither extraction changes the stored records.
NoteReport coverage
Define the coverage operation independently, then repeat it for the freshly nested respondent records:
library(tidyverse)
library(tidystata)
gss24 <- readRDS("gss24.rds")answer3_coverage <- function(x) {
summarise(
x,
respondents = n(),
educ_recorded = sum(!is.na(educ)),
maeduc_recorded = sum(!is.na(maeduc))
)
}
answer3_schooling <- gss24 |>
select(id, region, educ, maeduc)
answer3_nested <- answer3_schooling |>
nest(.by = region)
answer3_tables <- answer3_nested |>
pull(data)
answer3_first <- answer3_tables[[1]]
answer3_coverage(answer3_first)# A tibble: 1 × 3
respondents educ_recorded maeduc_recorded
<int> <int> <int>
1 616 611 525
answer3_report <- answer3_nested |>
mutate(coverage = purrr::map(data, answer3_coverage)) |>
select(region, coverage) |>
unnest(coverage) |>
mutate(maeduc_percent = 100 * maeduc_recorded / respondents)
answer3_report# A tibble: 4 × 5
region respondents educ_recorded maeduc_recorded maeduc_percent
<fct> <int> <int> <int> <dbl>
1 northeast 616 611 525 85.2
2 west 894 887 779 87.1
3 midwest 923 919 814 88.2
4 south 1553 1535 1327 85.4
Each percentage divides the number with recorded parent schooling by all respondents in that region. Here parent schooling means the source-defined mother’s or younger parent’s schooling in maeduc. Missing schooling stays in the respondent records and contributes to the denominator. If region were unknown, nesting would retain those respondents in a separate NA container. This extract has no missing region values; no extra geographic category has been created. These coverage percentages describe the sample without survey weights.
The function test returns a one-row report. Mapping it returns a list of those report tables; expanding them gives one region per row because each component has one row. Now test and collect a scalar percentage:
answer3_percent <- function(x) {
100 * sum(!is.na(x$educ)) / nrow(x)
}
answer3_percent(answer3_first)[1] 99.18831answer3_numeric <- answer3_nested |>
mutate(educ_percent = purrr::map_dbl(data, answer3_percent))
answer3_numeric |>
select(region, educ_percent)# A tibble: 4 × 2
region educ_percent
<fct> <dbl>
1 northeast 99.2
2 west 99.2
3 midwest 99.6
4 south 98.8
map_dbl() returns one double value per region, stored in educ_percent. This is the same recorded-respondent-schooling percentage as the report’s educ_recorded / respondents, multiplied by 100. It does not replace the report’s separate parent-schooling coverage.
For the diagnosis, define an operation returning one logical flag for every respondent:
answer3_flags <- function(x) {
!is.na(x$educ)
}
length(answer3_flags(answer3_first))[1] 616nrow(answer3_first)[1] 616The matching lengths show why this function does not return one value per region. The following deliberate error tests that mismatch:
purrr::map_dbl(answer3_tables, answer3_flags)Error in `purrr::map_dbl()`:
ℹ In index: 1.
Caused by error:
! Result must be length 1, not 616.
Each component result must have length one; a vector containing a flag for every respondent is too long. The repair is the tested answer3_percent() above: count the true flags and divide by that region’s respondent count before mapping. Use map() if the purpose is instead to retain the full flag vectors. Changing to map() alone would change the output shape without answering the one-percentage-per-region question.
NoteCheck a roundtrip
Use only the original data list column for the return to respondent rows. Then compare the recovered table with its selected input:
library(tidyverse)
library(tidystata)
gss24 <- readRDS("gss24.rds")answer4_schooling <- gss24 |>
select(id, region, educ, maeduc)
answer4_nested <- answer4_schooling |>
nest(.by = region)
answer4_restored <- answer4_nested |>
unnest(data) |>
select(id, region, educ, maeduc)
nrow(answer4_schooling) == nrow(answer4_restored)[1] TRUEncol(answer4_schooling) == ncol(answer4_restored)[1] TRUEisid(answer4_schooling, id)Variable `id` uniquely identifies the rows.
isid(answer4_restored, id)Variable `id` uniquely identifies the rows.
identical(
arrange(answer4_schooling, id),
arrange(answer4_restored, id)
)[1] TRUEidentical(
levels(answer4_schooling$region),
levels(answer4_restored$region)
)[1] TRUEExpanding data returns respondent rows: their predicted total is the sum of the inner counts, not the number of regions. The equality checks return TRUE, and both ID checks confirm unique, non-missing identifiers. Dimensions alone could hide a duplicated respondent replacing another or a changed schooling value. The sorted exact comparison checks records and their attributes, including missing values; the final comparison explicitly checks the region levels. We restore the selected table, rather than the other columns that were never included in nesting. gss24 remains the full original extract.