library(tidyverse)
library(tidystata)
gss24 <- readRDS("gss24.rds")15 Strings and factors
Survey answers such as marital status and confidence in institutions describe categories. A table needs readable category names and a useful display order. A questionnaire score needs a separate decision about which numbers represent which answers. This chapter distinguishes those tasks so that changing a label does not silently change an analysis.
By the end of this chapter, you will be able to:
- Create a factor and inspect its category levels.
- Search and transform text while retaining recorded names and codes.
- Change category order without rearranging respondents.
- Relabel or combine categories while preserving missing answers.
- Encode answers as labelled numeric scores, count available answers, and state a rule for combining items.
We use the prepared 2024 GSS extract, the column-creation method from Chapter 12, and the numeric recoding from Chapter 14. Frequency tables from Chapter 8 help us check each change. The GSS examples describe the sample without survey weights. A separate, contiguous text application reuses the captured aggregate country data because its names are character fields.
Prepare the extract following Appendix A, then restore the saved file in your project folder:
15.1 Create factors
A factor represents categories with a defined set of possible values, called levels. A character column stores text; a factor also records which categories are allowed and their order. The prepared GSS already uses factors for categorical answers.
First, recreate the numeric age groups from Section 14.4. This chapter starts from the saved extract rather than depending on objects left in your session:
people <- gss24 |>
select(id, age)
age_groups <- people |>
mutate(age_group = stata_recode(
age,
18:34 ~ 1,
35:64 ~ 2,
65:max ~ 3,
.default = NA
))The codes 1, 2, and 3 identify age bands, not ages in years. Now give these codes readable category labels by making a factor. The levels argument lists the codes in display order; labels supplies their corresponding text. Here c() combines each set of three values:
gss_ages <- age_groups |>
mutate(age_group = factor(
age_group, levels = c(1, 2, 3), labels = c("18-34", "35-64", "65+")
))
head(gss_ages)# A tibble: 6 × 3
id age age_group
<int> <dbl> <fct>
1 1 33 18-34
2 2 64 35-64
3 3 69 65+
4 4 19 18-34
5 5 70 65+
6 6 53 35-64
nrow(gss_ages)[1] 3986factor() returns the categorical values. All 3986 respondents remain in their original order. Changing a level order does not move rows or change anyone’s age. A missing age remains missing rather than becoming a fourth category.
levels() receives a factor and returns its category names in level order. Extract the new factor with the familiar $:
levels(gss_ages$age_group)[1] "18-34" "35-64" "65+" The three labels appear in the order we supplied. This list describes the factor’s categories, rather than the order in which respondents appear.
An explicitly supplied level list must cover the known input codes: a non-missing value absent from that list becomes missing. Check the codes and the correspondence between levels and labels. The order controls display and sorting; it does not establish equal distances between categories or turn category codes into years of schooling.
To see the categories alongside their frequencies, use the familiar tab():
gss_ages |>
tab(age_group)# A tibble: 4 × 6
age_group n percent cum valid valid_cum
<fct> <int> <dbl> <dbl> <dbl> <dbl>
1 18-34 983 24.7 24.7 25.5 25.5
2 35-64 1964 49.3 73.9 50.9 76.4
3 65+ 912 22.9 96.8 23.6 100
4 NA 127 3.19 100 NA NA
The age bands appear in their level order. The missing row represents unavailable ages; it is not an age band. Cumulative percentages follow the displayed order, so that order should make substantive sense.
15.2 Change category order
Suppose a presentation needs the oldest age band first. fct_relevel() from forcats, loaded by tidyverse, moves the named levels to the beginning in the order supplied. Levels not named follow in their existing order. Save its result in a new column so that the original ordering remains available:
age_display <- gss_ages |>
mutate(age_group_oldest_first = fct_relevel(age_group, "65+", "35-64"))
levels(age_display$age_group_oldest_first)[1] "65+" "35-64" "18-34"age_display |>
tab(age_group_oldest_first)# A tibble: 4 × 6
age_group_oldest_first n percent cum valid valid_cum
<fct> <int> <dbl> <dbl> <dbl> <dbl>
1 65+ 912 22.9 22.9 23.6 23.6
2 35-64 1964 49.3 72.2 50.9 74.5
3 18-34 983 24.7 96.8 25.5 100
4 NA 127 3.19 100 NA NA
The new order is 65+, 35-64, then 18-34. Each band still contains the same respondents, and the rows of age_display remain in their original order. arrange() sorts whole observations; fct_relevel() changes category order within a column. A factor with deliberately arranged levels is still an ordinary factor. R also has an ordered-factor class, which has additional consequences for comparisons and modeling; changing the display order here does not create that class.
15.3 Relabel and combine categories
A publication may need a fuller label than “65+”. fct_recode() changes level names. Write the new name on the left of = and the existing name on the right. Levels not named remain unchanged:
age_labels <- gss_ages |>
mutate(age_group_label = fct_recode(age_group, "65 and older" = "65+"))
age_labels |>
tab(age_group_label)# A tibble: 4 × 6
age_group_label n percent cum valid valid_cum
<fct> <int> <dbl> <dbl> <dbl> <dbl>
1 18-34 983 24.7 24.7 25.5 25.5
2 35-64 1964 49.3 73.9 50.9 76.4
3 65 and older 912 22.9 96.8 23.6 100
4 NA 127 3.19 100 NA NA
Only the label changes. The age-band membership and frequencies agree with the original factor, including its missing answers. A clearer label does not create a more precise measurement.
Combining categories changes their membership. fct_collapse() takes a new category name on the left and the old levels to combine on the right. Here c() supplies the two younger age bands:
age_bands <- gss_ages |>
mutate(age_band = fct_collapse(age_group, "Under 65" = c("18-34", "35-64")))
age_bands |>
tab(age_band)# A tibble: 3 × 6
age_band n percent cum valid valid_cum
<fct> <int> <dbl> <dbl> <dbl> <dbl>
1 Under 65 2947 73.9 73.9 76.4 76.4
2 65+ 912 22.9 96.8 23.6 100
3 NA 127 3.19 100 NA NA
“Under 65” now includes everyone previously in either younger band; the unnamed “65+” level remains. Unknown ages remain missing. We retain the original age_group column because the broader grouping discards the distinction between the two younger bands. Choose categories for the research question, and avoid combining substantively different answers merely to shorten a table.
15.4 Combine questionnaire items
The GSS asks about confidence in major companies, education, and medicine. We use these three items to demonstrate a score, without claiming they form a validated scale. A common spelling in variable names does not establish that questions belong together. Select these items because their meanings suit the teaching question:
confidence <- gss24 |>
select(id, conbus, coneduc, conmedic)
confidence# A tibble: 3,986 × 4
id conbus coneduc conmedic
<int> <fct> <fct> <fct>
1 1 a great deal a great deal a great deal
2 2 only some only some a great deal
3 3 only some a great deal a great deal
4 4 NA NA NA
5 5 only some hardly any only some
6 6 hardly any only some only some
7 7 only some only some a great deal
8 8 hardly any hardly any hardly any
9 9 only some only some only some
10 10 NA NA NA
# ℹ 3,976 more rows
The answers are factors. Define numeric scores explicitly: hardly any confidence is 1, only some is 2, and a great deal is 3. Higher values then mean greater confidence. Store the category names in the intended order, and apply that order to one item before converting any scores:
score_levels <- c("hardly any", "only some", "a great deal")
business_ordered <- factor(confidence$conbus, levels = score_levels)
levels(business_ordered)[1] "hardly any" "only some" "a great deal"The result lists hardly any, only some, then a great deal. The original conbus factor puts a great deal first, so its original order does not match the score direction we chose. The level list must cover all recorded answers: a value absent from it would become missing in factor().
Tidystata’s encode() receives a factor and returns labelled integer codes. The first supplied level receives 1, the second 2, and the third 3; the level text remains attached as value labels. It uses the factor order rather than choosing a score direction. Encode just the ordered business item:
business_score <- encode(business_ordered)
head(business_score)<labelled<integer>[6]>
[1] 3 2 2 NA 2 1
Labels:
value label
1 hardly any
2 only some
3 a great dealRecorded answers now have numeric scores with their meanings attached. Unknown answers remain missing. Neither assignment changes the original factor in confidence or gss24.
Apply these checked ordering and encoding steps separately to all three items:
confidence_scores <- confidence |>
mutate(
conbus = encode(factor(conbus, levels = score_levels)),
coneduc = encode(factor(coneduc, levels = score_levels)),
conmedic = encode(factor(conmedic, levels = score_levels))
)
head(confidence_scores)# A tibble: 6 × 4
id conbus coneduc conmedic
<int> <int+lbl> <int+lbl> <int+lbl>
1 1 3 [a great deal] 3 [a great deal] 3 [a great deal]
2 2 2 [only some] 2 [only some] 3 [a great deal]
3 3 2 [only some] 3 [a great deal] 3 [a great deal]
4 4 NA NA NA
5 5 2 [only some] 1 [hardly any] 2 [only some]
6 6 1 [hardly any] 2 [only some] 2 [only some]
The returned table keeps the same respondents and replaces the three factors with their scores. The source factors remain in confidence. Some questions were not asked on every ballot, so an unavailable answer need not indicate refusal.
15.4.1 Apply the same scoring rule to the battery
The three calls above use the same level order and conversion. After checking them separately, we can use across() from Chapter 13 to express the shared rule once:
confidence_scores_block <- confidence |>
mutate(across(
conbus:conmedic,
\(x) encode(factor(x, levels = score_levels))
))
identical(confidence_scores_block, confidence_scores)[1] TRUEAs in Section 13.2.1, \(x) supplies an unnamed function. For each selected column, across() supplies that column as x; the function orders its categories and encodes the result. It returns one score per original response. The comparison returns TRUE, including values, missingness, and value labels. The scores replace columns in the returned scoring table, while confidence retains the original answers. The common rule is justified by the three items’ shared response scale, not merely by their names. Do not reuse it for a battery containing differently worded or reverse-scored items without inspecting those definitions.
15.4.2 Require complete responses
Use row_non_miss() from Chapter 13 to count recorded answers before calculating a score. As before, count just the three items and exclude id:
confidence_coverage <- confidence_scores |>
mutate(n_answered = row_non_miss(conbus, coneduc, conmedic))
tab(confidence_coverage, n_answered)# A tibble: 4 × 6
n_answered n percent cum valid valid_cum
<int> <int> <dbl> <dbl> <dbl> <dbl>
1 0 1351 33.9 33.9 33.9 33.9
2 1 10 0.251 34.1 0.251 34.1
3 2 73 1.83 36.0 1.83 36.0
4 3 2552 64.0 100 64.0 100
Every respondent has a count from 0 through 3; 3 identifies complete responses. Zero available answers is not a confidence score of zero and does not establish why answers are unavailable. The scores and source respondents remain unchanged by this count.
For this demonstration, require all three answers. Tidystata’s row_mean() calculates one arithmetic mean per respondent across the supplied columns. It omits missing answers by default; na.rm = FALSE instead makes the score missing whenever any of the three answers is missing:
scored_confidence <- confidence_coverage |>
mutate(mean_score = row_mean(conbus, coneduc, conmedic, na.rm = FALSE))
scored_confidence |>
select(id, n_answered, mean_score) |>
head()# A tibble: 6 × 3
id n_answered mean_score
<int> <int> <dbl>
1 1 3 3
2 2 3 2.33
3 3 3 2.67
4 4 0 NA
5 5 3 1.67
6 6 3 1.67
The function combines values belonging to the same respondent, giving each item equal weight. With na.rm = FALSE, a missing item makes the score missing; we do not average two answers as though they were three. Complete scores run from 1 to 3. This complete-response rule is a teaching choice, not a recommendation for a validated confidence scale. The calculation neither fills source answers nor establishes equal spacing or measurement validity.
15.5 Work with recorded text
The GSS factor examples above concern fixed response categories. To work with names stored as character text, we reuse the captured Our World in Data (OWID) life-expectancy file from Chapter 7. Its rows describe aggregate entity-year statistics, not workshop participants. Keep this text application together; it does not change the source of the GSS scoring lesson.
Copy the book’s captured life-expectancy CSV into an examples folder inside your project. read_csv() returns a tibble and reports the types it inferred. We retain the original column names and measurements:
owid <- read_csv("examples/life-expectancy.csv")Rows: 21565 Columns: 4
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (2): Entity, Code
dbl (2): Year, Life expectancy
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
owid# A tibble: 21,565 × 4
Entity Code Year `Life expectancy`
<chr> <chr> <dbl> <dbl>
1 Afghanistan AFG 1950 28.2
2 Afghanistan AFG 1951 28.6
3 Afghanistan AFG 1952 29.0
4 Afghanistan AFG 1953 29.5
5 Afghanistan AFG 1954 29.7
6 Afghanistan AFG 1955 30.4
7 Afghanistan AFG 1956 30.8
8 Afghanistan AFG 1957 31.3
9 Afghanistan AFG 1958 31.8
10 Afghanistan AFG 1959 32.3
# ℹ 21,555 more rows
Each row records an entity in a year. Entity is the displayed name, Code is an identifying text field, and Year records the year. Some entities are aggregates or territories. The space in Life expectancy requires backticks when naming that column in code. The stringr functions below come from tidyverse, already loaded above.
15.6 Count characters
nchar() receives text and returns the number of characters inside its quotation marks:
text <- "hello"
nchar(text)[1] 5nchar("01")[1] 2The results are 5 and 2. Both the zero and the one in "01" count; quotation marks do not. Counting characters neither converts the text to numbers nor changes the original string.
As with arithmetic in Section 14.3, text operations act element by element. c() combines strings in their written order:
words <- c("one", "two", "three")
var_type(words)[1] "character"nchar(words)[1] 3 3 5paste("Word:", words)[1] "Word: one" "Word: two" "Word: three"The type is "character", and the character counts are 3, 3, and 5. The familiar paste() returns "Word: one", "Word: two", and "Word: three". Its single prefix applies to every input value, following the single-value repetition rule from Section 14.3. Text operations on a named column likewise return one result per row.
15.7 Prepare names for searching
Chapter 12 introduced numeric text, explicit conversion, paste(), spacing, and lowercase search forms. Those tools return transformed values without replacing the source unless assigned over it. Here we apply them to recorded country names. A description remains different from an identifier, and joined text can still conceal missing source components.
First select one year so that a name search does not repeat each entity across many years. Use the familiar str_squish() to supply a spacing rule in a separate column:
names_2023 <- owid |>
filter(Year == 2023) |>
select(Entity, Code, Year) |>
mutate(entity_clean = str_squish(Entity))This applies the rule without asserting that every recorded name needs correction. Now create a lowercase search form from that result:
names_2023 <- names_2023 |>
mutate(entity_lower = str_to_lower(entity_clean))
names_2023# A tibble: 261 × 5
Entity Code Year entity_clean entity_lower
<chr> <chr> <dbl> <chr> <chr>
1 Afghanistan AFG 2023 Afghanistan afghanistan
2 Africa OWID_AFR 2023 Africa africa
3 Albania ALB 2023 Albania albania
4 Algeria DZA 2023 Algeria algeria
5 American Samoa ASM 2023 American Samoa american samoa
6 Americas NA 2023 Americas americas
7 Andorra AND 2023 Andorra andorra
8 Angola AGO 2023 Angola angola
9 Anguilla AIA 2023 Anguilla anguilla
10 Antigua and Barbuda ATG 2023 Antigua and Barbuda antigua and barbuda
# ℹ 251 more rows
The new column provides a consistent search form; Entity remains suitable for display. Code remains text because its role is identification rather than arithmetic. Changing a code’s storage to a number can discard meaningful leading zeros in other datasets. Neither a digit-like appearance nor a short code makes a field a quantitative measurement.
Lowercasing is a specific transformation, not a way to reconcile different naming conventions. It cannot establish that two differently named records refer to the same entity. If names are used to combine datasets, inspect unmatched records and use an appropriate checked identifier, following Chapter 20.
15.8 Search names
Which names contain the word fragment United? Begin by defining what we want to match. fixed() constructs a literal search pattern, whose characters will be matched as written rather than interpreted as special pattern symbols:
united_pattern <- fixed("United")This saves a case-sensitive pattern; it has not yet searched any names. str_detect() receives strings and a pattern, then returns one logical result per string. Inspect that result before using it to select rows:
head(str_detect(names_2023$Entity, united_pattern))[1] FALSE FALSE FALSE FALSE FALSE FALSEEach displayed TRUE or FALSE refers to the corresponding entity name. Detection returns a result without changing those names. Inside the familiar filter(), its TRUE values retain matching rows:
united_names <- names_2023 |>
filter(str_detect(Entity, united_pattern))
united_names# A tibble: 4 × 5
Entity Code Year entity_clean entity_lower
<chr> <chr> <dbl> <chr> <chr>
1 United Arab Emirates ARE 2023 United Arab Emirates united arab…
2 United Kingdom GBR 2023 United Kingdom united king…
3 United States USA 2023 United States united stat…
4 United States Virgin Islands VIR 2023 United States Virgin Is… united stat…
The result contains 4 entity names in the selected year. Read the entire names: a shared fragment does not imply that the entities describe the same population. The original Entity, Code, and Year remain available to identify each match.
We can search the normalized column instead. The lowercase pattern must match its case:
names_2023 |>
filter(str_detect(entity_lower, fixed("united")))# A tibble: 4 × 5
Entity Code Year entity_clean entity_lower
<chr> <chr> <dbl> <chr> <chr>
1 United Arab Emirates ARE 2023 United Arab Emirates united arab…
2 United Kingdom GBR 2023 United Kingdom united king…
3 United States USA 2023 United States united stat…
4 United States Virgin Islands VIR 2023 United States Virgin Is… united stat…
This returns the same entities. We chose a consistent form for searching without changing the displayed names. A missing string would give an unknown detection result, NA; it would not prove that the pattern was absent. As in earlier filtering lessons, filter() retains only TRUE. Check unknown source names separately if they occur in the data you analyze, rather than replacing unknown matches with FALSE.
15.8.1 Match a position
A regular expression is a pattern whose symbols can describe positions or groups of characters. We need only a few symbols here. ^ means the start of a string, and $ means its end. Without fixed(), stringr interprets these as pattern instructions. Compare a name starting with United with a name ending in States:
starts_united <- names_2023 |>
filter(str_detect(Entity, "^United"))
ends_states <- names_2023 |>
filter(str_detect(Entity, "States$"))
starts_united# A tibble: 4 × 5
Entity Code Year entity_clean entity_lower
<chr> <chr> <dbl> <chr> <chr>
1 United Arab Emirates ARE 2023 United Arab Emirates united arab…
2 United Kingdom GBR 2023 United Kingdom united king…
3 United States USA 2023 United States united stat…
4 United States Virgin Islands VIR 2023 United States Virgin Is… united stat…
ends_states# A tibble: 1 × 5
Entity Code Year entity_clean entity_lower
<chr> <chr> <dbl> <chr> <chr>
1 United States USA 2023 United States united states
The first result has 4 rows; the second has 1. In particular, a name containing United States can continue with other words and therefore fail the end-of-string test. These differences are consequences of the search rule, not discrepancies in the records.
Choose literal matching when you want particular characters anywhere in a name. Add a position rule only when position is part of the question. Placing ^ inside fixed() would instead search for an actual caret character. Text tools follow the instructions we supply; they cannot infer the population we meant to select.
15.9 Replace and remove text
Text changes can make a search form or display label useful. str_replace_all() receives text, a search pattern, and replacement text. It returns a string with every match replaced; supplying fixed() makes the match literal. Shorten “and” to an ampersand only in a separate display column:
name_versions <- names_2023 |>
mutate(entity_display = str_replace_all(Entity, fixed(" and "), " & "))
name_versions |>
filter(Entity != entity_display)# A tibble: 11 × 6
Entity Code Year entity_clean entity_lower entity_display
<chr> <chr> <dbl> <chr> <chr> <chr>
1 Antigua and Barbuda ATG 2023 Antigua and… antigua and… Antigua & Bar…
2 Bonaire Sint Eustatius … BES 2023 Bonaire Sin… bonaire sin… Bonaire Sint …
3 Bosnia and Herzegovina BIH 2023 Bosnia and … bosnia and … Bosnia & Herz…
4 Latin America and the C… NA 2023 Latin Ameri… latin ameri… Latin America…
5 Saint Kitts and Nevis KNA 2023 Saint Kitts… saint kitts… Saint Kitts &…
6 Saint Pierre and Miquel… SPM 2023 Saint Pierr… saint pierr… Saint Pierre …
7 Saint Vincent and the G… VCT 2023 Saint Vince… saint vince… Saint Vincent…
8 Sao Tome and Principe STP 2023 Sao Tome an… sao tome an… Sao Tome & Pr…
9 Trinidad and Tobago TTO 2023 Trinidad an… trinidad an… Trinidad & To…
10 Turks and Caicos Islands TCA 2023 Turks and C… turks and c… Turks & Caico…
11 Wallis and Futuna WLF 2023 Wallis and … wallis and … Wallis & Futu…
The display column changes names containing the exact spaced word. The original Entity remains. str_remove() instead removes the first match and needs no replacement argument. Remove the literal qualifier ” (country)” only in a separate search column:
name_versions <- name_versions |>
mutate(entity_search = str_remove(Entity, fixed(" (country)")))
name_versions |>
filter(Entity != entity_search)# A tibble: 1 × 7
Entity Code Year entity_clean entity_lower entity_display entity_search
<chr> <chr> <dbl> <chr> <chr> <chr> <chr>
1 Micronesia… FSM 2023 Micronesia … micronesia … Micronesia (c… Micronesia
The output displays the affected names beside their originals. “Trinidad and Tobago” becomes “Trinidad & Tobago” only in the display form. “Micronesia (country)” loses its qualifier only in the search form. These changes do not establish that shortened names are unique or identify the same geography as another record. Preserve Entity and Code; a convenience label is not automatically a joining key.
We can apply one shared text operation to several columns with the across() method from Section 13.2. Lowercase versions of both names and codes allow lowercase search queries. Choose these two character columns explicitly and retain their original spelling:
name_searches <- names_2023 |>
mutate(across(c(Entity, Code), str_to_lower, .names = "{.col}_search"))
name_searches |>
filter(str_detect(Entity_search, fixed("united")) | Code_search == "usa")# A tibble: 4 × 7
Entity Code Year entity_clean entity_lower Entity_search Code_search
<chr> <chr> <dbl> <chr> <chr> <chr> <chr>
1 United Arab E… ARE 2023 United Arab… united arab… united arab … are
2 United Kingdom GBR 2023 United King… united king… united kingd… gbr
3 United States USA 2023 United Stat… united stat… united states usa
4 United States… VIR 2023 United Stat… united stat… united state… vir
Entity_search and Code_search are derived search forms. The recorded codes remain uppercase in Code, and a missing code stays missing. Lowercasing already consistent text may leave it unchanged; do not describe that as repairing an observed error. Apply an operation across columns only when their meanings support the same rule. We do not apply text cleaning to the numeric year or the life-expectancy measurement.
15.10 Review
factor()defines categories and their level order;levels()lists those levels.fct_relevel()changes category order without rearranging respondents.fct_recode()changes labels;fct_collapse()combines categories.encode()converts explicitly ordered factor levels to labelled numeric codes; choose score direction before encoding.row_mean()combines numeric items using an explicit missing-answer rule; count available items before interpreting the score.across()expresses a shared conversion after individual checks; response definitions still determine whether the rule is appropriate.str_detect(),fixed(), and anchors control searches;str_replace_all()andstr_remove()produce derived text forms.- Preserve original categories and unknown answers; joined text can conceal missing inputs. Category codes do not establish equal spacing, and a calculated score is not automatically a validated scale.
The next chapter, Chapter 16, applies the same practice of retaining source values to dates and calendar components. Chapter 17 later splits these recorded entity names into word lists. Later, Chapter 19 uses categories to calculate within groups and reshape questionnaire items. The optional programming part explains values, vectors, and other R data structures in more detail; return to it when you need to examine the mechanics behind these table operations.
15.11 Exercises
- Create the age bands, then make them display in reverse order. In separate new columns, relabel “18-34” as “Ages 18 to 34” and combine “35-64” with “65+” as “35 and older”. Which operation changes membership, which loses information, and which original columns should remain? Explain what happens to row order and missing ages.
- Encode the three confidence items with higher scores for greater confidence. Check that separate and block conversions agree. Count recorded answers and calculate a mean requiring all three. Inspect answer counts beside missing scores. Does zero recorded answers mean zero confidence? What would encoding the original factor order do to score direction?
- Clean the abstract string
" A short phrase "and search the captured OWID names in 2023 for lowercaseunited, retaining names and codes. Create separate display names replacing ” and ” with ” & “, remove” (country)” in a search form, and add lowercase name and code columns withacross(). Explain which field to display, which originals to preserve, and why the derived labels cannot be assumed to identify entities uniquely. - Compare the literal search
fixed("United"), the beginning rule"^United", and the ending rule"States$"in 2023. Predict why containingUnited Statesdoes not guarantee an ending match. What wouldfixed("^United")search for? Explain what detection and filtering would do with an unavailable name.
Worked answers
NoteOrder, labels, and membership
Prepare the extract following Appendix A, then restore the saved file in your project folder:
library(tidyverse)
library(tidystata)
gss24 <- readRDS("gss24.rds")people <- gss24 |>
select(id, age)
result <- people |>
mutate(age_group = stata_recode(
age,
18:34 ~ 1,
35:64 ~ 2,
65:max ~ 3,
.default = NA
)) |>
mutate(age_group = factor(
age_group, levels = c(3, 2, 1), labels = c("65+", "35-64", "18-34")
))
levels(result$age_group)[1] "65+" "35-64" "18-34"head(result)# A tibble: 6 × 3
id age age_group
<int> <dbl> <fct>
1 1 33 18-34
2 2 64 35-64
3 3 69 65+
4 4 19 18-34
5 5 70 65+
6 6 53 35-64
The level list now starts with code 3, labelled 65+. The rows remain in their original respondent order, recorded ages are unchanged, and unknown ages remain missing. A display order and a row order are different things.
Relabel a separate column without changing membership:
result <- result |>
mutate(age_group_label = fct_recode(age_group, "Ages 18 to 34" = "18-34"))
tab(result, age_group_label)# A tibble: 4 × 6
age_group_label n percent cum valid valid_cum
<fct> <int> <dbl> <dbl> <dbl> <dbl>
1 65+ 912 22.9 22.9 23.6 23.6
2 35-64 1964 49.3 72.2 50.9 74.5
3 Ages 18 to 34 983 24.7 96.8 25.5 100
4 NA 127 3.19 100 NA NA
Only that category’s label changes. Its members, the other categories, and missing answers are unchanged. The original factor remains in age_group.
Combine two bands while retaining the finer grouping:
result <- result |>
mutate(age_band = fct_collapse(age_group, "35 and older" = c("35-64", "65+")))
tab(result, age_band)# A tibble: 3 × 6
age_band n percent cum valid valid_cum
<fct> <int> <dbl> <dbl> <dbl> <dbl>
1 35 and older 2876 72.2 72.2 74.5 74.5
2 18-34 983 24.7 96.8 25.5 100
3 NA 127 3.19 100 NA NA
This band loses the distinction between respondents aged 35-64 and those aged 65 or older. Preserve age_group and the source age for the finer information. None of these operations rearranges respondents or assigns unknown ages to a known category. Reordering and relabelling retain membership; collapsing changes it.
NoteWho contributes to a score?
Prepare the extract following Appendix A, then restore the saved file in your project folder:
library(tidyverse)
library(tidystata)
gss24 <- readRDS("gss24.rds")confidence <- gss24 |>
select(id, conbus, coneduc, conmedic)
score_levels <- c("hardly any", "only some", "a great deal")
scores <- confidence |>
mutate(
conbus = encode(factor(conbus, levels = score_levels)),
coneduc = encode(factor(coneduc, levels = score_levels)),
conmedic = encode(factor(conmedic, levels = score_levels))
)
scores_block <- confidence |>
mutate(across(conbus:conmedic, \(x) encode(factor(x, levels = score_levels))))
identical(scores, scores_block)[1] TRUEresult <- scores |>
mutate(
n_answered = row_non_miss(conbus, coneduc, conmedic),
mean_score = row_mean(conbus, coneduc, conmedic, na.rm = FALSE)
)
tab(result, n_answered)# A tibble: 4 × 6
n_answered n percent cum valid valid_cum
<int> <int> <dbl> <dbl> <dbl> <dbl>
1 0 1351 33.9 33.9 33.9 33.9
2 1 10 0.251 34.1 0.251 34.1
3 2 73 1.83 36.0 1.83 36.0
4 3 2552 64.0 100 64.0 100
result |>
filter(is.na(mean_score)) |>
select(id, n_answered, mean_score) |>
head()# A tibble: 6 × 3
id n_answered mean_score
<int> <int> <dbl>
1 4 0 NA
2 10 0 NA
3 15 0 NA
4 16 0 NA
5 18 0 NA
6 20 0 NA
The comparison of separate and block conversions returns TRUE. Encoding the original factor order would assign the lowest code to a great deal and reverse our intended direction. The explicit level order assigns 1 to hardly any confidence, 2 to only some, and 3 to a great deal. encode() retains these meanings as value labels. row_non_miss() counts available answers, including a count of zero when all three are missing. The frequency table covers every respondent; zero recorded answers says nothing about their confidence level or the reason for missingness. Each displayed respondent has fewer than three recorded answers. row_mean() with na.rm = FALSE preserves that missingness, so the three-item mean is missing. No answer is replaced with zero, and two available answers are not treated as a complete response. The rule preserves unknown information without validating the proposed scale.
NoteKeep names for display
Start a fresh session with the saved public CSV. The new object names keep this answer independent of the main lesson.
library(tidyverse)
answer_owid <- read_csv("examples/life-expectancy.csv")Rows: 21565 Columns: 4
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (2): Entity, Code
dbl (2): Year, Life expectancy
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
answer_phrase <- str_squish(" A short phrase ")
answer_phrase[1] "A short phrase"answer_names <- answer_owid |>
filter(Year == 2023) |>
select(Entity, Code, Year) |>
mutate(entity_lower = str_to_lower(Entity)) |>
filter(str_detect(entity_lower, fixed("united")))
answer_names# A tibble: 4 × 4
Entity Code Year entity_lower
<chr> <chr> <dbl> <chr>
1 United Arab Emirates ARE 2023 united arab emirates
2 United Kingdom GBR 2023 united kingdom
3 United States USA 2023 united states
4 United States Virgin Islands VIR 2023 united states virgin islands
The abstract phrase loses extra spaces. The real-data result retains 4 names. Display Entity, whose original case helps readers recognize the names; use entity_lower for the specified search. Cleaning a search form does not establish that names are valid keys for a join.
answer_versions <- answer_owid |>
filter(Year == 2023) |>
select(Entity, Code, Year) |>
mutate(
entity_display = str_replace_all(Entity, fixed(" and "), " & "),
entity_search = str_remove(Entity, fixed(" (country)")),
across(c(Entity, Code), str_to_lower, .names = "{.col}_lower")
)
answer_versions |>
filter(Entity != entity_display | Entity != entity_search)# A tibble: 12 × 7
Entity Code Year entity_display entity_search Entity_lower Code_lower
<chr> <chr> <dbl> <chr> <chr> <chr> <chr>
1 Antigua and… ATG 2023 Antigua & Bar… Antigua and … antigua and… atg
2 Bonaire Sin… BES 2023 Bonaire Sint … Bonaire Sint… bonaire sin… bes
3 Bosnia and … BIH 2023 Bosnia & Herz… Bosnia and H… bosnia and … bih
4 Latin Ameri… NA 2023 Latin America… Latin Americ… latin ameri… NA
5 Micronesia … FSM 2023 Micronesia (c… Micronesia micronesia … fsm
6 Saint Kitts… KNA 2023 Saint Kitts &… Saint Kitts … saint kitts… kna
7 Saint Pierr… SPM 2023 Saint Pierre … Saint Pierre… saint pierr… spm
8 Saint Vince… VCT 2023 Saint Vincent… Saint Vincen… saint vince… vct
9 Sao Tome an… STP 2023 Sao Tome & Pr… Sao Tome and… sao tome an… stp
10 Trinidad an… TTO 2023 Trinidad & To… Trinidad and… trinidad an… tto
11 Turks and C… TCA 2023 Turks & Caico… Turks and Ca… turks and c… tca
12 Wallis and … WLF 2023 Wallis & Futu… Wallis and F… wallis and … wlf
The display and search columns serve stated purposes without changing any original name, code, or year. Derived labels need separate checks before they can be used as identifiers.
NoteExplain search boundaries
Reload the input in a fresh session and make each selection from the same annual records.
library(tidyverse)
answer_owid <- read_csv("examples/life-expectancy.csv")Rows: 21565 Columns: 4
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (2): Entity, Code
dbl (2): Year, Life expectancy
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
answer_annual <- answer_owid |>
filter(Year == 2023) |>
select(Entity, Code, Year)
answer_literal <- answer_annual |>
filter(str_detect(Entity, fixed("United")))
answer_beginning <- answer_annual |>
filter(str_detect(Entity, "^United"))
answer_ending <- answer_annual |>
filter(str_detect(Entity, "States$"))
answer_literal# A tibble: 4 × 3
Entity Code Year
<chr> <chr> <dbl>
1 United Arab Emirates ARE 2023
2 United Kingdom GBR 2023
3 United States USA 2023
4 United States Virgin Islands VIR 2023
answer_beginning# A tibble: 4 × 3
Entity Code Year
<chr> <chr> <dbl>
1 United Arab Emirates ARE 2023
2 United Kingdom GBR 2023
3 United States USA 2023
4 United States Virgin Islands VIR 2023
answer_ending# A tibble: 1 × 3
Entity Code Year
<chr> <chr> <dbl>
1 United States USA 2023
There are 4 literal matches, 4 beginning matches, and 1 ending matches. The first two happen to select the same records here, although the rules differ. United States Virgin Islands continues after States, so it cannot satisfy the end anchor. fixed("^United") would look for the literal caret followed by United anywhere in the name; it would not impose a beginning rule. Detection of an unavailable name returns NA, which filter() does not retain. Inspect unknown source names separately when coverage matters. These patterns describe characters and positions; they cannot determine an entity’s political status.