5 Pipe

We can now choose columns, move them, and order rows. A research task often needs several of these operations. The pipe, |>, connects them so that the code follows the order in which we describe the work.

By the end of this chapter, you will be able to read a pipeline, save its result, and inspect a step when the result is unexpected. The operations themselves come from Chapter 3 and Chapter 4.

5.1 Read the pipe as “and then”

Imagine making French fries: wash the potatoes, cut them into sticks, dry them, fry them, and season. The following blocks use imaginary functions to illustrate the syntax, so read them as a recipe rather than code to run. We could nest the steps inside one another:

season(fry(dry(cut_into_sticks(wash(potatoes))), oil = TRUE))

To follow the steps, we must read from the inside outward. The pipe, written |>, puts them in the order we would naturally describe them:

potatoes |>
  wash() |>
  cut_into_sticks() |>
  dry() |>
  fry(oil = TRUE) |>
  season()

Read |> as “and then”: each step passes its result to the next. In fry(oil = TRUE), the pipe supplies the potatoes, while oil = TRUE specifies an option. The function calls describe the operations; the pipes pass each result onward.

The native pipe |> is part of R (since R 4.1). For the calls in this chapter, it supplies the result on its left as the first argument of the function on its right. Other arguments, such as oil = TRUE in the recipe, stay inside that function’s parentheses.

Data preparation follows the same pattern. We can connect the table operations from the preceding chapters, passing a whole table from one step to the next.

5.2 Connect familiar table operations

Load the packages to work with the same lifeexp country data as before:

library(tidyverse)
library(tidystata)

Suppose we want country names, life expectancy, and water coverage, with water beside the name and high life expectancy first. We can retain each intermediate table:

country_view <- select(lifeexp, country, lexp, safewater)
water_beside_name <- relocate(country_view, safewater, .after = country)
country_ranked <- arrange(water_beside_name, desc(lexp), country)
head(country_ranked)

# A tibble: 6 × 3
  country     safewater  lexp
  <chr>           <dbl> <dbl>
1 Canada             99    79
2 France             97    79
3 Italy              95    79
4 Spain              99    79
5 Sweden             99    79
6 Switzerland        97    79

These functions take a table and return another table. That shared structure lets one operation use the result of the previous operation, like pieces with compatible connectors. The columns and row order can change while the result remains a tibble.

Each line uses the result from the preceding line. Every stored table remains available with all 68 countries; head() returns six rows for inspection. These intermediate names are useful when checking the work.

As in the nested French fries recipe, we can supply one function’s result directly as another’s argument:

country_nested <- arrange(
  relocate(select(lifeexp, country, lexp, safewater), safewater, .after = country),
  desc(lexp), country
)
head(country_nested)

# A tibble: 6 × 3
  country     safewater  lexp
  <chr>           <dbl> <dbl>
1 Canada             99    79
2 France             97    79
3 Italy              95    79
4 Spain              99    79
5 Sweden             99    79
6 Switzerland        97    79

Read this from the inside outward, just as with the nested recipe: select, move, then sort. It returns the same table as the named steps, but the order of operations is less visible.

Apply the first-argument rule to a familiar selection. These calls mean the same thing:

select(lifeexp, country, lexp)

# A tibble: 68 × 2
   country                 lexp
   <chr>                  <dbl>
 1 Albania                   74
 2 Armenia                   70
 3 Austria                   78
 4 Azerbaijan                64
 5 Belarus                   68
 6 Belgium                   77
 7 Bosnia and Herzegovina    74
 8 Bulgaria                  71
 9 Croatia                   72
10 Czechia                   75
# ℹ 58 more rows

lifeexp |>
  select(country, lexp)

# A tibble: 68 × 2
   country                 lexp
   <chr>                  <dbl>
 1 Albania                   74
 2 Armenia                   70
 3 Austria                   78
 4 Azerbaijan                64
 5 Belarus                   68
 6 Belgium                   77
 7 Bosnia and Herzegovina    74
 8 Bulgaria                  71
 9 Croatia                   72
10 Czechia                   75
# ℹ 58 more rows

In the second form, the pipe supplies lifeexp, so we do not write that argument again inside select(). Column names still identify the variables to keep. Use this connection to express our three steps as one pipeline:

country_pipeline <- lifeexp |>
  select(country, lexp, safewater) |>
  relocate(safewater, .after = country) |>
  arrange(desc(lexp), country)
head(country_pipeline)

# A tibble: 6 × 3
  country     safewater  lexp
  <chr>           <dbl> <dbl>
1 Canada             99    79
2 France             97    79
3 Italy              95    79
4 Spain              99    79
5 Sweden             99    79
6 Switzerland        97    79

Read this as: start with lifeexp, keep these columns, move water coverage, and then sort the rows. It returns the same table as the named steps. <- saves the final result; the source stays unchanged. The function calls do the work, while the pipes pass their results onward.

5.2.1 Keep a needed column until its work is done

Compatible table structure is only part of the requirement: the next step also needs particular columns. Suppose we want country names ordered by income but do not want income in the final table. Sort while gnppc is available, then drop it:

ranked_names <- lifeexp |>
  arrange(desc(gnppc), country) |>
  select(country, lexp)
head(ranked_names)

# A tibble: 6 × 2
  country        lexp
  <chr>         <dbl>
1 Switzerland      79
2 Norway           78
3 Denmark          76
4 United States    77
5 Sweden           79
6 Germany          77

The result is ordered by income, even though the final display contains only names and life expectancy. Selecting columns preserves that row order. Reversing the steps below intentionally fails:

lifeexp |>
  select(country, lexp) |>
  arrange(desc(gnppc), country)

Error in `arrange()`:
ℹ In argument: `..1 = gnppc`.
Caused by error:
! object 'gnppc' not found

The first result is still a tibble, but it no longer contains gnppc. The later sorting call cannot find its key. Restore that column to the selection, or sort before dropping it, as in the successful version. Moving operations can change whether a pipeline can run even when each step expects a table.

5.3 Save and check results

If the result is unexpected, run only the first part and inspect it. Saving an intermediate table gives us something concrete to check:

first_step <- lifeexp |>
  select(country, lexp, safewater)
ds(first_step)

[1] "country"   "lexp"      "safewater"

nrow(first_step)

[1] 68

There should be three named columns and 68 countries. If those are correct, add the next operation. ds() returns column names as text, not another country table. The next operation must accept the object it receives.

Now suppose we want to put lexp first, but insert ds() before relocate(). The following pipeline intentionally produces an error:

lifeexp |>
  ds() |>
  relocate(lexp)

Error in `UseMethod()`:
! no applicable method for 'relocate' applied to an object of class "character"

The message says that no relocate() method applies to an object of class "character": this function cannot move columns in the text returned by ds(). To find the problem, stop before the failing call and save the result under a name. This gives us something we can inspect separately:

column_names <- lifeexp |>
  ds()
column_names

[1] "region"    "country"   "popgrowth" "lexp"      "gnppc"     "safewater"

The result contains the six column names as text. To check what kind of object we now have, use tidystata’s var_type(). It takes one object and returns a description of its type, without changing it:

var_type(column_names)

[1] "character"

The type is "character". These are text values, not a table with a lexp column. relocate() needs a table, so it cannot continue from this result. To move life expectancy first, use lifeexp |> relocate(lexp) without inserting ds().

We can also pipe the names directly into the type check we just used:

lifeexp |>
  ds() |>
  var_type()

[1] "character"

This returns "character" again. If our goal is instead to count the variables, keep ds() and choose a function that counts text entries. length() counts the elements of a vector; here that means the number of column names:

lifeexp |>
  ds() |>
  length()

[1] 6

The result is 6, one entry per column name. A pipeline can continue after it leaves a tibble, provided the next function accepts the returned object. The pipe passes a result along; it does not convert that result into whatever the next function needs.

When a pipeline fails, run it up to the step before the error, save that result, and inspect it. Check both its type and its contents against what the next function requires. The intermediate objects from the table workflow are useful for exactly this purpose.

Recognizing older pipes

Older tidyverse code often uses %>%, supplied by magrittr and available after loading tidyverse. For these first-argument calls, lifeexp %>% select(country, lexp) gives the same result as lifeexp |> select(country, lexp). Their advanced rules differ, so check unfamiliar code before replacing one with the other. This book uses |>.

5.4 Optional: a table or a vector?

select() keeps table structure. pull() takes a table and a column name and returns that column as a vector. Compare the same water measurements in both forms:

water_table <- select(lifeexp, safewater)
water_vector <- pull(lifeexp, safewater)
var_type(water_table)

[1] "tibble"

var_type(water_vector)

[1] "double"

The results are "tibble" and "double". Both objects contain the same 68 measurements, with the same order and missing values. Their structures differ: water_table has rows and a named column; water_vector is a sequence of values. The $ syntax is another way to extract one column: write the table name, $, and the column name. Thus lifeexp$safewater returns the same vector as pull(lifeexp, safewater). Neither changes lifeexp. This distinction matters when the next function requires a table, as in Chapter 5.

5.5 Export the finished table

Assignment retained country_pipeline in the R session. To keep its values outside that session, we will write a file in the r4ss project folder created in Section 1.1.1. First check where R will resolve the filename. getwd() takes no input and returns the current working directory:

getwd()

The result should be the full path to your r4ss folder. If it differs, use setwd() with the full path to your own folder in quotation marks. For example, on Windows:

setwd("C:/Users/you/Documents/r4ss")

Replace that example path with the actual location of your folder before running it. Use forward slashes in R paths, including on Windows. On macOS or Linux, a full path will usually begin with /. setwd() changes this session’s working directory; it does not create a folder or move files. Run getwd() again to confirm the change. Check this whenever you start a file workflow in a new session.

A relative path, such as "country-ranked.csv", is resolved from the working directory. Here it names a file directly inside that folder. A full path identifies the location without relying on the working directory.

write_csv() takes a table and a quoted file path, and writes a comma-separated-values (CSV) file. It overwrites an existing file at that path; choose another name if you want to keep an earlier result. Run:

write_csv(country_pipeline, "country-ranked.csv")

Writing normally produces no printed table. To check whether a file exists at the path, file.exists() takes that path and returns TRUE or FALSE:

file.exists("country-ranked.csv")

Expect TRUE. If the result is FALSE, check the write error, spelling, and working directory. Existence alone does not establish that the contents are correct. read_csv() reads the file and returns a table; save that result under a new name:

saved_countries <- read_csv("country-ranked.csv")

The import message reports the detected column types. The names should be text and the two measurements numeric. Now inspect the returned table with familiar checks:

head(saved_countries)
ds(saved_countries)
nrow(saved_countries)

Expect 68 rows, with country, safewater, and lexp in that order, and the same country ordering as country_pipeline. The preview alone shows only the first six rows; the row count checks the full result. CSV preserves values, column names, and missing entries, but not the variable labels attached to the original columns. The later file lesson in Section 7.8 develops this distinction further.

The same workflow applies to your own data: import a file into a named table, inspect it, select and order the relevant columns and rows, and export the result under a new filename. Use your file’s variable names in place of the ones shown here. Keep the import and transformation commands in your script so that the saved result can be reproduced.

5.6 Review

|> passes a result to the next function; <- saves a result under a name.
select(), relocate(), and arrange() can be connected through named steps, nested calls, or pipes to express the same procedure.
ds() and nrow() check intermediate tables; retain required columns until their operations are complete.
var_type() identifies an object’s kind and length() counts vector entries; each function must accept the preceding result.
Optional: pull() extracts a vector, while select() keeps a table.
getwd() and setwd() locate the file workflow; relative paths begin at the working directory.
write_csv() exports, file.exists() checks the path, and read_csv() imports for inspection.

Next, Chapter 6 selects rows, beginning with the same country data. The GSS survey and its one-time preparation are introduced later in Chapter 8.

5.7 Exercises

Rewrite the income workflow from Chapter 4 as one pipeline: keep country, gnppc, and lexp, then order by descending income and country name. Save it as income_pipeline.
Inspect the column names and row count after just its selection step. Diagnose two failures: inserting ds() before arrange(), and selecting only country and lexp before sorting by gnppc. Explain how the problems differ and correct each one.
Export income_pipeline to "income-ranked.csv" in your checked project folder, check that the file exists, read it as saved_income, and inspect its first rows, column names, and total row count. What would you check if existence returned FALSE? Why does TRUE still require inspection of the contents?
Optional: A table or a vector? Run select(lifeexp, lexp) and pull(lifeexp, lexp). Use var_type() on each result. Which result keeps table structure? Which is the same kind of object as lifeexp$lexp?

Worked answers

Run the chapter package setup first.

Connect the income workflow

income_pipeline <- lifeexp |>
  select(country, gnppc, lexp) |>
  arrange(desc(gnppc), country)
head(income_pipeline)

# A tibble: 6 × 3
  country       gnppc  lexp
  <chr>         <dbl> <dbl>
1 Switzerland   46650    79
2 Norway        36120    78
3 Denmark       33440    76
4 United States 32040    77
5 Sweden        31100    79
6 Germany       28070    77

All 68 countries remain. The last row has unavailable income; saving the result does not change lifeexp.

Inspect an intermediate result

income_columns <- lifeexp |>
  select(country, gnppc, lexp)
ds(income_columns)

[1] "country" "gnppc"   "lexp"

nrow(income_columns)

[1] 68

The names are country, gnppc, and lexp; the row count is 68. A name listing is character data with no country records to sort, so remove ds() from the table pipeline and inspect names separately. A table containing only country and lexp has the right structure but lacks the gnppc key. Keep that key until after sorting:

income_names <- lifeexp |>
  arrange(desc(gnppc), country) |>
  select(country, lexp)
head(income_names)

# A tibble: 6 × 2
  country        lexp
  <chr>         <dbl>
1 Switzerland      79
2 Norway           78
3 Denmark          76
4 United States    77
5 Sweden           79
6 Germany          77

This returns names and life expectancy in income order. The two failures concern different requirements: object type and available columns.

Export and read back

Recreate income_pipeline if needed, then check your working directory. Change it using your actual project path if necessary, as taught above. These commands write a file, so choose a new filename if this one already contains work you want to retain.

income_pipeline <- lifeexp |>
  select(country, gnppc, lexp) |>
  arrange(desc(gnppc), country)
getwd()
write_csv(income_pipeline, "income-ranked.csv")
file.exists("income-ranked.csv")
saved_income <- read_csv("income-ranked.csv")
head(saved_income)
ds(saved_income)
nrow(saved_income)

Expect TRUE, the three names country, gnppc, and lexp, and 68 rows in income order. If the check returns FALSE, inspect any write error, the filename, and the working directory. A TRUE result establishes only existence; reading and inspecting the file checks whether it contains the intended table. Missing income remains missing and sorts after known values.

A table or a vector?

life_table <- select(lifeexp, lexp)
life_vector <- pull(lifeexp, lexp)
var_type(life_table)

[1] "tibble"

var_type(life_vector)

[1] "double"

var_type(lifeexp$lexp)

[1] "double"

The results are "tibble", "double", and "double". select() keeps a table with the named column lexp. pull() and $ return the same numeric vector, with one entry per country. None of these operations changes lifeexp.