The pins package helps you publish data sets, models, and other R objects, making it easy to share them across projects and with your colleagues. You can pin objects to a variety of “boards”, including local folders (to share on a networked drive or with dropbox), Posit Connect, Amazon S3, and more. This vignette will introduce you to the basics of pins.
Every pin lives in a pin board, so you must start by creating a pin board. In this vignette I’ll use a temporary board which is automatically deleted when your R session is over:
In real-life, you’d pick a board depending on how you want to share the data. Here are a few options:
Once you have a pin board, you can write data to it with
pin_write():
mtcars <- tibble::as_tibble(mtcars)
board %>% pin_write(mtcars, "mtcars")
#> Guessing `type = 'rds'`
#> Creating new version '20241106T055948Z-e5d8a'
#> Writing to pin 'mtcars'The first argument is the object to save (usually a data frame, but it can be any R object), and the second argument gives the “name” of the pin. The name is basically equivalent to a file name: you’ll use it when you later want to read the data from the pin. The only rule for a pin name is that it can’t contain slashes.
After you’ve pinned an object, you can read it back with
pin_read():
board %>% pin_read("mtcars")
#> # A tibble: 32 × 11
#> mpg cyl disp hp drat wt qsec vs am gear carb
#> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 21 6 160 110 3.9 2.62 16.5 0 1 4 4
#> 2 21 6 160 110 3.9 2.88 17.0 0 1 4 4
#> 3 22.8 4 108 93 3.85 2.32 18.6 1 1 4 1
#> 4 21.4 6 258 110 3.08 3.22 19.4 1 0 3 1
#> 5 18.7 8 360 175 3.15 3.44 17.0 0 0 3 2
#> 6 18.1 6 225 105 2.76 3.46 20.2 1 0 3 1
#> 7 14.3 8 360 245 3.21 3.57 15.8 0 0 3 4
#> 8 24.4 4 147. 62 3.69 3.19 20 1 0 4 2
#> 9 22.8 4 141. 95 3.92 3.15 22.9 1 0 4 2
#> 10 19.2 6 168. 123 3.92 3.44 18.3 1 0 4 4
#> # ℹ 22 more rowsYou don’t need to supply the file type when reading data from a pin because pins automatically stores the file type in the metadata.
As you can see from the output in the previous section, pins has
chosen to save this example data to an .rds file. But you
can choose another option depending on your goals:
type = "rds" uses writeRDS() to create a
binary R data file. It can save any R object (including trained models)
but it’s only readable from R, not other languages.type = "csv" uses write.csv() to create a
CSV file. CSVs are plain text and can be read easily by many
applications, but they only support simple columns (e.g. numbers,
strings), can take up a lot of disk space, and can be slow to read.type = "parquet" uses
nanoparquet::write_parquet() to create a Parquet file. Parquet is a modern,
language-independent, column-oriented file format for efficient data
storage and retrieval. Parquet is an excellent choice for storing
tabular data but requires the nanoparquet package.type = "arrow" uses arrow::write_feather()
to create an Arrow/Feather file.type = "json" uses jsonlite::write_json()
to create a JSON file. Pretty much every programming language can read
json files, but they only work well for nested lists.type = "qs" uses qs::qsave() to create a
binary R data file, like writeRDS(). This format achieves
faster read/write speeds than RDS, and compresses data more efficiently,
making it a good choice for larger objects. Read more on the qs package.Note that when the data lives elsewhere, pins takes care of downloading and caching so that it’s only re-downloaded when needed. That said, most boards transmit pins over HTTP, and this is going to be slow and possibly unreliable for very large pins. As a general rule of thumb, we don’t recommend using pins with files over 500 MB. If you find yourself routinely pinning data larger that this, you might need to reconsider your data engineering pipeline.
Storing your data/object as a pin works well when you write from a single source or process. It is not appropriate when multiple sources or processes need to write to the same pin; since the pins package reads and writes files, it cannot manage concurrent writes.
Every pin is accompanied by some metadata that you can access with
pin_meta():
board %>% pin_meta("mtcars")
#> List of 13
#> $ file : chr "mtcars.rds"
#> $ file_size : 'fs_bytes' int 901
#> $ pin_hash : chr "e5d8a8664904cd23"
#> $ type : chr "rds"
#> $ title : chr "mtcars: a pinned 32 x 11 data frame"
#> $ description: NULL
#> $ tags : NULL
#> $ urls : NULL
#> $ created : POSIXct[1:1], format: "2024-11-06 05:59:48"
#> $ api_version: int 1
#> $ user : list()
#> $ name : chr "mtcars"
#> $ local :List of 3
#> ..$ dir : 'fs_path' chr "/tmp/RtmpxSTUax/pins-15733c164ba4/mtcars/20241106T055948Z-e5d8a"
#> ..$ url : NULL
#> ..$ version: chr "20241106T055948Z-e5d8a"This shows you the metadata that’s generated by default. This includes:
title, a brief textual description of the
dataset.
an optional description, where you can provide more
details.
the date-time when the pin was created.
the file_size, in bytes, of the underlying
files.
a unique pin_hash that you can supply to
pin_read() to ensure that you’re reading exactly the data
that you expect.
When creating the pin, you can override the default description or provide additional metadata that is stored with the data:
board %>% pin_write(mtcars,
description = "Data extracted from the 1974 Motor Trend US magazine, and comprises fuel consumption and 10 aspects of automobile design and performance for 32 automobiles (1973–74 models).",
metadata = list(
source = "Henderson and Velleman (1981), Building multiple regression models interactively. Biometrics, 37, 391–411."
)
)
#> Using `name = 'mtcars'`
#> Guessing `type = 'rds'`
#> ! The hash of pin "mtcars" has not changed.
#> • Your pin will not be stored.
board %>% pin_meta("mtcars")
#> List of 13
#> $ file : chr "mtcars.rds"
#> $ file_size : 'fs_bytes' int 901
#> $ pin_hash : chr "e5d8a8664904cd23"
#> $ type : chr "rds"
#> $ title : chr "mtcars: a pinned 32 x 11 data frame"
#> $ description: NULL
#> $ tags : NULL
#> $ urls : NULL
#> $ created : POSIXct[1:1], format: "2024-11-06 05:59:48"
#> $ api_version: int 1
#> $ user : list()
#> $ name : chr "mtcars"
#> $ local :List of 3
#> ..$ dir : 'fs_path' chr "/tmp/RtmpxSTUax/pins-15733c164ba4/mtcars/20241106T055948Z-e5d8a"
#> ..$ url : NULL
#> ..$ version: chr "20241106T055948Z-e5d8a"While we’ll do our best to keep the automatically generated metadata
consistent over time, I’d recommend manually capturing anything you
really care about in metadata.
In many situations it’s useful to version pins, so that writing to an existing pin does not replace the existing data, but instead adds a new copy. There are two ways to turn versioning on:
When you create a board you can turn versioning on for every pin in that board:
When you write a pin, you can specifically request that versioning be turned on for that pin:
Most boards have versioning on by default. The primary exception is
board_folder() since that stores data on your computer, and
there’s no automated way to clean up the data you’re saving.
Once you have turned versioning on, every pin_write()
will create a new version:
board2 <- board_temp(versioned = TRUE)
board2 %>% pin_write(1:5, name = "x", type = "rds")
#> Creating new version '20241106T055949Z-0050f'
#> Writing to pin 'x'
board2 %>% pin_write(2:6, name = "x", type = "rds")
#> Creating new version '20241106T055949Z-2458e'
#> Writing to pin 'x'
board2 %>% pin_write(3:7, name = "x", type = "rds")
#> Creating new version '20241106T055949Z-8571d'
#> Writing to pin 'x'You can list all the available versions with
pin_versions():
board2 %>% pin_versions("x")
#> # A tibble: 3 × 3
#> version created hash
#> <chr> <dttm> <chr>
#> 1 20241106T055949Z-0050f 2024-11-06 05:59:49 0050f
#> 2 20241106T055949Z-2458e 2024-11-06 05:59:49 2458e
#> 3 20241106T055949Z-8571d 2024-11-06 05:59:49 8571dYou can delete a specific older version with
pin_version_delete() or sets of older versions with
pin_versions_prune().
By default, pin_read() will return the most recent
version:
But you can request an older version by supplying the
version argument:
So far we’ve focussed on pin_write() and
pin_read() which work with R objects. pins also provides
the lower-level pin_upload() and
pin_download() which work with files on disk. You can use
them to share types of data that are otherwise unsupported by pins.
pin_upload() works like pin_write() but
instead of an R object you give it a vector of paths. I’ll start by
creating a few files in the temp directory:
paths <- file.path(tempdir(), c("mtcars.csv", "alphabet.txt"))
write.csv(mtcars, paths[[1]])
writeLines(letters, paths[[2]])Now I can upload those to the board:
pin_download() returns a vector of paths:
board %>% pin_download("example")
#> [1] "/tmp/RtmpxSTUax/pins-15733c164ba4/example/20241106T055949Z-e9d42/mtcars.csv"
#> [2] "/tmp/RtmpxSTUax/pins-15733c164ba4/example/20241106T055949Z-e9d42/alphabet.txt"It’s now your job to handle them. You should treat these paths as internal implementation details — never modify them and never save them for use outside of pins.
Note that you can’t pin_read() something you pinned with
pin_upload():
board %>% pin_read("example")
#> Error in `object_read()`:
#> ! Cannot automatically read pin:
#> • Is your pin specified as a full path? Retrieve it with `pin_download()`
#> • Is your pin specified via a URL that is not a full path, such as a Posit
#> Connect vanity URL? Remember to include a trailing slash `/`But you can pin_download() something that you’ve pinned
with pin_write():
The primary purpose of pins is to make it easy to share data. But
pins is also designed to help you spend as little time as possible
downloading data. pin_read() and
pin_download() automatically cache remote pins: they
maintain a local copy of the data (so it’s fast) but always check that
it’s up-to-date (so your analysis doesn’t use stale data).
Wouldn’t it be nice if you could take advantage of this feature for
any dataset on the internet? That’s the idea behind
board_url() — you can assemble your own board from
datasets, wherever they live on the internet. For example, this code
creates a board containing a single pin, penguins, that
refers to some fun data I found on GitHub:
my_data <- board_url(c(
"penguins" = "https://raw.githubusercontent.com/allisonhorst/palmerpenguins/master/inst/extdata/penguins_raw.csv"
))You can read this data by combining pin_download() with
read.csv()1:
my_data %>%
pin_download("penguins") %>%
read.csv(check.names = FALSE) %>%
tibble::as_tibble()
#> # A tibble: 344 × 17
#> studyName `Sample Number` Species Region Island Stage `Individual ID`
#> <chr> <int> <chr> <chr> <chr> <chr> <chr>
#> 1 PAL0708 1 Adelie Penguin… Anvers Torge… Adul… N1A1
#> 2 PAL0708 2 Adelie Penguin… Anvers Torge… Adul… N1A2
#> 3 PAL0708 3 Adelie Penguin… Anvers Torge… Adul… N2A1
#> 4 PAL0708 4 Adelie Penguin… Anvers Torge… Adul… N2A2
#> 5 PAL0708 5 Adelie Penguin… Anvers Torge… Adul… N3A1
#> 6 PAL0708 6 Adelie Penguin… Anvers Torge… Adul… N3A2
#> 7 PAL0708 7 Adelie Penguin… Anvers Torge… Adul… N4A1
#> 8 PAL0708 8 Adelie Penguin… Anvers Torge… Adul… N4A2
#> 9 PAL0708 9 Adelie Penguin… Anvers Torge… Adul… N5A1
#> 10 PAL0708 10 Adelie Penguin… Anvers Torge… Adul… N5A2
#> # ℹ 334 more rows
#> # ℹ 10 more variables: `Clutch Completion` <chr>, `Date Egg` <chr>,
#> # `Culmen Length (mm)` <dbl>, `Culmen Depth (mm)` <dbl>,
#> # `Flipper Length (mm)` <int>, `Body Mass (g)` <int>, Sex <chr>,
#> # `Delta 15 N (o/oo)` <dbl>, `Delta 13 C (o/oo)` <dbl>, Comments <chr>board_url() requires a bit of work compared to using
download.file() or similar but it has a big payoff: the
data will only be re-downloaded when it changes.
Here I’m using read.csv() to the reduce the
dependencies of the pins package. For real code I’d recommend using
data.table::fread() or readr::read_csv().↩︎