Size a DuckDB connection to the host: RAM limit, disk-spill location and threads

Sets memory_limit to a fraction of the host's RAM, points temp_directory at the writable volume with the most free space so large joins spill gracefully, and sets threads from the usable cores. Mirror of omop_etl.resources.configure_resources in Python.

Usage

configure_resources(
  con,
  memory_fraction = 0.75,
  temp_directory = NULL,
  threads = NULL,
  use_available_ram = TRUE,
  reserve_threads = 1L,
  max_temp_directory_size = NULL
)

Arguments

con

DuckDB DBI connection (modified in place).

memory_fraction

Fraction of RAM DuckDB may use (default 0.75), in (0, 1].

temp_directory

Spill directory. Default: auto-pick the volume with the most free space.

threads

Thread count. Default: usable cores minus reserve_threads (at least 1).

use_available_ram

Base the limit on currently available RAM (default) instead of total RAM.

reserve_threads

Cores left to the OS when threads is not given.

max_temp_directory_size

Optional cap on spill size, e.g. "200GB".

Value

A list of class "ResourceConfig": total_ram_bytes, available_ram_bytes, memory_limit, temp_directory, temp_free_bytes, threads, notes. If RAM cannot be detected the DuckDB default memory_limit is left in place and a note says so.

Examples

con <- DBI::dbConnect(duckdb::duckdb())
#> duckdb keeps downloaded extensions and secrets in a temporary directory:
#> ℹ /tmp/RtmpFV3zps/duckdb
#> This is removed when the R session ends.
#> • Extensions are re-downloaded each session.
#> • Secrets are lost.
#> ℹ Run duckdb(shared_home = TRUE) (or create ~/.duckdb) to keep them (suitable for most users).
#> ℹ Run duckdb(shared_home = FALSE) to accept the temporary directory (and silence this message).
#> ℹ See ?duckdb_storage for details and alternatives.
cfg <- configure_resources(con, memory_fraction = 0.5, threads = 2)
cfg$threads
#> [1] 2
DBI::dbDisconnect(con, shutdown = TRUE)