Skip to contents

Stream a wide feature table into PatientLevelPrediction-shaped Arrow tables

Source: R/cohort_flow.R
as_plp_arrow.Rd

Non-null, non-zero numeric features become sparse (rowId, covariateId, covariateValue) rows computed by DuckDB and returned as Arrow tables, in memory: no Parquet or SQLite round-trip. One stable rowId is shared by cohorts, covariates and outcomes. Mirror of Python as_plp_arrow. Needs the arrow package. Turn the result into a plpData with plp_arrow_to_plp_data().

Usage

as_plp_arrow(
  con,
  features,
  id_col = "subject_id",
  outcome_col = NULL,
  date_col = "cohort_start_date",
  feature_cols = NULL,
  subprocess_worker = FALSE
)

Arguments

con

DuckDB DBI connection.

features

Data frame or DuckDB table name.

id_col, outcome_col, date_col

Column names (outcome_col / date_col optional).

feature_cols

Numeric columns to use (default: all numeric columns that are not ids / dates).

subprocess_worker

Logical. If TRUE (defensive on Windows), isolates pyarrow extraction in a separate worker execution environment to prevent in-process C++ DLL collision with R's arrow.dll.

Value

A list with Arrow tables cohorts, covariates, covariate_ref and (if outcome_col) outcomes.

Details

On Windows, loading R's arrow before python's pyarrow makes the latter fail to import in the same process ("DLL load failed while importing lib"), which breaks DeepPatientLevelPrediction. Import pyarrow first (reticulate::import("pyarrow.lib")) in sessions that use both.