Sparse concept matrix for ML / PLP-style models — extract_sparse_concept

Builds a high-dimensional sparse matrix of condition / drug / procedure concept counts inside a per-row lookback window, driven by a cohort parquet (one row per index event). It is the R counterpart of Python omop_etl.extract_sparse_concept_matrix() and follows the same rules, so the two return the same columns in the same order for the same data.

Usage

extract_sparse_concept_matrix(
  con,
  features_parquet,
  lookback_days = 365,
  min_patient_freq = 1,
  domains = c("condition", "drug"),
  value = "count",
  tokens = NULL,
  include_index_date = FALSE,
  person_col = NULL,
  index_date_col = NULL,
  outcome_col = NULL,
  concept_table = NULL,
  anchor_date = c("auto", "admit_date", "discharge_date"),
  washin_buffer_days = 0,
  washin_buffer_hours = 0
)

Arguments

con: Active DBI connection to DuckDB holding the OMOP CDM tables (read-only is fine). If the vocabulary lives in an attached central_vocab catalog, concept names are picked up automatically.
features_parquet: Path (or glob) of the cohort parquet, one row per index event, with a person id and an index date column; an outcome column is carried through if present. Columns are auto-detected (person_id/subject_id/patid; index_date/cohort_start_date/end_date/admit_date...; y/outcome_flag/label) or set with person_col, index_date_col, outcome_col.
lookback_days: Window length in days, or NULL for all prior history.
min_patient_freq: Keep a concept only if at least this many distinct patients have it in their window (computed on the cohort passed in). To apply a train-fitted vocabulary to a test cohort pass the train result's concepts$token as tokens instead.
domains: Any of "condition", "drug", "procedure".
value: "count" (records per concept) or "binary" (presence).
tokens: Optional character vector of tokens from a previous call. Fixes the columns and their order; min_patient_freq is then not applied and concepts outside it are dropped.
include_index_date: Also use events dated exactly on the index date (default FALSE).
person_col, index_date_col, outcome_col: Optional explicit column names in the parquet.
concept_table: Concept table to read names from (default: concept, then central_vocab).
anchor_date: Strategy for selecting index date column from cohort parquet: "auto" (default), "admit_date" (prioritizes admission date to prevent in-hospital leakage), or "discharge_date" (prioritizes discharge date).
washin_buffer_days: Post-anchor observation buffer days to include in the window (default 0).
washin_buffer_hours: Post-anchor observation buffer hours (e.g. 24 for 24h post-admission, default 0).

Value

A list with X (a Matrix::dgCMatrix, rows aligned to the parquet), concepts (one row per column of X: column_index (1-based), token, feature (syntactic name <domain>_<concept_id> used by as_tidymodels_data()), domain, concept_id, concept_name, domain_id, vocabulary_id, standard_concept, n_patients), cohort (person_id, index_date, optional y), y (or NULL) and params.

Details

The whole cohort is processed in a single set-based DuckDB query. By default a record is used only if it is dated strictly before the row's index date ([index - lookback_days, index)), so nothing at or after the index leaks into the features. Concept id 0 ("No matching concept") is always dropped.

Column j of X is the j-th token of concepts$token. Tokens use omop-learn's convention ("<concept_id> - <domain> - <concept_name>") and are sorted in C-locale order, which is what omop-learn's ConceptTokenizer does, so R and Python agree regardless of the R session's locale.

Examples

if (FALSE) { # \dontrun{
con <- DBI::dbConnect(duckdb::duckdb(), "omop_Temple.duckdb", read_only = TRUE)
DBI::dbExecute(con, "ATTACH 'central_vocabulary.duckdb' AS central_vocab (READ_ONLY);")
res <- extract_sparse_concept_matrix(con, "ederri_features_Temple.parquet", min_patient_freq = 50)
dim(res$X)
} # }