Vectorize symbol_to_df via gams.transfer instead of per-record iteration

symbol_to_df built DataFrames by iterating GDX records one at a time in pure Python (for rec in db[symbol]), which dominates runtime on high-cardinality symbols (e.g. PRO_YCRAGFST, millions of records for a full scenario). We replaced the internal extraction with gams.transfer.Container(system_directory=db.workspace.system_directory).read(db, symbols=[symbol]).records, which accepts the existing classic gams.GamsDatabase object directly, so the function’s signature and output shape are unchanged. gamsapi[transfer] was already a pinned dependency, so this introduced no new dependency. Measured ~62x speedup on an 81K-record symbol with identical row order to the old loop.

Considered options

  • Hybrid: keep the old loop for small symbols, vectorize only above a record-count threshold. Rejected — the vectorized path’s fixed per-call overhead (~0.26ms) is noise against real run times even for the smallest symbols in this codebase’s actual call patterns (dozens to low hundreds of calls per run), so a second code path would add maintenance cost for no practical benefit.

  • Keep the old loop as a fallback behind a flag. Rejected — there’s no realistic scenario where gams.transfer works but the classic API path is needed instead; a flag with no real fallback case is dead weight.

  • Include the element_text column that gams.transfer returns for Sets. Rejected for this change — it would widen the output shape for every Set-returning call site (create_set_columns does a hard positional column-count overwrite, so this isn’t additive-safe without further changes). Left as a separate, deliberately-scoped follow-up feature if wanted later.

Consequences

  • Domain columns come back from gams.transfer as category dtype; we cast them to object to preserve the exact dtype contract of the old implementation, since downstream code merges/concatenates symbol_to_df output across scenarios.

  • The classic API is still used for the pre-existing zero-records short-circuit and the type(db[symbol])-based dispatch; only record extraction moved to gams.transfer.