Polars market data Parquet guide

Lazy-scan date-partitioned OHLCV files, read only the columns and rows a query needs, and resample intraday bars -- with the actual outputs and timings from running each example on real MarketParquet files (Polars 1.43, 2-core / 2 GB VPS).

Why Polars fits this layout

MarketParquet ships one Snappy-compressed Parquet file per trading day (by_date/{asset}_{timeframe}/YYYY/YYYY-MM-DD.parquet), every symbol included -- a 2024 stock 1-minute day is ~22 MB and 1.28 million typed rows. Polars' lazy engine pushes column selection and filters down into the Parquet reader, so scan_parquet over a year of files touches only the bytes your query needs instead of loading 250 files into RAM.

One symbol, one day

  import polars as pl

  aapl = (
      pl.scan_parquet("stock_1min_2024-01-03.parquet")
        .filter(pl.col("symbol") == "AAPL")
        .select(["timestamp", "close", "volume"])
        .collect()
  )
  # 741 rows in ~12 ms
  # ┌─────────────────────┬────────┬────────┐
  # │ timestamp           ┆ close  ┆ volume │
  # │ datetime[μs]        ┆ f64    ┆ f64    │
  # ╞═════════════════════╪════════╪════════╡
  # │ 2024-01-03 04:00:00 ┆ 185.0  ┆ 791.0  │
  # │ 2024-01-03 04:01:00 ┆ 184.9  ┆ 1278.0 │

741 rows, not 960: bars exist only for minutes with trades, and the file covers the extended 04:00–19:59 Eastern session. Timestamps are bar opens, US/Eastern, timezone-naive -- attach a zone explicitly with .dt.replace_time_zone("America/New_York") if the rest of your stack is timezone-aware.

Cross-sectional screens

Full-day volume ranking across all 7,180 symbols runs in ~60 ms:

  top = (
      pl.scan_parquet("stock_1min_2024-01-03.parquet")
        .group_by("symbol")
        .agg(pl.col("volume").sum().alias("vol"))
        .sort("vol", descending=True)
        .head(5)
        .collect()
  )
  # ┌───────────────┬──────────┐
  # │ BETS-DELISTED ┆ 772.5M   │   <- delisted tickers are in the file,
  # │ NVDA          ┆ 244.5M   │      suffixed '-DELISTED'
  # │ TSLA          ┆ 102.7M   │
  # │ MARA          ┆  91.1M   │

That first row is the survivorship-bias point in miniature: the highest-volume stock that day no longer exists. Filter with pl.col("symbol").str.ends_with("-DELISTED").not_() only when you deliberately want today's survivors.

Resample 1-minute to 5-minute

  bars5 = (
      pl.scan_parquet("stock_1min_2024-01-03.parquet")
        .filter(pl.col("symbol") == "AAPL")
        .sort("timestamp")
        .group_by_dynamic("timestamp", every="5m")
        .agg(
            pl.col("open").first(),
            pl.col("high").max(),
            pl.col("low").min(),
            pl.col("close").last(),
            pl.col("volume").sum(),
        )
        .collect()
  )
  # ~11 ms
  # ┌─────────────────────┬────────┬────────┬────────┬────────┬────────┐
  # │ 2024-01-03 04:00:00 ┆ 185.0  ┆ 185.0  ┆ 184.9  ┆ 184.9  ┆ 3932.0 │
  # │ 2024-01-03 04:05:00 ┆ 184.85 ┆ 184.88 ┆ 184.73 ┆ 184.88 ┆ 1632.0 │

The .sort("timestamp") matters: group_by_dynamic requires sorted input, and file order is (timestamp, symbol) -- already sorted once you filter to one symbol, but be explicit after joins or concats. Also note MarketParquet ships native 5-minute, 30-minute, and 1-hour files -- resample yourself only for non-standard intervals.

Scan a year (or the whole archive)

  spy_2024 = (
      pl.scan_parquet("by_date/etf_daily/2024/*.parquet")
        .filter(pl.col("symbol") == "SPY")
        .select(["date", "close", "volume"])
        .sort("date")
        .collect()
  )

Note etf_daily: SPY and QQQ are ETFs, so they live in the etf_* datasets -- an empty frame from a stock_* scan is the most common first-day mistake. For scans too big for RAM (e.g. touching every 1-minute file), finish with .collect(engine="streaming") so Polars processes files in batches.

Edge case: legacy daily files

Daily files written since 2026-03-27 carry a typed date column; older archive files have a midnight timestamp column instead. When one scan spans both eras, insert the missing columns and coalesce:

  lf = pl.scan_parquet(
      "by_date/stock_daily/*/*.parquet",
      missing_columns="insert",          # tolerate both schema eras
  ).with_columns(
      pl.coalesce(pl.col("date"), pl.col("timestamp").dt.date()).alias("d")
  )
  # AAPL close: 2010-01-04 -> 7.6432 (214.01 as traded / 28 across
  # two later splits), 2026-07-24 -> 333.02. Split-adjusted, and
  # deliberately NOT dividend-adjusted -- see /data-quality.

Join daily context onto intraday bars

Because every file holds the full cross-section for one date, joining datasets is a plain join on symbol -- no per-ticker loops. A gap scanner, for example, joins yesterday's daily closes onto today's 1-minute opens:

  prev = (
      pl.scan_parquet("stock_daily_2024-01-02.parquet")
        .select("symbol", pl.col("close").alias("prev_close"))
  )
  gaps = (
      pl.scan_parquet("stock_1min_2024-01-03.parquet")
        .filter(pl.col("timestamp") == pl.datetime(2024, 1, 3, 9, 30))
        .join(prev, on="symbol")
        .with_columns(
            ((pl.col("open") / pl.col("prev_close") - 1) * 100)
                .round(2).alias("gap_pct")
        )
        .sort("gap_pct", descending=True)
        .collect()
  )

Both scans stay lazy until .collect(), so Polars reads just three columns from one file and five from the other. The same shape works for joining fundamentals, borrow lists, or your own signals -- anything keyed by symbol and date.

Pitfalls checklist

Empty result for SPY/QQQ -- they are in etf_*, not stock_*.
Missing minutes are not an error -- no trades, no bar. Use upsample or join against a generated session grid if you need density.
Naive Eastern timestamps -- do not .dt.convert_time_zone() before attaching the zone; replace first, convert second.
Volume is Float64 -- cast with .cast(pl.Int64) if you need integer share counts.
Delisted suffix -- BETS-DELISTED-style symbols are the point of the dataset; excluding them re-introduces survivorship bias.
Prices differ from Yahoo "Adj Close" -- these are split-adjusted traded levels, not dividend-smoothed total-return prices.

stock data hub · browse stock daily · intraday stock data · data quality & methodology · DuckDB guide · pricing