Backtrader with Parquet market data

backtrader wants one DataFrame per instrument with a datetime index and open/high/low/close/volume columns. MarketParquet ships one Parquet file per trading day with every symbol inside. This guide is the bridge: extract one symbol, shape it, feed it, run a strategy, on daily and on 5-minute bars.

What the files look like

Files live under by_date/{asset}_{timeframe}/YYYY/YYYY-MM-DD.parquet. Intraday files carry timestamp (US/Eastern bar open time), symbol, asset_type, open, high, low, close, volume. Daily files carry date instead of timestamp; daily files dated before 2026-03-27 still use the older timestamp name, so a multi-year daily read must unify the two. Stock and ETF prices are split-adjusted; dividends are not embedded, which is what you want for fills at the prices that actually traded.

Step 1: extract one symbol from daily files

DuckDB does the cross-file filter in one statement and returns a pandas DataFrame:

  import duckdb

  def load_daily(symbol, root="by_date/stock_daily"):
      return duckdb.sql(f"""
          SELECT COALESCE(date, CAST(timestamp AS DATE)) AS datetime,
                 open, high, low, close, volume
          FROM read_parquet('{root}/*/*.parquet', union_by_name=true)
          WHERE symbol = '{symbol}'
          ORDER BY 1
      """).df().set_index("datetime")

  aapl = load_daily("AAPL")
  print(aapl.tail())

union_by_name=true lets files with date and files with timestamp sit in one scan; COALESCE picks whichever is present. Reading a full 26-year history for one ticker takes seconds because DuckDB only touches the columns it needs and skips row groups whose symbol range cannot match.

Step 2: feed it to backtrader

  import backtrader as bt

  class SmaCross(bt.Strategy):
      params = dict(fast=20, slow=50)

      def __init__(self):
          fast = bt.ind.SMA(period=self.p.fast)
          slow = bt.ind.SMA(period=self.p.slow)
          self.cross = bt.ind.CrossOver(fast, slow)

      def next(self):
          if not self.position and self.cross > 0:
              self.buy()
          elif self.position and self.cross < 0:
              self.close()

  cerebro = bt.Cerebro()
  cerebro.adddata(bt.feeds.PandasData(dataname=aapl, openinterest=None), name="AAPL")
  cerebro.addstrategy(SmaCross)
  cerebro.broker.setcash(100_000)
  cerebro.broker.setcommission(commission=0.0005)
  result = cerebro.run()
  print(f"final value: {cerebro.broker.getvalue():,.0f}")

PandasData maps columns by name, so the DataFrame needs exactly open, high, low, close, volume and a datetime index. Passing openinterest=None tells the feed the column does not exist.

Step 3: intraday, 5-minute bars

Intraday files are large, so scan lazily with Polars and filter before collecting. Drop pre-market and after-hours bars unless the strategy trades them, and tell backtrader the bar size:

  import polars as pl

  bars = (
      pl.scan_parquet("by_date/stock_5min/2025/*.parquet")
        .filter(pl.col("symbol") == "AAPL")
        .select(["timestamp", "open", "high", "low", "close", "volume"])
        .sort("timestamp")
        .collect()
        .to_pandas()
        .set_index("timestamp")
  )
  regular = bars.between_time("09:30", "15:55")   # bar open times

  feed = bt.feeds.PandasData(
      dataname=regular,
      timeframe=bt.TimeFrame.Minutes,
      compression=5,
      openinterest=None,
  )
  cerebro = bt.Cerebro()
  cerebro.adddata(feed, name="AAPL-5m")
  cerebro.addstrategy(SmaCross, fast=12, slow=26)
  cerebro.run()

Timestamps are bar open times in US/Eastern, so the last regular bar of a 5-minute file opens at 15:55. A 1-minute feed is the same code with stock_1min and compression=1; expect roughly ten times the rows.

Step 4: more than one symbol

backtrader handles multiple feeds natively; add one PandasData per symbol and index them in the strategy with self.datas. Extract them in a single DuckDB query with WHERE symbol IN (...) and split the result with groupby("symbol") rather than scanning the files once per ticker.

  frame = duckdb.sql("""
      SELECT symbol, COALESCE(date, CAST(timestamp AS DATE)) AS datetime,
             open, high, low, close, volume
      FROM read_parquet('by_date/stock_daily/*/*.parquet', union_by_name=true)
      WHERE symbol IN ('AAPL', 'MSFT', 'NVDA')
      ORDER BY 2
  """).df()

  for symbol, df in frame.groupby("symbol"):
      feed = bt.feeds.PandasData(dataname=df.set_index("datetime").drop(columns="symbol"),
                                 openinterest=None)
      cerebro.adddata(feed, name=symbol)

Pitfalls

• Survivorship: the archive includes delisted tickers, so a universe built from today's index members is your choice, not a data limitation. Build the universe from the files for the backtest's start date.
• Session gaps: thin names skip days. backtrader tolerates that within a single feed; multi-feed strategies should align on the calendar of the most liquid feed.
• Adjusted vs traded prices: bars are split-adjusted only. Stops and limits execute at real traded levels; total return needs dividends added separately.
• Timezone: feed timestamps are naive US/Eastern. Do not localize to UTC before feeding, or session filters shift by four or five hours.

Related

vectorbt with Parquet data · DuckDB guide · Polars guide · AAPL data page · intraday stock data · pricing