[ Build a local market data lake ]

Keep downloaded OHLCV Parquet files in the same by-date structure MarketParquet uses.

[ Directory layout ]

MarketParquet uses one file per trading day with every symbol in that asset/timeframe. Keeping that layout locally makes cross-sectional and time-series loads straightforward.

  market-data/
    by_date/
      stock_daily/
        2024/
          2024-01-15.parquet
      stock_1min/
        2024/
          2024-01-15.parquet
      futures_daily/
        2024/
          2024-01-15.parquet

[ Download with the API ]

  import pathlib
  import requests

  API_KEY = "bt_YOUR_KEY"
  BASE = "https://marketparquet.com/api/v1"
  asset_type = "stock_daily"
  date = "2024-01-15"
  headers = {"Authorization": f"Bearer {API_KEY}"}

  meta = requests.get(f"{BASE}/download/{asset_type}/{date}", headers=headers).json()
  out = pathlib.Path("market-data/by_date") / asset_type / date[:4] / f"{date}.parquet"
  out.parent.mkdir(parents=True, exist_ok=True)
  out.write_bytes(requests.get(meta["download_url"]).content)

[ Query across the lake ]

  import duckdb

  con = duckdb.connect()
  df = con.execute("""
      SELECT symbol, date, close
      FROM 'market-data/by_date/stock_daily/2024/*.parquet'
      WHERE symbol = 'SPY'
      ORDER BY date
  """).fetchdf()

[ Related ]

stock data hub · browse stock daily · pricing · DuckDB guide · Polars guide