Stock market data as Parquet: datasets that exist and how to check one

Parquet is the obvious format for price history: typed columns, compression, and DuckDB, Polars or pandas reading it with no parsing step. Few providers actually ship it. This page lists what the commercial sources deliver, which free Parquet datasets exist, and the checks that tell a usable dataset from a broken one. Sources were checked on 2026-10-29.

What the vendors actually deliver

If your source is CSV, convert once and keep the Parquet. The conversion is mechanical (see the end of this page). Most of the effort goes into the checks below, which apply whatever the format.

Free Parquet datasets

None of these states a licence that clearly allows redistribution, so check before building a product on one. The AWS Open Data registry and Google's BigQuery public datasets had no US stock price data at all; the one market-data entry on AWS covers European exchanges and is deprecated. The free tier here is the daily files for roughly the last 365 days, plus a fixed intraday sample pack; see free historical stock data.

Seven checks before you trust one

1. What the timestamp means

Parquet stores a timestamp either as an instant in UTC (isAdjustedToUTC=true) or as a wall-clock reading with no zone (false). pandas writes a tz-aware New York column as UTC instants. DuckDB then reads it as TIMESTAMPTZ and displays it in the session time zone, so a 09:30 open shows as 13:30+00 under UTC. Check before you filter on 09:30:

  import pyarrow.parquet as pq
  print(pq.read_schema("file.parquet").field("timestamp").type)
  # timestamp[us]                        wall clock, no zone (this archive: New York)
  # timestamp[ns, tz=America/New_York]   UTC instants plus a zone label

Daily bars belong in a DATE column. A daily bar stored as midnight UTC shows up as the previous day in any US time zone.

2. Bar open or bar close

Find the first regular-session bar. If it is stamped 09:30, labels are bar opens; if 09:31, bar closes. Mixing the two shifts every intraday signal by one bar. Our measurements of how bars are labelled and built: OHLCV data.

3. Adjustment

Look at a known split. NVDA split 10-for-1 effective 2024-06-10. Split-adjusted data shows the 2024-06-07 close near 120.89 (this archive: 120.888); raw data shows 1,208.88. Then check whether dividends are folded in as well: a dividend-adjusted series drifts below the traded prices the further back you go.

4. Survivorship

Count the symbols on an old date and how many still trade. In this archive the 2010-01-04 daily file holds 5,876 stocks, and 3,516 of them no longer trade. A dataset where nearly every symbol is still listed today has dropped the failures, and a backtest on it looks better than the market was.

  SELECT count(DISTINCT symbol) FROM 'daily/2010-01-04.parquet';

5. Missing days

A US trading year has about 252 sessions (248 in 2001, when markets closed after 9/11). Count distinct dates per year. A short year means missing files, and silently missing days break return calculations.

  SELECT year(date) AS y, count(DISTINCT date) AS sessions
  FROM read_parquet('daily/*.parquet') GROUP BY y ORDER BY y;

6. Layout: partitioning and row groups

Parquet readers skip row groups whose min/max statistics rule out a filter. That only works if the data is sorted by the filtered column. In a test of 2 million rows and 5,000 symbols in 20 row groups, a symbol filter had to read all 20 when the data was unsorted and 1 when it was sorted by symbol. Check yours:

  SELECT row_group_id, row_group_num_rows, stats_min_value, stats_max_value
  FROM parquet_metadata('file.parquet')
  WHERE path_in_schema = 'symbol';

For the record, this archive's 2026-10-28 stock 1-minute file has two row groups (1,048,576 and 292,330 rows), and both span symbols A to ZYME. A symbol filter reads both. That costs little at one day per file, about 1.3 million rows, but it is the kind of thing this query tells you. Writer defaults differ: pandas and pyarrow use snappy and row groups of up to 1,048,576 rows, Polars uses zstd and 262,144 rows, DuckDB snappy and 122,880 rows.

Folders named date=2024-01-03/ (hive partitioning) let DuckDB and Polars skip whole folders on a date filter. One file per date with the date in the file name works just as well with a glob.

7. Licence and provenance

A dataset with no README, no stated source and no licence is a snapshot of someone's scrape. That is fine for a notebook, and a risk for anything you publish or sell.

Converting a CSV dump to Parquet

DuckDB does it in one statement. Declare the types instead of trusting inference, and sort by symbol and time so readers can skip row groups.

  COPY (
      SELECT CAST(ts AS TIMESTAMP) AS timestamp, symbol,
             open::DOUBLE AS open, high::DOUBLE AS high, low::DOUBLE AS low,
             close::DOUBLE AS close, volume::DOUBLE AS volume,
             CAST(ts AS DATE) AS date
      FROM read_csv('vendor/*.csv', header = true)
      ORDER BY symbol, timestamp
  ) TO 'bars' (FORMAT parquet, PARTITION_BY (date), COMPRESSION zstd);

Adjust the column names to the vendor's. Check the time zone of ts first: convert UTC to New York before the cast if the vendor stamps UTC.

Related

DuckDB stock data guide · Polars guide · build a local market data lake · OHLCV data explained · intraday data providers · data quality