Data quality & methodology

What is actually in the files: adjustment math, session rules, roll handling, missing-bar policy, the QA gates every file passes before it is published, and a public correction changelog. Stocks & ETFs current through 2026-09-09 · futures through 2026-09-09.

Sources & provenance

Bars are licensed from an institutional market-data vendor that builds them from exchange trade feeds; we do not scrape prices or synthesize bars ourselves. Every timeframe (1-minute, 5-minute, 30-minute, 1-hour, daily) is delivered natively by the vendor and published as-is -- we never resample one timeframe into another, which avoids the precision drift and session-boundary bugs that resampling introduces.

Active tickers are refreshed every trading day after the close. Recently delisted tickers are merged in from a separate delisted feed weekly, with a checkpointed merge ledger so an interrupted merge can resume without dropping or duplicating rows.

Universe & survivorship bias

The daily archive covers over 21,000 distinct symbols across stocks, ETFs, and futures. Roughly 10,000 of those no longer trade -- they were delisted, acquired, or merged -- and their full history stays in the archive. A single recent trading day contains about 6,500 active stocks; a single day in 2010 contains thousands of tickers that no longer exist.

Delisted tickers are marked with a -DELISTED suffix on the symbol, so one file gives you the tradeable universe exactly as it existed on that date while still letting you separate live from dead listings with a single filter. The 2010-01-04 daily file alone contains 3,516 suffixed symbols. This is what "survivorship-bias-free" means concretely: a momentum screen run on 2010 data sees the stocks that later went to zero, not just the winners that survived to today.

Sessions & timestamps

  • All timestamps are US/Eastern, stored timezone-naive. The timestamp column is the bar open time.
  • Stock and ETF intraday files cover the extended session: first bar 04:00 (premarket), last 1-minute bar opens 19:59 (post-market). Regular-session bars open 09:30–15:59.
  • Futures intraday files cover the near-24-hour Globex-style session: bars from 00:00 through 23:59 Eastern.
  • Daily files carry one row per symbol per trading day. Files written since 2026-03-27 use a date (DATE) column; older archive files use a midnight timestamp instead -- see the changelog below and the docs for a one-line COALESCE pattern that reads both.

Corporate actions: split adjustment

Stock and ETF prices are split-adjusted: when a company splits its stock, all historical prices are divided (and volumes multiplied) by the split ratio, so price series are continuous and percentage returns are correct across the split.

A worked example you can verify in the archive: AAPL closed at $214.01 on 2010-01-04 as traded. Apple has since split 7-for-1 (2014) and 4-for-1 (2020), a combined factor of 28. The 2010-01-04 daily file shows AAPL close = 7.6432 -- exactly 214.01 ÷ 28.

Dividends are not embedded in the price series. Prices are the levels at which the stock actually traded (divided by later split factors), not a total-return series. This is deliberate: dividend ("adjusted close") smoothing rewrites the prices your stops, limits, and percentage moves would have executed at, and it silently changes the entire history every time a new dividend is paid -- which breaks backtest reproducibility. If your strategy needs total returns, apply a dividend series on top; the price files stay stable underneath.

Futures methodology

Futures are delivered as continuous, ratio back-adjusted contracts under root symbols (ES, CL, GC, ZN, ...), about 130 active roots per day. At each contract roll, history is back-adjusted by ratio rather than by point offset, which preserves percentage returns across rolls and never produces negative prices deep in the history. Two consequences to be aware of: far-history price levels differ from what the then-front contract printed (true of any continuous series), and volume is the traded volume of the underlying contract, not interpolated.

Missing bars & zero volume

Intraday files contain a bar only for minutes in which the symbol actually traded. No synthetic bars are inserted and nothing is forward-filled: AAPL has 741 one-minute bars on 2024-01-03 out of a 960-minute extended session, because it did not trade in every premarket and post-market minute. Illiquid small-caps have far fewer. If your pipeline needs a dense time grid, reindex against the session calendar and decide explicitly how to fill -- the data will not make that decision for you.

A small number of zero-volume bars do occur (about 0.1% of rows in a typical 1-minute file); they come from the vendor feed (indicative or odd-lot-only prints) and are passed through unmodified rather than being silently deleted.

QA gates every file passes

Before a Parquet file is published, the ingestion pipeline enforces:

  • Feed integrity -- vendor responses are validated as real ZIP archives; an HTML error or maintenance page can never masquerade as an empty-but-successful update.
  • Symbol-count floors -- a day's file is rejected and retried if it contains fewer than 4,000 stocks (2,500 ETFs / 20 futures roots), which catches partial vendor builds.
  • Trailing-edge re-pull -- the most recent 4 trading days are re-downloaded on every nightly run, so a day first captured while the vendor was still assembling it is replaced by the complete version.
  • Split-adjustment heal -- every nightly run diffs the vendor's full trailing 30-day window against the published files and rewrites any date where a symbol's values changed, so retroactive split re-basing and vendor corrections land in the archive instead of being skipped. Splits reaching further back are verified against an independent data source and back-adjusted through full history, automatically for unambiguous cases.
  • Continuity monitoring -- adjacent-session close ratios are scanned nightly for split-shaped jumps (including early "pre-applied" adjustments the vendor occasionally publishes before a split's effective date, which we quarantine until the split is real).
  • Duplicate resolution -- exact duplicate rows are dropped; conflicting rows for the same (timestamp, symbol) key keep the vendor's latest correction.
  • Schema enforcement -- every file is written against a fixed Arrow schema (typed columns, microsecond timestamps, Snappy compression) and deterministically sorted by time then symbol.
  • Range sanity -- spot checks across published files show 0 rows with high < low.

Corrections & changelog

Vendor-side corrections inside the trailing 30-day window are picked up by the nightly re-pull; anything older is repaired by an explicit force-overwrite of the affected dates. Changes that alter published files are recorded here; the affected tickers are listed on the changelog:

  • 2026-09-09 -- Full history re-based across every timeframe for ARTL (1-for-9), CALC (1-for-5), CLGN (1-for-10), FGL (1-for-100), KAPA (1-for-7), NIVF (1-for-3) and TANH (1-for-50) -- effective 2026-08-31 to 2026-09-04. The table lists every ticker adjusted to date. [tickers]
  • 2026-09-08 -- APH's 2-for-1 split (effective 2026-09-03) is now applied across its full history in every timeframe. [tickers]
  • 2026-09-06 -- Full history re-based across every timeframe for RUSHA (1.5:1) and RUSHB (1.5:1) -- effective 2026-09-01 to 2026-09-01. The table lists every ticker adjusted to date. [tickers]
  • 2026-09-06 -- Complete price history added for tickers that stopped trading, stored under the -DELISTED suffix across all five stock timeframes. This week: ALOT, BCAB, BGICF. [tickers]
  • 2026-09-03 -- PN's 1-for-20 reverse split (effective 2026-04-13) is now applied across its full history in every timeframe, and WLDS's 1-for-3 (effective 2026-06-22) across history through that date. The table lists every ticker adjusted to date. [tickers]
  • 2026-09-02 -- Full history re-based across every timeframe for CURX (1-for-20), CYCU, ELAB, GAME, IMCC (1-for-30), INCR, LGVN, LIMN (1-for-50), NVX, SCNI, SKYE, VCIG, WHLR and YMT -- reverse splits effective 2026-08-20 to 2026-08-28. [tickers]
  • 2026-08-31 -- REAX completed its combination with RE/MAX and now trades as Real REMAX Group; its 1-for-10 share consolidation (effective 2026-08-25) is applied across REAX's full history in every timeframe. VMAR's 1-for-10 reverse split (effective 2026-08-26) likewise applied across full history. [tickers]
  • 2026-08-28 -- GLMD daily bars before 2023-05-15 now carry its 1-for-15 reverse split (intraday bars already did). BRNX 2026-07-27 re-based for its 1-for-6 (effective 08-13). BTOG 1-for-5 and SMTK 1-for-50 (both effective 08-21): full history re-based across every timeframe. [tickers]
  • 2026-08-27 -- Full history re-based across every timeframe for CDT, CIIT, CISS (1-for-40), CPOP, CRIS, FTFT, GLMD (1-for-8), HKIT, JEM, LBGJ, MSS, NIVF, NUWE, PAVS, SFBS (2-for-1), SGLY, WHLR and ZNB, mostly reverse splits effective 2026-07-06 to 2026-08-21. The table lists every ticker adjusted to date. [tickers]
  • 2026-08-25 -- The 1-minute to 1-hour bars for 2026-07-22, 07-23 and 07-24 now match the daily bars for NRDY (1-for-15), CISS (1-for-40) and GLMD (1-for-8); those three sessions had stayed at pre-split prices after the daily bars were adjusted. [tickers]
  • 2026-08-20 -- Splits announced but not yet effective (SFBS 2-for-1 on 2026-08-21, IESC 2-for-1 on 08-24, RUSHA/RUSHB 3-for-2 on 09-01, LNOK 2-for-1 on 09-10). Daily bars for 2026-08-03 to 08-06 (LNOK 07-27 to 08-10) and intraday bars through 08-18 are as-traded again; each history is adjusted once its split takes effect. [tickers]
  • 2026-08-19 -- XCH's announced 1-for-20 reverse split has not taken effect yet. Those 20 sessions carry as-traded prices and volumes again, continuous with 2026-08-17 onward; the history is adjusted once the split is effective. [tickers]
  • 2026-08-19 -- Full history re-based for 22 further tickers, mostly reverse splits effective 2026-07-20 to 2026-08-14 (e.g. MVIS 1-for-15, BYND 1-for-30). Also re-based: AEHL, ELPW, SXTC, THH for 2026-07-13 to 2026-07-17; IPW, LABX, SNDQ through their split dates. The table lists every ticker adjusted to date, newest first. [tickers]
  • 2026-08-16 -- Complete price history added for tickers that stopped trading, stored under the -DELISTED suffix across all five stock timeframes. This week: CCIX, ELSE, EA, GVHGF, STRS, VSEE. Runs every Sunday; the table below is regenerated after each run. [tickers]
  • 2026-08-15 -- Complete price history added for tickers that stopped trading, stored under the -DELISTED suffix across all five stock timeframes. [tickers]
  • 2026-08-12 -- Prices and volumes re-based across full history for 221 tickers, so each series is continuous through its split. Ratio reads new-for-old: 4-for-1 is a forward split, 1-for-10 a reverse split. Every ticker adjusted to date is listed in the table under the newest split entry. [tickers]
  • 2026-07-27 -- All tickers delisted in the trailing year merged across every stock timeframe. [tickers]
  • 2026-03-27 -- Daily files switched from a midnight timestamp column to a typed date column. Files written before that date keep the legacy column; both read cleanly with union_by_name / COALESCE. [tickers]
  • Ongoing -- weekly delisted merges (Saturdays) append final history for newly delisted tickers with the -DELISTED suffix; the nightly pipeline re-pulls the vendor's trailing 30-day window and rewrites any published day whose values changed.

Check files yourself

Every claim above is verifiable locally -- the checks below run in milliseconds with DuckDB: 

  import duckdb

  con = duckdb.connect()
  checks = con.execute("""
      SELECT
          COUNT(*)                                     AS rows,
          COUNT(DISTINCT symbol)                       AS symbols,
          COUNT(DISTINCT CASE WHEN symbol LIKE '%-DELISTED'
                              THEN symbol END)         AS delisted_symbols,
          SUM(CASE WHEN high < low THEN 1 ELSE 0 END)  AS bad_ranges,
          SUM(CASE WHEN volume = 0 THEN 1 ELSE 0 END)  AS zero_volume,
          MIN(timestamp)                               AS first_bar,
          MAX(timestamp)                               AS last_bar
      FROM 'stock_1min_2024-01-03.parquet'
  """).fetchdf()

Expected on that file: 1,280,735 rows, 7,180 symbols, 1,194 delisted symbols, 0 bad ranges, first bar 04:00, last bar 19:59.

Related

stock data hub · browse stock daily · delisted stock data · continuous futures · DuckDB guide · pricing