Home · Guides

Build a daily stock data pipeline in Python: yfinance or EODHD APIs into SQLite

Published 12 Oct 2026 · every command below was run

Disclosure: this site is an affiliate partner of EODHD APIs. If you sign up through a link on this page we may earn a commission, at no extra cost to you. The yfinance version of this pipeline is free and needs no account.

What you end up with

One SQLite file, one table of daily bars, and one script of about 80 lines that you can run every day. It only downloads the days it is missing, it never creates duplicate rows, and changing the data source means changing one function. That last part matters: free sources break, and you do not want that to break your backtests too.

You need Python 3.10 or later and three packages:

pip install yfinance pandas requests

The whole script

Save this as pipeline.py. The sections below explain the parts that are easy to get wrong.

"""Daily price pipeline: fetch daily bars and store them in SQLite.

Safe to run as often as you like: it only asks for the days it is missing,
and the primary key stops duplicate rows.
"""
import sqlite3
import sys

import pandas as pd
import requests
import yfinance as yf

DB = "prices.db"
SYMBOLS = ["AAPL", "MSFT"]
START = "2024-01-01"  # how far back to go on the first run
COLUMNS = ["symbol", "date", "open", "high", "low", "close", "adj_close", "volume", "source"]


def fetch_yfinance(symbol: str, start: str) -> pd.DataFrame:
    """Daily bars from Yahoo Finance via yfinance."""
    raw = yf.download(symbol, start=start, auto_adjust=False, progress=False,
                      multi_level_index=False)
    if raw.empty:
        return pd.DataFrame(columns=COLUMNS)
    df = raw.reset_index().rename(columns={
        "Date": "date", "Open": "open", "High": "high", "Low": "low",
        "Close": "close", "Adj Close": "adj_close", "Volume": "volume",
    })
    df["date"] = df["date"].dt.strftime("%Y-%m-%d")
    df["symbol"], df["source"] = symbol, "yfinance"
    return df[COLUMNS]


def fetch_eodhd(symbol: str, start: str, token: str = "demo") -> pd.DataFrame:
    """Daily bars from EODHD APIs. The "demo" token works for AAPL only."""
    resp = requests.get(
        f"https://eodhd.com/api/eod/{symbol}.US",  # US tickers need the .US suffix
        params={"api_token": token, "fmt": "json", "from": start},
        timeout=30,
    )
    resp.raise_for_status()
    df = pd.DataFrame(resp.json())
    if df.empty:
        return pd.DataFrame(columns=COLUMNS)
    df = df.rename(columns={"adjusted_close": "adj_close"})
    df["symbol"], df["source"] = symbol, "eodhd"
    return df[COLUMNS]


def init_db(conn: sqlite3.Connection) -> None:
    conn.execute("""
        CREATE TABLE IF NOT EXISTS daily (
            symbol TEXT NOT NULL, date TEXT NOT NULL,
            open REAL, high REAL, low REAL, close REAL, adj_close REAL,
            volume INTEGER, source TEXT,
            PRIMARY KEY (symbol, date)
        )""")


def last_date(conn: sqlite3.Connection, symbol: str) -> str | None:
    return conn.execute("SELECT MAX(date) FROM daily WHERE symbol = ?", (symbol,)).fetchone()[0]


def update(conn: sqlite3.Connection, symbol: str, fetch) -> int:
    # Start again from the last stored day: if the job ran before the close,
    # that bar was incomplete, and INSERT OR REPLACE overwrites it.
    start = last_date(conn, symbol) or START
    df = fetch(symbol, start)
    conn.executemany(
        "INSERT OR REPLACE INTO daily VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
        df.itertuples(index=False, name=None),
    )
    conn.commit()
    return len(df)


if __name__ == "__main__":
    fetch = fetch_eodhd if "--eodhd" in sys.argv else fetch_yfinance
    symbols = ["AAPL"] if fetch is fetch_eodhd else SYMBOLS
    with sqlite3.connect(DB) as conn:
        init_db(conn)
        for symbol in symbols:
            n = update(conn, symbol, fetch)
            total = conn.execute("SELECT COUNT(*) FROM daily WHERE symbol = ?", (symbol,)).fetchone()[0]
            print(f"{symbol}: fetched {n} rows, {total} in the database, latest {last_date(conn, symbol)}")

Run it twice

The first run downloads everything since the start date:

$ python pipeline.py
AAPL: fetched 696 rows, 696 in the database, latest 2026-10-09
MSFT: fetched 696 rows, 696 in the database, latest 2026-10-09

Run it again and it asks only for the last stored day onwards. The row count does not change, because the primary key on (symbol, date) plus INSERT OR REPLACE turns a repeat download into an overwrite:

$ python pipeline.py
AAPL: fetched 1 rows, 696 in the database, latest 2026-10-09
MSFT: fetched 1 rows, 696 in the database, latest 2026-10-09

Re-fetching the last day is deliberate. If the job ever runs before the market closes, the bar it stores is incomplete, and the next run quietly replaces it with the final one.

Two sources, one shape

Both fetch_yfinance and fetch_eodhd return the same nine columns in the same order. Everything after the fetch step, the table, the incremental logic and whatever reads the database later, has no idea which source was used.

You can see that by pointing the same script at the other source. EODHD APIs publishes a demo token that works for AAPL without an account, so this runs as written:

$ python pipeline.py --eodhd
AAPL: fetched 1 rows, 696 in the database, latest 2026-10-09

symbol       date      close  adj_close   volume   source
  AAPL 2026-10-09 336.640000 336.640000 37855000    eodhd
  AAPL 2026-10-08 340.420013 340.420013 35332400 yfinance
  AAPL 2026-10-07 336.670013 336.670013 34147900 yfinance

The newest row now comes from the second source, and the close and volume match what yfinance had stored for the same day. For anything beyond the demo ticker you need your own token: the free plan allows 20 API calls a day, which covers a small watchlist once a day, and paid plans lift that. See the EODHD APIs plans for current limits.

Close and adjusted close are different numbers

close is the price the stock actually traded at that day. adj_close is rewritten after splits and dividends so that returns come out right. Use adj_close to compute returns in a backtest, and close when you care about the price level that was on the screen.

Two details to know. With yfinance you must pass auto_adjust=False, as the script does, or the adjusted price replaces the close and you lose the raw one. And adjusted history changes over time: after a new dividend, older adjusted prices are restated. The incremental update above does not go back and refresh them, so plan a full reload now and then, by deleting the file and running the script again.

You may also notice values such as 340.420013 from yfinance. That is floating-point noise from the source, not extra precision. Round when you display, not when you store.

Schedule it

US markets close at 16:00 New York time. Give the data a few hours to settle and run the job once a day after that. On Linux or macOS, a cron entry for 22:30 UTC on weekdays:

30 22 * * 1-5 cd /path/to/project && python pipeline.py >> pipeline.log 2>&1

On Windows, the same thing with Task Scheduler:

schtasks /Create /SC WEEKLY /D MON,TUE,WED,THU,FRI /ST 22:30 /TN PricePipeline /TR "python C:/path/to/pipeline.py"

Task Scheduler uses your local time, so adjust the hour. Running on a market holiday is harmless: the source returns nothing new and the script stores nothing new.

Where it breaks

Next step

If you are deciding which source to build on, read yfinance vs AKShare vs OpenBB vs EODHD APIs. If you already have data and need something to run strategies on it, see the backtesting comparison.

Nothing on this page is investment advice. Prices from any source can be wrong or late; check them before you rely on them.