How to Process Archived Web Pages with Python for Historical Analysis

A price disappears from a website. A public notice changes its wording. A product page quietly drops a feature. Saved web pages can preserve those details, but a folder full of HTML files does not explain what changed or when. Python helps turn those files into a historical dataset you can inspect, compare, and trace back to its sources.

This tutorial builds a small workflow around Beautiful Soup, capture timestamps, and CSV exports. You will keep the original files intact, record where each capture came from, extract useful fields, and preserve enough context to check your findings later.

Decide What Each Historical Record Represents

Before writing a parser, choose the question your dataset should answer. You might track a subscription price, changes to a policy heading, or the availability of a service. A narrow question makes extraction easier and gives you a reason to keep particular fields.

For this example, each record represents one captured product page. The output will contain its title, displayed price, capture time, original address, archive reference, and local filename. A file hash will help confirm which saved bytes produced the record.

Keep three kinds of time separate. The capture time describes when an archive recorded the response. A publication date belongs to the page’s content. Your download time describes when you retrieved the saved file. These dates can differ by years.

A capture can establish that particular content was present in that archived response. It does not establish the exact moment someone published or changed that content. If two captures differ, the change happened somewhere between those observations, subject to gaps and capture limitations.

Prepare a Small Local Python Workspace

Create a project directory with a captures folder for your saved HTML. Put your Python script and a manifest CSV beside that folder. Keeping these pieces together makes the workflow easier to rerun or share.

Install Beautiful Soup in your project environment:

python -m pip install beautifulsoup4

The remaining examples use Python’s standard library. Beautiful Soup will parse the markup with the built-in html.parser backend. That choice keeps the setup small. For a larger project, record your Python version and package versions because parser behavior can affect difficult or malformed documents.

Open a few saved files before building selectors. Check whether they contain the content you expect. A successful download might contain an archive error screen, a redirect notice, or a page that depends on JavaScript for its main content.

Beautiful Soup reads the HTML you give it. It does not run JavaScript or retrieve missing API responses. A blank price in the saved markup therefore needs investigation before you treat it as a historical fact.

Collect Source Files With Consistent Capture Dates

Source collection deserves the same care as parsing. An archive downloader can fit into the file collection stage before Python processing begins. Check the retrieved files against their archive references, then record their provenance in your manifest.

Use a consistent date policy. For a monthly study, you might choose the nearest available capture to the first day of each month within a seven-day tolerance. Record the actual selected capture time, even when it differs from your target date.

Apply that rule across every page in the study. If no suitable capture exists, preserve the gap. Quietly substituting a capture from another month can make unrelated observations appear directly comparable.

Follow this collection sequence so each file enters the dataset with useful context:

1. Define the pages, target dates, and acceptable capture window.
2. Retrieve available captures and preserve the downloaded originals.
3. Record each original address, archive reference, and actual capture timestamp.
4. Check that the saved HTML contains the intended page.
5. Assign a stable local filename and add it to the manifest.

Keep an untouched copy of every input file. If you later remove archive navigation or repair malformed markup, write the result to a separate location. Your extraction should remain traceable to the original saved bytes.

Build a Manifest Before Extracting Content

A manifest connects local files with their historical context. Save the following as manifest.csv. The addresses are illustrative placeholders, so replace them with the actual source and capture references from your collection.

filename,original_url,archive_url,captured_at
product-20200101.html,ORIGINAL_ADDRESS_A,CAPTURE_REFERENCE_A,20200101123000
product-20200201.html,ORIGINAL_ADDRESS_A,CAPTURE_REFERENCE_B,2020-02-01T12:45:00Z
product-20200301.html,ORIGINAL_ADDRESS_A,CAPTURE_REFERENCE_C,2020-03-01T13:15:00+00:00

This example accepts compact archive timestamps and ISO-style timestamps with explicit offsets. Only treat a compact timestamp as UTC when its source format defines that convention. Do not infer capture time from a file’s modification time, which can change during copying or downloading.

Normalize Capture Timestamps Before Comparing Pages

Historical records become difficult to compare when their timestamps use mixed formats. Convert accepted inputs into timezone-aware UTC values and export them in one format.

The function below rejects timestamps without timezone information. It also checks the length of the compact form before parsing it. Add this code to process_captures.py:

import re
from datetime import datetime, timezone


def normalize_capture_time(value):
    value = value.strip()

    if re.fullmatch(r"[0-9]{14}", value):
        parsed = datetime.strptime(value, "%Y%m%d%H%M%S")
        parsed = parsed.replace(tzinfo=timezone.utc)
    else:
        if value.endswith("Z"):
            value = value[:-1] + "+00:00"

        parsed = datetime.fromisoformat(value)

        if parsed.tzinfo is None:
            raise ValueError(
                "Capture timestamp must include a timezone"
            )

    return (
        parsed.astimezone(timezone.utc)
        .isoformat(timespec="seconds")
        .replace("+00:00", "Z")
    )

For example, 2020-03-01T08:15:00-05:00 becomes 2020-03-01T13:15:00Z. The instant stays the same. Only its representation changes.

Attaching UTC with replace() is appropriate in the compact branch because this workflow requires that input format to represent UTC already. It would be incorrect to attach UTC to an unknown local time and call that a conversion.

Keep the original timestamp string in the exported data too. That small addition makes timezone mistakes easier to audit. If your source provides only a calendar date, preserve that lower precision in a separate field instead of inventing a capture time.

Extract Stable Fields With Beautiful Soup

Start with one representative capture and identify the elements you need. The sample parser expects a product heading inside main and a displayed price inside an element with the class price. Replace those selectors after inspecting your files.

Append this code to the same script:

from bs4 import BeautifulSoup


def extract_fields(raw_html):
    soup = BeautifulSoup(raw_html, "html.parser")

    for element in soup.select("script, style"):
        element.decompose()

    title_element = soup.select_one("main h1")
    price_element = soup.select_one("main .price")

    title = (
        title_element.get_text(" ", strip=True)
        if title_element is not None
        else ""
    )

    price_text = (
        price_element.get_text(" ", strip=True)
        if price_element is not None
        else ""
    )

    issues = []

    if not title:
        issues.append("missing_title")

    if not price_text:
        issues.append("missing_price")

    return {
        "title": title,
        "price_text": price_text,
        "review_notes": ";".join(issues),
    }

Passing bytes into Beautiful Soup lets it attempt encoding detection from the document. Check accented characters and currency symbols in the output. If the original response headers supply an encoding that the saved HTML lacks, preserve that information and handle it explicitly.

The space separator in get_text(" ", strip=True) helps keep adjacent text fragments from running together. It also reduces formatting noise while retaining readable content.

Keep the displayed price as text during this first pass. A value can include a currency symbol, a billing interval, or a regional number format. Removing every character except digits can turn a useful observation into an ambiguous number.

Account for Layout Changes Across Years

A selector that works in March may fail in April because the site changed its template. Missing extraction results do not automatically mean the underlying information disappeared.

Review captures from the beginning, middle, and end of your date range. If layouts differ, create explicit extraction rules for each known template. Record a parser version when maintaining a long-running dataset.

Avoid broad fallback selectors until you inspect what they return. Selecting the first heading anywhere in the document could capture an archive banner or an error message. A blank field with a review note is easier to correct than a convincing but unrelated value.

Export Historical Records With Their Provenance

The final stage reads the manifest, normalizes timestamps, parses each file, and writes the combined records. It also hashes every readable input. Matching SHA-256 values provide strong evidence that two records used identical file bytes, although a hash alone does not establish authenticity.

Append the following code below the earlier functions:

import csv
import hashlib
from pathlib import Path


BASE_DIR = Path(__file__).resolve().parent
CAPTURE_DIR = (BASE_DIR / "captures").resolve()

FIELDNAMES = [
    "filename",
    "original_url",
    "archive_url",
    "captured_at_raw",
    "captured_at_utc",
    "sha256",
    "title",
    "price_text",
    "review_notes",
]


def process_manifest():
    rows = []

    with (BASE_DIR / "manifest.csv").open(
        encoding="utf-8", newline=""
    ) as source:
        reader = csv.DictReader(source)

        for item in reader:
            row = {
                field: "" for field in FIELDNAMES
            }
            row.update({
                "filename": item["filename"],
                "original_url": item["original_url"],
                "archive_url": item["archive_url"],
                "captured_at_raw": item["captured_at"],
            })

            try:
                row["captured_at_utc"] = (
                    normalize_capture_time(item["captured_at"])
                )

                path = (
                    CAPTURE_DIR / item["filename"]
                ).resolve()

                if CAPTURE_DIR not in path.parents:
                    raise ValueError(
                        "File must be inside captures directory"
                    )

                raw_html = path.read_bytes()
                row["sha256"] = hashlib.sha256(
                    raw_html
                ).hexdigest()

                row.update(extract_fields(raw_html))

            except (OSError, ValueError) as error:
                row["review_notes"] = (
                    f"processing_error: {error}"
                )

            rows.append(row)

    rows.sort(key=lambda row: (
        row["captured_at_utc"],
        row["filename"],
    ))

    with (BASE_DIR / "historical_data.csv").open(
        "w", encoding="utf-8", newline=""
    ) as output:
        writer = csv.DictWriter(
            output, fieldnames=FIELDNAMES
        )
        writer.writeheader()
        writer.writerows(rows)


if __name__ == "__main__":
    process_manifest()

Run the complete script from your terminal:

python process_captures.py

The resulting historical_data.csv contains one row per manifest entry, including entries with processing errors. That keeps failures visible instead of silently removing them from the historical record.

The CSV writer handles commas, quotation marks, and line breaks inside extracted values. Opening output files with newline="" lets the CSV module handle its own line endings.

Check the Dataset Before Drawing Historical Claims

Review every row with a nonempty review_notes field. Then compare several successful rows against their saved HTML. Confirm that the title and price came from the intended content and that each timestamp matches its capture reference.

Check record counts too. The number of exported rows should match the number of manifest entries. Repeated hashes can reveal duplicate downloads, while different hashes can result from small markup changes that leave your selected fields untouched.

Interpret absence carefully. A missing price might reflect a new layout, a failed capture, or content loaded through JavaScript. Preserve these possibilities until you have inspected the evidence.

When reporting a change, describe your observations precisely. “The February capture shows a higher displayed price than the January capture” is supported by those records. A claim about the exact date of the increase needs more evidence.

Keep Every Finding Connected to Its Capture

A useful historical dataset lets another person follow your work backward. From a CSV row, they should be able to identify the capture, locate the saved file, check its bytes, and understand how you extracted the value.

Start with a small batch and inspect it closely. Once the timestamp rules and selectors behave reliably, process the larger collection. The result is more than extracted text. It is a set of dated observations with a clear path back to the pages that support them.