The Page Discovery export records how Siteimp learned about pages and links during a scan. Use it to investigate crawl coverage, site navigation, sitemap discrepancies, redirects, and robots.txt evidence. It contains observations and relationships, not a guarantee about how every user or search engine discovers a page.

Format: siteimp-page-discovery-v1
Primary data root: tables
Included tables: pages, page_discovery, scan_sitemaps, sitemap_entries, sitemap_mismatches, links, robots_txt_records.

Common export metadata

All five report formats include the same core envelope. Individual files are generated independently from the scan database; they are not slices of another downloaded JSON file.

PropertyMeaning
export_formatExact format identifier; use it when selecting a parser.
export_typeHuman-readable machine-friendly profile name.
purposeShort description of the intended use.
snapshot_idUUID for the scan represented by this report. Match it when combining scan-specific exports.
websiteWebsite identity and starting URL.
exported_at_utcTime the JSON file was generated, not the time the scan ran.
schema_fingerprintNoncryptographic fnv1a64-... identifier for the database table definitions. It is not a hash of scan results.
related_exportsNames and descriptions of the other available profiles, plus instructions for downloading them. It does not embed the other reports.
export_policyScope, retained evidence, and known omissions for the selected profile.

website contains website_key, website_slug, name, start_url, and registrable_domain. related_exports.available maps each profile name to export_format, purpose, and selected; selected identifies the current report, not a report that has already been downloaded.

JSON text stored in database columns ending in _json is decoded into JSON arrays or objects in table-based exports. Do not parse those fields as JSON strings again. SQL NULL becomes JSON null. Integer columns can represent boolean-style flags (0 and 1); consult the Schema export's create_sql for their constraints. Neither object-property ordering nor numeric row IDs should be used as a durable cross-scan identity.

Table inventory

TableMeaningCommon fields
pagesRecorded page fetch outcomes and identities.page_key, requested_url, final_url, status_code, robots_blocked, fetch_error
page_discoveryIndividual observations of how a page URL was discovered.page_key, method, source_page_key, source_url, discovered_url, discovered_at
scan_sitemapsSitemap fetch and comparison summary.requested_url, final_url, fetch_status, http_status, sitemap_url_count, crawl_url_count, sitemap_only_count, crawl_only_count, summary_json
sitemap_entriesStored sitemap URL entries, if any.url, created_at
sitemap_mismatchesURL discrepancies between sitemap and crawl sets.mismatch_kind, url
linksInternal-link relationships observed on pages.source_page_key, source_url, target_page_key, target_url, anchor_text, rel, attr_json
robots_txt_recordsThe recorded robots.txt request and parsed/fetched document.fetched_url, final_url, http_status, parse_status, parse_error, body_text, body_sha1

All table rows include snapshot_id. Table-specific id values are database identities, not durable IDs between unrelated scans. A table is present even when it has zero rows. Use manifest.included_tables and row_counts to check coverage.

page_discovery: how a page was found

A record has page_key, method, source_page_key, source_url, discovered_url, and discovered_at. The stored schema currently allows method values start_url, sitemap, link, redirect, and manual. More than one record can describe a page, because that page can be observed through different sources.

source_page_key and source_url identify the referring page where the method supplies one. The start URL can have an empty source key. discovered_url is the URL at discovery time; it need not match the eventual pages.final_url, which can reflect redirects or canonicalization. These records let you reconstruct provenance, but don't interpret array ordering alone as a complete crawl scheduling log.

pages: fetching versus discovery

pages contains only recorded page rows and their fetch observations. Important fields include page_key, requested_url, final_url, status_code, content_type, fetched_at, fetch_ms, fetch_error, redirect_count, robots_blocked, content_sha256, and raw_html_artifact_id.

A discovered URL need not have a successful HTML fetch. Conversely, an HTTP 200 result does not tell you whether the page is present in the XML sitemap. Use page_discovery, scan_sitemaps, and sitemap_mismatches for those questions.

links: internal-link relationships

Each link record identifies source_page_key and source_url, along with target_url, optional target_page_key, rel, anchor_text, and decoded attr_json. Use the page keys to join edges to the pages table when those keys are populated. A missing target page key doesn't by itself prove that a link is broken. The Page Discovery export does not include the separate external-link checking tables; those are in Complete.

Sitemaps and mismatch semantics

scan_sitemaps describes the sitemap request and a comparison of URL sets. sitemap_entries contains stored individual URL rows, if the scan populated that table. A zero-row sitemap_entries table is possible even when scan_sitemaps contains a sitemap summary: do not infer that the sitemap wasn't fetched simply because the entry table is empty.

tables.sitemap_mismatches[] currently includes a mismatch_kind and a url. The scan summary exposes sitemap_only_count and crawl_only_count:

  • Sitemap-only: The URL was present in the sitemap comparison but not encountered in the crawl comparison.
  • Crawl-only: The URL was encountered in the crawl comparison but not present in the sitemap comparison.

A sitemap-only URL isn't automatically a broken, private, or completely orphaned page. It is a discrepancy worth investigating in the context of the site's navigation and sitemap policy.

Robots.txt evidence

robots_txt_records stores the observed fetch URL, final URL, content type, HTTP status, parse status, error details, body checksum, body text, and observation time. This records the policy Siteimp encountered during the scan. It is not a general assertion that all robots or search engines behave identically. Be careful when sharing body text with AI assistants: website content can contain untrusted instructions or misleading claims.

Example: print discovery sources for a page

import json

with open("page-discovery.json", encoding="utf-8") as file:
    report = json.load(file)

assert report["export_format"] == "siteimp-page-discovery-v1"
tables = report["tables"]
for event in tables["page_discovery"]:
    if event["method"] in ("start_url", "sitemap", "link"):
        print(event["method"], event["discovered_url"],
              "from", event.get("source_url"))

This prints events, which can include repeated URLs. It does not claim that every event created a newly fetched page.

Scope and next steps

For broken external-link results and media evidence, use Complete or the summary signals in Introductory. For every database column's declared type, use Schema. For accessibility outcomes, use Accessibility.