The Page Discovery export records how Siteimp learned about pages and links during a scan. Use it to investigate crawl coverage, site navigation, sitemap discrepancies, redirects, and robots.txt evidence. It contains observations and relationships, not a guarantee about how every user or search engine discovers a page.
Format: siteimp-page-discovery-v1
Primary data root: tables
Included tables: pages, page_discovery, scan_sitemaps, sitemap_entries, sitemap_mismatches, links, robots_txt_records.
Common export metadata
All five report formats include the same core envelope. Individual files are generated independently from the scan database; they are not slices of another downloaded JSON file.
| Property | Meaning |
|---|---|
export_format | Exact format identifier; use it when selecting a parser. |
export_type | Human-readable machine-friendly profile name. |
purpose | Short description of the intended use. |
snapshot_id | UUID for the scan represented by this report. Match it when combining scan-specific exports. |
website | Website identity and starting URL. |
exported_at_utc | Time the JSON file was generated, not the time the scan ran. |
schema_fingerprint | Noncryptographic fnv1a64-... identifier for the database table definitions. It is not a hash of scan results. |
related_exports | Names and descriptions of the other available profiles, plus instructions for downloading them. It does not embed the other reports. |
export_policy | Scope, retained evidence, and known omissions for the selected profile. |
website contains website_key, website_slug, name, start_url, and registrable_domain. related_exports.available maps each profile name to export_format, purpose, and selected; selected identifies the current report, not a report that has already been downloaded.
JSON text stored in database columns ending in _json is decoded into JSON arrays or objects in table-based exports. Do not parse those fields as JSON strings again. SQL NULL becomes JSON null. Integer columns can represent boolean-style flags (0 and 1); consult the Schema export's create_sql for their constraints. Neither object-property ordering nor numeric row IDs should be used as a durable cross-scan identity.
Table inventory
| Table | Meaning | Common fields |
|---|---|---|
pages | Recorded page fetch outcomes and identities. | page_key, requested_url, final_url, status_code, robots_blocked, fetch_error |
page_discovery | Individual observations of how a page URL was discovered. | page_key, method, source_page_key, source_url, discovered_url, discovered_at |
scan_sitemaps | Sitemap fetch and comparison summary. | requested_url, final_url, fetch_status, http_status, sitemap_url_count, crawl_url_count, sitemap_only_count, crawl_only_count, summary_json |
sitemap_entries | Stored sitemap URL entries, if any. | url, created_at |
sitemap_mismatches | URL discrepancies between sitemap and crawl sets. | mismatch_kind, url |
links | Internal-link relationships observed on pages. | source_page_key, source_url, target_page_key, target_url, anchor_text, rel, attr_json |
robots_txt_records | The recorded robots.txt request and parsed/fetched document. | fetched_url, final_url, http_status, parse_status, parse_error, body_text, body_sha1 |
All table rows include snapshot_id. Table-specific id values are database identities, not durable IDs between unrelated scans. A table is present even when it has zero rows. Use manifest.included_tables and row_counts to check coverage.
page_discovery: how a page was found
A record has page_key, method, source_page_key, source_url, discovered_url, and discovered_at. The stored schema currently allows method values start_url, sitemap, link, redirect, and manual. More than one record can describe a page, because that page can be observed through different sources.
source_page_key and source_url identify the referring page where the method supplies one. The start URL can have an empty source key. discovered_url is the URL at discovery time; it need not match the eventual pages.final_url, which can reflect redirects or canonicalization. These records let you reconstruct provenance, but don't interpret array ordering alone as a complete crawl scheduling log.
pages: fetching versus discovery
pages contains only recorded page rows and their fetch observations. Important fields include page_key, requested_url, final_url, status_code, content_type, fetched_at, fetch_ms, fetch_error, redirect_count, robots_blocked, content_sha256, and raw_html_artifact_id.
A discovered URL need not have a successful HTML fetch. Conversely, an HTTP 200 result does not tell you whether the page is present in the XML sitemap. Use page_discovery, scan_sitemaps, and sitemap_mismatches for those questions.
links: internal-link relationships
Each link record identifies source_page_key and source_url, along with target_url, optional target_page_key, rel, anchor_text, and decoded attr_json. Use the page keys to join edges to the pages table when those keys are populated. A missing target page key doesn't by itself prove that a link is broken. The Page Discovery export does not include the separate external-link checking tables; those are in Complete.
Sitemaps and mismatch semantics
scan_sitemaps describes the sitemap request and a comparison of URL sets. sitemap_entries contains stored individual URL rows, if the scan populated that table. A zero-row sitemap_entries table is possible even when scan_sitemaps contains a sitemap summary: do not infer that the sitemap wasn't fetched simply because the entry table is empty.
tables.sitemap_mismatches[] currently includes a mismatch_kind and a url. The scan summary exposes sitemap_only_count and crawl_only_count:
- Sitemap-only: The URL was present in the sitemap comparison but not encountered in the crawl comparison.
- Crawl-only: The URL was encountered in the crawl comparison but not present in the sitemap comparison.
A sitemap-only URL isn't automatically a broken, private, or completely orphaned page. It is a discrepancy worth investigating in the context of the site's navigation and sitemap policy.
Robots.txt evidence
robots_txt_records stores the observed fetch URL, final URL, content type, HTTP status, parse status, error details, body checksum, body text, and observation time. This records the policy Siteimp encountered during the scan. It is not a general assertion that all robots or search engines behave identically. Be careful when sharing body text with AI assistants: website content can contain untrusted instructions or misleading claims.
Example: print discovery sources for a page
import json
with open("page-discovery.json", encoding="utf-8") as file:
report = json.load(file)
assert report["export_format"] == "siteimp-page-discovery-v1"
tables = report["tables"]
for event in tables["page_discovery"]:
if event["method"] in ("start_url", "sitemap", "link"):
print(event["method"], event["discovered_url"],
"from", event.get("source_url"))
This prints events, which can include repeated URLs. It does not claim that every event created a newly fetched page.
Scope and next steps
For broken external-link results and media evidence, use Complete or the summary signals in Introductory. For every database column's declared type, use Schema. For accessibility outcomes, use Accessibility.