The Add website page creates a saved website record in your local Siteimp workspace. It asks for the minimum information Siteimp needs before it can prepare the site for scanning: a clear name and a full start URL.

This page also previews the site’s current robots.txt file. The preview is there to explain what Siteimp sees before the website is saved, including whether the file has a Siteimp group, a wildcard group, allow rules, disallow rules, crawl delay, sitemap locations, and raw robots.txt content.

Adding a website does not run a scan. It saves the website first. After that, you can verify ownership, review robots rules and crawl settings, then run the first scan from the Website dashboard.

What this page is for

The Add website page answers one practical question:

What site should Siteimp know about before it starts collecting evidence?

A website record gives Siteimp a stable local home for:

  • the website name
  • the primary start URL
  • the registrable domain
  • crawl settings for this site
  • ownership status
  • future scan history
  • future monitoring setup

This keeps setup separate from scanning. Siteimp can save the website first, then use the Website dashboard and Scan Results pages for deeper work.

What you can do here

Add a website name

The Name field is the label Siteimp uses in your dashboard and scan history. It does not have to match the domain exactly. Use a name that will make sense when you return to the app later.

Good examples:

  • Formimp
  • StorePhotos.ca
  • Client marketing site
  • Main company website

The name is for your local workspace. It helps you recognize the site quickly, especially once you have more than one website in Siteimp.

Enter the start URL

The Start URL is the primary URL Siteimp should begin from. Use the full address, including https:// or http://.

For example:

https://example.com/

The start URL matters because Siteimp uses it to understand the website’s starting point. It also tells Siteimp where to look for robots.txt before the first scan.

Preview robots.txt

The Robots.txt preview card appears beside the form. Before a valid start URL is entered, it waits for a start URL. Once the start URL is valid, Siteimp checks for a robots.txt file and summarizes what it finds.

The preview can show:

  • whether robots.txt was found
  • whether a Siteimp group was found
  • whether a wildcard group was found
  • whether either group includes visible Allow or Disallow rules
  • whether a Siteimp-specific root block appears
  • whether a wildcard root block appears
  • the fetched URL
  • the final location, if redirects were involved
  • crawl delay
  • sitemap locations
  • raw robots.txt content

The preview is a setup aid. Siteimp checks robots.txt again before a scan starts, so the scan uses the current policy at scan time.

Siteimp groups and wildcard groups

A robots.txt file can include rules for specific crawlers and rules for all crawlers.

Siteimp uses two concepts in the preview:

  • Siteimp group means rules under User-agent: Siteimp.
  • Wildcard group means rules under User-agent: *.

The repeated “group” language is intentional. It helps you see which part of robots.txt Siteimp is reading.

No Siteimp group

If the preview says No Siteimp group, the site did not publish a User-agent: Siteimp group.

In that case, Siteimp uses the wildcard group below when deciding whether the URL can be scanned.

Siteimp group found

If the preview says Siteimp group found, the site has a User-agent: Siteimp group.

A Siteimp group can contain Allow, Disallow, or crawl-delay rules that apply specifically to Siteimp.

Siteimp hard block

If the Siteimp group contains a matching Disallow rule, Siteimp treats that as a hard stop.

A Siteimp-specific block is always respected. Ownership override does not bypass it.

For example:

User-agent: Siteimp
Disallow: /

This tells Siteimp not to scan the site.

Wildcard group found

If the preview says Wildcard group found, the site has a User-agent: * group. This group applies to general automated access.

For example:

User-agent: *
Allow: /

or:

User-agent: *
Disallow: /private

Wildcard block

If the wildcard group contains a matching Disallow rule, Siteimp treats that as a general robots block.

A wildcard block is respected unless the website has verified ownership and owner override is active.

For example:

User-agent: *
Disallow: /

This blocks ordinary strict scanning. After ownership is verified, owner override can allow Siteimp to scan a site you own or control.

Siteimp Allow does not bypass wildcard Disallow

A Siteimp-specific Allow does not bypass a wildcard Disallow.

For example:

User-agent: Siteimp
Allow: /

User-agent: *
Disallow: /

Siteimp still sees the wildcard block. Verified ownership and owner override are needed before scanning can continue.

This keeps the Siteimp group focused on the clearest site-owner signal: an explicit Siteimp block.

Set optional crawl settings

The Crawl settings card lets you set draft website-specific crawl behavior before the website exists.

These values are optional. Leaving them blank means Siteimp can use the broader app defaults.

The available draft settings may include:

  • crawl delay override
  • crawl jitter override
  • max concurrency override

These settings are about politeness and scan behavior. A delay controls the minimum wait between requests. Jitter adds a small random extra delay. Concurrency controls how many crawl requests can run at the same time when that option is shown.

On this setup page, these are draft values. They are applied right after the website is created.

Review ownership status

Before the website is saved, the Ownership card explains that ownership verification is not active yet. This is expected. Siteimp needs a saved website record before it can generate a verification challenge.

Ownership verification matters because it can unlock owner override for wildcard robots blocks on a site you own or control.

Before save, the card explains the available methods:

  • Meta Tag, usually the fastest option
  • DNS TXT, a strong long-term option

After save, the card becomes live and can generate verification challenges.

How to use this page

A good setup flow is:

  1. Enter a clear website name.
  2. Enter the full start URL.
  3. Review the robots.txt preview when it appears.
  4. Leave crawl settings blank unless you already know you want custom behavior.
  5. Save the website.
  6. Verify ownership now, or continue to the Website dashboard and verify later.
  7. Run the first scan from the Website dashboard when you are ready.

For most websites, the safest path is to fill in the name and start URL first, review the side cards, then save without changing advanced settings.

Required fields

Name

The name is required because Siteimp needs a readable label for the website in your local dashboard and scan history.

This field is part of your local workspace.

Start URL

The start URL is required because Siteimp needs a primary address for the website.

The URL must be a complete address. Use:

https://example.com/

instead of:

example.com

A full URL helps Siteimp avoid guessing which scheme, host, or path you meant.

What happens when you save

When you choose Save website, Siteimp validates the form first.

If the fields are valid, Siteimp:

  1. creates the website record
  2. applies any draft crawl settings
  3. reloads the saved website details
  4. switches the page into the Website created state

The created state shows a website summary with:

  • name
  • start URL
  • registrable domain
  • website key

At this point, the website is part of your Websites dashboard and ready for scan setup.

Website created state

After the website is saved, the page changes from a form into a confirmation view.

From there, you can:

  • choose Add another to reset the form and add another website
  • choose Continue to dashboard to open the new Website dashboard
  • use the now-active Ownership card to verify the website
  • review the robots preview for the saved start URL

The ownership card is open by default after creation because verification is often the next useful setup step.

Ownership verification

Ownership verification proves that you control the website.

Siteimp supports two verification methods:

Meta Tag

Meta tag verification is usually the fastest option. Siteimp generates a meta tag that you add to your homepage HTML. After adding the tag, choose Verify now.

The tag must be present in the server-delivered HTML. If the tag is added later by client-side JavaScript, Siteimp may not be able to verify it.

DNS TXT

DNS TXT verification is a strong long-term option. Siteimp generates a TXT record name and value that you add to your DNS settings. After the record is in place, choose Verify now.

DNS changes can take time to propagate. If verification does not pass right away, wait a little longer and try again.

Owner override and scan policy

Until ownership is verified, owner override is not available.

After ownership is verified, Siteimp can let you choose how future scans behave:

  • Respect robots.txt keeps scans in strict mode.
  • Use owner override allows verified-owner scanning through wildcard robots blocks.

Owner override applies to wildcard robots blocks only. It does not bypass a Siteimp-specific block.

For most sites, respecting robots.txt is the safest default. Owner override is for websites you own or control where you intentionally want Siteimp to scan with verified-owner permissions.

Robots.txt preview states

Waiting for start URL

Before a start URL is entered, the preview waits. This is normal. Siteimp cannot know where to look for robots.txt until it has a valid website address.

Checking robots.txt

After you enter a valid start URL, Siteimp waits briefly and then checks for robots.txt. This small delay helps avoid checking while you are still typing.

Robots found

This means Siteimp found a robots.txt file and created a preview. Open the details to inspect fetched location, groups, rules, sitemap entries, and raw content.

No robots.txt found

This means Siteimp did not find a robots.txt file at the expected location.

That does not automatically mean the site cannot be scanned. It means the site did not publish a robots file where Siteimp looked.

Robots.txt needs review

This means the file was fetched, but the parser could not interpret it cleanly. Open the details and review the raw content before relying on the preview.

Robots preview unavailable

This means Siteimp could not preview robots.txt at that moment. The cause may be temporary network trouble, a server response problem, or an unreachable URL.

The preview is still only a guide. Scan startup checks robots.txt again before running.

Crawl settings

The crawl settings card is for website-specific politeness rules.

Crawl delay override

This is the minimum wait between requests before jitter is applied. A larger delay makes the crawl gentler but slower.

Crawl jitter override

This adds random extra delay per request. Jitter can help avoid sending requests in an overly mechanical pattern.

Max concurrency override

When shown, this controls the maximum number of simultaneous crawl requests. Using 1 is the safest polite default.

Leaving fields blank

Blank fields mean “do not set a website-specific override.” In that case, Siteimp can use its app-level defaults instead.

Validation messages

Website name needed

This message appears when you try to save without entering a name.

Add a short name for the website, then save again.

Start URL needed

This message appears when you try to save without entering a start URL.

Enter the primary URL Siteimp should start from. Include the full address.

Start URL needs a full address

This message appears when the start URL is not a complete URL.

Use a full address with http:// or https:// at the front.

For example:

https://example.com/

Troubleshooting

Website was not saved

If Siteimp says Website was not saved, the app could not create the website record.

Try these steps:

  1. Confirm the name is filled in.
  2. Confirm the start URL is a complete URL.
  3. Check whether the website already exists in your workspace.
  4. Try saving again.
  5. Contact support if the error continues.

Crawl settings not saved

If crawl settings fail to save, the website-specific override may not have been applied.

Check that numeric fields contain whole numbers. Delay and jitter must be 0 or higher. Concurrency, when available, must be 1 or higher.

You can still save the website without custom crawl settings and adjust them later from the Website dashboard or settings area.

Ownership not verified yet

This means Siteimp checked the ownership challenge but did not find what it needed yet.

For meta tag verification:

  • confirm the tag was added to the homepage
  • confirm it is present in the server-delivered HTML
  • try verifying again

For DNS TXT verification:

  • confirm the TXT record name and value match the generated challenge
  • wait for DNS propagation
  • try verifying again

Ownership check failed

This means Siteimp could not complete the ownership check.

The cause may be a network problem, DNS issue, server issue, or unexpected response. Review the error shown in the app, then try again. If it keeps failing, contact support with the website name, verification method, and what happened.

Robots preview unavailable

This means Siteimp could not preview robots.txt right now. You can still save the website and try again later from the Website dashboard. Scan startup also checks robots.txt before running.

Where to go next

After adding a website, the next destination is usually the Website dashboard.

From there, you can:

  • review the saved website
  • verify ownership if you did not do it here
  • review robots rules and crawl settings
  • run the first scan
  • use Scan Results as the evidence hub
  • set up monitoring when you are ready

The Add website page is the setup step. The Website dashboard is where the website becomes operational. The Scan Results pages are where Siteimp starts showing deeper evidence.