The Add website page creates a saved website record in your local Siteimp workspace. It asks for the minimum information Siteimp needs before it can prepare the site for scanning: a clear name and a full start URL.
This page also previews the site’s current robots.txt file. The preview is there to explain what Siteimp sees before the website is saved, including whether the file has a Siteimp group, a wildcard group, allow rules, disallow rules, crawl delay, sitemap locations, and raw robots.txt content.
Adding a website does not run a scan. It saves the website first. After that, you can verify ownership, review robots rules and crawl settings, then run the first scan from the Website dashboard.
What this page is for
The Add website page answers one practical question:
What site should Siteimp know about before it starts collecting evidence?
A website record gives Siteimp a stable local home for:
- the website name
- the primary start URL
- the registrable domain
- crawl settings for this site
- ownership status
- future scan history
- future monitoring setup
This keeps setup separate from scanning. Siteimp can save the website first, then use the Website dashboard and Scan Results pages for deeper work.
What you can do here
Add a website name
The Name field is the label Siteimp uses in your dashboard and scan history. It does not have to match the domain exactly. Use a name that will make sense when you return to the app later.
Good examples:
FormimpStorePhotos.caClient marketing siteMain company website
The name is for your local workspace. It helps you recognize the site quickly, especially once you have more than one website in Siteimp.
Enter the start URL
The Start URL is the primary URL Siteimp should begin from. Use the full address, including https:// or http://.
For example:
https://example.com/
The start URL matters because Siteimp uses it to understand the website’s starting point. It also tells Siteimp where to look for robots.txt before the first scan.
Preview robots.txt
The Robots.txt preview card appears beside the form. Before a valid start URL is entered, it waits for a start URL. Once the start URL is valid, Siteimp checks for a robots.txt file and summarizes what it finds.
The preview can show:
- whether
robots.txtwas found - whether a Siteimp group was found
- whether a wildcard group was found
- whether either group includes visible
AlloworDisallowrules - whether a Siteimp-specific root block appears
- whether a wildcard root block appears
- the fetched URL
- the final location, if redirects were involved
- crawl delay
- sitemap locations
- raw
robots.txtcontent
The preview is a setup aid. Siteimp checks robots.txt again before a scan starts, so the scan uses the current policy at scan time.
Siteimp groups and wildcard groups
A robots.txt file can include rules for specific crawlers and rules for all crawlers.
Siteimp uses two concepts in the preview:
- Siteimp group means rules under
User-agent: Siteimp. - Wildcard group means rules under
User-agent: *.
The repeated “group” language is intentional. It helps you see which part of robots.txt Siteimp is reading.
No Siteimp group
If the preview says No Siteimp group, the site did not publish a User-agent: Siteimp group.
In that case, Siteimp uses the wildcard group below when deciding whether the URL can be scanned.
Siteimp group found
If the preview says Siteimp group found, the site has a User-agent: Siteimp group.
A Siteimp group can contain Allow, Disallow, or crawl-delay rules that apply specifically to Siteimp.
Siteimp hard block
If the Siteimp group contains a matching Disallow rule, Siteimp treats that as a hard stop.
A Siteimp-specific block is always respected. Ownership override does not bypass it.
For example:
User-agent: Siteimp
Disallow: /
This tells Siteimp not to scan the site.
Wildcard group found
If the preview says Wildcard group found, the site has a User-agent: * group. This group applies to general automated access.
For example:
User-agent: *
Allow: /
or:
User-agent: *
Disallow: /private
Wildcard block
If the wildcard group contains a matching Disallow rule, Siteimp treats that as a general robots block.
A wildcard block is respected unless the website has verified ownership and owner override is active.
For example:
User-agent: *
Disallow: /
This blocks ordinary strict scanning. After ownership is verified, owner override can allow Siteimp to scan a site you own or control.
Siteimp Allow does not bypass wildcard Disallow
A Siteimp-specific Allow does not bypass a wildcard Disallow.
For example:
User-agent: Siteimp
Allow: /
User-agent: *
Disallow: /
Siteimp still sees the wildcard block. Verified ownership and owner override are needed before scanning can continue.
This keeps the Siteimp group focused on the clearest site-owner signal: an explicit Siteimp block.
Set optional crawl settings
The Crawl settings card lets you prepare optional website-specific delay and jitter settings before the website is saved. Leave either field blank to inherit the applicable app default.
- Crawl delay override: base wait after a worker request, measured in milliseconds. The minimum supported value is 200 ms.
- Crawl jitter override: random additional wait. 0 ms is valid but strongly discouraged, since fixed, predictable timing can make automated traffic easier for bot protection to recognize.
Worker concurrency is handled internally by Siteimp and is not an editable setting on this page.
These are draft values until you create the website. Choosing Save settings on the card updates the draft, and the associated InfoDock message refers to that saved draft. The values are applied after you choose Save website. The website can also be created without custom crawl settings.
Crawl-speed warnings and InfoDock
Siteimp warns about aggressive settings while you edit them and repeats the warning in InfoDock after you choose Save settings. The notification confirms what was saved and describes the possible consequences. A strong caution remains visible until dismissed.
| Effective delay or jitter | Guidance |
|---|---|
| Delay of 1,000 ms or more, with nonzero jitter | Normal save confirmation |
| Delay of 500–999 ms | Caution: faster crawling can increase server load and rate limiting |
| Delay of 200–499 ms | Strong caution: aggressive crawling can strain a website and trigger rate limits |
| Jitter of 0 ms at any delay | Strong caution: fixed, predictable timing can make automated requests easier for bot protection to identify |
| Delay below 200 ms | Not supported; the setting cannot be saved |
Warnings depend on the effective combination of delay and jitter. For example, 200 ms delay with 0 ms jitter produces a strong caution about both server load and predictable timing. Verification can change the warning language, but it does not relax the 200 ms floor. Only use aggressive settings with the website operator's permission.
A configured delay is a base wait after a worker request, before random jitter is added. Concurrent workers and browser subrequests can still overlap. It is not a guarantee that every HTTP request reaching the website is spaced by that interval.
Review ownership status
Before the website is saved, the Ownership card explains the available verification methods. Siteimp needs a saved website record before it can generate a challenge.
- Meta Tag: add a generated verification tag to the server-delivered homepage HTML.
- DNS TXT: publish the generated TXT record at the specified domain and allow time for DNS propagation.
After saving, the card becomes active. You can generate a challenge, choose Verify now, and view the result and last-check details. A successful check marks the website Verified.
Verification has two separate purposes. It makes Use owner override available for wildcard robots.txt blocks, and it informs crawl-speed safety guidance. Verifying ownership does not automatically turn on owner override or remove scan-speed limits. The default scan policy remains Respect robots.txt until you explicitly change it.
How to use this page
A good setup flow is:
- Enter a clear website name.
- Enter the full start URL.
- Review the
robots.txtpreview when it appears. - Leave crawl settings blank unless you already know you want custom behavior.
- Save the website.
- Verify ownership now, or continue to the Website dashboard and verify later.
- Run the first scan from the Website dashboard when you are ready.
For most websites, the safest path is to fill in the name and start URL first, review the side cards, then save without changing advanced settings.
Required fields
Name
The name is required because Siteimp needs a readable label for the website in your local dashboard and scan history.
This field is part of your local workspace.
Start URL
The start URL is required because Siteimp needs a primary address for the website.
The URL must be a complete address. Use:
https://example.com/
instead of:
example.com
A full URL helps Siteimp avoid guessing which scheme, host, or path you meant.
What happens when you save
When you choose Save website, Siteimp validates the form first.
If the fields are valid, Siteimp:
- creates the website record
- applies any draft crawl settings
- reloads the saved website details
- switches the page into the Website created state
The created state shows a website summary with:
- name
- start URL
- registrable domain
- website key
At this point, the website is part of your Websites dashboard and ready for scan setup.
Website created state
After the website is saved, the page changes from a form into a confirmation view.
From there, you can:
- choose Add another to reset the form and add another website
- choose Continue to dashboard to open the new Website dashboard
- use the now-active Ownership card to verify the website
- review the robots preview for the saved start URL
The ownership card is open by default after creation because verification is often the next useful setup step.
Ownership verification
Ownership verification checks that you control a website. Siteimp supports Meta Tag and DNS TXT methods. Either method can mark the website as verified, and the status card shows the verification method, latest check result, and check detail.
Meta Tag
Generate a meta tag challenge, add the tag to the homepage's server-delivered HTML, then choose Verify now. A tag inserted only by client-side JavaScript may not be detected.
DNS TXT
Generate a DNS TXT challenge and add the exact record name and value to your DNS settings. Wait for propagation, then choose Verify now.
You can check ownership again later. Generating a new token does not, by itself, revoke an already verified website. A failed later check can appear in the latest-result details while previously verified status is retained.
Owner override and scan policy
Until ownership is verified, Use owner override is unavailable. Once verified, you can choose between:
- Respect robots.txt, which keeps the strict robots policy.
- Use owner override, which allows verified-owner scanning through applicable wildcard robots blocks.
Owner override does not bypass a Siteimp-specific block. Verification alone does not enable override, and neither verification nor override changes the 200 ms minimum crawl delay. A verified owner can still configure an overly aggressive crawl, so speed warnings remain relevant.
Robots.txt preview states
Waiting for start URL
Before a start URL is entered, the preview waits. This is normal. Siteimp cannot know where to look for robots.txt until it has a valid website address.
Checking robots.txt
After you enter a valid start URL, Siteimp waits briefly and then checks for robots.txt. This small delay helps avoid checking while you are still typing.
Robots found
This means Siteimp found a robots.txt file and created a preview. Open the details to inspect fetched location, groups, rules, sitemap entries, and raw content.
No robots.txt found
This means Siteimp did not find a robots.txt file at the expected location.
That does not automatically mean the site cannot be scanned. It means the site did not publish a robots file where Siteimp looked.
Robots.txt needs review
This means the file was fetched, but the parser could not interpret it cleanly. Open the details and review the raw content before relying on the preview.
Robots preview unavailable
This means Siteimp could not preview robots.txt at that moment. The cause may be temporary network trouble, a server response problem, or an unreachable URL.
The preview is still only a guide. Scan startup checks robots.txt again before running.
Crawl settings
Crawl delay override
The minimum supported delay is 200 ms. It is a base wait applied after a worker request, before jitter. Longer waits generally reduce request pressure, but concurrent workers and browser subrequests can still overlap.
Crawl jitter override
Jitter adds random extra delay. 0 ms is permitted but strongly discouraged because fixed request timing can be easier for bot protection systems to detect.
Leaving fields blank
Blank fields mean no website-specific override for that field. Siteimp inherits the app default, or the built-in fallback when no app default is set.
The crawl settings card shows warnings while editing, and InfoDock displays a confirmation and any relevant caution when draft settings are saved. Draft changes are applied when the website is created. The thresholds and warnings are described in Set optional crawl settings above.
Validation messages
Website name needed
This message appears when you try to save without entering a name.
Add a short name for the website, then save again.
Start URL needed
This message appears when you try to save without entering a start URL.
Enter the primary URL Siteimp should start from. Include the full address.
Start URL needs a full address
This message appears when the start URL is not a complete URL.
Use a full address with http:// or https:// at the front.
For example:
https://example.com/
Troubleshooting
Website was not saved
If Siteimp says Website was not saved, the app could not create the website record.
Try these steps:
- Confirm the name is filled in.
- Confirm the start URL is a complete URL.
- Check whether the website already exists in your workspace.
- Try saving again.
- Contact support if the error continues.
Crawl settings not saved
If crawl settings fail to save, the website-specific override may not have been applied.
Check that numeric fields contain whole numbers. Delay must be a whole number of 200 ms or higher. Jitter must be a whole number of 0 ms or higher. A zero-jitter setting can be saved but triggers a strong caution in InfoDock.
You can still save the website without custom crawl settings and adjust them later from the Website dashboard or settings area.
Ownership not verified yet
This means Siteimp checked the ownership challenge but did not find what it needed yet.
For meta tag verification:
- confirm the tag was added to the homepage
- confirm it is present in the server-delivered HTML
- try verifying again
For DNS TXT verification:
- confirm the TXT record name and value match the generated challenge
- wait for DNS propagation
- try verifying again
Ownership check failed
This means Siteimp could not complete the ownership check.
The cause may be a network problem, DNS issue, server issue, or unexpected response. Review the error shown in the app, then try again. If it keeps failing, contact support with the website name, verification method, and what happened.
Robots preview unavailable
This means Siteimp could not preview robots.txt right now. You can still save the website and try again later from the Website dashboard. Scan startup also checks robots.txt before running.
Where to go next
After adding a website, the next destination is usually the Website dashboard.
From there, you can:
- review the saved website
- verify ownership if you did not do it here
- review robots rules and crawl settings
- run the first scan
- use Scan Results as the evidence hub
- set up monitoring when you are ready
The Add website page is the setup step. The Website dashboard is where the website becomes operational. The Scan Results pages are where Siteimp starts showing deeper evidence.