Sometimes in product development, a theoretically low risk can result in a set of changes that run deep into how your product works. Over the last few months, this happened with Siteimp and it sent me down one of the stranger product development rabbit holes I've gone down.
By design, Siteimp fully respects a website's robots.txt. An ownership verification mode has been built in for a long time so that website owners could override general robots.txt restrictions on websites they control. But I hadn't really thought of a need to fully block Siteimp (and Siteimp only) until a good friend pointed out a very low risk of Siteimp being used as an annoyance. Most sites on competent modern hosting are unlikely to notice a normal Siteimp scan... heck if your overall performance is impacted by Siteimp, that's a symptom of something deeper so it should be taken as a sign to dig deeper into your performance.
But, it was a very good point and so the idea of a killswitch started to take shape. You'll note that I've called it a killswitch - that is because I'm a nerd and this is one of my very few opportunities to look tough. But it's also a good term because now, a website owner can simply add:
User-agent: Siteimp
Disallow: /That rule is absolute. Ownership verification does not override it. If a website owner explicitly tells Siteimp to stop, Siteimp stops.
It was a simple change... but the thinking behind it was the hard part because it led down a deeper security path. Siteimp is obscure... but it's a set of very powerful and well optimized bots. Any tool that can crawl websites quickly can be abused if someone automates it badly enough. Siteimp is designed for owners and developers, but the web does not get to assume every user will behave as intended. It wouldn't be a good scraper but it could be a heck of a distributed nuisance to solve. Now... there's something deep inside of it so that if it ever becomes a nuisance a website owner can kill it off.
However... that opens up another scarier attack. Whenever I write any kind of bot, robots.txt respect is built in first. But not everyone thinks like I do. And user agents are about as easy to change as your socks so I have to think about something a lot worse. Siteimp is obscure and for now, that's a good thing. But if it ever becomes a big enough tool that people know about it, it also becomes a tool that less ethical people could impersonate.
That's the deeper problem with robots.txt. It can tell cooperative software what to do, but it can't prove that the software is who it claims to be. Someone else can send User-agent: Siteimp while behaving nothing like Siteimp and there isn't really anything Siteimp itself can do to stop them from borrowing its name.
I'm not sure there's really a way to fix this, though I am sure that if it ever comes up a far bigger company will impose the fix. So that will be fun. But, I'm thinking about it, am fully aware of what Siteimp really is under the hood and constantly think through keeping my users and the overall web safe.
Build your own products and make the world a faster, more accessible place, they said. It will be fun...