I'm working on a markdown link replacer for npm and some my tests are failing at random because too many requests are being issued at the same time. Now I probably need to put some kind of limiter in there controlling how fast it issues requests, and a test server.
But it got me wondering if there was a free test page to test web bots or crawlers on. I tried 'Test Page For Crawlers', but even it rejected my hammering. So is there a page on the internet that lets you test bots on it? I imagine every site must have their limits (DDOS), but there could be a site built for this.
Is there a free page to test web readers or bots on?
Edit: I'm looking for a website that can receive the hammering from multiple integration tests. I think the Selenium test page is just for direct dom manipulation from a single request rather than pulling the web page rapidly over and over. I could be wrong.