0

I'm working on a markdown link replacer for npm and some my tests are failing at random because too many requests are being issued at the same time. Now I probably need to put some kind of limiter in there controlling how fast it issues requests, and a test server.

But it got me wondering if there was a free test page to test web bots or crawlers on. I tried 'Test Page For Crawlers', but even it rejected my hammering. So is there a page on the internet that lets you test bots on it? I imagine every site must have their limits (DDOS), but there could be a site built for this.

Is there a free page to test web readers or bots on?

Edit: I'm looking for a website that can receive the hammering from multiple integration tests. I think the Selenium test page is just for direct dom manipulation from a single request rather than pulling the web page rapidly over and over. I could be wrong.

Improve this question
2
  • Possible duplicate of What is a good website to hack around using Selenium? – AdrianHHH May 20 '16 at 12:42
  • 2
    Link above has websites to use for people who are learning selenium, and expect very light use. Seems (am I right, @MarkRogers ?) like OP's problem is different: OP's own website is not able to handle the load? If you are asking if it is OK to sent so many requests to someone's website that the owner might suspect DDOS attack, you already know your answer: hammer your own website and let bystanders alone. – Peter M. - stands for Monica May 20 '16 at 14:15
2

Spin up your own test environment and test against that. Let's face it, a public facing web page that's designed to attract DDOS testing isn't going to be a valid test candidate.

The reason for this is that you'll not know how many other users are hitting a public server or the spec of the environment. If you ran 100 simultaneous users and the site went down, you'd not know why. Was there an issue with your crawler that meant it put a greater load on the server than you expected? Did it trip any security features? Perhaps there's another tester running 5,000 threads which are interfering with your calls?

No, when testing you should try to control both sides of the scenario so you can more accurately understand the behaviour you're seeing.

I know it's less convenient but when has testing ever been about convenience? :)

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.