With the caveat that this answer isn't Selenium specific.
You aren't necessarily doing anything explicitly wrong by having a 'large' number of tests running in one suite. For example for back end testing some of the suites I run have millions of test cases. One notable test suite I support has over 200 million tests as part of it's run(Granted this is back end testing, data driven and not in selenium).
But, a very important but, we don't run all 200 million in parallel because testing has to exist, for the most part, in the real world and very few systems can even run that many tests at once let alone support that many actions being tested against them at once. I don't see you running these in parallel so I'm assuming that you are indeed running these in serial. Even so, a large number of serial actions against a product, system or site can impact performance in a way that can cause test failures. Additionally your tests may be interacting with each other in unexpected ways (changing the expected state behind the scenes.)
There are a couple of things you can do here to narrow down your issue:
- Do the same tests always fail when run in the larger test suite? If yes then you potentially have a couple of culprits:
- Try changing the order of your tests. Try to narrow down if your tests are not truly atomic and automated. This should help you identify the case where a specific test or tests are impacting other tests unexpectedly. It's worth pointing out, if your tests are doing what they should this could be a bug in the system itself.
- Try spacing out your tests by adding waits. It's possible that a specific test is putting your system under load and causing issues. Spacing out actions can assist in identifying whether this is a specific test problem or a larger load problem. This leads nicely into...
- The thing you are testing may not be capable of handling the load you are sending it. If you are testing in a staging environment the load expectations may be much lower than a production environment. If you are testing your production environment you may have specific actions that decrease your system's performance. You may just be hitting it too hard or too fast.
- Typically speaking, Selenium is not a tool for testing load or performance and if load/performance are getting in the way of functional testing you should rate limit your tests in some way to see if this improves the results of your tests.
- Your tests may be doing actions behind the scenes that cause issues. If you authenticate, for example, every time you do an action you could be unintentionally causing very high load for your tests. Actions that make sense to be memoized(that aren't under test, that are expensive, whose data doesn't change over the course of your test, and that you use often) should have their data chached in some way.
Finally - this may be an underlying bug that occurs when disparate actions are performed against your product. The best way to catch something like this would be to identify the components of your test suite that trigger this (if test X always fails, figure out how many or what other tests can repeatedly cause this.) Bugs like this are very difficult to find and even harder to prove, you may (while keeping this issue in mind for further triage and research) have to do any of the steps above to test the individual functionalities of your product.