I have come across an issue that affects a common component in our website. Instead of having to go through manually and identify each occurrence, I was going to look into putting a web scraper together that could go through and identify every place a drop down appears.

Has anyone taken this approach for identifying common components or have a better way of doing so?

1 Answer 1


This approach might work well for particular cases (for example when the component is a drop-down and you are sure that it behaves identically everywhere), but in general I would avoid identifying common components by web-scrapping. Below is some justification:

  • The components might only look like they are the same, but in fact the behavior assigned by a JS can be different so test could not be applicable for all of them
  • The "components" might be complex enough. So you'll have to put a lot of effort to make your code have a clear vision on what the "component is and reliably detect where the component starts and where it ends in your DOM.
  • Web scrapping is not that trivial task itself, since to reach some of the pages you should assure some conditions are met. Also there could be several components in DOM at once but displayed depending on certain conditions also.

Better way is to ask your devs since they do really know which components are common for which pages. You will save the time for coding and get the reliable results.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.