I'm currently trying to automate the testing of the visual appearance of a web application. I'm using a combination of openCV and tesseract OCR to find text intersection with Element borders with quite some success but the second Text overlaps with text tesseract fails to recognise characters and therefor fails to properly outline text boxes. I'm using seleniums WebElement.getRectangle() function to try and look for Elements intersecting into each other but not only is it slow but also shows a lot of false positives because of hidden text nodes.

Since the problem boils down to a binary Image classification (overlapping text or not overlapping text) I've created a simple classification model in Tensorflow which doesn't produce good results though since even after I mined all the data our bug tracker / google gave I'm still far away from having a solid set to train it on.

Now to the heart of my question: Since I do not believe that I'm the first person to try to get rid of the task to look through hundreds upon hundreds of screenshots only to make sure everything looks right does anyone know of a huge enough data set of common visual errors to efficiently train an convoluted neural network with it?

  • Try asking this question on Artificial Intelligence and Data Science, you may get a reply there. – Yu Zhang Jul 19 '17 at 2:04
  • I've spent some time pondering on which SE to ask for this and I'm fairly certain that if such a dataset exists our profession is pretty much the only one that could get any use out of it so I assume that the highest probability of finding someone who worked with it would be here. I will try others but I'm almost certain that if I won't get any results here i won't get any results at all. – Daniel Jul 19 '17 at 6:29
  • I was not pushing you away; it was more of a suggestion. If my comment offended you in any way, I apologize. – Yu Zhang Jul 19 '17 at 6:54
  • 1
    No, not at all. I really shouldn't comment before my first coffee. I'm just trying to say that the problem is pretty specific and I do not think any other SE would be suited. Yet the guys over at Data Science might have an idea how to best mine for data if no such set exists (though I dearly hope it does). – Daniel Jul 19 '17 at 9:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.