1

I am working on an npm package scraper library. I would like to add integration tests but I am not sure the best way to approach it.

I have an examples folder which shows how the scraper works with real world sites, and I want to add integration tests to each to prove that they work. I want to use a library called nock, which can record requests and store them as fixtures, for each external endpoint, but I am not sure if I can store fixtures (which are essentially html files) in an open source library. Similarly, I want to store snapshots of what the scraper returns from the site, but I do not know if I can store snapshots of parsed data from a website in an open source repository.

To ask in the most general way, is it frowned upon to store scraped website's content in an open source library?

Improve this question

2 Answers 2

Reset to default
1

IMHO, firstly, you should create JSON schema to test all your JSON files in an actual state with this JSON schema. This is very important to keep the structure of JSON in the correct way.

Secondly, you need to create html file with all possible dom element variations to cover all cases. Then create JSON file with selectors. And then start the local web server (nodejs, python, etc.) to host html file and as a source for integration tests.

Summary:

  • create JSON schema
  • create an etalon html file, JSON file with all selectors
  • write integration tests (puppeteer, cypress, etc.)
Improve this answer
1
  • thanks for answering, I do have testing for the json schema, though it is done using typescript, not JSON Schema, and the npm package nock allows me to mock node's internal http library so I don't actually need a headless browser. I was hoping to avoid manually creating html pages but I believe you are right, that is the best way to test this system
    – andykais
    Dec 27, 2018 at 1:37
0

credit to @OlegDovger for the important bits, but I am answering my own question because I believe I have an improved solution to my initial question.

I decided against testing & caching external endpoints, since they contain more complex webpages than I need, and would therefore be tedious to look through for debugging. Instead, I manually created simple html pages and used the npm library nock to mock out the urls for said html pages. I then ran the scraper as normal. By mocking node's internal http library, I do not need puppeteer or another headless browser just to return some fake data from urls.

In my question I asked for good way to write integration scraper tests, but I ended up writing functional tests instead, which are much more reliable and easier to understand what I am trying to accomplish with tiny html files.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.