Writing Integration Tests programmatically vs using Postman for REST APIs

Question

I am developing REST APIs in my project and using .net core 3.1.

Various best practices suggest that we should write Unit Tests as well as Integration Tests for our project.

I agree with the point of writing unit tests. But do we really need to write Integration Tests when we can do the same with the Postman API tool as well? What will be the benefits if I write Integration Tests programmatically in .net core by creating a separate project or I should only write Unit Tests and continue to do Integration Tests using Postman?

Answer 1

In overall it looks more a question regarding the code-based vs tool-based testing. And for the low-level, technology facing tests, such as integration tests, the code-based approach would give you:

1. A little bit more overhead while writing the tests. You'll have to write the code itself (obviously), and potentially manually write mocks for the components that are out of testing scope. It depends what integration you want to test and what are the components of your system.
2. But on the other side, you'll get the ability to debug your tests with the codebase, and as a result better maintainability and development experience. In case there would be a need to find the root cause of a failure, debugging via IDE and going through the codebase would be faster and more convenient.
3. You'll write integration tests in the same language as your main SUT. It's easier to stick to the same technology stack for system support.
4. Easier integration with CI/CD
5. Free of vendor locking. Postman is a commercial tool, and as any of the commercial tools it has it's own hidden costs.

As for Postman:

1. It's easier and faster to test an API from scratch, especially when you lack programming skills. So when you have a limited amount of time and need to quickly test the basic scenarios for endpoints - it's absolutely ok to use the tool. I use it for local development and debugging, for example.
2. It might be easier to mock some endpoints if it's needed.
3. You won't be able to debug your own code (API controllers), so the defect localization speed would increase. More time, more effort.
4. Relatively more overhead with CI/CD integration comparing to regular unit testing frameworks.
5. You may need to buy a license, depending on your needs.
6. The more tests you'll have - the harder and less convenient it would be to maintain, outside the IDE.

So it's up to you what to choose. My personal choice is using free Postman for basic testing while development, and later covering controllers with integration tests written in code.

Answer 2

I used to write integration tests in Postman and after a couple of tens of requests, it became very difficult to maintain. Mostly because:

• accessing other resources like a DB was hard
• it was hard to separate test data from actual requests
• it was a bit harder and definitely more criptic to use a data driven approach
• you still need to use the GUI interface in Postman, which slows you down compared to the ease of basically an IDE
• versioning of your tests is harder in Postman - it has some built-in feature (which I don't know well, but if I'm not mistaken, it's mostly a part of the paid Postman version), but if you want to use git, you'll likely end up exporting your tests from Postman which basically creates one huge json file; again, there's very little separation in this approach

I kept on going in Postman for a while and then basically threw it all away and rewrote it all programatically.

The way I use Postman now is this:

• I need to check something very quickly
• I need to perform exploratory testing

So, I see these tho options (programatical tests vs. Postman) as being geared towards different purposes.

Answer 3

Integration tests is a really vague term, but in general it is something that stands between the very isolated unit tests and the full end to end or system tests.

Putting terms aside you could want to test how your sub-systems work together while other sub-systems are mocked or disabled. This has the benefits of having faster tests, being able to better pinpoint problems, and being able to exercise the system in ways that are not possible from the outside. In a bigger system with many sub-systems this is a huge benefit, in a smaller monolithic system maybe less.

Postman will exercise your system from the outside, and have relatively limited capabilities for test logic, reporting or integration with other tools so it could be beneficial to have some tests which are "enhanced" unit tests- testing a few sub-systems together, requiring minimal outside support and still very flexible.