Use a mock.
Mock 3rd party API using MockServer or WireMock, if you're in the Java world.
Your test will call getNumUsers() method
which in turn will call mock of your 3rd API
and the mock will simulate returning HTTP 404 response back to the getNumerUsers().
Then your test will verify if the method can handle the 404 response, e.g., whether it throws ...
No if it can be done in Unit tests
The answer however is not a yes / no as this is not really a binary choice.
When possible use Unit tests. If you are testing 'based on this information from this interface...", i.e. business logic, then you can mock or stub that interface to provide that specific data for that condition.
In many organizations not all ...
As already mentioned in your TechBeacon article, teams indeed often spend too much at the top of the test automation pyramid. In general, the pyramid is a good rule of thumb, but (as always) it depends on the project. Many systems are well designed so that the GUI is just some glue between the user and the underlying APIs, which is why such systems often don'...
Here are some fun examples:
Doors - https://youtu.be/VPSE13NLPnM
Sink - https://youtu.be/RMYRTJuY8Ss
Dryer - https://youtu.be/ajUB4-m-y2o
Door Lock - https://www.youtube.com/watch?v=0GypdsJulKE
I'll add more as I find them
I am working under the assumption that you would be familiar with the language the tools are being written in.
If the team you are on already is writing unit tests a good place to start might be with them. Talk to the developers that are writing the unit tests and get an idea for how the underlying code is working. In the process of doing this start ...
You have some choices here. You can go with formal definitions like the ISTQB definition, or you can treat the definitions the way your organization handles them. Software testing is a field where definitions tend to vary depending on the understanding of the team and how they work.
That said, in my experience interface testing can be one of two things:
I'll assume the question is this:
Why measure how much of the SUT's code is covered by integration tests?
and not this:
Why measure how much of the integration test code is covered?
I think you should avoid measuring code coverage for integration tests, for two reasons:
Unit tests are a much more effective way to exercise code than integration tests....
In a nutshell,
Big bang testing involves real software components after each component has been finished; all of the components have never been tested before.
Top down testing involves lower level stubs software components. Low as in abstract level, the more abstract a software concept / component is the higher level it has.
You are ...
TL;DR: Yes, but keep in mind that Selenium is only part of the integration test tool chain.
If we define integration testing as something like this:
Integration testing is the phase in software testing in which individual
software modules are combined and tested as a group.
Then Selenium is only part of the integration testing phase, since it does not ...
Use the test pyramid as mentioned by Peter Masiar.
My contribution here is help answer the question of exactly how to decide the 20% and 80% figures that Peter mentions.
My approach to this is to make sure that the UI testing focuses on the functionality of the UI itself. In other words, from a user perspective the text and links are correctly displayed ...
Martin Fowler has a good article just about that
Integration tests collect modules together and test them as a subsystem in order to verify that they collaborate as intended to achieve some larger piece of behaviour.
Whenever some consumer couples to the interface of a component to make use of its behaviour, a contract is formed ...
You don't need to add any additional dependency.
Start doing some preparatory refactoring in order to:
Program to an interface instead of a concrete object;
Break the getNumUsers() into two functions, one to fetch the data itself and another to extract the data from the DTO.
Then you can implement a stub which simulates the 404 return value.
(As a side-...
We thought of some sort of limited coverage measurement - checking if
collaboration points like app to database, app to external service are
covered by tests.
Most code coverage tools allow filtering by file name or location; you can create multiple reports or checks for certain areas of your codebase. Here is how Jacoco does
Naturally, these would not have ...
What do you think?
Should you eat one meal at a time and 2 to 3 meals a day?
should you eat sufficient meals for 30 days at once and enjoy the remaining 29 days?
Keep things as simple as possible! Combining everything and testing it all at once may make things too complicated to handle...
Consider test pyramid. Using Pareto principle (also known as 80/20 rule), you can get 80% of benefits from testing by focusing on unit test with 20% of the effort. And 80% benefit of the remainder if you focus on API/service test.
So of course you need "some" UI-level test, 96% of effort should be lower: 80% in unit and 16% (80% of the remaining 20%) in API/...
The one and only correct answer to this question do not exist. It depends on many factors. It depends on the software development life cycle model, on the team and developer to tester ratio and so on.
There are 5 main testing levels that we can highlight:
Unit testing - It is basically done by the developers to make sure that their code is working fine. ...
I think you fell on the Integrated Tests scam.
Integration tests are about "determining if independently developed units of software work correctly when they are connected to each other".
Legos can be a metaphor for it:
Let's say you want to create a big block, half blue, and half red.
You can make these independent checks if:
The first half is ...
There are a lot of ways to answer the first question.
Should this specific feature be tested?
That depends on how important the feature is and whether it is possible that the feature was broken by a change since the last time the feature was tested.
Should the association between permissions and users be tested?
Should this specific role be ...
As others have said, if there isn't a consistent definition of what constitutes top priority across all teams, there will be disagreement over what constitutes a top priority bug.
That said, I have a few thoughts to offer:
Will this bug break core functionality if it is introduced to production? If the answer is "yes" then priority should be higher.
It sounds as if you want to avoid giving QA a build that falls apart right away. One way to address this is to add a smoke test to the development cycle. There are other SQA questions that cover smoke testing, e.g. Smoke testing by Dev team. I won't try to repeat that information here.
One goal of smoke-testing is to encourage developers to deliver QA ...
Some people distinguish between
integration (some parts are used for real, some are mocked) and
system/end-to-end testing (where is no mocking at all).
For other people, system, integration , and end-to-end are synonyms.
Usually end-to-end testing uses just basic scenarios (Pareto principle: 20% of use cases cover 80% of functionality), and unit testing ...
Let's start from your initial mandate. "Automated testing" can be a pretty vast land to explore. Leave the unit tests as they are for now and focus on the doubts you are having for integration tests. I presume here that you have none.
By looking at Wikipedia's definition of "integration testing":
Integration testing (sometimes called integration and
In overall it looks more a question regarding the code-based vs tool-based testing.
And for the low-level, technology facing tests, such as integration tests, the code-based approach would give you:
A little bit more overhead while writing the tests. You'll have to write the code itself (obviously), and potentially manually write mocks for the components ...
Some relevant guidelines:
The goal is confidence, not coverage. If you have good coverage with well-written unit tests then a tiny number of high-level tests can be enough to have confidence that the overall system will Just Work™.
Having both unit and integration tests for the same functionality is a waste of time in terms of writing the tests, maintaining ...
I agree with everything in Dan's answer. There are a few options on how to automate testing of the web service layer. You can use gui tools such as SOAPUI or JMeter, or you can use libraries such as Rest Sharp or a java alternative. I often prefer to start with an API or library for my automation, but recently I've been having success with JMeter and ...
Seems that you want to have a tool which can translate requirements/test cases (written by business people with no experience in programming) into executable code.
In our practice FitNesse is not that tool. We started with FitNesse, and keep maintaining existing tests and even adding new when feasible, but these tests are written by developers, not business ...
You need to take a step back and look at the whole picture- why did the dev lead objected to the P0 setting ?
It is not because "it is not in production" (well, it is but there is a higher level reason for that) but because the company doesn't have an agreed and accepted definition for priority levels across all teams.
I take a look into the ISTQB glossary and found this two definitions:
interface testing: An integration test type that is concerned with
testing the interfaces between components or systems.
integration testing: Testing performed to expose defects in the
interfaces and in the interactions between integrated components or
systems. See also ...
As user246 says, whether you automate your permissions tests depends on the application and whether it can be properly tested that way. The question of what you test and how always depends on the application, its user base, and how critical each feature is to that user base.
There are no best practices. This is something that a lot of people find difficult ...
Here's my perspective:
tests a single unit - usually at the routine level (that is, only one unit of code such as a function is tested). Unit tests are typically code-based, written by developers, and executed whenever the application is compiled or built.
tests a single chunk of functionality in isolation, such as testing the ...