I would agree with Laurent, and we have faced a very similar situation. Our team tests the APIs for data transport and parsing. The developer unit/component level tests were all against mock objects and data stores.
Unit tests are not always comprehensive so we built a 'fake server' to emulate various web services that we used for running additional ...
Yes, we use log monitoring as part of our testing process. Each manual tester is asked to monitor the logs during test (tail -f), to look for hidden errors/exceptions during testing.
Also, our UI will pop-up a dialog if a client side exception is thrown. These usually cause automated tests to fail, and starts the investigation.
Finally, we have a script ...
I think "acceptance tests" and "mocking services layers" don't go along. My feeling is that when you do acceptance/functional testing, then you run your whole system in a manner close to the way it will be used although you try to do it in a QA/preprod environment to be confident of the state of your system. If, for the reasons you mention, you can not do ...
This looks fine. BDD feature files is meant to be at a level to give useful information to product folks.
In each step you can call lower level steps if you want to give more specific information (or more reusable actions)
For this one for example
Given I am signed in
It might just call within it
// enter the username
I like to keep the acceptance tests as close to the code as possible.
If your acceptance tests test from a user perspective I would locate them in the code repository of the website or mobile app.
If the same acceptance tests are used by both the website and the mobile app you could opt for creating a separate repository, but I think I would rather ...
How to store code in a repository has many factors.
How many people are working on this project?
Who is responsible for testing (devs only, devs and qa)?
What testing methods are in use: unit tests, integration tests, UI tests?
Is there a difference between end-to-end and full stack testing?
Are you the only one working on this project and are responsible ...
There are a few clarifications needed here, before I can help you out in the right direction, as per your query.
1- Are you working in an Agile/Scrum model?
2- Why are the 'Requirements' unclear?
3- Why do the user stories change so often?
4- Why are the user stories so vast?
I think you may find that your testing is hampered, and you are being set up ...
As always, it depends. If there's a requirement that the link contain a label then obviously the test needs to check that a label exists (whether automated or not).
Here are some of the possible things I'd consider with this scenario:
Is the code dynamically generated or static? For a static "once and done" page, the mark 1 human eyeball might be ...
Web applications are a little bit different in that the web server deals with multiple, unrelated clients. If you are looking for a pattern, it is not enough to know that page Y followed page X -- you also need to know whether those events occurred in the same session. Sometimes this information is available in the logs; sometimes, not. Of course in a NAT'...
I'm not sure there is a specific term for this. Anyways you can always come up with a term of your own and consult with your team whether that sounds appropriate for what you are doing.
I guess you can call it "Maintenance Testing" where you have a team continuously testing the live site for maintaining the proper execution of the site along with the design....
This is the kind of scenario where I'd use a mix of automated and manual testing - manual testing using the iPad, because one thing GUI automation always does is simulate the human inputs. I've found that simulation often has crucial differences from actual input.
For automation, the crucial question here is where the data sent from the ios app is stored. ...
the answer to your first question is YES. You can definitely use Excel for creating acceptance tests. There are different types of acceptance testing:
User Acceptance test
Operational Acceptance test
Contract Acceptance testing
Compliance acceptance testing
So, I am assuming that you are planning to create acceptance tests for User Acceptance testing.
Cucumber won't magically make tests from your Gherkin features.
With all those tools you still have to write the step definitions and the code that actually makes the tests happen and maintain it. None of the BDD tools will help you do that and most of them make it hard to exploratory test.
If the acceptance is a sprint level its going to be harder to have ...
Every team works out its own process for handling bugs that escape to acceptance testing (or beyond).
The method I prefer works this way:
Any bug found during acceptance testing or later is triaged and prioritized.
Bugs are treated at the same level as user stories, and both are tracked in the same system.
If the bug is complex, it becomes a ...
Sure you could use any of these or others out there or do your own. To start I think you should ask a few questions and get those answers.
What test output is needed?
This could be...just test it and call it good or it could be some kind of report which would indicate the test needs to produce results that match that criteria.
How can I functionally ...
Working software is the primary measure of progress
Customer collaboration over contract negotiation
These two phrases from the Agile Manifesto say that success is only achieved when the software solves the customers problems. And as I imagine you have experienced that when picking a fast-food meal, often we don't know what we want or how to solve things - ...
Q: How do you handle errors that occur in Acceptance Testing, in parallel with running / upcoming sprints?
Ans: Note the first agile value which says "individuals and interactions over processes and tools". So there is no need to waiting for the story to get completely build to do an Acceptance testing. I can provide a tip from my experience, check with ...
Benefits of gherkin non-automated:
Has rigid structure (given/when/then, steps and tables)
Can be more familiar to QA and DEV
Easy to put to source control and review changes as textual diffs
Can be eventually supplied with a glue which would execute those specs against an implementation or a model of such. Although most of the time that would still ...
As an alternative to feature-based testing, you can also test your SUT based on processes/use cases. Another way is to group coherent test cases into test suites. Then, it's OK to have dependencies between test cases, but test suites should still be independent.
However, IMHO, all these things are suboptimal—for all the reasons you've already mentioned. ...
In my experience, you can achieve a good balance with the following ATDD approach for each use case you are test driving:
Write your first test by interacting with the over-the-wire API of the microservice (e.g. REST API, JMS queue). This test will give you confidence that the "happy path" is working end to end.
Use service virtualization to provide over-...
The other thing that many companies neglect in Agile, Extreme and even Waterfall working is that you need to have a test section on board as well and while the developers are working to fulfill a given story point, feature or requirement the test team should be developing the tests, (in extreme these need to be developed before the developers start), and ...
The main difference between Acceptance and System testing is that Acceptance testing should be (in ideal world) defined before coding started. It is a suite of requirements/use cases agreed with customer, which must be passed to accept product. It is a part of "business". System test - it is a part of personal responsibility/quality of project QA. So, you ...
In addition to what Kate said, here's one thing I always consider when writing up scenarios:
I am writing them up as an user of the application, so I need to provide all the details that the user would be aware of, and not much more than that. The rest can be handled by the code behind it.
We rely extensively on log analysis, but not for Web applications but for testing embedded code (from low level drivers to applications).
As the presentation says one of the first decisions you'll have to take is online vs. offline , this changes your strategy completely.
Our Perl framework has a simplified API for analyzing offline logs, and/or handling of ...
You could call this testing in production. There's been a lot of discussion of that - see for example:
Testing in Production: How we combined tests with monitoring, Jonathan Hare-Winton & Sam Cutler, The Guardian