11
Welcome to SQA, I think you've got a few questions going on, so I'll try and answer them in sections.
Whats Travis/Continous Integration
Travis-CI is one of a number of build automation tools that help developers understand if their application is working by building, testing and reporting on the build sequences the developer has configured. Frequently ...
9
As already mentioned in your TechBeacon article, teams indeed often spend too much at the top of the test automation pyramid. In general, the pyramid is a good rule of thumb, but (as always) it depends on the project. Many systems are well designed so that the GUI is just some glue between the user and the underlying APIs, which is why such systems often don'...
7
As a general rule, retesting in the staging environment is done more as an integration/sanity test than a full retest because the QA environment will likely have different code (due to other changes that aren't being pushed to the staging environment yet).
The usual considerations are:
The staging environment is kept as close to production as possible, and ...
7
The short version: Don't sweat on automation or test cases. Chances are you won't get time to build it because there's too much that has to be manually tested now. Instead focus on building light-weight, easy to maintain documentation of the key elements of the software you're testing.
The longer version:
This is my world. I'm the sole tester in a team of ...
7
It all depends on what you're configuring, and if the configuration mechanism itself was tested before.
If it's something small, like a timeout interval, number of retries, etc, then I'd agree it likely doesn't need full testing before changing the config in prod.
On the other hand, if it's bigger/riskier (i.e. changing a DB connection string to a new ...
7
I don't see CI/CD as something different then non CI/CD process, tests are simply being ran by a non human entity.
Do you need a test plan (or plan your tests) ? of course you do, you need to think what goes into automation and whether something should be done manually or tested in production (A/B tested, telemetry), analyze risks related to testing this ...
7
Do you still need a Test Plan when using CI/CD?
YES you do. Because a test plan will tell you at least the thing that you did not specify in your question: how often do you plan to run the tests? Which tests will run with which periodicity?
Maybe there is a common practice to do it every night. Is it feasible for your project? Which alternatives do you ...
6
In recent versions of Windows, Microsoft decided to give services their own hidden desktop, and so even if you check "interact with desktop" and running as current user, the service still is referring to a hidden background desktop. There is no way to change this, as Microsoft did this on purpose. So, for example, to run Selenium tests from a executor, you ...
6
There are an awful lot of variables to consider here. But here are some points all the same.
Manual Testing vs Automated Testing This point alone is probably worthy of a book and is one of the most frequent questions I see. E.g. I'm new to a project, what should I automate? Things to consider here are...
Does take a long time to execute the manual test?...
6
In my experience, when the lack of automated regression in a delivery pipeline becomes technical debt, you have exactly one option.
Treat automated regression tests as code and sprint on them the same way you sprint on new features and bug fixes.
What that comes down to is that if your team(s) didn't set this up at the start, you will have to either waste ...
5
What you want is called an continuous integration (CI) server. A very popular one is called Jenkins (jenkins-ci.org), though there are others both open source and commercial.
You can configure it to watch for checkins, and to build a project. If the build succeeds, it can trigger tests to be run. If the tests pass, you can trigger other jobs such as ...
5
What do you think?
Should you eat one meal at a time and 2 to 3 meals a day?
Or
should you eat sufficient meals for 30 days at once and enjoy the remaining 29 days?
Keep things as simple as possible! Combining everything and testing it all at once may make things too complicated to handle...
5
Consider test pyramid. Using Pareto principle (also known as 80/20 rule), you can get 80% of benefits from testing by focusing on unit test with 20% of the effort. And 80% benefit of the remainder if you focus on API/service test.
So of course you need "some" UI-level test, 96% of effort should be lower: 80% in unit and 16% (80% of the remaining 20%) in API/...
5
Use the test pyramid as mentioned by Peter Masiar.
My contribution here is help answer the question of exactly how to decide the 20% and 80% figures that Peter mentions.
My approach to this is to make sure that the UI testing focuses on the functionality of the UI itself. In other words, from a user perspective the text and links are correctly displayed ...
4
The short answer: No, it is not crazy, as long as you take the proper precautions.
The longer answer:
There's nothing inherently wrong with running UI automation from a CI server. That said, there are a number of factors you need to take into account.
Don't block CI - You should be limiting the continuous run suites to a basic smoke test run on the CI ...
4
It depends. Part of me is concerned with the tone of the question, to be honest. I'd be concerned that automated testing is being done to the detriment of manual testing, although some types of software would probably do OK with only automated testing.
Yes, you absolutely should automate that which you can. And some types of software development, and ...
3
Selenium Builder is the evolution of the Selenium IDE. Its tooling page has a video (11 minutes) about using it with Travis-CI. Although I haven't watched it I expect it will answer most of your questions.
To simplify test building (for non developers) I would suggest using something like Cucumber or another BDD framework, you can then make a lot of ...
answered Jun 10 '14 at 11:08
Niels van Reijmersdal
25.4k2 gold badges39 silver badges92 bronze badges
3
Figured it out. The host IP in the VM configuration window in Jenkins has to be the IPv4 address of the host-only adapter (in this case, 192.168.120.1). I also changed the host port in the port forwarding rule from 3022 to 22.
For historical reference, the username and password fields (which appear when the "Advanced" button under the "Host" field in the ...
3
The most efficient tool for testing the appearance of a website is still the human eye and brain.
That said, if you want or need to automate, image comparison is probably the least effective method because even with fuzzy logic it can generate far too many false positives. All it takes is a change in hardware (a different monitor or video card - this will ...
3
Apologies in advance for a quick/simplistic answer - it's crunch time here!
My current client have automated around 80% of regression testing, the rest is done manually with an offshore team, where the basic rule of thumb is:
"If you change B, test A and C"
i.e., everything around the change - there's no point in your regression team (or the automated ...
3
First of all, coverage isn't coverage. There're several coverage criteria, but most used are:
Function coverage – Has each function (or subroutine) in the program been called?
Statement coverage – Has each statement in the program been executed?
Branch coverage – Has each branch (also called DD-path) of each control structure (such as in if and ...
3
In a nutshell:
Continuous integration (CI): continuously integrate (small) software changes.
Continuous delivery (CD): continuously deliver these changes.
Continuous testing (CT): continuously test these changes.
Jenkins is just one of many servers (others are, for instance, Bamboo or Travis CI) you can use to implement CI/CD/CT.
Integration testing ...
3
Integration Testing
Integration testing is about taking the units of code developed with unit testing and combine them together into the desired system. It (can) also mean testing software by including dependencies such as database, screen, communication, micro-services, etc. This is as opposed to unit testing something and mocking and stubbing out all ...
3
Yes, I think you can group lack of test automation under technical debt.
We don't have time for test automation, so we do it manual
This sounds like "Deliberate" and "Reckless" on the Technical Debt Quadrants.
Worse would be "We don't know how to automate this testing", which is "Inadvertent" and "Reckless". Go get some skills ;-)
We know we should ...
answered Nov 21 '17 at 10:19
Niels van Reijmersdal
25.4k2 gold badges39 silver badges92 bronze badges
3
Continuous Integration
Let me explain in increasing levels of detail:
A place to run your tests.
A remote server that you can push code to that runs the automated tests
Continuous integration usually refers to two parts of the development process. First when users work in branches it runs the tests for their branches. Secondly is is part of a devops ...
3
Regression testing should be automated (not manual) and CI is very valuable tool for automated regression testing.
So I assume you already have automated regression tests, and you are considering if you need CI to run them: Automation with CI vs automation without CI
Without CI, you need to remember to run your test. And when you forgot to run them, and ...
2
A quick Google shows plenty of people doing this, including This page from WatirMelon ( if you dont know of this site could be worth your while reading it, lots of good stuff on it )
and an example of using ci_reporter is here
2
According to this blog post it is not possible to run tests from xcodebuild but there is a work around. This problem exists for xcode 4.0 - 4.3, I'm not sure about 4.4
2
Similar question posted on Software Testing Club
Robotium looks like it help fulfill some of your needs? (props to Stephen Janaway)
2
You want to know why they do this and or if it is justified.
It is difficult to answer without knowing more about the project length, product, technology and deployment environment and the acceptable defect goals.
For instance there could be a number of valid reasons. It may be the exact same code, but the deployment process may be unreliable. It may also ...
Only top voted, non community-wiki answers of a minimum length are eligible
