11
Welcome to SQA, I think you've got a few questions going on, so I'll try and answer them in sections.
Whats Travis/Continous Integration
Travis-CI is one of a number of build automation tools that help developers understand if their application is working by building, testing and reporting on the build sequences the developer has configured. Frequently ...
9
As already mentioned in your TechBeacon article, teams indeed often spend too much at the top of the test automation pyramid. In general, the pyramid is a good rule of thumb, but (as always) it depends on the project. Many systems are well designed so that the GUI is just some glue between the user and the underlying APIs, which is why such systems often don'...
9
I don't see CI/CD as something different than non CI/CD process, tests are simply being run by a non human entity.
Do you need a test plan (or plan your tests) ? of course you do, you need to think what goes into automation and whether something should be done manually or tested in production (A/B tested, telemetry), analyze risks related to testing this ...
7
As a general rule, retesting in the staging environment is done more as an integration/sanity test than a full retest because the QA environment will likely have different code (due to other changes that aren't being pushed to the staging environment yet).
The usual considerations are:
The staging environment is kept as close to production as possible, and ...
7
The short version: Don't sweat on automation or test cases. Chances are you won't get time to build it because there's too much that has to be manually tested now. Instead focus on building light-weight, easy to maintain documentation of the key elements of the software you're testing.
The longer version:
This is my world. I'm the sole tester in a team of ...
7
It all depends on what you're configuring, and if the configuration mechanism itself was tested before.
If it's something small, like a timeout interval, number of retries, etc, then I'd agree it likely doesn't need full testing before changing the config in prod.
On the other hand, if it's bigger/riskier (i.e. changing a DB connection string to a new ...
7
There are an awful lot of variables to consider here. But here are some points all the same.
Manual Testing vs Automated Testing This point alone is probably worthy of a book and is one of the most frequent questions I see. E.g. I'm new to a project, what should I automate? Things to consider here are...
Does take a long time to execute the manual test?...
7
Do you still need a Test Plan when using CI/CD?
YES you do. Because a test plan will tell you at least the thing that you did not specify in your question: how often do you plan to run the tests? Which tests will run with which periodicity?
Maybe there is a common practice to do it every night. Is it feasible for your project? Which alternatives do you ...
6
In my experience, when the lack of automated regression in a delivery pipeline becomes technical debt, you have exactly one option.
Treat automated regression tests as code and sprint on them the same way you sprint on new features and bug fixes.
What that comes down to is that if your team(s) didn't set this up at the start, you will have to either waste ...
6
Use the test pyramid as mentioned by Peter Masiar.
My contribution here is help answer the question of exactly how to decide the 20% and 80% figures that Peter mentions.
My approach to this is to make sure that the UI testing focuses on the functionality of the UI itself. In other words, from a user perspective the text and links are correctly displayed ...
5
What do you think?
Should you eat one meal at a time and 2 to 3 meals a day?
Or
should you eat sufficient meals for 30 days at once and enjoy the remaining 29 days?
Keep things as simple as possible! Combining everything and testing it all at once may make things too complicated to handle...
5
What you want is called an continuous integration (CI) server. A very popular one is called Jenkins (jenkins-ci.org), though there are others both open source and commercial.
You can configure it to watch for checkins, and to build a project. If the build succeeds, it can trigger tests to be run. If the tests pass, you can trigger other jobs such as ...
5
Consider test pyramid. Using Pareto principle (also known as 80/20 rule), you can get 80% of benefits from testing by focusing on unit test with 20% of the effort. And 80% benefit of the remainder if you focus on API/service test.
So of course you need "some" UI-level test, 96% of effort should be lower: 80% in unit and 16% (80% of the remaining 20%) in API/...
4
The most efficient tool for testing the appearance of a website is still the human eye and brain.
That said, if you want or need to automate, image comparison is probably the least effective method because even with fuzzy logic it can generate far too many false positives. All it takes is a change in hardware (a different monitor or video card - this will ...
4
The short answer: No, it is not crazy, as long as you take the proper precautions.
The longer answer:
There's nothing inherently wrong with running UI automation from a CI server. That said, there are a number of factors you need to take into account.
Don't block CI - You should be limiting the continuous run suites to a basic smoke test run on the CI ...
4
It depends. Part of me is concerned with the tone of the question, to be honest. I'd be concerned that automated testing is being done to the detriment of manual testing, although some types of software would probably do OK with only automated testing.
Yes, you absolutely should automate that which you can. And some types of software development, and ...
3
Selenium Builder is the evolution of the Selenium IDE. Its tooling page has a video (11 minutes) about using it with Travis-CI. Although I haven't watched it I expect it will answer most of your questions.
To simplify test building (for non developers) I would suggest using something like Cucumber or another BDD framework, you can then make a lot of ...
answered Jun 10 '14 at 11:08
Niels van Reijmersdal
31.7k4 gold badges52 silver badges118 bronze badges
3
Apologies in advance for a quick/simplistic answer - it's crunch time here!
My current client have automated around 80% of regression testing, the rest is done manually with an offshore team, where the basic rule of thumb is:
"If you change B, test A and C"
i.e., everything around the change - there's no point in your regression team (or the automated ...
3
First of all, coverage isn't coverage. There're several coverage criteria, but most used are:
Function coverage – Has each function (or subroutine) in the program been called?
Statement coverage – Has each statement in the program been executed?
Branch coverage – Has each branch (also called DD-path) of each control structure (such as in if and ...
3
In a nutshell:
Continuous integration (CI): continuously integrate (small) software changes.
Continuous delivery (CD): continuously deliver these changes.
Continuous testing (CT): continuously test these changes.
Jenkins is just one of many servers (others are, for instance, Bamboo or Travis CI) you can use to implement CI/CD/CT.
Integration testing ...
3
Integration Testing
Integration testing is about taking the units of code developed with unit testing and combine them together into the desired system. It (can) also mean testing software by including dependencies such as database, screen, communication, micro-services, etc. This is as opposed to unit testing something and mocking and stubbing out all ...
3
Yes, I think you can group lack of test automation under technical debt.
We don't have time for test automation, so we do it manual
This sounds like "Deliberate" and "Reckless" on the Technical Debt Quadrants.
Worse would be "We don't know how to automate this testing", which is "Inadvertent" and "Reckless". Go get some skills ;-)
We know we should ...
answered Nov 21 '17 at 10:19
Niels van Reijmersdal
31.7k4 gold badges52 silver badges118 bronze badges
3
Continuous Integration
Let me explain in increasing levels of detail:
A place to run your tests.
A remote server that you can push code to that runs the automated tests
Continuous integration usually refers to two parts of the development process. First when users work in branches it runs the tests for their branches. Secondly is is part of a devops ...
3
Regression testing should be automated (not manual) and CI is very valuable tool for automated regression testing.
So I assume you already have automated regression tests, and you are considering if you need CI to run them: Automation with CI vs automation without CI
Without CI, you need to remember to run your test. And when you forgot to run them, and ...
2
You want to know why they do this and or if it is justified.
It is difficult to answer without knowing more about the project length, product, technology and deployment environment and the acceptable defect goals.
For instance there could be a number of valid reasons. It may be the exact same code, but the deployment process may be unreliable. It may also ...
2
If you're looking for TestComplete training, you probably want to look at the SmartBear website (http://www.smartbear.com). They have a number of webinars available at any time. There's also more intense training available: http://smartbear.com/products/qa-tools/automated-testing-tools/features/training-certification/
I've never done any of the training ...
2
We use Jenkins to orchestrate our CI and Continuous Delivery.
This training course looks pretty good: http://www.cloudbees.com/training.cb
2
It really comes down to overall reliability and speed.
If you can guarantee that you have 24/7 access to your 'test server' via the Internet, with guaranteed > 10Mbps download/upload speeds, and you can protect 100% from external attacks (e.g. DoS) then the Internet might be the way to go.
But, if you are like most of the world that experiences ...
2
It should be hosted where everyone who needs to test can access it.
Important is that everyone can test in isolation, you don't want different processes or users interfering with each others tests runs.
As you tagged this question with "continuous integration" I would like to add that for automated tests I love to spin up a test environment from scratch so ...
answered Mar 14 '14 at 20:22
Niels van Reijmersdal
31.7k4 gold badges52 silver badges118 bronze badges
2
There is still an open ticket to get the enhancement you want.
To elaborate on Adurbe's answer:
You could drop a .sh file onto your server and then call it via Bamboo. Check out this StackOverflow question and look over the answers to find what will be best for you.
Only top voted, non community-wiki answers of a minimum length are eligible