Questions tagged [continuous-integration]

Continuous Integration, aka CI refers to the development practice of integrating work from team members frequently, but is also sometimes used as shorthand for the automated build systems that support that practice.

Filter by
Sorted by
Tagged with
10
votes
3answers
2k views

Tests in CI: Are API + Integration Tests the most suitable?

In my experience, automated GUI tests may take too long for a build of Jenkins (even parallelized) and may demand a lot of maintenance. On the other hand API + Integration tests are way faster, do not ...
9
votes
8answers
924 views

Is manual regression testing an technical debt for agile teams aiming for continuous delivery?

Background: Following are not my assumptions or expectations but in my last couple of interactions with different project teams, I am seeing these patterns where it is expected (from testing by the ...
9
votes
4answers
2k views

Do you still need a Test Plan when adopting Continuous Delivery?

As a test manager, I need to review a Test Plan for the upcoming release. I also need to review the test results. The tests are both automated and manual. As an organization, we would like to move to ...
9
votes
4answers
266 views

What should a QA in software be worried about when a small configuration deployment is being pushed without testing?

In my limited experience, configuration changes (small 1-3 line changes to a project config file), either completely break the program or do not seem to bring noticeable changes to the user-experience....
9
votes
3answers
19k views

Selenium Tests run in the background when TeamCity CI is run as a Windows service

Our test suite is written in Python and makes use of the WebDriver Python bindings. We are looking into moving to TeamCity for continuous integration. The setup I have in mind is a build triggered for ...
9
votes
8answers
6k views

Visual Regression Testing

For testing the visual appearance of a website the use of tools like selenium (functional surface test) is not sufficient because it doesn´t check the visual appearance and/or the CSS-Code directly. ...
8
votes
1answer
1k views

Continuous Integration and testing environment and configuration for Android

I am an Android developer but I am not satisfied with my testing and Continuous Integration environment and configuration. Someone could recommend me how to configure the best setup and workflow for ...
7
votes
1answer
7k views

Trying to understand what Travis CI does and when it should be used

I am very new to Git and I am planning to contribute to some open-source project on GitHub after discovering a small error in it. Upon forking it and fixing the error, I purposed a pull request and I ...
6
votes
8answers
350 views

What to do with Limited Time/Testers - What is most Important?

In an ideal world we'd have Acceptance Tests-->Unit Tests-->Integration Tests-->Test Plans and Automated GUI Tests (With Selenium or something). But where does it start to get cut out when you just ...
6
votes
5answers
187 views

How does Continuous Integration add value to Automation Testing?

How do I gain the value from CI tools as an Automation Tester? What are the advantages to using such tools and how do they help me and the company in the development process ?
6
votes
2answers
252 views

Is it a good practice to run functional tests with coverage?

Does it make sense to run once in a while, say, on a weekly basis, integration/functional tests against special build to have a coverage report. If anyone here has some practical experience of using ...
6
votes
2answers
12k views

What is the correct way to configure the VirtualBox plugin on Jenkins?

I am trying to get the VirtualBox plugin working with Jenkins. My plan is then to hook up the Selenium plugin as well and get a grid running on the VirtualBox cloud. When I follow the instructions on ...
5
votes
2answers
138 views

When to mark a build as failed?

Currently, we do a lot of things when there is a application build job triggered (the trigger can be manual, or there is a commit to the develop/trunk branch, or there is a Pull Request created or ...
5
votes
2answers
71 views

Version labels in continuous integration teams

How do people handle version labels while performing manual testing when working in a continuous integration environments. In Jira there is a ‘releases’ feature that allows a version to be assigned to ...
5
votes
1answer
2k views

xcodebuild does not run any unit tests when launched from console

My first question here so please bear with me. It may be off topic, if so - close the question. I have setup Jenkins with git and Xcode plugins so that I would be able to build an iOS project on CI ...
5
votes
2answers
618 views

Validate Quality Cobol code - Jenkins, Sonarqube/SonarLint and Rational/Topaz?

In my work as a tester, I was asked to find a solution to validate the quality of Cobol code. For that I was asked to investigate Jenkins, Sonarqube/SonarLint and Rational/Topaz. Starting point - I ...
4
votes
4answers
2k views

Scope of regression testing

Had a question about what other folks do for regression testing. Let me lay out what we currently have and what I think it should be: We have smoke tests, which are just some minimum tests for build ...
4
votes
2answers
1k views

Difference between Integration Testing and Continuous Integration Testing

In my college, I learned about Integration testing in which we combine the whole code together and then test it at once after we are done with unit testing. A few days back, I heard about Jenkins, ...
4
votes
3answers
57 views

Automated tests in CI: same or different repo as SUT?

I'm currently writing API and UI automated tests for a website. My goal is to plug these tests in the CI workflow (Jenkins is used in this organization). The workflow would be: if there's a new commit ...
4
votes
1answer
457 views

Running unit tests on production build?

In one of the projects I am involved with, the team is running a continuous integration/continuous deployment and in development they have a bunch of unit tests as part of the nightly builds. When ...
4
votes
2answers
2k views

Open source deployment validation tool

Asked this on stackoverflow and earned myself the tumbleweed badge... Anyone here know of any? I'm looking for an Open source tool to help with deployment validation for mainly web based applications ...
4
votes
1answer
3k views

How do I automate front-end testing for a web app with Javascript?

I am introducing myself to devops engineering and I'm stuck on "how to automate testing for web applications". I consider that backend and api test automation are relatively simple and the concept is ...
4
votes
1answer
2k views

Remote deployment on WAS using websphere deployer plugin of jenkins

I want to deploy an application on websphere application server(8.5.5) from Jenkins using websphere deployer plugin. I have attached the two required jars. I can telnet the WAS server from jenkins ...
4
votes
0answers
800 views

Jenkins stops getting my unit test output from xcodebuild [closed]

I've got Jenkins set up to build my application test target and execute tests in the iOS simulator like this answer says. When I execute the build in Jenkins, the iOS simulator launches and runs all ...
3
votes
3answers
4k views

Selenium IDE and Selenese scripts in Continuous Integration

Apologies if this is not the forum for such a generic question. As our organisation moves to an Agile project methodology, we need to be able to rapidly automate applications as they are built up ...
3
votes
1answer
365 views

Non-headless Automation Testing on CI Server?

So im in a bit of a conundrum. I have a Rails project which i've written Automated tests for in Capybara using ChromeDriver/Selenium. It works great, but is quite long. I also have headless chrome ...
3
votes
1answer
96 views

Supporting multiple versions of test scripts

We're running our own CI/performance tests system for some other team's software library as they've yet to decide on something beyond nightly tarballs and our requirements are much more stringent. Due ...
3
votes
1answer
195 views

Automated testing in cloud

I have a Windows application using OpenGL and automated E2E test environment based on Sikuli framework. Tests run locally and are not available outside of my company network. As we have two different ...
3
votes
2answers
1k views

Functional/Black Box tests automation library on Python

Summary: I have to test embedded software which runs on qemu virtual machine or on real device and can write its output to stdout. So I can check the state of software only by parsing stdout. To ...
3
votes
1answer
1k views

How to set up stages within test plan to use the same build_dir on Atlassian Bamboo?

I have a few libraries that needs to be tested under multiple versions of python. I have defined each library as a plan. For each plan I have two stages: Setup (Source Code Checkout + "make install")...
3
votes
0answers
458 views

SonarQube: Auto assign issues doesn't work

Currently, I have an integration between GIT, Jenkins and SonarQube as described in this pseudo workflow: commit----> AWS CodeCommit (GIT) <-> Jenkins <-> SonarQube Scanner for Maven &...
2
votes
2answers
121 views

cross-OS automated testing

First, please leave your stones down: I am a windows-oriented developer and I have very limited knowledge in testing tools world - especially in those that works on multiple platforms. I have been ...
2
votes
4answers
200 views

Automatically only run the fast tests?

Christmas List: Manually I nark tests as being in category "long running". I'm lazy and don't want people to have to do this. Are there any testing or CI frameworks out there that can run only tests ...
2
votes
2answers
1k views

Include Watir test cases in a Continuous Integration system

Did anyone succeed deploying Watir test cases to a continuous integration system such as Team City or Hudson? I'm trying to find a way to have the test scripts executed and have the results ...
2
votes
1answer
87 views

Integrating Protractor to Octopus deployment (CI/CD Integration)

I have a batch file with below content: %1 %2 --params.validFile=%3 --params.invalidFile=%4 I am starting it through power shell script: $path= "<filePath>\start.bat" $args= '"<filePath&...
2
votes
1answer
1k views

Jenkins: failing builds on warnings

I've been reading the book "Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Addison-Wesley Signature Series (Fowler))" And these paragraphs ...
2
votes
2answers
49 views

Manage UI Tests with each branch

Well, we are currently working on three branches: dev, staging, and master My UI tests are running on continued integration pointing only to the dev branch and to the dev application as well. The ...
2
votes
2answers
163 views

CI for Selenium Project

I am not fully sure how to fully build Continuous Integration for Selenium project, so I would like to ask the following: Is it true that the only difference between a Maven project and a Java ...
2
votes
1answer
890 views

How to move from test automation to devops? [duplicate]

I have a background as a developer, but have been working on test automation here in the UK for the last years. I work a lot with Java, Webdriver and Jenkins/continuous deivery - but I am very ...
2
votes
2answers
187 views

CSS visual regression - do I need to store screenshots in git repository?

There is a technique to check if something visually not broken in HTML and CSS markup - visual regression testing. We do following steps: Check everything is ok. Create a test "reference" (creating *...
2
votes
1answer
109 views

Regarding the Build job in jenkins

When I am running maven goal in Jenkins which runs soap tests which are developed using groovy, whenever there is a test failure then all the other tests are failed without any test execution from job....
2
votes
1answer
39 views

Multiple pull request reviewers from different timezones

Our QA team is not a large one, but consists of full or partially remote, full or part time team members which, as it currently stands, live in different countries and in different timezones. We are ...
2
votes
4answers
263 views

Selenium tests as a part of a build stage in a continuous integration pipeline

There is a statement from microsoft documentation about Selenium testing in a continuous integration pipeline: To include the Selenium test as part of a build, the source code must be in version ...
2
votes
3answers
850 views

Best practices to organize the automated tests in a continuous delivery world [closed]

I am trying to organize my automated tests in the source control [ GIT] - mostly Jprofiler/Jmeter, Selenium and SOAP tests. What are the industry standards for organizing/managing the Automated ...
2
votes
1answer
1k views

Combining coverage from unit tests and functional tests/Selenium

Has anyone had experience getting a combined code coverage measure across both unit tests and functional/UI automation tests? I'm in a Java stack with Cobertura, Jenkins and Ant. I can successfully ...
2
votes
1answer
1k views

Configuring a framework using Selenium Builder, GitHub and Jenkins

Ok, so I have posted a couple of other questions here that have been unclear so, apologies for another along similar lines. I am a little further along now but am still blocked from progressing. I ...
2
votes
1answer
128 views

Test automation for .NET / Java Project relying on app container

I am looking to a testing / continuous integration tool which can help me with the following: 1) Inside a Maven project, allow me to run a JUnit test case that, using some API, deploy an application ...
1
vote
7answers
3k views

Do you need to re-verify all bugs in the staging environment?

At my company the QA department will verify all of the bugs fixed in builds deployed to our QA environment. We might deploy like 10 or so builds over a few weeks and they verify that the bug fixes are ...
1
vote
2answers
71 views

Integration Testing requires consecutive builds?

If I have several code-level integration tests, and my integration strategy is to integrate the components in a bottom-up manner, should I perform a build for each integration level of the modules? Or ...
1
vote
3answers
46 views

When nightly testing integrations with external systems, how should random failures of the external system be handled?

Our product can integrate with a variety of external systems. We support 5 different databases for storing business data, 5 different version control repositories, 5 different bug trackers,... We also ...