I've been using the Canary build (nightly build) of Chrome for a month now. So far the quality has been high and it has worked without any major issues. Can someone describe the automated testing that is performed on each release?
I believe the Canary builds are not checked manually before release, so the automated testing must be extensive.
http://blog.chromium.org/2011/05/adding-more-yellow-to-mac-color-scheme.html
Ok Chrome is built on top of the WebKit open source engine that was first built by Apple for Safari. The first round of tests are run as part of that project, and they are documented at http://www.webkit.org/quality/testing.html
The WebKit Tests Before patches can land in any of the frameworks in the repository, the layout regression tests must pass. To run these tests, execute the run-webkit-tests script.
The script will dump the render trees for all of the pages and diff the results against the expected correct results. If no differences are found, then the patch has passed the tests. If any tests fail, then the patch cannot be committed until the discrepancies in the tests are resolved.
Regression tests must be run on Leopard, since the expected results checked in were generated on Leopard.
The JavaScriptCore Tests If you are making changes to JavaScriptCore, there is an additional test suite you must run before landing changes. This is the Mozilla JavaScript test suite.
What's covered by the JavaScript tests?
The JavaScript tests cover the functionality of the core JavaScript engine. This includes the following JavaScript objects: Arrays, Booleans, Dates, Functions, Global Object, Math, Numbers, Objects, Regular Expressions and Strings.
In addition, the JavaScript tests cover parsing, lexical conventions, expressions, statements, type conversion, and exception handling. How to run the tests Execute the run-javascriptcore-tests script. The script will run all the tests and summarize how the results differ from what is currently expected.
What just happened After all the test runs have finished the results of tests are saved to actual.html. The script the compares these results from your local tree against what is expected to pass/fail from the tip of tree. If there are any regressions caused by your changes you'll be made aware of them. If you fixed a bug that caused an existing failure, you'll also be made aware of what specific test your fix affected.
Google then takes webkit, and builds it into Chrome, and runs a whole suite more tests which are documented here: http://dev.chromium.org/developers/testing
Here is an non-exhaustive probably-outdated list of some of the tests currently used by Chromium: automated_ui_tests - Uses automation like ui_tests, but runs on separate reliability bots.
base_unittests - Tests the base module.
browser_tests - Tests the browser UI, but in the same process rather than from a separate process like the ui_tests.
installer_unittests - Tests the Chromium installer.
interactive_ui_tests - Uses automation like ui_tests, but tests interactive features like find-in-page and tab dragging that need simulated user events. These run on separate bots where there is always a logged-in user so the interactive events don't fail.
ipc_tests - Tests the IPC subsystem for communication between browser, renderer, and plugin processes.
layout_tests - Part of WebKit. We run these tests using test_shell (see below). mini_installer_test - Tests the mini installer.
net_perftests - Performance tests for the disk cache and cookie storage.
net_unittests - Tests the network module.
npapi_layout_test_plugin - A NPAPI plugin used with the layout tests.
npapi_test_plugin - A NPAPI plugin used with the plugin_tests and ui_tests.
perf_tests - Tests performance for some submodules - JSON, Safe Browsing, URL parsing, etc.
plugin_tests - Tests the plugin subsystem.
reliability_tests - Tests to verify Chromium recovery after hanging or crashing of renderers.
security_tests - A set of security tests for Chromium.
selenium_tests - A driver for running selenium tests.
startup_tests - Test startup performance of Chromium.
tab_switching_test - Test tab switching functionality.
test_chrome_plugin test_shell - Test Shell is a standalone application for running the renderer (WebKit).
test_shell_tests - A collection of tests within the Test Shell.
ui_tests - Tests which launch Chromium in a separate process and control it through Automation.
ui_unittests - Unit tests for the /src/ui module.
unit_tests - The kitchen sink for unit tests. These tests cover several modules within Chromium.
So that is the testing that is done on every build of Chrome.
