SASQAG Monthly Meeting "Signals in Testing"/How Google Testing Software

Tonight,  I attended SASQAG Monthly Meeting. Jason Arbon from give a talk about "Signals in Testing".   Jason used to be SQL Server WinFS team, then he went to Bing, and join Google later.  After he and James finished their famous book "How Google Testing Software" (I already order the book), Both of them leave Google (we joked about this. twice tonigt).

   I will share you the slides later. The key point is that the way of testing software is changing, and we need be "quick" to develop and release feature.  There are a couple of points he is making about how we can do quickly.
·         Google called engineer at search department as search quality engineer, not tester, not developer.
·         Chrome upgrades every  six weeks (in the background when you running apps under your OS), and in the end, most of the users are usingt he latest version.  Compared with the long release cycle of IE, they can ship feature more frequently.  They use Crowd sourcing testing to allow customer to install beta version of the Brower and report issues. Roll out of the browser to end user is incrementally.  They also collect a lot of statistics about how user use the feature to improve their product. Some feature will be cut when no one use it.
·         10 Minutes Test plan.  Test activities just focus on the "risk" feature (depending on how many people use the feature, and how flaky of the code).  In this way, you only test what is the important. Security and Speed are features, not dimension and they are the most important features we tested in Chrome.  For us,  it is the same thing,  If we propose some change, we just focus on what is changed, what is the goal, and spend more time on verify that we achieve the goal.   Thinking about we have millions of tests today at SQL Server,  if we don’t carefully thinking about reducing  test cost, we will pay big cost later.  So focusing on important area, write less tests is the key.
·         He mentioned that Google throw away tons of automated UI tests (using Selenium), but rather to let dev. wrote a lot of JavaScript unit tests. For example, instead of using UI test automation to test the web page of Gmail, dev just write unit tests using JavaScript Unit Testing Framework to test the Gmail API.   His big argument is that don't automate  the regression tests, the cost of maintain such tests are huge if you are active developing.  You just need to build a fast and reliable BVT tests and run every time.  I think in term our Fabric regression tests,  we might use similar approach.  If the regression test is hard to write, hard to maintain, we don’t need to write such test for the bug we found.  s well), but has lack chance to finding new bugs.
·         Dev is the quality owner.  Dev Lead  (who is also technique lead for the project) decide everything about the project include Testing and Quality.
·         As a result,  Google try to integrate tester into product team, i.e., tester report to Dev lead. And the lead decide how many tester he/she want.
·         He spent a lot of time talking about the raise of mobile device and mobile app. He believe this is a very hot area, also very challenge for testing.  He shows a couple of interesting finding for Games in both iPhone and Andriod. For example, why the same app received poor rate in Android, but not IPhone (because the fragmentation of Android, it is hard to test all kinds of devices). 
·         Finally, he saw us a slide about the future of tester's career (and what is happening inside Google).  The slide is shocking.
 You can find out many of the resources at
NOTE: I tried to remember what I heard. Some of the contents might not be correct, but the general idea is the same.
Comments (0)

Skip to main content