Published 5 years ago
The Test Automation Challenge
It’s been said that software is “eating the world.” More and more, critical systems that were once controlled mechanically, or by people, are coming to depend on code. This was perhaps never clearer than in the summer of 2015, when on a single day, United Airlines grounded its fleet because of a problem with its departure-management system; trading was suspended on the New York Stock Exchange after an upgrade; the front page of The Wall Street Journal’s website crashed; and Seattle’s 911 system went down again, this time because a different router failed. The simultaneous failure of so many software systems smelled at first of a coordinated cyber attack. Almost more frightening was the realization, late in the day, that it was just a coincidence. [1]
The digital transformation is bringing the software everywhere: it “occupies” socioeconomic, physical, and infrastructure systems otherwise controlled mechanically or by people. This "invasion" takes the form of the factorial growth of distributed software applications and systems running and interacting without human intervention. The nonconformities - outcomes that are not the correct ones - and vulnerabilities - incorrect reactions to faulty inputs - of these applications, even the most ordinary ones, are unrecoverable in real time now that humans are not in the loop anymore, and have strongly adverse consequences at all levels: business, political, societal, environmental.
Nonconformities (failures) and vulnerabilities (flaws) are caused by software defects (faults) that are always the result of human errors. The flexibility of the software - changing a small chunk of code changes radically the way it works - is its leading success factor, but also its Achilles heel. The software developer faces two tough challenges:
- firstly, she must understand the business needs - all the aspects of the desired dynamic behavior of the system to be implemented - and think them as requirements,
- secondly, she must implement by coding, that is author a static text - the program - that generates the system behavior when running on a computer.
To address the first challenge, the software developer faces an arduous communication hurdle with her patron [2]. To take the second, while drafting the program text, she must step through it one instruction at a time in her head, thinking the way the computer would, and keeping track of the intermediate computation results, including the interactions with the environment [3].
Business understanding and software design require different capabilities, competencies, and attitudes, and a fundamental semantic gap marks the transition from the former to the latter. Model-driven and formal method approaches to software engineering can improve and even guarantee the consistency of the program text, but do not ensure its compliance with the business requirements, and say nothing about their understanding. Errors of communication, comprehension, design, and coding are unavoidable and grow with the complexity of the applications and their environments. Only functional test, i.e., the search, detection, and diagnosis of failures and flaws can assist to check the compliance of the software behavior with the requirements and to localize and fix software (or requirement) defects. Functional test is
- black-box test of individual components of the distributed application under test, namely transmitting nominal (correct) or fault-based (incorrect) inputs to the application and gathering and evaluating its outcomes, and
- gray-box test of distributed applications, i.e., transmitting inputs to application components and observing and assessing the following exchanges between them and with the environment.
Note that code verification and white-box testing, performed in conjunction with software development, can help to improve code quality (e.g., finding uncovered program paths) and to find coding errors (e.g., infinite loops), but does not help at all to examine the compliance of the system implemented by the code with the requirements.
Functional test does not prove the absence of failures and flaws, but, if performed intelligently, enables to search for them
- accurately - avoiding false positives, i.e., failed tests where there are no software faults, and false negatives, i.e., passed tests that hide software faults,
- effectively - with test cases that have high failure/flaw discovery potential,
- efficiently - by prioritizing the test runs to find the critical failures and flaws as soon as possible,
and strengthen the application trustworthiness. Software testing has come from a supporting function to one that is critical for business performance [4].
Unfortunately, software testing turns out to be a tough challenge. It is hard, expensive in labor and equipment (up to > 30% of IT spend [5]), and time-consuming. 85% of the test is done entirely manually, 15% with the support of partial test mechanization. The most critical test tasks - e.g., test input design, test oracle determination [6], test outcome evaluation, test run prioritization, test campaign planning - are human-based, knowledge-intensive, and burdensome. Manual testing is tedious, low-rewarding, inaccurate, ineffective, and inefficient. Only prominent bodies (large enterprises and big government organizations) can afford to implement ad hoc, custom-built, high-code, and complex test systems that mechanize just some routine test tasks for a particular application. These systems are unstable, brittle, and expensive to design, develop, and maintain - often more than the applications that they test - by rare highly-skilled professionals [7]. On the other hand, about 50% of the actors with test obligations (Mid Caps, Startups, SMEs, freelancers, but also departments of large bodies) perform near-to-zero testing for lack of awareness, skills, budget, and tools [8].
The situation described above is at the origin of the emergence of a global test automation market [9]. ‘Test automation’ names the process of using specialized software that performs various testing activities, and the global market for test automation is expected to grow at a CAGR of 15.4 percent between 2017 and 2025, reaching $ 109.69 billion in 2025, from $ 30.45 billion in 2016 [10]. This market is segmented into products and services, the latter being split into professional services and outsourcing services. The service segment accounts for the majority of market share and records maximum growth thanks to the preference of most mainstream customers for outsourcing services. Functional (conformance and vulnerability) test holds the most substantial portion of the test automation market.
An essential factor that fuels the development of the test automation market is the growth of the mobile app market. Note in this regard that the test of mobile applications only concerns the “last mile.” Thanks to the spread of cloud computing [11] and to the unlimited possibilities of combining services via APIs [12], the global ecosystem of "back-office" services feeding the mobile applications is growing at a factorial pace, and is essential in domains like energy, transport & logistics, water, and waste, the vital infrastructures of the ecological transition [13]. Analysts believe that in a few years the test of human-machine interfaces will ceil to 15% of the testing activity, the rest being devoted to the APIs of the background services. The crowdsourced testing approach that is emerging recently [13] is utterly inadequate for API testing and will be stuck to the HMI testing market (Websites and mobile apps).
Although the concept of test automation has been around for more than 15 years,
the level of automation of test activities is still very low (between 14–18% for different activities). This low level of automation is the number-one bottleneck for maturing testing in enterprises [14].
Test automation has become a buzzword that indicates heterogeneous and rudimentary (the term mechanization could be more appropriate) methods and tools:
- the simple test execution (transmission of predefined stimuli and collection without evaluation of the test outcomes),
- the test run evaluation with manually produced static oracles,
- the generation of test input and oracle skeletons from the service signatures, and, more recently,
- the model-driven generation of test input contents.
The current test automation tools package these mechanisms, and the tester, to use these tools, has to write code (possibly, a lot of code).
- to build the test data (the predefined test inputs and oracles),
- to implement the test tasks (generation, execution, evaluation), and
- to integrate these tasks in the test process.
In summary, the accelerated digital transformation requires the adoption of Agile and DevOps but makes more stringent the primary business goal of the absence of failures and flaws in production. As a result, it engenders the conflicting injunction to obtain ‘quality at speed,' but Agile and DevOps without true test automation turn out to 'speed without quality.' Hence we come back to the question: What is test automation?
Notes
[1] Somer, J. (2017, September). The Coming Software Apocalypse. The Atlantic.
[2] As reported by Somer (Somer, J. (2017, September). The Coming Software Apocalypse. The Atlantic.), Ms. Nancy G. Leveson, a renowned MIT software-safety expert, said: “The problem is that software engineers don’t understand the problem they’re trying to solve, and don’t care to.”
[3] The distinguished Dutch computer scientist Edsger Dijkstra wrote in 1988 that the software engineer “has to be able to think in terms of conceptual hierarchies that are much deeper than a single mind ever needed to face before.” [EDW 1036].
[4] From the World Quality Report 2018-2019 (WQR 2018-19) - annual report on Testing and Quality Assurance, sponsored Capgemini, Sogeti et Microfocus, that is a reference for IT professionals. The tenth edition (2018-19) is based on a survey of 1700 executives from across ten different sectors and 32 countries.
[5] The imprecise evaluation of the QA and test share of IT budgets is due to several causes. One significant cause is that with the widespread adoption of Agile and DevOps approaches, early test activities are usually counted as development activities (WQR 2018-19).
[6] The term ‘oracle’ designates either a dynamic mechanism for evaluating a test run or a static representation of the expected outcome that can be compared with the actual outcome. The oracle problem is considered unanimously as the hardest testing problem.
[7] In 2005, Microsoft introduced the SDET (Software Development Engineer in Test) acronym to qualify a software engineer that combines software testing and development to automate the testing itself. Apple and Amazon use the SDET job title, while Google prefers SET (Software Engineer in Test) for a similar role. Large platforms, faced with enormous complexity - Google 40000 developers handle 2 billion lines of code, that is an average of 50000 lines per developer - have the will and the ability to hire all these rare professionals, above all by the wages they propose.
[8] Forrsights Developer Survey, Q1 2013.
[9] Gartner review on test automation.
[10] Transparency Market Research.
[11] Bloom, N., and Pierri, N. (2018). Research: Cloud Computing Is Helping Smaller, Newer Firms Compete. Harvard Business Review.
[12] Brynjolfsson, E., & McAfee, A. (2014). The second machine age: Work, progress, and prosperity in a time of brilliant technologies. WW Norton & Company.
[13] Mazzucato, M. and McPherson, M. (2018). The green NewDeal: A bold mission-oriented approach. IIPP Policy Brief, UCL.
[14] WQR 2018-19. This objective has been placed at the top by respondents in all the recent reports (WQR 2014, 15, 16, 17).
(published on the simplYTestify blog)
