Practice makes man perfect – so goes the old adage. This is even more true for tests. While there could be few scripts to be executed on-demand for certain situations, as a rule of thumb attempt should be made to include automation test scripts with a continuous integration system. If there is no continuous integration system in place, it is absolutely a great idea to invest time and energy in creating one!

A good frequency of test execution helps in quickly establishing bad-builds when an error occurs. Comparing the test reports of successive runs, one can quickly ascertain when a test case first failed. This can save tremendous amount of time otherwise wasted in tracking the source of bugs.

Test results as Execution Log and Trace

As software ages, new features are introduced and sometimes these could break existing functionality. A daily build integration of test scripts preempts such avoidable errors. A test execution report with all “greens” and no “reds” or “yellows” is a tremendous confidence booster as well as discipline inducer for the development team. Knowing that a failed test in automated build will result in rejection of the task at hand, instills the discipline of practicing good software development practices in the team. And – such a report is a direct and very accurate indicator of the health of the software being developed.

A test script which is executed with the daily build matures early and proves to be a very important first gate for common errors. Without going through the rigor of being executed frequently a test script will remain weak and purposeless.

This concludes our multi-part article on Test Automation Systemic Investment Plan. Stay tuned for many such topics in future.

Since test automation scripts would run in most of the cases in scheduled mode unsupervised without any manual intervention, it becomes imperative to have solid tracing and logging mechanisms built into the automation framework so that if anything goes wrong, it is still possible to recreate the error and make meaningful logical inferences about the source of the error.

The Long Story

All remains well when nothing goes wrong. But that unfortunately is not the case in real world. In real world, disks get full, memory becomes corrupt, file-handles become invalid, power goes off – in short – so many things which “should” not happen, indeed happen. This reality of our world is best expressed by Murphy’s Law – “If anything can go wrong, it will.”

And when things go wrong, we want to set them right. Further, being human, we would want to know the circumstances which led to the error and take precautionary measures for future.

Such “Holmes”-like investigation is never easy. More often than not exact sequence of events leading to the error or unknown. In other cases, same conditions do not give the same error!

So what could aid the poor test-developer in debugging such errors when they occur?

Early investment in tracing and logging mechanisms

While the choice of exact mechanism for logging and tracing will depend on multiple factors like platform of development, underlying operating system, language/compiler support, system constraints amongst many other, there are few guidelines which could help in making the right choices.

1. First and foremost, if tracing and logging effort is included in architecture and design, this in itself is a big right step. This would ensure that implementation is natively available when required.
2. On platforms like .Net and Java, tools like log4net , log4j, slf4j, Enterprise Library could be of immense use. These are well crafted feature rich libraries which make the task of logging a breeze. Before trying to “re-invent the wheel” – it will be a good idea to explore these libraries and ascertain requirements against their feature sets.
3. As we have discussed before keeping the logging implementation simple is a good idea.
4. Keeping in mind the probability of Murphy’s Law, conditions which should never occur, should always be logged.
5. It will be worth-while to invest in a screenshot utility so that errors are captured at UI level. Experience says that these screenshots become immensely useful evidence in debugging process later.
6. All popular logging frameworks have notion of log-levels and while writing test-logic it is imperative to use the log levels judiciously. Too less of log will be lost in woods, too much of log will be noise. Log levels help by providing a throttling mechanism so that at runtime we can  tweak what messages we really want to be logged and what we want to be filtered depending on our current situation.

Not all good things in a developer’s life are free. Investing in tracing and logging mechanism is in reality an architectural decision which should get translated in non-intrusive design elements, further making its way into actual implementation. If tracing and logging is not given its due importance, and is meted with second-grade treatment by adding as an after-thought to the implementation, the results will fall short of desired and will not be adequate enough to be a strategic aid in debugging.

Continued…

Continuing our conversation about tips for better software test automation, this time we would talk about reuse.

Given the fact that many test automation initiatives suffer from time crunch, this aspect becomes even more important to understand. While it is very tempting to want to write your own framework for everything you need, it is not practical. If we pause and think, we can emerge over the urge to write our own libraries suited uniquely to task at hand.

While there are legitimate scenarios where such an approach might be required, more often than not, it is counter-productive. The contrast between generic and specific implementation is too stark, yet too subtle at the same time. It is advisable to seek the middle ground in our work.

As a test automation developer it is important to first learn what is already available in terms of frameworks, libraries and refer to other people’s documented experience. Only when we are sure that an existing implementation will not suffice for more than 80% percent of what we need, we should look for custom/specific solutions. Because in most of the cases, 80% requirement match is a large enough ground to start with.

Free/Open Source Tools Available

There are multiple competing tools available in both commercial and open source categories and we can evaluate them before embarking on our own implementation journey. Following is a brief list of open-source/free to use tools/frameworks available for various types of automation needs -

1. Selenium Framework

Selenium is a suite of tools to automate web browsers across many platforms.

2. Robot Framework

Robot Framework is a generic test automation framework for acceptance testing and acceptance test-driven development (ATDD). It has easy-to-use tabular test data syntax and utilizes the keyword-driven testing approach. Its testing capabilities can be extended by test libraries implemented either with Python or Java, and users can create new keywords from existing ones using the same syntax that is used for creating test cases. Robot Framework is open source software released under Apache License 2.0.

3. Fitnesse

Fitness is an acceptance testing framework and also duals as a fully integrated standalone wiki.

4. Watir

Watir, pronounced water, is an open-source (BSD) family of Ruby libraries for automating web browsers. It allows you to write tests that are easy to read and maintain. It is simple and flexible.

5. Watin

Inspired by Watir development of WatiN started in December 2005 to make a similar kind of Web Application Testing possible for the .Net languages. Since then WatiN has grown into an easy to use, feature rich and stable framework.

6. Frank

Frank allows you to write structured text test/acceptance tests/requirements (using Cucumber) and have them execute against your iOS application.

7. MonkeyTalk

MonkeyTalk is an integrated environment for recording, customizing, running and managing test suites for native mobile applications. It supports iOS (iPhone + iPad) & Andriod.

8. Microsoft UI Automation Library

Microsoft UI Automation is the new accessibility framework for Microsoft Windows, available on all operating systems that support Windows Presentation Foundation (WPF). UI Automation provides programmatic access to most user interface (UI) elements on the desktop, enabling assistive technology products such as screen readers to provide information about the UI to end users and to manipulate the UI by means other than standard input. UI Automation also allows automated test scripts to interact with the UI.

9. White

White is a framework for automating rich client applications based on Win32, WinForms, WPF, Silverlight and SWT (Java) platforms. It makes use of Microsoft UI Automation Library.

There are many commercially available tools too which can help in your own test-automation adventure!

Again, remember, if you really wish to re-invent the wheel, you can, but the actual fun begins when you start putting wheels together and build a car!

Testing SaaS apps is challenging.  From our experience of testing several SaaS apps, we categorize the testing strategy in five key areas. This article describes those areas and provides a quick checklist of what are the different things that needs to be tested.

1. Functional Testing Checklist

This is the standard functional testing to validate if the app is doing what it is supposed to do.

• Conduct rigorous Manual tests as per defined test plans, keeping the end user in mind
• Conduct Exploratory tests based on existing or new test cases
• Conduct Browser compatibility testing to check performance of the application on different web browsers
• Conduct Regression testing on every release, minor upgrade, an integration or data migration.
• Automate Functional and Regression tests
• Conduct tests in target environment – whether it is your data center or the Cloud.
• Conduct reliability testing to find the total defects of the application and thus reduce the number of failures, during real time deployment.

2.Infrastructure and Security Testing

The goal of this is to test the underlying infrastructure and security of the app.

• Test the security of the SaaS application for typical web application security issues such as HTTP header injection, Cross Site scripting (XSS), SQL Injection etc.
• Test security of the network where SaaS application is being deployed
• Test possible scenarios of security attacks/threats
• Test the application with respect to access privileges with the corresponding job roles (especially in a multi-tenant environment)
• Test the security, integrity & accessibility of test data (especially in a multi-tenant environment )
• Determine situations that could make the SaaS application vulnerable
• Test compliance with Payment Card Industry Data Security Standard (PCI Compliance)
• Maintain logs of security warnings, errors and requests from unreliable sources

3.Performance and Scalability Testing

A critical area of SaaS apps testing, and is really the foundation of business growth.

• Conduct scalability tests to determine the capacity of the application to scale up or down as per requirements
• Conduct volume testing for your data
• Conduct availability testing for a planned period of time
• Conduct load testing under normal as well as peak load conditions in multiple environments

4.Integration and API testing

Success of SaaS apps lie in how well you have thought of scenarios where third party developers can build their own apps using your APIs, and add value to your product. So testing all the APIs for functionality, security, usability, performance and completeness of documentation is critical to make them successful.

5.  Operational Testing

This area is intended for the operations team whose objective is to make sure the apps are working fine, and take care of customer service & billing. Usually, there are tools that are built as part of the product which help the operational team members to monitor, track and analyze for issues. The areas to look for:

• Application, Services, App Server, Platforms (OS), Databases and Data Center Level Logs/Alerts/Warnings/Errors for functionality and performance.
• Billing and Customer Support Tools, especially for integration

In this article, we will look at the specific challenges faced by QA organizations of SaaS companies.  In our next article, we will highlight the testing strategy to address these challenges.

1. Handling Changes through Frequent Releases - Every time the Application is upgraded, the users have to understand the impact of the change, validate it against the existing system & ensure that the impact on the existing features of the application is minimal. Managing and executing all these activities within a short time span (1-2 weeks) is challenging.  When SaaS upgrades involve interface upgrading, compatibility and integration issues across old and new interfaces crop up for the subscribers.  Live upgrades being simulated or tested on the SaaS application impedes the activity of the existing users.
2. Security Testing - Maintaining data security, accessibility & integrity on a single SaaS application across multiple tenants. To understand individual privacy requirements, privilege levels, behavioural patterns and provide adequate privacy to the data can be a daunting task.
3. Integration Challenges - When subscribers integrate their internal enterprise applications with SaaS, inbound and outbound data integration validations from client networks to the SaaS providers is needed. In such cases it is very difficult to conduct thorough validation simultaneously ensuring 100% data security and privacy
4. Data Migration Issues  – Data migration across different SaaS applications or from other applications to SaaS can be challenging in terms of time taken for understanding the requirements and the exhaustive integration validation processes
5. Licensing:   The SaaS app licensing may vary by functionality, usage (such as volume of transactions or amount of specific data) or # of named/concurrent users.  All this needs to be tested across every release.
6. Performance testing: Successfully modelling the most-used business transactions, application usage and user mix may require greater diligence than an on-premise application.

What can make a good test a weak one? What can make a simple test an effective one?

Data.

Or Test Data to be precise.

A test and its test-data are like body and soul. But like in real life, soul often gets neglected. While test developers focus a lot of their energies in automating most complex of test scenarios, same extent of attention often gets denied to test data. This neglect can manifest in different ways -

Common pit-falls in test data implementation

a. Didn’t know, didn’t do

This means the test developer was not aware of the significance of test data and missed to implement any special strategy for the same.

b. Strong Coupling with Logic

Another mistake which happens is tight coupling with test logic. The flow gets intermingled with setting/reading of data and gets strongly embedded in the DNA of the test. It becomes difficult to see test separately from its data. For ex, test to check if account gets locked after multiple repeated attempts can suffer from this issue when the number of attempts is hard-coded in the test. In future when this business condition changes, test becomes invalid. Sometimes changing such hard-coded data has severe impact on the over-all test and results may get misinterpreted.

c. Randomize Test Data

Quite often test data is left to random data generators. While this can be useful in certain stress testing scenarios, it can be counter-productive to rich test executions which require meaningful data.

d. Repeated Data

Each test run gets executed with same test data every time. This monotonous repeat execution may fail to detect any bugs which are associated with usage of data since some of the data-flows may never get tested.

So what do we do? Few tips from our Experience…

So what should be the approach for proper data handling in test automation scenarios? Well, the key lies in moderation and mix-match of approaches.

a. Give importance to test data

First and foremost, give that importance to test data it deserves. Invest in test-data planning when you plan for test automation scenarios.

b. Separation of Concerns

Keep test logic separate from the data. Rather than testing for 3 attempts, test for n attempts where n could be picked based on a test strategy.

c. Randomize with caution.

Instead of having a randomizer as sole test data provider, implement a test data strategy factory. This will provide multiple implementations of test data providers. A random test data generator can be one of the implementations and other could be implemented as required. This will ensure that test logic is not tightly bound with test data and can change independently of test data.

d. Change test data.

If you invest in a test data strategy factory, this will be an added advantage since there could be a test data strategy to modify data for each test run or for any run based on some rules relevant to the scenario at hand.

There are multiple programming techniques which could aid in effective test data planning and execution. Important thing to remember is that, test data should be given equal importance if not more, as actual test logic. This approach will yield multiple benefits in future in terms of re-usable test data which makes tests more useful & effective with each run.

To be continued…

Writing Test Automation code traditionally has been a low-level activity. Find a window, find a control, set the value, click the button – and then – find the window, find the control, check the value, and so on. More often than not, a major portion of effort goes in what we call “Infrastructure” code than “Heartbeat” code. Core logic of the test is cluttered with plumbing to deal with the interaction with the application user-interface.

As a rule of thumb we have observed that in order to test X lines of code, it takes at least 2.5 X lines of code to test basic flow and about 4-5 X lines for reasonable coverage. Now given the scale of modern rich UI applications, complexity of required tests can be anyone’s guess.

Many test automation attempts start with a straight translation of manual testing steps, interlaced with API calls to identify forms, fields and other controls. This approach often produces quick results with few test cases getting automated in very short period of time. However like any other quick n’ dirty solutions, if not corrected at early stage results in reams of test-automation logic which is monolithic in structure and extremely fragile to maintain.

So what can help us here?

Power of abstractions.

Abstractions

In our daily life we all use so many gadgets – laptops, cars, watches, refrigerators, etc, without ever worrying about the complexities of the underlying subsystems. Each of these devices provide specific features by combining many smaller sub-systems behind the scenes. This layering makes it easy to install a program on a laptop, adjust time in a watch, make ice-cubes in refrigerator and so on.

Similarly, while programming test automation if we write test logic in terms of high level widgets instead of the underlying representations, it will make our tests a lot more readable and extremely amenable to change. After all,

LoginForm.UserName = “UserName”

is lot easier on brain cells to understand than

Windows.Application.Find(“My Application”).FindWindow(“LoginForm”).FindControl(“UserName”).SetValue(“UserName”)*
*Not actual API, just representational

Concept of Zoom

Another way to get this concept is look at the concept of Zoom. Zoom let us look at the same landscape at different levels of understanding. Each level of zoom will show entities relevant at that level.

In test automation parlance, this means that before even thinking of writing test flows, we should think of entities participating in these flows. Tests should be viewed as interactions between these entities rather than as sequential steps. This approach immediately adds up to ROI on the effort because it is easier to device and test complex interaction between various entities than writing complex sequential scripts. This approach instantly shifts attention to tests over data by separating the two, data being input to interaction than being primary test driving factor.

To be continued…

1.   Treat Test Automation like a Software Development project:

We have seen that Test Automation projects have greater success if they are treated with the same discipline and rigour of software development. A successful testing automation project:

1. Focuses on design (reusability, extensibility etc.)
2. Takes a framework driven approach
3. Has team members who are experts in writing code and not just testing or test automation tools.
4. Has a clear cut development and governance plan
5. Maintains detailed documentation
6. Prepares a plan to track bugs and rectify them
7. Identifies reusable functions in automation scripts

2.   Set measurable goals for your Test Automation project:

It is essential to set clear, quantifiable objectives for the test automation project. These goals should clearly bring out:

1. Percentage of test cases to be automated in various Test Suites
2. % Improvement in the test coverage
3. % reduction in the time to release the build
4. Reduction of test cycle time of new releases by improving their time-to-market (management-level goal)

For testing metrics like those above,  tools such as Automated Execution Efficiency Ratio (AEER) could be used.

 3.  Choose a testing tool that answers your Automation needs:

Selecting the right tools is obviously a very important foundation for success in your Test Automation project. We suggest employing the following criteria in selecting the right test tool:

1. The tool must posses a built- in facility to run multiple test cases
2. The tool should be able to classify tests being run, basing on set criteria  while simultaneously running multiple test cases
3. The tool should be able to run the desired test cases without human intervention
4. The tool should be able to generate execution reports or logs  at the end of each  test case
5. The tool should be able to compare two values, so that the difference between real and expected results can be estimated.

4.  Get Your Design of Test Automation Framework Right

Test Automation framework allows us to define the scripting methodology to automate tests, creates mechanism to hook into the application under test,  execute the tests and report the results.  Framework establishes a common standard for the team to achieve the goals.

While designing a test automation framework, the key goal is maintainability. The tests will change and good automation framework will support and facilitate change. If there is any change in any test, it should be very easy to change the automation suite, and in some cases not required to change at all. In addition, performance is also important. In a typical automated test suite, several tests will be executed. If each test can be executed faster by a second or so, it will matter! Ultimately we are talking about cutting down the time to test!

The following list will serve as a checklist (not exhaustive) to design your test automation framework:

1. Define your scripting standards for business components, system communications, data check points, loggers, reporters etc.
2. Hide the complexities and critical aspects of the code from Test engineers and expose them to the implemented Libraries and tests.
3. Separate Scripts and Data – automated tests scripts must be separated from the input data store (could be in text, XML or Excel files). Ideally only input data gets manipulated for testing with multiple values and the script does not change.
4. Think through the applications future enhancements and modifications before building your test framework library, and build in the extensibility in it.
5. Use the standard process for script versioning.
6. Your framework will have components/libraries to handle

• Application Functionality
• Types and sources of Input data
• Communication between utilities, components and systems such as communication between distributed components,  check points, communication with logger etc
• Tool Extending capabilities – features which are not supported by the tool that you have selected.

5.  Ensure high quality of Test Data:

A good investment of time and effort should go into creating well-structured test data during the early stages of the automation testing project. This makes the process of writing automated tests easier, in the later stages of the project life-cycle. The quality of test data also determines its reusability and maintainability.

6.  Build the right team and invest in training them

Apart from the right tools and a proven testing process, a testing automation project requires a dedicated, and skilled team who are trained to understand the complexity of the project and work accordingly . The challenge in such scenarios is to find skilled, dynamic  personnel who should be trained to:

1. Handle ambiguous scenarios and test cases
2. Conduct manual testing along with developers
3. Write automated regression tests
4. Execute automated regression packages.
5. Handle integration and performance testing
6. Plan and gather requirements

7.  Conduct early and frequent tests

The more frequent and earlier the testing automation process is applied in the project life-cycle, the earlier are the bugs detected in the process. The earlier the bugs are identified, the more economical it is to remove them from the system at a preliminary stage.  This process also helps refine the automated test suite for  further enhancements.

Continuing the discussion about our distilled experience in test automation, we will continue our discussion, starting with the basics. Anyone who can type a semi-colon, can write code. Writing code is easy. Writing good is not. And writing good code which can test other code is even more difficult. Common mistakes to avoid when embarking on the test automation journey are:

A. BUFD (Big Up Front Design)

In today’s ever evolving world, it has never helped developers to create huge design artifacts in the very beginning only to dis-mantle, re-do, re-work and waste lot of time (and money) in the process. This holds true even more for test code. Investing great amount of time in designing and creating testing frameworks which are not immediately put to use in a live test script is bound to put pressure on the test engineers as time-lines shrink and delivery deadlines converge. Result will be beautiful, unusable test code and, even with good deal of investment in test-automation, poorly tested application. Needless to say this will result in a spiral situation further worsening as budgets will be questioned by stake-holders and will shrink since it will be very difficult to demonstrate value-add from code which is still in the kitchen. No customer will pay for a dish as long as it is in kitchen!

B. NUFD (No Up Front Design)

This is contradictory to BUFD, but equally catasthropic. Directly jumping into creating test scripts without thinking through aspects of maintainability, “extensibility”, code-reuse, SOLID design principles & simplicity, will result in poorly developed scripts which would be bug-ridden, fragile to change, and will be tough to evolve as application goes through churn and evolution cycles. This could be fairly tricky situation situation to handle if there are multiple test developers involved in the development of scripts.

Side Note: It is a balancing act to invest right amount of time and energies into design of test-code. What is just enough is a tough question to ask, tougher to answer. Simple design choices are the best. As Kelly Johnson wisely said – “Keep it Simple, Stupid”, KISS principle is the best yard-stick to tackle this tough decision. If it requires reams of comment lines to explain that pretty little hack, it is probably not worth to keep it in.

C. Poor Structure

Good (test) code looks good too. If it is messy looking, it is highly likely it is messy implementation too! Clean well thought folder structure, name-spaces, packages, etc depending on the language used go a long way in making code readable & maintainable by posterity.

Contd…

Abstract

The need for automating functional tests for software has been well established over recent couple of years with ever growing application complexity and ever shrinking customer release cycles. While putting time and effort in any kind of test automation helps in the application life-cycle, we have seen that a systematic investment plan in developing right test automation recipes and frameworks yields significant savings of time and money.

Writing code for automating test cases is  still writing code at the end of the day and faces similar challenges as traditional software development, plus a few more unique to the world of testing. Being considered mostly as a support activity in software development life-cycle, test automation efforts usually fall short of what is required and desired.

This white-paper discusses the need for a regular & systematic investment in automating functional test cases for any application and provides an in-sight into how such a planned effort leads to significant savings of money & time (which again is equivalent to money).

Typical Software Test Automation

Traditionally software test automation is seen as a project activity, with efforts focused on a particular application/system at hand. While this approach makes it easy to initiate the automation effort, it does not promote benefits of reuse of design ideas, implementation and architecture.

As the application goes through churn and evolution, so does the need for updating and keeping the test automation suite. Complexity in the application has a cascading affect on the complexity of the automation code, sometimes even more to account for the plumbing required to automate newer complex functionality.

Like any other piece of written code, test automation logic can also suffer from defects and this further leads to another dimension to the already complicated puzzle.

Test Automation SIP

Having faced similar situations in our multiple engagements, we realised that it is best to treat test automation as an umbrella activity in the application development and invest little, but invest consistently in the automation code.

We have captured the essence of our multi-man-month experience in developing test automation suites for multitude of applications of varying sizes and complexity in this white paper.

1. Test code is first-class citizen like application code

Test code is as important as the bits which make any application. So it is subject to same rules of engagement – coding standards, naming conventions, design patterns, architectural best practices & well maintained documentation.

2. Abstractions help

While it is legitimate to write test automation logic using low level objects like buttons, text-boxes, it makes tremendous difference when such objects are encapsulated and abstracted to represent high level objects in the eco-system like forms, widgets, etc.

3. Test is as strong as the test data

While test automation engineers focus their time and energies into developing test scripts, the data which is fed to these scripts often remains ignored. Probability of a test catching a bug in the system increases manifolds if the test data is intelligently randomized.

4. Do not reinvent the wheel

Many software vendors have published libraries for UI automation of applications being developed on their platform. Microsoft has made automation API integral part of newer versions of .Net framework. Microsoft UI Automation is the new accessibility framework for Microsoft Windows, available on all operating systems that support Windows Presentation Foundation (WPF).

UI Automation provides programmatic access to most user interface (UI) elements on the desktop, enabling assistive technology products such as screen readers to provide information about the UI to end users and to manipulate the UI by means other than standard input. UI Automation also allows automated test scripts to interact with the UI.

5. Invest in tracing and logging in test automation logic

Since test automation scripts would run in most of the cases in scheduled mode unsupervised without any manual intervention, it becomes imperative to have solid tracing and logging mechanisms built into the automation framework so that if anything goes wrong, it is still possible to recreate the error and make meaningful logical inferences about the source of the error.
Another important piece to be made available to test developer is a screenshot API so that script can take screenshots when things do not go as planned.

6. Integrate with the daily build

A test script which is executed with the daily build matures early and proves to be a very important first gate for common errors. Without going through the rigor of being executed frequently a test script will remain weak and purposeless.

Contd…