The Twelve Steps of a Managed Testing Workflow

The Twelve Steps of a Managed Testing Workflow

- And how can help

The Testing Workflow can be broken down into twelve steps. By creating steps it is easier to define processes that need to occur prior to, during and exiting each step. Some projects may not require all of the steps, but, on average, most projects will fit into this model. T-WorMS is designed to help out all along the way by providing support for the entire testing workflow, from start to finish and back to start again. T-WorMS works for little projects, big projects, projects with adequate schedule time and projects that are frantic. T-WorMS is designed to work in the real world where the requirements may be sketchy, the schedule ludicrous, the expectations inflated and historical information limited.

We have enough to do without having to use a painful and annoying tool set. T-WorMS provides a simple interface that allows testers, analysts, automators and managers to create their attack plan, design and write the actual tests, document those tests, collect summary information and track run time statistics in a reportable format. It provides a simple and straightforward solution that lets you painlessly create and follow a Testing Workflow without adding a burden to the test effort.

The twelve steps are described below, including an explanation of how T-WorMS can assist as you progress through the workflow.

Step 1 - Setup and Administration

There is a certain amount of administration work that is required for every project. The project has to be defined, resources allocated and the test approach determined. If you are working in an environment that uses formal test plans, a lot of this information can be found in that document. Test plans should contain the strategic approach to the testing - defining the rules of the game for all the players. By setting the rules and the expectations at the beginning of the project, the project will run more smoothly with less surprises and better participation by the project team.

Setting the Framework

Any project requires some amount of setup in the beginning and administration throughout. It's best to start a project with these steps as this will set the playing field for the rest of the project and the project team. In using T-WorMS to track your test project, several setup steps are necessary. These include defining the database that will be used to store the test information, defining the project context in which you will be working, selecting various configuration options (for example, require password for login, always require entering time metrics, always require a description of changes made). Within the project, you need to define the releases within that project and the builds/cycles within those releases. T-WorMS tracks Test Cases based on the Release to which they are assigned (Test Planner function) and the Build/Cycle in which they are run (Defect tracking and Test Results) and the Environment in which they run.

Categories

You may also want to define valid test environments and valid test categories to further define the Test Cases. Using Releases and Cycles to divide up the testing can be very helpful for determining how the testing is progressing, allocating resources and checking test completion information. For example, some Cycles of testing may be conducted in certain environments. For a Windows application, you might run Cycles 1, 3 and 5 on Windows XP and Cycles 2 and 4 on Windows 2000. By classifying the testing into Cycles/Builds within releases, planning your test effort will be easier and your metrics will be more precise and more useful.

The use of the compile label for creating Builds may also be useful for you. This can help the developers determine when a problem was introduced and what Build experienced the problem. Rather than creating Cycles, which is usually a testing term, the project team may find it easier to use the Build number to identify the software that is being tested. T-WorMS accommodates either method of classification of tests and by allowing you to partition your testing into meaningful sessions, multiple runs of Test Cases can be easily tracked across changes in Cycle/Build and/or changes in the Test Environment allowing you to compare the results of multiple executions of the same Test Case. The lists of Cycle/Build, Environment and Release can be customized to best suit your organization's needs.

Automation

Another setup requirement is needed for automation. All automation tools, commercial, open-source or home-grown can be supported by T-WorMS. For each tool T-WorMS has a database entry which contains the script templates to be used with that tool. These templates are designed to speed the development of automation and remove the majority of the programming effort required to effectively use the tool.

Environments

There are two concepts of environments in T-WorMS: the environment required by a Test Case and the environment in which the Test Case was executed. It is very important for an organization to determine how it will define and maintain the various test environments throughout the life of the project. T-WorMS is designed to provide the maximum flexibility for the test team.

The required environment is used when a Test Case is defined. Some Test Cases require specific test environments to run successfully. If the Test Case requires a specific environment, then it should be indicated when the Test Case is defined. If no specific environment is needed, "none" can be used in the definition. For example, a Test Case that requires a Windows O/S may have its defined environment is "Windows O/S". For a test that requires Windows NT, the Test Case should define Windows NT as the required environment.

The execution environment is used to indicate the exact environment in which a Test Case was executed. Depending on the complexity of your test environment, these definitions may include client, server, database, web server and hardware versions. In determining the level of detail to put into the available environment list, consider the information you need to accurately review and understand the test metrics. Will you need to know what proportion of the Test Cases was run in certain environments using certain servers or network configurations? If so, those environments need to be defined so the tester can select the proper execution environment when the Test Case execution information is recorded.

Defining environments requires a tradeoff between gathering the detailed information needed for accurate analysis vs. the overhead required to create, maintain and document numerous different configurations. This has to be an organization decision. Environments are specific to each project/application.

Version Control

One of the "Best Practices" that all QA teams should adopt is source control for all documentation. While many organizations believe this, few actually adopt it - often due to the cost associated with purchasing the tools. T-WorMS contains a full-featured file repository system that allows an organization to control its documentation in a quality way. You must define the categories of files that will be stored by T-WorMS. Once stored in the repository, the files become controlled and must be checked out for modification and checked back into the repository for storage. Old versions are maintained in the repository along with the current version. While a file is checked out it cannot be changed by anyone else. Any type of file can be stored by T-WorMS, including image files, script files, specification documents, email files, anything that is considered a "file" by the Windows operating system. File categories are defined for each project/application.

Step 2 - Finding the Requirements

Sometimes the most challenging (or frustrating?) part of a testing project is figuring out what you are supposed to be testing. Organizations with formal software requirements are already working toward a healthy Software Development Lifecycle. Whether your requirements are available in a nice formal document or whether they are scribbled on the back of an envelope, T-WorMS can help tie your testing and test planning back to the requirements or specifications that were used to develop the software. T-WorMS also works in the case of a testing project for which there are no written requirements.

Whether your requirements are good, bad or non-existent, it is important that the test effort be able to track back to the features that exist in the software. Obviously it's easier to determine the features that should be there if you have requirements, but even if you don't, software has features. A feature can be thought of as a single piece of functionality. For example, a feature of a word processing application is the ability to Save a File. Within the feature of Save a File, you could create Test Cases to test if there are security restrictions (saving to an unauthorized location), load/stress issues (size constraints), performance issues and you would also test to be sure that the feature actually worked (functionality testing). Under the feature Save a File, a number of Test Cases would exist to thoroughly test if the feature behaves correctly and handles error conditions correctly.

All software can be divided into features. Embedded software has a set number of inputs and outputs that it is expected to handle. Those can be considered "features". Networking software similarly has conditions with which it must work - those also are features. Similarly, hardware has features. With or without the requirements, you can determine the features of the software by determining what it can do and thinking about what it should and shouldn't do. Conversely, features could also be used to represent testing areas, such as functional tests, usability tests, etc. Features are a way of categorizing the testing. Use them anyway you want, just be consistent.

Step 3 - Developing the Feature Hierarchy

Features and sub-features form a hierarchical structure that allows the test organization to logically define the way that the testing of system requirements is to be done. Each feature can correspond to a related set of requirements (explicit or implicit) that are defined for the system; sub-features further clarify each requirement. Test Cases are then defined to test the features and sub-features and any inherent risk items associated with those features/sub-features.

Creating a Hierarchy

Features and Sub-Features are hierarchical in nature. A Feature is always at root level, describing a testable section of the system. Sub-Features below each Feature sub-divide the Feature into logical sub-divisions. This allows the QA Engineer to create a hierarchical structure of small, testable sub-divisions. By breaking each Feature into smaller sections, we are much more likely to capture all scenarios that need to be tested, and ensure that ownership of these sections is clearly defined.

A simple guideline when creating the Feature Hierarchy is to break each feature down to a level where Test Cases can easily be defined. For example, if you are told to test Microsoft Word, the task is overwhelming. If you are told to design Test Cases for all the file manipulation functions, it's still a big job. If you are told to develop Test Cases for the Save As functionality, the task is now defined well enough that you can retire to your corner and work out your test cases. If you think of the Feature Hierarchy as an outline of the product with the lowest levels of the outline defining testable components, you'll be well on your way. There is no limit to the depth of Sub-Features that can be defined for a Feature. The tree can be thought of as:

Feature

Sub-Feature

where each nested Sub-Feature is a further refinement of its parent. The lowest Sub-Feature houses the Test Cases that are specific for that Sub-Feature. Higher level tests, for example integration tests, could exist at a higher Sub-Feature level.

Organizing Test Cases

The Feature Hierarchy is used to organize the individual Test Cases. Most organizations have no trouble writing Test Cases, but once they are written it's difficult to assess gaps in the Test Case coverage. By assigning Test Cases to features and by cataloging them into the Feature Hierarchy, auditing the coverage becomes a simple task. Also, when new features are added to the product or added to later releases of the product, it's a simple matter to add Test Cases to the hierarchy to cover the new features.

Defining Features

Each Feature and Sub-Feature in T-WorMS has a number of fields to define it. Once a feature is defined, a Test Case can be associated with it. Test Cases can be added to a Feature or Sub-Feature at any level. A Feature can also be associated with a Feature Defect. The Feature Defect capability is used to set all the Test Cases associated with the feature to "Suspended". This is particularly useful when a Feature is not yet fully defined, is being significantly changed or is blocked because of a problem or incomplete definition. The Feature Defect capability makes it easy for the QA Engineer to develop tests before Features are fully defined or working. If you are working in an Agile or XP environment where the tests are defined before the coding commences, T-WorMS can track those tests and allow them to be enabled when the code is implemented by removing the Feature Defect associated with the Feature.

Versioning

Features, like Test Cases, are versioned. Anytime a Feature is edited, its version is incremented. All previous versions can be accessed from the Feature History table. This can provide valuable metrics regarding the frequency of changes to Features/Requirements and can be used to track effort associated with those changes.

Traceability

Traceability is an important component in any good SDLC. Traceability from requirements to defined Features to Test Cases to results is part of this critical aspect of a managed project. T-WorMS provides this traceability capability by allowing the user to reference the specification document (via file reference or link) and the spot within the document that pertains to the designated Feature/Sub-Feature. Once the Specification/Requirement is tied to the Feature, T-WorMS associates the Sub-Features to the parent Features, the Test Cases to the Sub-Features and the results (and data if DiRT is used) to the Test Case. Complete traceability is tracked and maintained by the system.

Step 4 - Evaluating and Assigning the Risks

Risk is a very real concept in the software testing world. It should be used to prioritize what is tested and the order in which it is tested. Every feature of the software has a risk associated with it. What happens if the feature doesn't work? What does the failure of this feature mean to the business? Could the failure of this feature have a technical impact on other features? Is this feature likely to fail because it has recently been re-written? All these risk factors need to be evaluated in order to determine the priority of the testing effort.

Gathering Risk Information

Risk cannot be determined strictly within the confines of the software testing and development organizations. Business risk can only be assessed by the business unit that has the information to determine the risk to the business. Technical risk can be determined by the technical team. Project risk can be determined by the Project Manager or the project team. Specific test case risk can be determined by the test team. It is important to include all interested and knowledgeable parties in the determination of the risk levels associated with each feature. Risk should be used as a scheduling tool. When time runs out, it is most important to have mitigated the risk for the highest priority items. If lower risk items have to go untested or receive only minimal testing, it is a project decision to incur that risk in favor of meeting a schedule.

Cutting Corners Carefully

Quality Assurance groups are often forced into cutting corners to meet schedule dates and to make up for schedule time lost in earlier efforts on the project. If the development effort takes longer than was scheduled, and the end date doesn't move, testing time will be squeezed. It is up to the project team to set the priorities for the testing. Once the priorities are set and agreed to, the test group will be cutting the corners selected by the entire team. This allows the schedule to be met with the least risk as indicated by those who set the risk levels. By getting risk level input from all knowledgeable parties, the risk vs. schedule decision can be an informed and calculated effort to safely meet the scheduled date with an acceptable level of risk. Risk determination is a project team responsibility, not a QA responsibility.

Risk should be assigned at the lowest Feature/Sub-Feature level that is meaningful. When Test Cases are later associated with the Sub-Features, they inherit the risk that was assigned to the Sub-Feature. The risk can be further refined for the specific test case by adding a Test Case Risk value. This risk number is then used to prioritize the tests and to evaluate the metrics to verify that the pre-established risk mitigation was met by the testing that was conducted.

Risk Categories and Levels

There are five levels of risk, low to critical, with critical being the highest or most risky. Each of these levels are assigned a numeric value from 1 to 5, 1 = low. There are four categories of risk supported in T-WorMS, three at the (Sub-)Feature level and one at the Test Case level:

Business Risk - What does it mean to the business if this Feature/Sub-Feature doesn't work correctly? How important is the Feature/Sub-Feature to the customer? This information should be supplied by a business analyst, customer representative or domain expert.

Technical Risk - What is the inherent risk of this item? Does it need more testing because it has recently changed or may adversely affect other components? Is the code unusually complex and therefore more error prone? This information should be supplied by the testing and development teams.

Project Risk - How important is this feature to the project as a whole? Is this the main feature of the project? This information is supplied by the Project Manager or project team.

Test Case Risk – Is this Test Case more important that the others for this Feature? If so, you can add the Test Case risk in the Test Definition which will prioritize this Test Case within the Feature.

The cumulative risk of a Feature/Sub-Feature is determined by multiplying the three risk values together resulting in a value between 1 (ho-hum, who cares) to 125 (Yikes!). This multiplication is done automatically by T-WorMS and is shown as the Feature Risk value for the Test Case.

When, Who and Why

Risk values should be gathered at the same time the Feature list is reviewed. This will ensure that all Features/Sub-Features are identified and that all risk values are accurately assigned. This requires a Project Team review meeting. The completed list of Features/Sub-Features with the risk values can be considered the Quality Risk Analysis. This is a critical step in determining the scope of the testing project and draws the Project Team into the hard tradeoff decisions of schedule vs. quality.

Step 5 - Creating Test Cases

A Test Case represents a single test that will be executed against the system to prove that a Feature or Sub-Feature is working correctly (or to mitigate the risk of its not working.) Each Test Case is "owned" by a Feature or Sub-Feature; this is represented in the hierarchical tree by placing each Test Case node directly under its owning (Sub-)Feature.

A Test Case should be created by a QA Engineer to mitigate risk. In doing an assessment of a (Sub-)Feature for the system, the QA Engineer looks for ways to verify proper execution as well as ways in which the (Sub-)Feature could fail (usually resulting in a cost to the organization.) Once these ways are identified, the Engineer creates tests that will exercise that particular part of the system and validate that the operation was correct. A portion of the testing cycle consists of running all these Test Cases; in general, the fewer Test Cases that fail, the closer the system is to being complete.

Independence

The Test Case is an important deliverable in any SDLC. Test Cases should be independent from each other. Independence allows Test Cases to be executed based on their priorities rather than due to interdependence from another Test Case that must be executed. Independence in design requires planning in order to minimize the effort to create and maintain the Test Cases. It is also important to keep the Test Cases independent from the data that will be used during execution. By maintaining separate data files (via DiRT or some other tool) for each Test Case, additional tests can be added easily by adding additional data files rather than adding Test Cases. This helps to keep the test library clean while allowing the data files to proliferate, setting up an easy path for future automation efforts.

T-WorMS allows the tester to add Test Cases to a project by associating them with a Feature/Sub-Feature. The Test Script itself (the detailed steps that are executed by the manual tester) can also be stored by T-WorMS.

Avoiding Redundancy

Test Cases are created in T-WorMS by defining Test Case information that is saved for that Test Case. Test Cases are often re-used within a system. This is sometimes done because identical functionality exists in several places within the product (like pressing the Cancel button). In this case, the identical test case should exist in several areas on the Feature list. If the copies are not maintained in the list, an inaccurate count will be made for the time it takes to truly test the system.

If there are 10 occurrences of the Cancel button and each are tested the same way, you will still have to run 10 Test Cases to validate the functionality in each place (assuming you can't guarantee the code is identical and is called in an identical fashion in each case). Maintaining 10 copies of the same Test Case is not efficient. T-WorMS provides the ability to maintain Test Cases with common elements to limit the maintenance effort while still providing accurate scheduling and execution information. If an exact copy of the Test Case is needed in the hierarchy, but maintaining multiple copies will be burdensome, one copy of the Test Case can be maintained and can be made a pre-requisite for the "placeholders" that will merely be stubs.

Similarly, the situation may occur when most of the Test Case is common, but there are a few unique features also (for example, on one screen, the Cancel should result in a special message box being displayed). T-WorMS also handles this case by supporting the concept of specific and common fields within Test Cases. You may also have the instance where you want to copy a Test Case, but don't want it to be related to the original Test Case. This capability is also available.

A key concept of the Test Case record is that it is split into two different parts: a common part that is shared or linked between all Test Cases that are linked together, and the specific part that is unique for this specific Test Case. For a Test Case that was created through the standard interface (or a deep copy creation), both parts are actually unique to the Test Case. The common part of the Test Case will only be shared when it is created via the linked copy process. This scheme allows a QA Engineer to set up Test Cases to minimize the amount of maintenance they take. By linked copying sets of Test Cases, changing a common field on one changes all of the linked copies. Both the specific and common parts of a Test Case are versioned.

Ordering and Prioritizing

There are two ways to affect the order in which a Test Case will be executed. Feature Risk for the Test Case is inherited from the (Sub-)Feature to which it is attached. This Risk level should be used to prioritize the Test Cases when time limits exist that will prohibit running all the Test Cases. Further risk classification can be made by using the Test Case Risk value to determine the relative importance of the Test Case for that (Sub-)Feature.

The second way to affect order is by indicating Relative Order for the Test Case. This field contains a value from 1 to 100, defaulting to 50. Relative order can be used to indicate tests that should be run before others. For example, you might want to run the tests that generate data before you run a report test. For automated Test Cases, this value will be used when scheduling the tests to be run. For manual Test Cases, this field is used as a guideline for the test executor.

Relative ordering is useful when using Pre-Requisite Test Cases. Assigning a lower relative order to the Pre-Requisite Test Case will cause it to be placed on the run list before the requiring Test Case. Pre-Requisites should be kept to a minimum because running a Test Case will also require running the Pre-Requisites. If the Pre-Requisite list gets long, prioritizing the Test Cases becomes meaningless. Test Cases defined as Pre-Requisites will be required to have run and passed if the Test Case is automated. If a Pre-Requisite Test Case has not been run, the Test Case having the Pre-Requisite will not be run and will be marked Blocked. If the Test Case is manual, the Pre-Requisite list is used as a guideline for the tester executing the Test Case. The Test Planner can check to be sure that all Pre-Requisite requirements are met and notify the user if there are any unfulfilled Pre-Requisites in the Planner. Although Pre-Requisite Test Cases do not directly affect ordering, they may affect the order in which the tester selects to execute the Test Cases.

One additional use of the Pre-Requisite capability is to help keep Test Cases independent. Test Cases often require the same setup steps, or require that you drive the software to a certain point before testing can begin (for example, you have to log on). These preliminary steps or setup steps can be captured into their own Test Cases. These preliminary Test Cases can then be declared as Pre-Requisites for a Test Case, giving maximum independence between Test Cases, providing easy maintenance, and still giving an accurate view of exactly what steps are needed to complete a given Test Case.

In addition to make the best use of these ordering features, there is also an ability to indicate how often a Test Case should be executed. The Execution Frequency feature allows the tester to indicate a value from 1 (run for every build) to 5 (run only for complete regression testing). This value is a guideline for the manual tester when creating Test Planners. The value is used by automation to run the requested level and everything lower automatically.

Status Settings

There are four status settings for Test Cases: Test is Runnable, Test is Suspended, Test Needs Maintenance and Test Needs Review. These are used to control and monitor the Test Cases in the system. One of the main problems with Test Case libraries is keeping the Test Cases up to date. This is particularly problematic in an environment where the software is changing rapidly.

The Test is Runnable flag indicates the Test Case is ready to go and should produce the expected results. Test Cases that are still under development can be created in T-WorMS and are not counted as runnable Test Cases until they are declared "Runnable". Once the status is set to Runnable, versioning of the Test Case is started. An organization may decide to sometimes remove the Runnable status to disable a Test Case for some reason (which must be indicated when that status is changed). A Test Case that is not Runnable will be excluded from the automation run list, but is eligible to be added to a Test Planner. At run time, the manual tester can check to see why the Test Case is not Runnable.

A Test Case is set to "Suspended" if a Defect is logged against the (Sub-)Feature chain it tests or against the Test Case itself. In some cases, an SUT Defect may be sufficiently catastrophic to warrant the suspension of the Test Case until the Defect is closed. Anytime there is an SUT Defect noted, the tester has the option to "Suspend" the Test Case that found it. Once all suspending Defects are closed, the flag is removed.

Test Needs Maintenance indicates that changes to the Test Case may be needed prior to its execution. This may be because the software it tests has changed or it may be due to changes needed in the Test Case itself. If the maintenance needs are such that the Test Case doesn't work correctly anymore, the Test Case should also be set to "Suspended" by creating a Defect against it.

Test Needs Review can be used for environments where there is a formal review process before Test Cases are put into the general test system. Reports are available to list Test Cases that are awaiting review. Each reviewer is notified that a review has been requested via their Tickler. As reviewers complete their review, they are removed from the list for the Test Case. When all reviews are completed, the owner of the Test Case is notified by Tickler and must accept the reviews before the Test Case review needed status is removed. The Test Case owner also has the option to terminate the outstanding reviews by accepting those that have been completed and deleting the rest. This eliminates the possibility of Test Cases being held up eternally because of irresponsible reviewers.

Other Test Case Information

In addition to the general fields, T-WorMS also tracks some additional data about a Test Case:

Execution mode - Designates if a Test Case is run manually, run via the automated process or is it currently manual but is a candidate for future automation. Only an Automator can set the execution mode to Automated.

Timing Estimates - This information includes setup time, execution time and teardown time. The estimates will be used during scheduling with the Test Planner. Comparison of the estimates with the actual results can be used to improve the accuracy of the estimate values.

Test Environment - Test Cases may be written for a specific environment (only Windows NT) or for a set of environments (all Windows platforms). It is important that this target environment be indicated to ensure that a test is not invalidated by being run in an unsupported environment. Environments can include specific hardware configurations, specific software configurations, specific software versions, or whatever makes sense to your organization.

Attached Files - Files from the repository can be attached to Test Cases. These files may contain data that will be used during the test, expected results, screen snapshots or whatever is useful to the QA Engineer for this particular Test Case. Test Cases can have any number of files attached to them and files can be attached to many Test Cases. Files are stored in the T-WorMS repository which has a rich set of features and provides versioning support. T-WorMS can download all attached files for a Test Case to the local workstation for both manual and automated tests.

Step 6 - Creating Test Data

The data that is fed into tests determines how effective and complete a Test Case will be when executed. Creating test data is often a casual practice where data is created on the fly and not recorded unless a bug is found. This can cause several problems. Test repeatability by different testers is often lost. If a tester is the only one working on a particular area of the software, that person may get in a rut and always use the same test data.

Keeping the Data

One good way to prevent the data rut is to create and maintain the test data separately from the test case. When the data can be viewed as sets of inputs, the QA Engineer will quickly realize that varying data sets are needed to thoroughly test the software. One data set may include all valid data. Others may incorporate invalid data for any or all fields. Some will want to investigate the boundary conditions. Others may use data combinations that will induce certain errors. By thinking of the data sets as a group used to thoroughly exercise a test case (also known as a data-driven test case), a more comprehensive set of data will be created. Once you've gone to all the trouble to create these data sets, you want to keep and re-use them.

Creating the Data

So how can you create this data? T-WorMS includes a component called DiRT - Data Repository for Testing. DiRT can be used to create sets of data (records) for a particular test case or set of test cases. To use DiRT, you initially define the data fields (including type, size, valid values, and other attributes for each) then populate those fields with sets of data to create data records. In an automation environment, the GUI map information should also be used to further define the fields for DiRT.

Once the fields are defined, you can create as many data sets as you want. When the tester is ready to execute a test case, the data is ready. The data can either be viewed via DiRT or printed out for reference as the Test Case is being executed. An automated test case can pick up the data directly from DiRT, via the T-WorMS automation functionality (TAC T-API), and use it to perform the automated test.

Storing and Versioning the Data

In addition to providing an easy interface to create and access the data, DiRT can also be used to version and store the data for audit or recording purposes. This can be particularly useful for comparison between versions or releases of the software (Did this fail before? Did we ever try to do this?). Storing the data can also be very valuable if you've been able to obtain actual customer data that you would like to check on every release. If you work in an environment where auditors review your test cases, you'll be able to show exactly what data was used with each of the test cases you ran.

The integrity of the test data determines the integrity of the test cases. Creating good test data is a significant investment. DiRT lets you create, maintain and re-use this investment.

Step 7 - Adding Automation

T-WorMS has been designed to more closely integrate an organization's automation efforts with its overall testing strategy. This integration begins with defining Test Cases, allowing QA Engineers to think more about what the test does and less about how it will be executed.

Automation Candidates

During the test definition process or at any time later, the QA Engineer may designate a Test Case as an automation candidate. This is an advisory designation only. All automation candidate Test Cases will be evaluated by the automation engineers; if a Test Case does not appear to be well suited to automation, the Automators can discuss those issues with the QA Engineers. Until such time as a Test Case is actually automated, T-WorMS will continue to treat it as a manual test. In all cases, the organization should automate only those cases where there is an economic justification, unless there are other, overriding concerns (e.g. safety.)

The first step for the Automator is to get the list of all the Test Cases that should or might be automated. This is done by using a variety of filters to get the list of work for the Automator. Once the likely Test Cases have been obtained, the Automator can start with the automation effort. Before jumping into what the Automator will do, it's worth reviewing a couple of automation concepts.

T-WorMS supports both the framework and data-driven architectures allowing an organization to use the automation techniques best suited for their needs. In the next major release, T-WorMS will additionally support keyword-driven automation. By allowing the co-existence of multiple automation architectures, the existing automation investment will be preserved.

The Framework Architecture

A Framework Architecture is designed to support large scale automation efforts. The heart of the framework architecture as implemented by T-WorMS is the script template. A script template is a file that is generated for each stand-alone Test Case that is to be automated. Each different automation tool has its own unique template type supplied by T-WorMS. The template contains functionality that performs the following tasks:

· Downloading all files that are attached to the Test Case

· Downloading DiRT data

· External logging, including the ability to capture the screen image at any time

· Pre-Requisite support; the ability to automatically defer the execution of a Test Case if it is dependent on Test Cases that did not run or failed

· Error handling to ensure that failed Test Cases do not stop the suite

· The ability to start and stop the system under test

· And more…

After T-WorMS generates a script template, the Automator will enter that template into the testing store and enter the actual automation code that executes the Test Case. Once the Test Case has been unit tested, the Automator uses T-WorMS functionality to set the Test Case executable. Now, when T-WorMS generates a run list this test may be set to run.

Data-Driven Concepts

The Data-Driven (DD) part of T-WorMS functionality refers to the built-in ability to define a single parent script that can execute one or more child scripts exercising multiple data-sets.

Often, much of the testing that is done on a system consists of doing the same steps many times, using different data each time. This topic is fully explained in the architecture document referenced above.

DD leverages written code to give much more testing for the dollar spent. A Test Case may be designated as run via DD technology. By assigning the Test Case both a parent script and a child script, we avoid the need to have a single dedicated automated script.

T-WorMS facilitates DD technology is several different ways, Including:

· The ability to generate both parent and child DD template scripts. A parent script is designed to call a child script (or multiple child scripts) multiple times. Each time the child is called, the data for the specific Test Case is downloaded to the workstation; while the child script is run over and over, the data used and the Test Case that is run is different each time.

· The ability to designate a Test Case as DD and allow selection of the parent and child script for that test.

· Integration with DiRT for storing the DD data.

· Special functionality to run DD tests (see Run Automation Concepts)

If a Test Case is not DD, it is considered a "singleton" in that it exists by itself and is self-contained.

OK, so how do you actually automate a Test Case?

Once you've selected the Test Cases you want to work with, then you have to do something with them. The actual automation of a Test Script requires the use of an automation tool. Any commercially available or home-brewed automation tool is suitable (as long as it can call .dlls). What T-WorMS supplies is the framework to handle the administrative details of the tests, to create the log files, to record run time information and to marry the automation information with the T-WorMS system. So, the actual automation is done by writing the automation script in the selected automation language. T-WorMS generates the wrapper for the script and the driver that will execute the wrapper.

The Automator informs T-WorMS about the status of automated tests through a variety of flags that are set for each Test Case. These flags inform the system of the progress of the automation effort which will culminate in an automated test that is declared Executable. The flags are designed to inform about the initial automation effort and to also inform the system and users about any maintenance effort that is needed. One of the big problems with test automation systems is maintaining the scripts, keeping track of what needs to be maintained and keeping a clean run list. T-WorMS does all the administrative work to keep this information readily available and current so the Automator can concentrate on the work of actually automating the Test Case.

Using Status Flags

The status flags and settings that are used by T-WorMS are explained below. Many of these flags allow the Automator to filter the Test Cases that are available for editing.

Test Data-Driven - When this flag is set to TRUE, the Automator must also designate the parent script for the DD tests, and the actual child script that will execute the test.

Automation Script Created - This flag shows that the Automation script template has physically been created by the Automator. If the Test Case is a singleton, it is created by clicking on the "Create Script Template" action button. If the Test Case is DD, then this flag may be toggled manually (once the DD parent and child scripts are created, there is no need to recreate them).

Automated Test Executable - This flag is the final flag that must be set to TRUE for the automated Test Case to be placed on the run list. This flag indicates that the Test Script has been completed and unit tested and is ready for execution. Essentially, this flag is to automation as the "Test is Runnable" flag is to manual testing.

Suspend Test (used in conjunction with the "Reason for Suspension" field) - This flag should be set to TRUE when the Test Case (manual or automated) should not be run. This may be due to a Test Case defect or Feature defect, or it may be toggled manually in the Define Automation screen. In some cases, an SUT Defect may be sufficiently catastrophic to warrant the suspension of the Test Case until the Defect is closed. Anytime there is an SUT Defect noted, the tester has the option to Suspend the Test Case that found it. When set to TRUE, this flag prevents the Test Case from being included in the automation run list.

Test Definition has Changed - This flag is used to notify the Automator that a change has been made to the Test Definition. This is informational only. The Automator should confirm that the automation is still OK to run. The flag is automatically reset when the Automator edits the Test Case or may be manually reset by the Automator if no changes are warranted.

Setting the DiRT record. This value defaults to 0 when a test is created; if DiRT data is to be passed into the Test Case, it is passed in by creating a DiRT record and setting the DiRT key value in this field.

Reason for Suspension: Anytime the Suspend flag is TRUE, there must be a reason for it. Likewise, when a Test Case defect or feature defect is opened for this Test Case, the reason will be placed here. It is possible to have several different reasons in this control.

Step 8 - Planning the Test Execution

Figuring out the requirements, identifying the features, assigning the risk, creating the Test Cases, determining automation requirements and creating the test data are all critical preparation steps. All this work leads us to planning the actual execution of the tests. If we've done our design right, we have good coverage of all the important risk areas. We have Test Cases that will mitigate all the identified risks of the project. We also have estimates for how long each Test Case will take, considering setup/teardown and execution time. What we need now is a way to fit all the desired Test Cases into the time we're allotted. Piece of cake, right?

Picking the Right Tests

So much for fantasy. The problem that we usually face at this point is too many tests and not enough time. This is why we went to all the trouble to prioritize our tests. To create the Test Planner, we select the Test Cases that we should execute and allocate time for any planned eXploratory testing. In the Test Planner window, the selected Test Cases are listed and the total time for all the selected Test Cases is shown. This allows the planner to add and subtract Test Cases to mitigate the required risk while still fitting the testing into the schedule. Mitigating the proper risk may require adding more time to the schedule. Reducing the schedule may require the project team to accept a higher level of risk. Either way, the tradeoff is clearly shown and the planner can add and remove Test Cases until the desired balance is reached.

The Test Planner function provides an easy interface to plan the testing. It also provides an easy to understand list of what is being tested and what is being excluded due to schedule or other restrictions. Test planning is a critical process and one in which the rest of the organization must be educated so they understand the demands and importance of testing to provide risk mitigation. By providing a clear and objective list of what "should" be tested and what actually "is" being tested, intelligent schedule decisions can be made by the team. The QA group will no longer be sent off with the instructions to "do the best you can". We can now provide the team with all the data needed to make a team decision regarding which corners will be cut while still providing the accepted level of risk mitigation to the project.

Tactical Planning

The Test Planner capability in T-WorMS is not intended to replace formal test planning that may be done within your organization. Formal test planning should include creating a strategic document (which can be stored in the T-WorMS repository) that defines the rules of the test game and the interactions with other groups. This test plan document should be used as a communication device to provide the framework (the who, what, when, how, where) of the test project.

In a well-controlled SDLC environment, the Test Plan itself can be written as soon as the project requirements are defined. Since it's a strategic document, the details that will be identified later in the project should not substantially change the plan itself. The T-WorMS Test Planner capability provides the ability to do the tactical planning - laying out exactly which tests and eXploratory sessions will be run when and by whom. The reporting capability of Test Planner provides a view of the tests and sessions that have been executed for that Test Planner. These results can be compared to the expected schedule to ascertain if the project is on schedule or not early enough to take corrective action.

When a Planner is created, it is defined in terms of the Project, Release, and QA Engineer. Optionally, Cycle/Build and Environment can also be specified for the tests. If the optional values are not supplied, the Planner will be valid for any Cycle/Build and/or Environment. A project may have many Test Planners if there are multiple cycles and multiple environments. The granularity of test planning that your organization needs depends on the complexity of the software, the size of the organization and the schedule. Test planning should be kept as lightweight as possible to avoid overwhelming maintenance effort, yet should contain enough detail to allow control of the project. The right balance has to be determined and standards established for your organization.

Once the Planner is created, it has to be populated with tests and pertinent eXploratory sessions. The available tests are those that meet the selection criteria specified in the filtering. The filtering option is powerful, allowing multiple selection criteria to be entered to refine the list of Test Cases as needed for the project. Test Cases are selected from the list of available tests and added to the plan. Test Cases can be added and removed to produce a Planner that mitigates the desired level of risk and fits the schedule criteria.

Step 9 - Determining Schedule Time

One of the hardest parts of coordinating and managing a test project is creating and monitoring the schedule. And that's in the good world! Often we are not given the option to create the schedule - we're given a project and an end date and we have to figure out what to do with it. It's ideal if you get to create your own schedule, but reality sometimes interferes with that ideal situation.

Making a Reasonable Schedule

Whether we're given a schedule or get to make one, the first step is to determine what we "should" test. Only by determining this can we make an accurate estimation of the total risk of the project and weigh that against the risk mitigation that's possible for the given schedule. One of the problems in the software testing industry is the tendency to do the best we can in the time we have. This is done by cutting corners, minimizing testing is some areas and making decisions regarding acceptable risk to the organization. While this is a commendable effort, it is sometimes rewarded with outraged accusations that begin with "How could you miss that?!?". Sound familiar? The reason this happens is because we're trying to do the impossible - we're cutting corners to make the test effort fit in the schedule while the rest of the organization is assuming we're testing 100% of the software. It's time to start making this clear. The first step is to make a reasonable schedule.

What's a reasonable schedule? A schedule that provides enough time to complete the testing that "should" be done. To "complete" the testing means conducting both the necessary scripted testing and eXploratory testing as well as investigating any anomalies discovered. The capabilities that fall within the "should" list are all the features of the software that the user is likely to use, tested in the likely environments. The exact determination of the tests that fit this criteria depend on your industry and your application. Some industries demand that the software be tested for the "could" environment, which is everything that could possibly be done to and with the software in every possible configuration. If you are working in a "could" environment, your list of Test Cases will obviously be longer. Once we've figured out our environment, then we go on to looking at the Test Cases we have.

Test Case development is often an on-going effort. An organization usually only writes the Test Cases it intends to execute. This may not be anywhere near covering 100% of the functionality of the software. A determination of the delta between what is written, what should be written and what could be written, needs to be determined if any coverage assessment is going to be possible. Just because you have 100 Test Cases and you ran them all doesn't mean you got 100% coverage. What if you needed 1000 Test Cases to meet the "should" conditions and 5000 to meet the "could" conditions? Your statistics will be highly skewed if you are reporting you've completely 100% testing. You haven't. You've completed 10% testing. And this may be the right thing to do because 98% of the bugs may lie in the areas covered by the 100 Test Cases. The point here is that you need to create a schedule that provides an accurate view of what you "should" be doing versus what you are doing: the "must" cases.

How do you make a schedule with T-WorMS? Use the Test Planner function. This allows you to include the tests you "must" run and compare that to the list of what you "should" be running if you had enough time and money. You can even create a "could" list for comparison purposes. In some industries, the "must" and the "could" lists are the same because of the criticality of the software. Your "must" versus "should" or "could" lists will give the project team an accurate view of the test coverage and the risk that is incurred by the current schedule. Your list may show that your testing is mitigating all high level risk items and so is sufficient. Great! Now you just have to monitor how many bugs are discovered by the field (or whoever is downstream from you) to determine if you are providing sufficient test coverage. If your list shows that your risk mitigation will not meet the organization's standards, it's time to take the issue to the project team and decide if you should lengthen the schedule, reduce project scope or accept the higher level of risk.

Battling Schedule Fiction

The validity of the schedule information depends on the accuracy of the individual Test Case expectation information. Each Test Case should be defined with its expected execution time and its expected setup/teardown time. By picking the Test Cases via the Test Planner feature, you can determine which cases can be run within the available time based on the expected duration of each test case (setup time + execution time + teardown time).

In addition to the timing on the individual Test Cases, there is also an ability to assign an expected bug yield to a Test Plan. The expected bug yield is the number of bugs, on average, the Test Cases in this plan would be expected to find. This is usually derived by using historical bug yield information. This number can be assigned to a Test Planner by hand or it can be assigned to the Test Planner based on the Expected Bug Yield for the Project/Release. If automatic assignment is selected, T-WorMS takes the total number of expected bugs and divides it equally among all the Test Planners for the Project/Release based on the number of Test Cases in each Planner. For example, if a normal test cycle for a Project/Release finds 200 bugs and you have the following Test Planners, you would see the Expected Bug Yield automatically distributed as shown:

Test Planner 1 (4 cases) = 8 bugs

Test Planner 2 (6 cases) = 12 bugs

Test Planner 3 (5 cases) = 10 bugs

Test Planner 4 (8 cases) = 16 bugs

Test Planner 5 (2 cases) = 4 bugs

Test Planner 6 (10 cases) = 20 bugs

Test Planner 7 (15 cases) = 30 bugs

Test Planner 8 (25 cases) = 50 bugs

Test Planner 9 (10 cases) = 20 bugs

Test Planner 10 (15 cases) = 30 bugs

If you don't expect an even distribution of bugs, you might choose to enter the values yourself to indicate the expected yield from each Test Planner. This is particularly useful when some features are more complex, more unstable or just traditionally more bug infested.

By recording the Expected Bug Yield, you can provide a more accurate schedule. A good part of the time spent testing is actually spent researching bugs, not executing tests. This bug investigation time is based on an average number (say five hours) that an "average" QA Engineer will spend investigating an "average" bug. Obviously a typo would take considerably less time. A nasty, intermittent problem might take much more. This average bug investigation time is entered via the Admin functions. The bug investigation time is multiplied by the expected bug yield to produce the investigation time that is added to the execution time for a Test Planner. This gives a more accurate number for how long it really will take to run a Test Planner.

This will give you a very accurate schedule (even more so the next time when the expected values have been adjusted to reflect reality). It's a good idea to make at least two Test Planners for the project, one showing the "should" list and one showing the "must" list and the difference in the risk mitigation between the two.

Monitoring the Plan

Planning is one important aspect of scheduling. The second aspect is monitoring the on-going testing to see if you are staying on schedule. Again, the Test Planner function provides the list of the Test Cases to be run and how long they should take. There is a reporting capability associated with the Test Planner that allows you to see which Test Cases have been run. This affords a real time view of the testing and provides current information on the execution of the Test Planner.

Recycling Information

The third step to good scheduling is to feed the information learned back into the system. The expected time values of each Test Case should be evaluated against the actual values and the expectations should be updated to be more accurate. The accuracy of the expected values will determine the accuracy of the overall schedule. T-WorMS provides this capability by allowing you to view all actual time metrics for each Test Case.

While making a good schedule won't ensure you have adequate time to execute it, it does set the guidelines for the project team and provides objective information for the basis of discussion when the corners do need to be cut. Our goal is to provide accurate information to the decision makers. A valid, objective, understandable schedule is a big part of that vital information.

Step 10 - Executing Tests

Finally! We've now figured out the features, completed our Quality Risk Analysis, created Test Cases and Test Data, automated what we could and planned our attack. Now all we have to do is run the Test Cases. If we've done all the planning right, this is the easiest step because all the preparatory work has been done. We know we are running the right Test Cases at the right time to get the maximum coverage while staying on schedule.

MAT – Manual Assistant for Testing

First let's look at the manual tests. Automation specifics are discussed below. The manual tests are run by the manual tester on the machine of choice. In order to be sure the tests are valid, the tester must verify the Test Data to be used (either stored in DiRT or stored outside the T-WorMS system), verify the environment specified in the Test Case and complete any setup instructions including running any indicated Pre-Requisite cases. The tester must obtain the Test Steps for the Test Case, either by requesting a report or viewing the steps from MAT. If the Test Steps are in an external file held in the repository, MAT can download those files to the local machine for viewing. Once the QA Engineer is armed with the Setup, Test Steps and Test Data, the testing can begin.

It's time to meet MAT. MAT is another component of the T-WorMS system and is designed to provide test execution support for manual testing with minimal impact to the running system. MAT can be shut down when testing commences to minimize impact on the system under test and will be returned to the same state on startup. MAT can also assist with exploratory testing by providing an easy interface to enter test procedures and findings for the exploratory session. Exploratory session statistics are then tracked with regular test case results to provide a clear picture of all the testing that occurs in a test cycle. MAT also allows the QA Engineer to enter the test results and capture screen shots and output data as needed. The screen shots, or other test artifacts, are saved in a temporary directory until the QA Engineer indicates the test is complete. If the QA Engineer wishes to retain the test artifacts, the selected artifacts will be zipped up and put into a network directory. If the QA Engineer selects to discard the data, it is deleted.

The QA Engineer also has the opportunity when using DiRT to record the actual test data that was used for this execution of the Test Case. If requested, the data is versioned and stored within the DiRT repository where it can later be re-used for comparison purposes or audit requirements. If not, the data is retained but not versioned and can be overwritten by subsequent changes.

Automation Specifics

So what's different about the automated Test Cases? To the system, lots of things. To the tester who's running the tests, there are several noteworthy differences. First of all, an automated test cannot be run unless it has been set to "Executable". That means that first of all the Automator has to have automated the test and set its status to "Automated". This indicates that automation is complete and the test case has been unit tested by the Automator. The automation test case must also have the status of "Executable. These status flags are set by the Automator under the "Define Automation" tab after script creation or modification.

It's important to remember that once a Test Case has been Automated, it may occasionally have to be run manually because of a problem with the automation. If the Test Case is in your Planner, be sure to verify that it "Can be run". If it can't, MAT can tell the tester why. This may indicate that the test should be run manually for the current test cycle.

Pre-Requisites (PreReqs) are handled differently for automated Test Cases than for manual test cases. For manual test cases, the Pre-Requisite notation is there to help the tester when planning and executing tests. For the automated test cases, PreReq dependencies are enforced at run time. When a Test Case with PreReqs is started, the run-time system will go out and check the run records for this run for evidence that each PreReq test actually ran and passed. If one or more PreReqs are found to not have run, or to have failed, the Test Case is not allowed to run. Instead, it is logged as "Blocked" and the missing PreReqs are noted in the log.

File Attachments

Attaching files to Test Cases was discussed in Step 5. One particular note here for automated test cases. For automation purposes, all attached files are automatically downloaded to the local workstation where the automation is running. This occurs just before the Test Case starts to run. This eliminates any chance that the data will not be available at the time the test is run. If files are needed for manual testing, they can be downloaded automatically by MAT.

Creating the Run List

When selecting the test cases to run, there are two main considerations: which tests can run at this time and which tests should run at this time. The larger the set of tests, the harder this problem becomes. T-WorMS is designed to help the QA Engineer determine which automated tests are available and which tests are not. The system also provides a list of the available tests "Potential Run List" from which the QA Engineer can select the Test Cases for this execution run.

In order to create the Potential Run List, T-WorMS filters through all of the Test Cases that are available and determines which ones may be run. It automatically filters out the Test Cases that are not available due to defect records being open against them. It filters out those Test Cases that are not capable of running because they are "Not Runnable", "Not Executable", or "Suspended". Then, there are optional filters that can be set based on the type of run that is planned. These include:

Test environment: the tests can be filtered to choose only those that are designed to run in a certain environment.
Test Owners: the tests available may be constrained by the name of the owner - this allows private runs of owned automation.
Automation Level: the tests may be filtered by the priority of their run. The automation level is a value that is defined for each test, indicating how often it needs to be run. From 1, meaning that the test should be run every build, to 5, meaning that the test needs only to be run once per cycle for full regression testing. Selecting a value adds a filter to the Test Cases to only populate the potential run list with Test Cases that are at that level or lower. For example, if the user decides to set the Auto Level to 3, then T-WorMS will only select tests with a value of 1, 2 or 3 for the potential run list.
Filter by sub-string: Finally, the QA Engineer may decide to select only those Test Cases that have a specific sub-string in their name. This is useful only if there is a strong naming convention in the test process for the organization. By adding meaningful prefixes and/or suffixes to test names, the person running the automation may select different sets of tests for the potential run list.

There are several other ways to generate a Potential Run List for automation. These ways include using a batch file with the names of the tests to be run or creating the list based on date last changed.

Once the Potential Run List has been generated, it is time to generate the Actual Run List. This is the list of all Test Cases derived from the Potential Run List that have been selected to run in this suite. A Test Case is selected to run from the potential list by checking the checkbox in front of the Test Case. All of the tests may be selected using the context menu of the Automation Tree. There is also an option to select the Actual Run List from the set of previously failed automation tests. This is particularly useful when automation is being developed and tested.

Once the tests to be run are selected, the actual run list is generated by clicking on the "Generate Run List" action button. The actual run list contains the names of the Singleton Test Cases that are selected to run, and also the names of the parent tests for all selected Data-Driven Test Cases.

There are several other ways to create an actual run list. These include requesting to run all tests that were failed or blocked in a previous automation run, selecting a specific pre-built automation set, and selecting all automated test cases that are in a particular Test Planner.

The actual run list will be used to generate all the files needed on the workstation where the test will be run. This includes all the DD tests that are chosen to run under the listed parent. Order of execution may be selected by the user at this time. You may select to run all the DD Test Cases at the beginning or end of the suite. Within that choice, you might also request that the test cases be run in relative order (determined by the relative order value of 0 - 100 set when the test case was defined) or in Sub-Feature order which sorts the test cases in the order in which they appear in the hierarchy.

This mechanism is designed to allow the person selecting the tests to simply pick the tests that they want to run. T-WorMS will handle setting up all of the files and artifacts needed to actually make the automation run.

In order to maximize the data we gather and the accuracy of the information about the run, there are several run settings that are used. The run initiator records the Release, Cycle/Build, and Execution Environment by using the drop down menus. This is critical information that will be associated with the run information for each Test Case executed in this automation run. The executor can also request that the run log be loaded or printed upon run completion and can request the log information to be recorded in verbose mode. There is also an option to set and record a timer for the run. This is used when delayed start is needed - for example, you want to run the tests after everyone has gone home and you don't particularly want to hang around and start them yourself.

Execution

When running the automation suite by starting it from T-WorMS, the automation tool must be launched via the command line (all commercial tools have this start option.) The Automator must teach T-WorMS how to set up the command line correctly by defining where the tool executable is located and entering the command line that will be used to start execution. This information is entered via a dialog box accessible from the Run Automation tab.

Once the setup in complete, the user pushes the run button and the automation suite is either executed immediately or will be executed at the designated time if the timer option is used. When the automation is launched, T-WorMS shuts itself down to preserve memory.

The automation driver that maintains the log also maintains a list of the files to be executed. When execution is completed successfully, the file is deleted from the list. In this way, if there is an interruption in the test, it can be restarted from where it left off without unnecessarily repeating tests that already passed.

Recording Test Results

One of the most important tasks in testing is to accurately enter the results of the tests. This doesn't just mean logging bugs, because if all we do is record bugs found, we never get credit for all the risk mitigation work we've done by running Test Cases that might not have found bugs - at least not this time. What we want to implement is a consistent method for reporting test results that doesn't require an extensive amount of effort on the part of the QA Engineer or the Automator. We also want a system that integrates the results of the manual and automated tests to provide a complete picture of the test status of the project. And, we want the data to be immediately available for reporting once it has been entered. Don't want much, do we? We need an accurate picture of the testing and we need to get it without burdening the testers with awkward reporting systems. And, because statistics have to be presented in a readable format, we need the ability to access the data to make pretty charts and not be locked into the reporting system available in the tool.

So, how can T-WorMS help us with this endeavor? T-WorMS provides all the capabilities discussed above. Let's take a closer look at the test results aspects of T-WoRMs.

There are four possible result states for a Test Case that was scheduled to be run:

Passed - the Test Case performed correctly, matching the expected results exactly

Failed - the Test Case was run, but failed to meet its expected results or generated an error during its run

Blocked - the Test Case was not run to completion, but the failure to complete was due to a problem external to the Test Case

Not Run - the Test Case was not even attempted. Test Cases that are in the Test Planner that do not have run results records will be designated as "Not Run" for metrics purposes.

In addition to the completion status, the results screen for manual tests also accepts the following actual information:

Set Up Time - the actual number of minutes it took to set up the environment so the test could be run

Execution Time - the actual number of minutes it took to run the Test Case.

Tear Down Time - the actual number of minutes that it took to restore the environment after the test was run

Defects Recorded - the reference number and description of any defects that were found in the SUT by running this Test Case

Actual Time for Defect Investigation – the actual number of minutes spent hunting down and isolating a Defect in the SUT

Execution Date - the date the test was run

Issues - this is a free-form field the tester can use to enter any information (including embedded bitmaps) that will help a viewer to understand what happened during the Test Case run

Comments - this free-form field allows the entry of any information in almost any format. This control should be used as directed the organization; one good use would be suggestions for improving the Test Case.

These values can then be compared with the estimated values to determine if there were unexpected schedule-impacting issues. The actual numbers can also be used to improve the accuracy of the estimations so the next schedule will be even more accurate. The maturity and accuracy of the data will evolve over time.

Each Test Case run record is unique based on Test Case/Release/Build/Environment. A Test Case should not be run more than one time covering exactly the same code in exactly the same environment unless there was an issue with it on the prior run. If this is the case, the existing run record should be edited rather than a new record being created. This makes metrics more accurate and helps to keep testing efficient. If it is more informative to create a new run record in the above situation, the metrics will look only at the latest run record for the Test Case/Release/Build/Environment. The internal process of the organization can determine the most efficient method.

Automation results can be viewed via the Test Evaluation tab which shows the automation log, the runtime statistics, the saved artifacts and any errors found in the automation. The Test Evaluation tab can be used by anyone to review the tests that ran and look at the data produced and captured. Results from automated Test Cases are entered automatically by the run-time system. All results, whether entered manually or via automation are used in metrics evaluation by T-WorMS and may be viewed from the Test Results screen of the Test Definition tab.

Defects

When a bug is found in the SUT by a Test Case, a defect should be logged in the organization's bug tracking system. In addition, the defect should be noted in T-WorMS so the metrics can be gathered for the bugs found by the testing. While logging a defect that is found during testing, the tester can request that the Test Case be Suspended.

Summary

The T-WorMS execution environment is designed to assist the QA Engineer while the actual testing is occurring and when it is concluded. The tester needs easy access to Test Case information and requirements and a simple interface for entering results and capturing artifacts. DiRT handles the test data creation and storage, MAT provides a lightweight interface and T-WorMS maintains the captured data. The easy to use, low overhead interface encourages the tester to enter timely and accurate test result information that feeds into the project metrics. Accurate metrics are only possible if test results are entered consistently and accurately. Since the metrics drive the reports that will be used to determine risk mitigation, test coverage and test effectiveness, results entering has to become a habit for each QA Engineer. Why test if no one can interpret the results and use those results to improve the quality of the product? T-WorMS provides an easy interface that supports consistent and accurate reporting of the test execution results.

Step 11 - Determining Risk Mitigation

There's no point in testing if there isn't a goal. Is the goal to find bugs? No, that's a side effect. The goal is to mitigate the risks to the organization that would occur if untested software were released. This is the value that quality assurance adds to the organization, the identification and mitigation of risks.

In order to determine risk, a quality risk analysis (QRA) must be conducted. This can be done at varying depths and gather varying amounts of information. The goal of the QRA is to determine the risks within the software and to assign a risk factor. Ideally this analysis is done at the same level of detail used to create Test Cases. Creating the Feature Hierarchy is the first step in creating the quality risk analysis. Meeting with the project team and assigning risk to each item in the feature hierarchy is the next step. Once the hierarchy is created, you have an analysis of the risk in the software. The Test Planner is used to create the risk mitigation plan by selecting the Test Cases that will be run, thus mitigating the risks covered by those Test Cases. Now it's time to determine the risk mitigation that was accomplished by the actual, successful execution of the Test Cases in the plan. This list can be determined by looking at the reporting function for Test Plans and viewing the run information for those Test Cases in the plan.

It is important to produce a list of the risk mitigation as well as the list of bugs found. Even if no bugs were found, significant effort went into running Test Cases. The result of running the Test Cases was the mitigation of the risks within those areas of the software. In order to provide an accurate picture of the testing effort and the contribution to the organization as a whole, both a list of found bugs and a list of mitigated risks must be produced. Probably just as interesting as risk mitigated is the risk that was not mitigated.

It's important to look at the Test Cases that were planned but not run and determine if the resulting outstanding risk is acceptable to the organization. Scheduled tests may not be run because of equipment availability, software availability, blocking problems and myriad other issues. Part of managing the test process is determining what was not tested as well as what was tested. This information can be fed back into the scheduling process for the next project. Untested areas are also excellent points to monitor to see if the resultant bugs that are found downstream from testing were consistent with the risk level assigned to those areas.

Step 12 - Reviewing the Results

No matter how great a job you do testing the software, there may still be some bugs lurking. How many times has a developer told you he just fixed the last bug in the software? You probably laughed. QA Engineers know that 100% testing is unlikely and may be infeasible. And, quite honestly, it may be unnecessary. What we want to do is test until we have mitigated enough risk that the remainder is acceptable to the project team.

In order to know we've reached this point, we must have created our hierarchy, assigned the risk ratings, designed and prioritized our Test Cases/Test Data and executed the planned tests. In this way, we will have mitigated the inherent risks in the software. So, we just have to do what we and the project team planned to do. But that's not all we have to do to finish our job. We also have to generate reports - progress and summary reports - throughout the project. We also have to look at the defects that we have found and see if we need to enhance, or in the case of exploratory sessions, create Test Cases to further examine bug conditions.

The best way to teach the organization about the importance of testing is to educate them about what testing really is. This means providing objective data that shows the tests that were conducted, the risks that were mitigated and the results of those tests. It also means providing data about the tests that were not conducted, due to time or quality issues. This data feeds into the project team's decision on the quality and shipment-readiness of the project.

Metrics are currently implemented on the basis of a particular release within the selected project. Within that release, data can be gathered based on cycle/build, by QA Engineer and by Environment. Within the selected categories, statistics are gathered based on the manual and automated tests that were planned, run, % of total, the results of the execution and the current status of the Test Cases (for example, number of Test Cases needing maintenance).

Realistically, you can't report data you don't have. Fortunately, if you've been using T-WorMS all along, you've got the data you need. Let's look at the report options. T-WorMS gathers a lot of information. It also provides the tools you need to filter through and organize the data so it is meaningful to your organization. In most cases, you can gather the information from T-WorMS and export it to Excel or some other tool of choice to format it and create pretty charts.