TAC
WorM (Workflow Management) Features
·
Administration Settings
o The administrator has a variety of configuration
options to support customized processes
§
Allow anyone to
delete Test Cases vs. allow only those with admin[1]
authority to delete them
§
Allow login with
no password vs. password required
§
Allow Feature
Tree restructuring through drag and drop of nodes (moving Sub-Features and Test
Cases to new parents)
§
Force changer of
Test Cases to enter the reason for changes vs. allow it but don’t force it.
§
Force users to
enter time metrics for running test cases vs. allow but don’t require it
·
Test Case setup
time
·
Test Case
execution time
·
Test Case tear
down time
·
Defect
investigation time
§
Encrypt
Repository files vs. no encryption
§
Allow limited
automatic login to TAC Manual Assistant for Testing (TAC MAT): when running a
test, allow the system to “remember” the user’s login for X hours.
§
Tickler default
settings when first installed (allows an organization to customize the way
Tickler works for all users.)
·
Show Test Cases
assigned for review by this user
·
Show Test Cases
owned by this user awaiting review
·
Show Test Cases
owned by this user awaiting maintenance
·
Show automated
test results owned by this user that are unhandled (i.e. no one has signed off
on Failed and/or blocked results.)
·
Enter number of
days backwards we should check for Unhandled automation results.
·
Hours between
Tickler checks (every time WorM is started, it checks to see if Tickler should
be run automatically. If there are X
hours since the last time it was run, it will be triggered.)
·
Turn automatic
execution of Tickler off / on.
§
Defect default
processing
·
Number of
expected defects in this release (optional)
·
Allocate defects
o By proportion: As Test Planner (TP) objects are
created, the number of defects they are ‘assigned’ to find varies
proportionally with the number of expected defects vs. the number of Test Cases
in the TP.
o Manually by the test manager, taking into account
risk areas, etc.
·
Estimated
Standard Investigation Time for defects (average number of hours and minutes
that it is expected to take to investigate / document / process each defect
(for the test team only.)
§
File Repository
Path (location on a server that the File Repository will be stored.
§
Artifact Save
Base Path: a location on the server to be used as the base directory for
storing zipped up artifact[2]
files for both automated and manually run Test Cases.
o
§
Access
§
SQL Server
o Multiple Projects / Applications are supported
§
There is
complete separation of name spaces
o Ownership can be transferred from one user to another
§
A single object
can be transferred
§
All owned
objects can be transferred
§
A user can be
deleted from the database
o Lists can be configured to simplify and customize
selections for users. Available lists:
§
Apps / Projects
§
Automation Sets
(groups of automated tests run as a single entity)
§
Supported
Automation Tools (commercial or home brew)
§
Releases (per
App / Project)
§
Builds [Cycles]
(per Release)
§
Test
Environments
§
File Categories
(for the File Repository cataloguing)
§
Data-Driven
script names (Parent and Child)
§
Test Types
(classification of different Test Cases)
§
User Names /
Roles (accessible only to those with Administrator privledges.)
·
Features /
Sub-Features
o T-WorMS' hierarchical Feature Tree allows an
organization to easily map requirements to the software features and
sub-features that represent them.
o The Feature Hierarchy is used to organize Test Cases
into a logical and maintainable order
o Features can be added and deleted as the project
evolves, allowing early start with no penalties for changes
o Features can be hot linked or reference linked back
to requirements documents to provide traceability
§
Direct
connection to Requirements Management tools may be supported in the future
through their APIs.
§
Controls for
both a Requirements Document and the location in the document are provided.
o Quantitative risk assignment is supported in three
dimensions
§
Business risk
§
Technical Risk
§
Project risk
o Feature history table shows evolution of feature. Each change to a Feature is tracked by
requiring (Sub-)Feature changer to enter the reason for the change.
o Each Feature / Sub-Feature is an object owned by the
person who created it (may be transferred to another user)
§
Assigning a
(Sub-)Feature to another person may be recursive (i.e. the ownership of all
sub-features and tests below the transferred one may also optionally be
transferred to the new owner if they are owned by the original person.)
o Feature defect mechanism allows the suspension of
associated Test Cases from running when a feature is broken, not available or
not ready for test
o Features may be deleted, but only if they are a leaf
node (i.e. no Sub-Features or Test Cases are directly owned by the
(Sub-)Feature.
o Test Case reports (in Excel) can be created for all
Test Cases directly owned by the (Sub-)Feature.
o All tests directly owned by a (Sub-)Feature can be
marked for review
§
May be
recursively applied to all Test Cases in the hierarchy below the selected
(Sub-)Feature.
§
Allows the
selection of reviewers and date for review (see description on Test Case review
below.)
o Owner of (Sub-)Feature and the user who last changed
the Feature are tracked.
o Feature Description (a required field) is a Rich Text
Edit that allows formatting the text and inclusion of images, objects, etc.)
o Future release will allow reversion to earlier
versions of the (Sub-)Feature
·
Test Cases
o Test cases are "assigned" to (Sub-)Features
o Risk factor is automatically inherited from owning
(Sub-)Feature
o Test information fields are highly configurable to
provide more or less data and control as needed
§
Includes several
Rich Text Edit fields to allow individualized formatting and inclusion of
images if desired
·
Detailed
Description (Common[3]
) [Optional]
·
Test Comments
(Common) [Optional]
·
Test Specific
(Specific) [Optional]
·
Expected Results
(Common) [Optional]
§
Standard Edit
fields
·
Estimated[4]
setup time for Test Case (Specific) [Optional]
·
Estimated tear
down time (Specific) [Optional]
·
Estimated
execution time (Specific) [Optional]
·
DiRT record[5]
associated with this Test Case (Specific) [Optional]
·
Relative Order
o An integer between 0..100
o Defaults to 50
o This field is used to indicate relative execution location
for the Test Case
o For automated Test Cases, in general, the smaller
this number, the earlier in the suite the Test Case will be run[6].
o For manual Test Cases, this order is merely advisory.
§
Drop-down List
boxes
·
Test Environment
(Specific) [Optional]
·
Test Type
(Specific) [Optional]
o Multiple modes for Test Cases are supported:
§
Manual
·
Execution is
expected to be performed by manual testers
§
Automation
Candidate
·
Execution is
eventually expected to be performed though automated means.
·
Until such time
as the Test Case is actually automated, it is treated identically to the manual
tests.
·
All automated
tests start out as Automation Candidate; a test owner may not directly change
the execution mode to Automated. Only a
user with Automator authority may change a Test Case from Automation Candidate
to Automated.
§
Automated
·
Execution will
be performed via some kind of automation tool rather than manually.
·
Automated test
cases are still owned by the manual tester; Automators do not take ownership
simply because they created an automated way to execute them.
o Flags are used to indicate Test Case status
throughout life cycle
§
Needs
Maintenance
·
Set by a QA
Engineer or Automator to indicate that the Test Case needs to have work
performed on it.
·
This does not
imply that the test case may not be run.
If the Test Case is broken to the extent that it cannot be run, it
should be either suspended or marked as NOT Runnable.
§
Needs Review
·
Indicates that
this Test Case has been assigned to one or more users for review.
·
A review must be
accepted by the owner of the test (or other authorized person) for the Needs
Review Flag to be toggled off.
§
Runnable[7]
·
This flag
indicates that the test case is ready to run.
·
If the Test Case
is manual or an Automation Candidate, when this flag is set to TRUE (checked),
the test will be reported in Test Planners as ready to execute (assuming that
it is not Suspended (see below.)
·
If the Test Case
is automated, it will automatically be available to be executed via automation
assuming that the Executable flag is also checked and the Test Case is not
Suspended.
·
When a Test Case
is created, this flag defaults to FALSE.
·
The user may
edit the Test Case any number of times without tracking changes as long as this
flag remains FALSE.
·
Once the Test
Case is ready for use, the Runnable flag should be set to TRUE.
·
Once this is
done, each time the Test Case is edited, the reason for change must be entered
and the Test Case is versioned on each change.
·
Once versioning
starts it does not stop, even if this flag is toggled back off.
·
This flag
generally should not be set to FALSE (unchecked) once it has been set to
TRUE.
o However, local processes may be designed to use this
flag to control execution. It is better
to use the Defect / Suspension mechanism to control the actual running of the
test.
·
If there are
problems with the Test Case, a Defect record should be opened against it (see
below.)
o If the Test Case should not be executed due to a
defect in the Test Case, the Defect Record should be set to Suspend the Test
Case.
§
Executable
·
This flag is
only used for Automated Test Cases and only the Automator may toggle it.
·
If this flag is
set to TRUE (checked) the Test Case is eligible for automated execution
assuming that it is also set to Runnable and is NOT Suspended.
·
This flag is
initialized to FALSE.
·
It should be
toggled to TRUE after a Test Case is fully automated and the automation has
been unit tested.
·
This flag
generally should not be set to FALSE after it has been set to TRUE.
o However, local processes may be designed to use this
flag to control execution. It is better
to use the Defect / Suspension mechanism to control the actual running of the
automated test.
§
Suspended
·
This flag is
automatically toggled from off to on based on the Defect mechanism.
·
Only the
Automator may manually change this flag in the Define Automation screen.
·
Creating a Test
Defect record against a Test Case does not automatically suspend the test. There is a setting in the Defect record to
indicate whether the Test Case should be suspended or not.
·
Creating a
Feature Defect record against any (Sub-)Feature in the hierarchy above a Test
Case may suspend that Test Case.
o Creating a (Sub-)Feature Defect Record for the
(Sub-)Feature that directly owns the Test Case will always cause it to be
Suspended.
·
Closing a Defect
Record (either Test or Feature) does not necessarily un-suspend the Test Case.
o The Test Case suspension remains until such time as
ALL defect records [that called for Test Case suspension against it] are
closed.
o Tests may have files attached to them (automatically
made available at run time from the File Repository)
§
For tests that
are automated, the run-time system will automatically download the attached
files to the local workstation before the test runs.
§
For manual and
automation candidate Test Cases, the MAT tool will automatically download all
attached files when the test is started.
o A tabular manual script can be stored as part of the
test object
§
A script is a
three column table
·
Test Step
·
Data to use
·
Expected result
§
Scripts are versioned
each time they are changed.
§
In the next
version of WorM, a user will be able to revert the script back to an earlier
version.
§
A script may be
printed out in tabular form.
o Test history table shows all changes made to the Test
Case
§
After the test
is marked Runnable, each change to the Test Case cause it to be versioned.
§
In the next
version of WorM, a user will be empowered to revert a Test Case back to an
earlier version.
o Integrated Test Case review field makes good process
easy
§
The owner of a
Test Case or any authorized user (Test Lead, Manager, etc.) may assign one or
more people to review the Test Case.
§
Each person who
is assigned to the review gets notification in their Tickler report of their
assignment.
§
A specific due
date is assigned for the review.
§
As the Test Case
is reviewed, the user adds comments to a running history showing all past
review comments with timestamps.
§
After completing
the review, the reviewer's name is automatically removed from the Review list.
§
When the last
reviewer name is removed from the list (i.e. the last review has been
performed), a note indicating the reviews are done is placed in the Tickler
report of the Test Case owner.
§
The owner (or
authorized person) may accept the existing reviews and remove the Test Case
from review at any time.
§
New reviewers
may be assigned at any time.
o Pre-requisite tests can be identified to ensure that
a Test Case is only executed when all of its pre-requisite tests have been
passed.
§
For automation,
pre-requisites are enforced.
·
If an automated
test has defined pre-requisites, the run-time system checks to make sure that
each and every one of the listed pre-requisite Test Cases both has run and has
passed before allowing the Test Case to run.
·
If one or more
pre-requisite tests did not run or did not pass, the Test Case will not be
allowed to be run. Instead, it will be
marked as blocked and logged as such.
§
For manual Test
Cases, pre-requisites are only advisory.
§
The interface
makes pre-requisite Test Case selection easy.
·
A warning is
given at selection time if the relative execution order is reversed (i.e. the
pre-requisite test has a larger relative order than the Test Case being
edited.)
o Execution frequency guidelines can be set
§
Each Test Case
is set to have an execution frequency between 0 (execute every build) and 4
(run only when running Full Regression.)
§
For automation,
the execution frequency is used as a filter.
·
All automated
Test Cases that are theoretically available for running are filtered to select
only those Test Cases that are included in the specified execution frequency.
·
This scheme
allows a user to slice and dice the entire automation test base to select tests
to run.
o The owner of a Test Case can recommend that it be run
in one or more Automation Sets.
§
An Auto Set is a
group of automated Test Cases that is run as a single entity.
§
The Automator
actually selects which Test Cases go in which Auto Sets.
§
The
recommendation of a test owner is advisory only, as there may be other criteria
known to the Automator in actually selecting the tests.
o Reports
§
A report may be
created to show all of the Run Results ever created for this Test Case (both
automated and manual.)
§
A button is
available to see the time metrics recorded for all manual runs of a Test
Case. Pressing the button generates this
report, allowing the user to refine the times to more closely resemble reality.
§
A report (in
Excel) can be generated showing all fields in the Test Case record.
§
See the main
Report section below for a complete list of all the T-WorMS reports.
o DiRT is integrated with WorM to collect, store and
track Test Data.
§
A single DiRT
record from a Virtual Table may be attached to a Test Case.
§
Because a DiRT
record may contain other DiRT records to any desired recursive level, an almost
infinite amount of data may be attached to any Test Case as a single reference.
§
There is a
button in the Test Case definition screen to automatically open up the DiRT
tool (if not already open) and bring up the attached record in Edit mode.
§
There is also a
button to bring up the record in View only mode.
§
See DiRT section
for more information.
o Ownership of a Test Case may be transferred to
another user.
o The Test Owner and Last Changer are both listed in
the interface.
o A Test Case can be deleted under certain
circumstances:
§
If a user has
Admin permissions
§
If the Admin
allows, a QA Engineer may delete Test Cases up until the time they are marked
Runnable.
§
When a Test Case
is deleted, it is placed in an archive.
·
In the next
version of WorM, a user will be able to un-archive a Test Case.
o From a Test Case, a user may elect to go to the
Automation Test Evaluation screen.
Pressing this button causes all automated run results for the Test Case
to be loaded into the Evaluation Tree and highlights the first one.
o A button is available and enabled when the Test Case
is Linked to other Test Cases. Pressing
the button brings up a report that shows all Test Cases that are linked to it.
·
Feature Tree
o In the Test Definition screen, this tree view control
contains the hierarchical structure containing all of the project Features,
Sub-Features, and Test Cases that are defined in the project (subject to
filtering.)
o A filter dialog may be invoked to filter the test
cases.
§
(Sub-)Features
are not filtered. Instead, when Test
Cases are filtered, all Features and Sub-Features that own no Test Cases are
removed from the tree automatically.
§
Filtering may be
done by creating a SQL predicate in the Filter dialog.
o Drag and Drop capabilities
§
A number of
actions may be performed in the tree via Drag and Drop (these may also be
performed via right mouse click menus)
·
A Test case or
branch starting from any Sub-Feature may be moved to another owning
(Sub-)Feature by drag and drop with no keyboard key pressed.
·
A Test Case or
branch starting from any Sub-Feature may be link copied to any other
Sub-Feature by Drag and Drop with the Ctrl key pressed.
o Each Sub-Feature is created new in the new position.
o Each test case will be link copied (the Specific
portion of the Test Case is created new, while the Common portion of the test
is merely shared.)
o The original test case is unchanged (other than a
link to it being added)
·
A Test Case or
branch starting from any Sub-Feature may be cloned to another Sub-Feature. A clone means that an entire new Test Case is
created for each one that is moved.
There is no linking involved.
This is done by Drag and Dropping the branch with the Shift key pressed.
·
A Test Case may
be moved to a different position under its parent.
o This is only available when the tree is not sorted
alphabetically.
o The tree has a right mouse menu, containing items to
control the following:
§
The tree may be
ordered alphabetically, or it may be ordered in the same order it was created
in.
§
The tree may be
expanded or collapsed.
§
The tree may be
refreshed (reloaded).
§
An individual
branch of the tree may be expanded.
§
A user may
search for a test case by
·
Specific Key
value
·
Sub-string in
the Test Case name
§
A user may
duplicate an existing Test Case
·
A completely new
Test Case is created (no linking is done)
·
The new Test
Case is placed as the last test belonging to the Sub-Feature if the tree is not
sorted alphabetically.
·
The name of the
new Test Case is created by appending [Dup] to the original name.
·
Test Defects
o WorM contains a complete defect tracking mechanism to
manage defects against the Test Case or Feature objects.
o When a defect record is opened against a Test Case,
the user can Suspend the Test Case from running (either automated or manual.)
§
Any number of
defect records may be opened against an individual Test Case.
§
If any of the
defect records are in the open state and set to suspend the Test Case, then the
Test Case will be suspended.
§
Closing a defect
record causes the system to re-evaluate whether it should be suspended or not.
§
For automation,
suspending a Test Case precludes it from being run, even if all other
conditions are such that it should be run.
§
For manual
tests, suspending a test does not remove it from the Test Planner; however,
when a Test Planner is opened in MAT, it is clearly marked as being
suspended. The reason for the suspension
may be called up in a report; the user may then run the test anyway if needed.
§
When creating a
defect record and suspending a Test Case, the user may elect to suspend all
Test Cases linked to the base Test Case if desired.
o This defect tool is not meant to be used for defects
found in the System Under Test (SUT.)
Instead, it is used to manage execution, maintenance, and metrics for
the testing scaffold.
·
Test Planner
o Each tester can create test planners for any portion
of their work.
o Each test plan has four specifications
§
Tester name
§
Release
§
Build / Cycle
(optional)
§
Environment
being tested (optional)
o Test Cases are added to the test plan from a
hierarchical tree
§
Initially, the
tree contains all test cases owned by the user.
·
It may be
filtered using the same mechanism as the main tree
§
Any test cases
may be added to the Test Planner, not just those owned by the user. This allows a Test Lead or Manager to add or
remove Test Cases from an individual Test Planner to do load balancing.
o The Test Planner Test Cases are rendered in two
different tracking controls.
§
A hierarchical
tree, with the Feature / Sub-Feature hierarchy
§
A List view in
tabular form, showing just the Test Cases
§
Any action
(addition, removal, etc.) from one control also affects the other control.
o Test Cases may be removed from the Test Planner.
§
Removing a Test
Case does a provisional removal of the Test Case
·
It is marked for
removal, but not removed until the Test Planner is saved.
·
Its estimated
time values (see below) are removed from the aggregate to show what the
estimated time will be if they are actually removed.
·
This allows
“What If” estimates to be made allowing the Test Planner to be tailored for
actual conditions.
o Test Planner keeps a running total of “expected” time
values of Test Cases in the plan, allowing each user to tailor their plan for
available time.
§
Expected values
include
·
Setup time
·
Teardown time
·
Execution time
o Test planners can be sorted
§
By risk value,
so most important tests are prioritized
§
By Common or
Specific Key values
§
By Test Case
name
o eXploratory Testing sessions may be added to or
removed from the Test Planner (see eXploratory Testing below.)
§
The allocated
time for the eXploratory session is added to the Test Planner to help calculate
the total testing time the Test Planner represents.
o A button may be pressed to check for Test Case
Pre-requisites.
§
Any Test Cases
that have pre-requisite Test Cases that are not in the Test Planner are
flagged, ensuring all dependencies of the Test Cases are taken into account.
o A button can be pressed to jump to the Test
Definition for any Test Case in the Test Planner.
o Ownership of a Test Planner can be transferred.
·
Test Results
o Simple interface exists for recording test results
for manual testing.
o Automated as well as manual testing run results
records are available to be viewed in this interface.
o A Test Case may be marked as Passed, Failed, Blocked,
or Not Run.
o Each Test Case run can be categorized by:
§
Release
§
Cycle / Build
§
Environment
§
Date of
execution
o Tester is provided with simple interface to enter
actual timing metrics.
§
Setup time
§
Execution time
§
Tear down time
o Actual test results can be compared to expected test
results.
o Free-form Issues and Comments fields allow capturing
any test anomalies or issues.
§
Each control is
a Rich Text Edit allowing formatting and inclusion of images if desired
o Defect reference is provided to link defects found
and recorded in the central bug tracking system to the T-WorMS Test Case
§
Metric
calculations can be done if these are utilized, but they are optional
·
Amount of time
spent discovering, refining, and documenting the defect can be tracked
directly.
§
Defects may be
added or edited.
§
In the future,
may link to specific defect tracking tools if useful.
o Test Artifacts Files may be stored using this
interface.
§
When reviewing a
test results record, the artifact file may be unzipped directly using WinZip
(if it is on the workstation) or it may be downloaded to the Temporary Manual
Directory and unzipped automatically (not using WinZip.)
·
Tickler Report
o User may receive an automatically generated report of
task information including:
§
Tests owned by
the logged in user where review is complete but not yet accepted.
§
Owned tests
still waiting for review
§
Tests assigned
for review to the logged in user
§
Owned tests
needing maintenance
§
Unhandled[8]
automated test run results
·
Any Test Cases
that failed or were blocked in an automated test run within the last X days
o The report is configurable by user
§
Individual
sections may be switched on or off in the Options dialog (See below)
o This report is automatically generated by WorM on
startup.
§
It may be set to
only start up every so many X hours, so it does not pop up every time WorM is
started.
§
It may also
manually be generated by clicking on the menu or toolbar button.
·
File Repository
o Protected storage of all test deliverable files is
provided
o All files are kept in industry standard Zip files on
the server
o Archives may be password protected if desired.
o All file actions in Repository are logged.
o Change control and serialization is enforced
§
A file must be
checked out to modify it
·
When it is
checked out, no one else may check it out
·
It may be
downloaded for Read only use while it is checked out
§
Each
modification versions the file, with all versions of the file saved in the
archive.
·
In the next
version, a user will be able access previous versions of the file.
§
Each time a file
is checked in, the user must give a reason for the change. These time stamped reasons are kept in a
running list.
§
A comment may be
entered for any file without checking it out.
§
The file
description may be edited without checking it out
§
When a file is
checked out, anyone who clicks on that file can see who has it checked out.
o Files are displayed using configurable categories.
o Files may be added to the Repository in batch mode:
however, each file that is entered must have a description given for that file.
o Files may be shared by any number of Test Cases.
§
Any number of
files may be attached to any Test Case
§
A file may be
attached to the Linked portion of a Test Case, which de facto attaches it to
each different Linked Test Case
§
A file may be
attached to the Specific portion of the Test Case, making it private from any
Linked siblings.
o A file (and its archive) may be deleted from the
Repository only if it is not attached to any Test Cases. As soon as it is attached, it may not be
deleted.
o A file may be checked out by any user. If that user fails to check the file back in,
the file may be forcibly checked back in (essentially, this severs the tie to
the checked out file, and makes the last version active again.)
o A file may be checked in by the person who has it
checked out.
§
A reason for the
change must be entered at this time
o A file may be replaced by another file
§
This has the
same effect of checking in a new version of the file without first checking it
out. The file is versioned, the version
number is incremented, and the new version is stored. A reason for change must be made at this
time.
o A file may be downloaded to the local workstation without
checking it out.
§
Obviously, any
changes to this file will never be reflected in the Repository
§
If requested,
the file may be opened on the local workstation automatically.
o A report may be generated showing all of the Test
Cases that any individual file is attached to.
o When an automated Test Case is run, all files
attached to the test case are downloaded automatically to the local workstation
on which the automation is running.
o When the MAT starts a manual Test Case, it
automatically downloads the files to the local workstation.
·
Reports Screens
o A variety of customizable reports are supplied for
all facets of the testing process
o The data can be exported to a tool like Excel to
customize the look and feel
·
Metrics
o An authorized user can get a real-time snap shot of
the project at any time
o Separate metrics are gathered for
§
Test Plan
execution
·
Manual
·
Automated
§
Entire Project
·
Manual
·
Automated
o Metrics are gathered to assess the state of the
project
§
How many tests
are awaiting review
§
How many need
maintenance
§
How many are
affected by defects (Feature and Test)
·
User Options
o Show warning / Nag messages or don’t
o Script Report Type: either Excel or Text
o When testing a command line object WorM can try to
automatically open up the result file in Notepad if requested.
o Tickler options as enumerated above.
·
eXploratory
Sessions
o An eXploratory Session is a planned chunk of
unscripted testing.
§
WorM helps an
organization be more structured while performing unscripted testing.
o An eXploratory session is categorized according to
the following criteria:
§
QA Enginner
§
Release
§
Build / Cycle
(Optional)
§
Environment
(Optional)
o These criteria correspond to the categorizations of
the Test Planner.
§
An eXploratory
Session is designed to be imported into a Test Planner. This allows the tester to plan up front for
how much unscripted testing is going to be done.
o Each session has an estimated amount of time
assigned, based on hours and minutes.
§
Actual time
spent inside a session is recorded through MAT, or it may be entered through
this interface.
o Each session may have a symbolic name assigned.
o Each session has a Charter.
§
The Charter is a
textual description of what the session is supposed to be testing.
§
Usually this
description indicates
·
General areas to
be tested
·
Methodology to
be used
·
Test data
specifics, if applicable
o Each session has a Notes field.
§
This field is a
Rich Text Edit that gives a persistent area to put formatted notes during the
actual testing.
o Each session has a reference to the artifacts file
that is zipped up and stored as described above in Run Results.
o SUT Defects may be entered in this screen and
attached to any session. The interface
is similar to the Run Results defect entry screen.
§
By tracking the
defects that are actually found during an eXploratory Session, we can
·
Track the
usefulness of the session though metrics.
·
Determine which
of the actions taken during the eXploratory session should be scripted into
physical Test Cases for future releases.
o The real value of defining an eXploratory session is
using it in MAT. A full description of
how it is used there is found below.
TAC
MAT (Manual Assistant for Testing) Features
DiRT
(Data Repository for Testing) Features
Automation
and TAC T-API (Test Automation Programming Interface) Features
·
Define
Automation Screen
o Automator interaction with manual testers is
facilitated to simplify process
o Automatic creation of script templates for any
automation tool for test cases is provided
o Simplified creation of both singleton and data-driven
(DD) automated testing is supported
o Tests can be assigned to Automation Sets based on
organizational needs and tester recommendations
o A single point of control exists for removing /
reading test cases from active status
o DiRT is fully integrated to simplify data access and capture
for automation
o Running scripts have full access to any command line
functionality during run
·
Automation Integration
o TAC T-API (Test & Automation Consulting LLC Test
– Automation Programming Interface.) is used to supply integration with
commercial tools
§
The software
consists of sets of DLLs, database tables, Excel spread-sheets, native
automation tool code that has been developed over the last 10 years.
§
The software has
been customized for most commercial automation tools
§
Features have
been included to solve weaknesses of each tool
o A sophisticated external logging system is included
o Screen and window snap-shots are available on demand
and may be saved off to use as reference images
o The tool automatically enforces test pre-requisites
and will not allow a test case to run if all of its pre-requisites have not run
and passed
o Exception handling and recovery is built in
o All artifacts created by the test running are
automatically zipped and stored
§
Complete
evaluation after the run (if needed) is facilitated
·
Run Automation
Screen
o All automation candidate tests are automatically
reviewed to ensure only tests that should run are run
o Tests are selected to run based on a variety of
selection options
o The automation suite can be started now or by timer
without leaving WorM
o The automation suite is designed to work with almost
any commercial tool
o Suites can be restarted after they are stopped (either voluntarily or
due to catastrophic failure) without causing completed tests to be re-run
o All tests that were failed or blocked on a previous
suite run can be re-run easily
·
Test Evaluation
Screen
o Test owners may easily evaluate their automated test
runs
o Real-time access is provided to show all tests run
throughout the organization from any workstation – easy filtering to find any
run
o Simple access is provided to
§
Logs created at
run time
§
Screen
snap-shots taken during run – and reference images with comparison utilities
and composite image creation
§
All files
created during the test run
§
Re-run any
command-line tools that were run
§
Complete memory
stats for the workstation the test was run on
·
Total memory
·
Snap-shot of
memory before Test Case ran
·
Snap-shot of
memory after Test Case ran
o The ability to flag each result record as handled /
unhandled, allowing effective triage of results.
[1] There are 4 levels of security authority on uses. Developer, QA Engineer, Automator, and Administrator. Permissions for them will be discussed below.
[2] Artifacts are defined as files created during the test process that are worthy of being saved. These may include: screen snap shots, output files generated by command line tools, partial product files generated by the testing process, logging information for the workstation, etc. Artifacts are made available to users who are reviewing run results.
[3] A linked Test Case is one that shares some fields with other Test Cases to which it is linked. Linking Test Cases is performed in the Feature Tree and described below.) The ability to link Test Cases is to provide an easy way to manage related test cases where the functionality of the tests is repeated in a different context. By creating a base test, and then Link Copying it to other places, a user may simplify the maintenance of all the Test Cases. Each Test Case field is defined as either being Common (shared with all other Linked Test Cases) or Specific (visible by only a single Test Case and not shared with any other.)
[4] When recording Test Results, the user may enter the actual times taken during the execution. This allows the estimates to be refined from release to release.
[5] A Data Repository for Testing (DiRT) record is a way of linking an unlimited amount of data directly to the Test Case. This topic will be covered below.
[6] The relative order scheme was chosen to avoid having to micro-manage the execution order of test cases in automation. The default value is 50. All tests that have the same Relative Order are executed in unspecified order. To ensure that one Test Case runs before another Test Case, simply give it a lower Relative Order. In general, automated test cases should be designed not to have too many dependencies on other tests. However, since there are sometimes dependencies, this scheme allows them to be easily managed.
[7] For manual test cases, two flags control whether a Test Case is available for execution. If the Runnable flag is FALSE (unchecked) OR the Suspended Flag is TRUE, the Test Case is deemed as not being available for execution. This allows a user to control the testing proactively.
[8] Unhandled refers to a failed or a blocked test that has not yet been looked at or investigated as to why it did not pass. Either an Automator or the test owner may look at the automation run result record and mark it as Handled if they understand (and react to) the failure reason.