Points of Interest for Demonstration of Functionality

Epic Systems, Inc.

 

Epic is currently in the process of evaluating automated testing software for the primary purposes of regression and functionality testing of our enterprise medical software.  Currently, we have a development lifecycle lasting about a year per version, with smaller updates released in between.  We are looking for a program that can provide full regression testing for each of our smaller update releases, as well as provide functionality and regression testing for the version currently under development.  Due to integration between our products, scripts will be managed at the enterprise level to allow multiple teams to access developed scripts.  We currently have a system for tracking development and problems found in the code, so at this time we are not interested in defect tracking or development management tools.

In regards to regression testing the update releases, scripts would need to be written to allow for thorough coverage of program functionality, and a way to report back any errors discovered in the functionality.  For testing the version currently under development, the scripts will need to be able to identify an object if it has changed position on the form or if the label of the field has been changed.  In addition, the program should allow for the development of error handling and recovery scripts.  For example, if there is an error or a crash in the application under test, this can be recorded and testing can be resumed.  The results can then be summarized as to the stability of the section of the program tested, or as to the stability of the program as a whole.

In summary, we are looking primarily for the ability to have robust regression and functionality scripting.  We would have a majority if not all of the automated tests developed by testers who know our application and how to design tests but are not trained programmers. For the most part, these would be technical testers (that is, testers who have learned the scripting language but are not trained programmers), but some tests should also be able to be created by those who do not have experience with the programming language.

 

Summary of Points of Interest For Demo:

  1. The workflow for constructing a new automated test
    1. If it involves scripting at the test case level (or record/edit/playback):

                                                               i.      How the record/playback features work

                                                              ii.      Features (such as wizards) that aid in editing scripts or creating new ones

                                                            iii.      Ease of reading/editing/writing scripting language used

    1. If a method other than record/edit/playback enables testers who are not trained programmers to construct or assist in construction automated tests:

                                                               i.      How it works from the user’s perspective

                                                              ii.      What the underlying architecture is that makes it possible

                                                            iii.      How the system enables the user to verify that a test will run correctly

  1. How objects are recognized
    1. How objects are mapped
    2. How the properties of an object are identified
  2. The recommended process for updating automated tests from one version to the next
    1. How automated tests using objects that have been removed are identified for revision
    2. How changed objects are identified and the automated tests that use them are revised or identified for revision

                                                               i.      How cases are handled in which changes have been made to both the object type (e.g., edit field to list box) and the object name.

  1. How automated tests are run, individually and in groups
  2. How errors and crashes are handled when tests are run
    1. How the system reports on errors or crashes to facilitate further investigation
    2. How the system recovers and continues to run either the current or the next test
  3. How the automated test effort is managed at an enterprise level
    1. How current and previous versions of automated tests are managed
    2. How the system tracks the status of automated test construction: tests that are ready to be run, tests that need to be revised, and (if possible) tests that need to be constructed
    3. How the system tracks and reports on the status of automated test execution: which tests have been run and with what results