Hello All,
I want to discuss and get feedback about my ideas for creating a custom testing framework for our applications that leverages the Coded UI Test in Visual Studio 2010. I need to evaluate my current design.
Definition: A UI testing platformis a system that provides interaction with the UI in an application under test. Coded UI Testing and Quicktest Pro are examples.
Definition: A custom testing frameworkis a layer or several layers of basis functionality that provides levels of indirection between the testing platform and actual tests that will be executed. The tests call functionality from the custom testing framework, which manipulates the UI using the UI testing platform or provides custom functionality of some sort. These levels of indirection are meant to enhance maintainability and robustness of Coded UI Tests.
In investigating Coded UI Testing I have found that recording of individual tests was often leading to very similar steps and UIMaps in each test. I have already gone throughhttp://msdn.microsoft.com/en-us/library/ff398056%28v=VS.100%29.aspx and have implemented multiple UIMaps. There is still a problem recording actions on a per-test basis though since for every test I am generating identical functionality. An example of this would be the logical group of actions called “open a project” which is a composite of many sub steps and is used in almost every test. It is therefore desirable to have a single method recorded or built by hand and stored in some library of functions that can be invoked when required. I would call these methods that implement logical groups “test components”.
Definition: A test componentis a test method existing in some layer of the custom testing framework that uses one or more UIMaps to manipulate UI elements to accomplish some specific task such as clicking an individual button, or to accomplish a logical group of actions such as opening a project. Test components can be identified by their common patterns in many tests.
There are two types of test component. An “atomic test component” might perform some number of actions against the UI, but will never reference other components. That is, an atomic test component only uses the UIMap to manipulate UI elements. The second type of test component is a composite test component that is built from atomic components and composite components. Each test component will require access to one or more UIMap(s) to function. Since each test component will be implemented as a “test method”, the exception handling mechanism that provides results for tests can still be used and will filter up through the hierarchy of components in the custom testing framework. Since a test component is either atomic or is a collection of test components, and any test component is handled in the same way whether it is atomic or a collection, this is somewhat analogous to the composite design pattern except not in the context of object construction. A good example of a composite component is the “open a project” test component. This method would take the name of the project to open as a parameter and use atomic and potentially other composite components to execute the logical group of operations that open a project. The UIMaps used by such components are a little bit of a concern.
The UIMap used by the “open a project” method also exists in the library and as such is not duplicated in multiple tests. Thus at the beginning of most tests, the test author will have to insert a call to the “open a project” method. Unfortunately as far as I can see at this point, this seems to break the test recording model since the Coded UI Test builder doesn’t provide a way to indicate a call to a method in the library instead of recording a new UIMap and method.
Ultimately the goal is to provide a comprehensive set of test components implementing logical groups of actions such that a test creator can avoid duplicating functionality and quickly assemble a test by calling the appropriate components. The testers here have limited development experience, but are in the process of learning the basics. Hopefully in the long term, anyone from the most experienced developer to the programming challenged will be able to create automated tests that are maintainable and robust.
With this model in mind, the recording functionality in VS2010 will be used only for creating the library of logical groups of actions (test components) and for per-test functionality that has not been included in the library at the time of test creation. This is ok, but it is note ideal and it would be nice to be able to extend the Coded UI Test Builder to add calls to the component library when required. A more serious problem lies in asserting or checkpointing values using the test creation.
While there may be some common assertions or checkpoints included in the library of components, for all tests there exist test specific expected values. These test specific expected values must be stored in each test or in some structure defined to hold them; perhaps a CSV file. Since the UIMaps for everything lie in the application specific library of methods, using the assertion creator of VS2010 will not reference the library UIMaps and will thus try to create a per test UIMap that may duplicate some of the library UIMap for that portion of the application under test. This may even be desirable in some cases, but it still creates a potential maintainability problem. In most cases it would be better if the assertion recorder and generator could generate expected values in a UIMap that subclasses (inheritance) one of the library UIMaps. Another useful way to do things would be to allow the assertion generator to add expected values to some file such as a CSV or Excel spreadsheet. The UIMap subclass situation would be ideal since a test can call its recorded assertion methods that exist local to the test, and these methods would be able to use the preexisting UIMap to verify expected values. As far as I know, using the Coded UI Test Builder in this way is not possible at this time and as such, assertion methods need to be provided via the library of test functionality. Somehow the test specific expected values need to be accounted for, but they cannot be recorded without duplicating UIMaps; undermining the usefulness of the recording tool.
I must make a design decision and balance the tradeoffs of maintainability and ease of use in whatever system I implement. Making a component-wise library of functionality will provide maintainability and scalability, but will require much more development time initially and could potentially be more difficult to use for non-developers. That said, I believe maintainability and robustness of the system is the highest priority for our automated testing system due to personnel constraints. I do not want to get bogged down maintaining old tests; I would rather build new tests and have to perform simple fixes to components used in older tests.
I have defined three levels of testing functionality as follows, each of which is a “test project” inside a “testing solution”. Each test project is compiled to a .dll and is referenced in the next higher level of functionality. These levels are as follows,
I would like to know what anyone thinks about my design, and if anyone else has tried something similar. I am concerned about this design cutting out much of the recording of actual tests, but at this point I feel that once a large number of tests are created, this framework will make the tests more maintainable. Please give me your feedback and any ideas you have. Any input is welcome! I am trying to learn how to maximize the effectiveness of my testing project, and how to leverage VS2010 properly.
I want to discuss and get feedback about my ideas for creating a custom testing framework for our applications that leverages the Coded UI Test in Visual Studio 2010. I need to evaluate my current design.
Definition: A UI testing platformis a system that provides interaction with the UI in an application under test. Coded UI Testing and Quicktest Pro are examples.
Definition: A custom testing frameworkis a layer or several layers of basis functionality that provides levels of indirection between the testing platform and actual tests that will be executed. The tests call functionality from the custom testing framework, which manipulates the UI using the UI testing platform or provides custom functionality of some sort. These levels of indirection are meant to enhance maintainability and robustness of Coded UI Tests.
In investigating Coded UI Testing I have found that recording of individual tests was often leading to very similar steps and UIMaps in each test. I have already gone throughhttp://msdn.microsoft.com/en-us/library/ff398056%28v=VS.100%29.aspx and have implemented multiple UIMaps. There is still a problem recording actions on a per-test basis though since for every test I am generating identical functionality. An example of this would be the logical group of actions called “open a project” which is a composite of many sub steps and is used in almost every test. It is therefore desirable to have a single method recorded or built by hand and stored in some library of functions that can be invoked when required. I would call these methods that implement logical groups “test components”.
Definition: A test componentis a test method existing in some layer of the custom testing framework that uses one or more UIMaps to manipulate UI elements to accomplish some specific task such as clicking an individual button, or to accomplish a logical group of actions such as opening a project. Test components can be identified by their common patterns in many tests.
There are two types of test component. An “atomic test component” might perform some number of actions against the UI, but will never reference other components. That is, an atomic test component only uses the UIMap to manipulate UI elements. The second type of test component is a composite test component that is built from atomic components and composite components. Each test component will require access to one or more UIMap(s) to function. Since each test component will be implemented as a “test method”, the exception handling mechanism that provides results for tests can still be used and will filter up through the hierarchy of components in the custom testing framework. Since a test component is either atomic or is a collection of test components, and any test component is handled in the same way whether it is atomic or a collection, this is somewhat analogous to the composite design pattern except not in the context of object construction. A good example of a composite component is the “open a project” test component. This method would take the name of the project to open as a parameter and use atomic and potentially other composite components to execute the logical group of operations that open a project. The UIMaps used by such components are a little bit of a concern.
The UIMap used by the “open a project” method also exists in the library and as such is not duplicated in multiple tests. Thus at the beginning of most tests, the test author will have to insert a call to the “open a project” method. Unfortunately as far as I can see at this point, this seems to break the test recording model since the Coded UI Test builder doesn’t provide a way to indicate a call to a method in the library instead of recording a new UIMap and method.
Ultimately the goal is to provide a comprehensive set of test components implementing logical groups of actions such that a test creator can avoid duplicating functionality and quickly assemble a test by calling the appropriate components. The testers here have limited development experience, but are in the process of learning the basics. Hopefully in the long term, anyone from the most experienced developer to the programming challenged will be able to create automated tests that are maintainable and robust.
With this model in mind, the recording functionality in VS2010 will be used only for creating the library of logical groups of actions (test components) and for per-test functionality that has not been included in the library at the time of test creation. This is ok, but it is note ideal and it would be nice to be able to extend the Coded UI Test Builder to add calls to the component library when required. A more serious problem lies in asserting or checkpointing values using the test creation.
While there may be some common assertions or checkpoints included in the library of components, for all tests there exist test specific expected values. These test specific expected values must be stored in each test or in some structure defined to hold them; perhaps a CSV file. Since the UIMaps for everything lie in the application specific library of methods, using the assertion creator of VS2010 will not reference the library UIMaps and will thus try to create a per test UIMap that may duplicate some of the library UIMap for that portion of the application under test. This may even be desirable in some cases, but it still creates a potential maintainability problem. In most cases it would be better if the assertion recorder and generator could generate expected values in a UIMap that subclasses (inheritance) one of the library UIMaps. Another useful way to do things would be to allow the assertion generator to add expected values to some file such as a CSV or Excel spreadsheet. The UIMap subclass situation would be ideal since a test can call its recorded assertion methods that exist local to the test, and these methods would be able to use the preexisting UIMap to verify expected values. As far as I know, using the Coded UI Test Builder in this way is not possible at this time and as such, assertion methods need to be provided via the library of test functionality. Somehow the test specific expected values need to be accounted for, but they cannot be recorded without duplicating UIMaps; undermining the usefulness of the recording tool.
I must make a design decision and balance the tradeoffs of maintainability and ease of use in whatever system I implement. Making a component-wise library of functionality will provide maintainability and scalability, but will require much more development time initially and could potentially be more difficult to use for non-developers. That said, I believe maintainability and robustness of the system is the highest priority for our automated testing system due to personnel constraints. I do not want to get bogged down maintaining old tests; I would rather build new tests and have to perform simple fixes to components used in older tests.
I have defined three levels of testing functionality as follows, each of which is a “test project” inside a “testing solution”. Each test project is compiled to a .dll and is referenced in the next higher level of functionality. These levels are as follows,
1
. The top level (level 1) is the common functionality that will be used in testing all of our software products. Examples of this are something like bitmap checkpointing, which any individual test might need to use.
2. The middle level (level 2) is the application dependent test functionality (test component) provider. This level is where all UIMaps exist and where the definitions of recorded or hand built atomic components and composite components exist.
3. The bottom level (level 3) contains the actual tests that will be executed. These tests will call test components from level 2 to perform some common actions such as “opening a project” and will also allow use of recording for per-test steps. These per-test recordings will allow the components from level 2 to be glued together to accomplish tasks for which there has been no test component defined in level 2. If it is found that all or part of these per-test recordings are being duplicated across different tests, then the common pattern can be extracted and will be a strong candidate for promotion to level 2.
No comments:
Post a Comment