Wednesday, 25 June 2008

Test framework brief description

Here is a brief description what happens when running tests using my test automation framework (and some future plans).

Data model:
Each test suite has one or more test scenarios which has one or more test cases.
When running test suites, a unique test run is created which has one or more test case execution queues.
All data is in a MS SQL Server database.

In automation GUI:
1. Select run mode, New or Resume (If resume select a test run which is not completed).
2. Select one or more test suites if New.
3. For each test suite select test level, test days duration, test environment and test tool if New.
4. For each test tool instance, select global run options like reporting, e-mail etc.
5. Press run button for each test tool instance which is not aleady running.
6. Test run is started for each test tool instance via Scheduled tasks.

For each test tool:
7a. Execute each test case in queue until all is executed or cancelled (could happen if a prevoius nested test case fails).

In test run report GUI:
7b. Follow test run progress during run-time (now also reporting at step basis).

In test case dashboard GUI:
7c. Latest test case result are shown during run-time for each version och configuration.

Known limits:
Parallel execution is not implemented (you cannot use multiple test tool instances for a test run in order to decrease test execution time).

Supported tools:
QTP
TestBatchRunner (home made tool for running java batch jobs)
TestAPIRunner (home made tool for Web Service, XML API and http requests)

Future plans:
Implement support for more tools (thinking of trying Selenium).
Learn Java and OO (natural language choice since our AUT is written in Java and allowing for more collaboration with Development).
Rewrite test framework using OO-concept in Java (currently in VBScript and JavaScript) if Selenium works OK.

6 comments:

Anonymous said...

Hi,

I am following your post for quite some time. This is great post. I am trying to follow the framework you had developed but being new (only 2 years of exp), I am not clearly getting this Framework. Can you please share a working demo of your framework. This will be a great learning exp.

Regards,
Sanjeev
sm25578@gmail.com

Stefan said...

Hi,

I could try to do a "proof-of-concept" using excel instead of SQL Server but that would take a while to accomplish and since our release schedule this autumn is quite heavy you have to be very patient I'm afraid.

However, there are a lot of framework elements in my previous posts, especially those about XML and my custom report which you can check out meanwhile...

Regards,

/Stefan

Alex Ikhelis said...

Hi Stefan!

Good framework design of currently modern distributed environment (server-clients, web UI tests management). I would also recommend to switch to OOP and real language like Java + Selenium for GUI testing. Also you might consider to look at Ruby/Watir + ruby-on-rails for efficient development of framework UI and db_layer.

If you do not mind, I wonder:
1) how is interaction between server and clients is implemented? E.g. how QTP tool on a client workstation gets command to start and execute specific test suite?
2) What GUI do you use? did I get correctly that you use web UI + MS SQL?

Stefan said...

Hi and thanks for your comment!

Q: 1) how is interaction between server and clients is implemented? E.g. how QTP tool on a client workstation gets command to start and execute specific test suite?

A: Via a web GUI, you simple set a "run"-value for a test tool instance (column TEST_RUN in table TEST_RUN_OPTION) + in the same GUI mark available test suites that shall be included in the test run. If the test tool instance is scheduled, the test run starts within a minute (you could also start manually be starting the test driver script). Once started, a test run "queue" (TEST_RUN_RESULT_ROW) is created based in choosen test suite(s), test level and test days.

Q: 2) What GUI do you use? did I get correctly that you use web UI + MS SQL?

A: Yes, I use simple HTML-pages with client-based Java- and VBScript + MS SQL Server. I also have some HTML-pages created in MS ACCESS 2003 but since it is no longer supported by Microsoft I am replacing those one by one. You find examples of the GUIs in different posts here.

Good framework design of currently modern distributed environment (server-clients, web UI tests management). I would also recommend to switch to OOP and real language like Java + Selenium for GUI testing. Also you might consider to look at Ruby/Watir + ruby-on-rails for efficient development of framework UI and db_layer.I am have a long term goal of using Java (see other posts) and I tried Selenium but droped it because it could not handle modal pop-ups. I am currently checking out Watij which seems a bit promising.

Regards,

Stefan

Alex Ikhelis said...

Hi Stefan,

Thanks for your reply. In my Q#1 above I was actually interested in another thing... (sorry if I did not get it). Let's say you have a QTP client (QTP installed on a remote workstation), which you would like to use for runnin certain test suite. You mark the test suite and appropriate test client id in your GUI. What happens next? How command "Run this suite" comes from the server to QTP instance? I assume you have some code wrapping your QTP instance and checking for available test suites to run?

Regarding Watij - it probably might be a good choice for testing in Java. However Watir (ancestor) is more robust and its user community is very impressive and higly recommended. I do not know Selenium well, but guess there should be solutions to handle your pop-ups.

Thanks and good luck,
Alex

Stefan said...

Hi,

Q: 1) Let's say you have a QTP client (QTP installed on a remote workstation), which you would like to use for runnin certain test suite. You mark the test suite and appropriate test client id in your GUI. What happens next? How command "Run this suite" comes from the server to QTP instance? I assume you have some code wrapping your QTP instance and checking for available test suites to run?

A: Each test tool instance (including QTP and other tools) have a scheduled task which checks each minute if there are tests to run by running a shared test suite driver script. All test management data are in the Test tool db.

I will post my progress with Watij rather soon I guess...

Regards

/Stefan