Stefan Thelenius about Software Testing

QTP: And I thought I knew QTP by Tarun Lalwani

2011-11-08T20:39:00.000+01:00

Tarun Lalwani, one of the leading experts in the automation test tool QuickTest Professional™ (QTP) has recently released his second book: And I thought I knew QTP.

This is most likely a unique book in this context: A technical novel! At least I have not read anything like it or heard about something simular to this concept.

The story is about Nurat, an Indian consultant with 7 years experience of QTP. While considering a new career, he is called to a series of interviews for one the top companies in the IT-business. Those interviews are about every aspect of QTP you can think of (and some you did not knew about I can assure you).

The question-answer format in combination with some practical hands-on tips makes this a great book about QTP whether you work (or want to) as an automation consultant or with other automation releated tasks such as framwork design for example.

I got a feeling when reading this book that it has some elements of self-biography but it is my personal reflection. I guess only Tarun himself could measure with Nurat. Without reviling the end of this book I probably would not have passed these interviews since I thought I knew QTP :-).

Test case model - refactoring part 2

2011-06-19T10:27:00.001+02:00

How to report function coverage using SBTM? Maybe the model should look like this:

- Posted using BlogPress from my iPhone

Testing dashboard part 3

2011-05-06T16:27:00.002+02:00

I am still a severe dashboard addict...and this new gauge from google is so cool (rendered within the browser).

What can it be used for in an SQA context?

Severe bugs per day? Suggestions welcome...I must have it somewhere in our release status testing dashboard...

Test case model - refactoring

2011-04-30T23:58:00.000+02:00

After very fruitful sessions with James Bach and addQ a lot of ideas are spinning in my head.

I am definetly refactoring my test case model so Test Charters and Test sessions are included...

Maybe it should look like this:

- Posted using BlogPress from my iPhone

Pairwise test case design part four

2011-03-17T13:25:00.003+01:00

New version of my PICT web shell is now available here.

I got a request for allowing PICT option values so in the new version it is possible to set option values in an input field. In addition I have also implemented a parameter for setting the timeout value for creation of the output file (in previous version it was a hard coded value of 5 seconds).

Pairwise test case design part three

2011-02-27T16:06:00.004+01:00

I discovered the wonderful project CodeMirror about a year ago where you can design a text editor in browser with syntax highlightning...

...the other day when I edited a model file for PICT.exe I got the idea of a deluxe version of my old PICT web shell. For more background about PICT and Pairwise test case design, please see my first post about it.

Key features:
+ Syntax highlight for PICT model file content
+ Test cases output file content visable in GUI
+ Export to Excel

Download and enjoy!

Long Time no...

2010-12-20T22:02:00.001+01:00

...post

Well since I am with iPhone nowadays maybe (no promises) I will post more often...I have some small projects I might publish soon, stay tuned...

- Posted using BlogPress from my iPhone

Some source files are back...

2010-10-13T13:39:00.001+02:00

...please let me know if there are some files missing or so...

All source files...

2010-08-01T10:25:00.002+02:00

are currently deleted by my file host provider...don't know why really but I guess you sometimes get what you pay for (free service vs quality)...however I will try to restore the content within a month...

JavaScript or VBScript?

2010-03-04T13:45:00.003+01:00

A relevant question...

Lately I am more into JavaScript...why?

One of the most important reason for me is that the error handling functionality is better in JavaScript (try...catch)...

JavaScript seems also like a better long-time solution since it is more common than VBScript these days...

SQL: Database size management

2010-01-26T14:15:00.002+01:00

Keep up good test effeciency often requires several test environments with various configuration and test data setups etc. Add the dimension multiple AUT release versions and you probably need more test sites than you can possible dream of.

To handle multiple test environments properly requires technical knowledge, some programming skills and a good releationship with your system administrator :-)

A very good idea is to have a test environment dashboard for survaillance of critical parameters. One of those parameters is database size. For large enterprise systems with high transaction rate, production-like copies often becomes quite large, hence you have to do some preventing actions in order to keep your disk quota. Some well targeted delete scripts in combination with shrink commands often does the trick. But how to check the current size?

Fortunatly MS SQL Server has some built-in Store procedures for this:


--Get total database size in MB including log file
sp_spaceused

database_name  database_size      unallocated space
-------------- ------------------ ------------------
MyDatabase     31877.31 MB        5382.25 MB


--Get remaining disk size for each drive on the database server
xp_fixeddrives

drive MB free
----- -----------
C     8011
D     90660
E     23057

--Get disk file name, size and drive
select filename, (SUM(size)*8)/1024 As FileSizeInMB FROM master.dbo.sysaltfiles
group by filename

filename            FileSizeInMB
-----------------------    ------------
D:\MyDatabase.mdf        5806
D:\MyDatabase_log.ldf    542

SQL: Select your table rows gently...

2009-12-04T14:48:00.002+01:00

How to get all rows in a table?

Simple answer:


SELECT * FROM TABLE

BUT, a better practise is to

1. Do a count first if you suspect it is a large table
2. Use (nolock) in order not to lock the table during the query execution

Syntax examples for MS SQL Server:


--Count rows without locking table
SELECT COUNT(*) FROM TABLE(NOLOCK)

--Get all rows without locking table
SELECT * FROM TABLE(NOLOCK)

--Get the 1000 last entries based on primary key (assuming it is column #1)
SELECT TOP 1000 * FROM TABLE(NOLOCK) ORDER BY 1 DESC

Windows 7: Three steps back...

2009-11-05T12:02:00.003+01:00

...since my new Laptop has "freezed" three times during the last couple of weeks. Hardware reset has been the only option since the screen has became black and the activity handler does not respond within 10 minutes (which is so long my patience holds in these situations)...

No traces whatsoever what could be the cause in the log files...Dell hardware, Windows7 or a combination?

Windows 7: One step forward...

2009-11-01T14:22:00.002+01:00

...to faster start-up time (compared to XP)

I recently switched laptop to a Dell with Windows7 onboard. So far I think it works pretty well...I really like the preview open documents functionality and the activity bar improvments which is a small productivity booster for those of us that works with many open windows.

Although it was a bit sad to retire my old IBM T43...the most reliable and robust PC I have ever had (so far...)

SQL: MS SQL Server 2008 Management Studio - First impression

2009-09-29T21:58:00.002+02:00

For those of you who are not familiar working with MS SQL Server product family, the program Mangagement Studio includes SQL Query and Object Explorer among a lot of other useful features for administration of databases. As a tester, knowing SQL and AUT Database objects is essential if you want to be really successful in your work in my opinion. Last week I upgraded to the 2008 version and found two new features:

1. Debugging SQL/Store Procedures
2. Intellisense writing SQL

Need I say more?

Testing Dashboard part 2

2009-08-14T15:19:00.003+02:00

As you already may know...I like dashboards (probably too much...)

After I was being aware of the Google Chart API and I also found this JavaScript implementation, I am thinking of using the Google-o-meters showing different coverage metrics in my auto-generated real-time test report which is based on both on subjective conclusions as well as logged regression test result data...

Building dashboards is rather simple, the hard part is to collect proper test status data...

QTP: QuickTest Professional Unplugged by Tarun Lalwani

2009-07-01T08:20:00.003+02:00

Tarun Lalwani's book QuickTest Professional Unplugged covers QTP basic as well as advanced features.

I have used QTP since 2001 (version 6.x) and if I have had this book from the start, a lot of hours and frustration had been saved.

Tarun has prior the book, contributed to the QTP community with numerous of high quality articles and forum post replies over the years. I recall an old article about Descriptive programming (DP) which made me switch from OR to DP Big-bang style (I don't regret it...).

My favorite parts in the book are the chapters regarding RegExp, IE/Word/Excel objects, HTML DOM and of course the advanced section at the end. But this book include a lot of other QTP/VBScript features as well and is well suited both for newbies (you should have some programming experience at least) and advanced users.

This is a must have for any person using QTP and/or VBScript-based automation!

Watij: Brief progress report

2009-06-10T16:21:00.003+02:00

I have been evaluating Watij lately and here are my conclusions so far for some automation features important to me:

IDE

Since I am from the QTP/VBScript world I still struggle a bit with the Java syntax and OO...furtunatly my IDE (IntelliJ IDEA) has very good refactoring features (and other develop-friendly features as well) so it is not necessary to make perfect code/structure from the start.

GUI test execution speed

Not as fast as QTP but very close. Since the source code is available I managed to change a hard-coded timeout property and I guess there are several others that you could tweek for faster performance.

Test object model

Quite simular to Descriptive programing in QTP.

QTP example: Browser("Google").Page("Google").WebEdit("q").Set "Watij"
Watij example: ie.textField(name,"q").set("Watij");

Have not explored all possibilities with dynamic test object handling in Watij yet but at least getting all links in a frame into an array was very simple.

Community

The QTP community is awesome and Watij's dito seems ok but rather small.

Pop-up handling

QTP is very good handling pop-ups but have a weakness running such actions in a locked workstation. Watij has so far also been very good at this (except for an authentication dialog which I handled with a quick and dirty java robot solution) and IT WORKS IN LOCKED MODE! This feature alone allows unattended GUI tests being run 24x7...This seems to good to be true so I will do some more testing on this using remote desktop -> server...

Test framework - Multiple tool support

2009-05-14T11:14:00.005+02:00

To design a robust and cost effective automation framework is a complex task...

This morning a read a good post about Programming Paradigms in Test Automation which I can recommend reading.

I have on my agenda to switch from VBScript to Java in my framework. I have realized that this process is going to take a while since I have not yet found a decent tool for the GUI testing part...

I am currently evaluating Watij and so far it seems really promising since it can deal with pop-ups much better than Selenium. However, test execution speed is not that good when running Watij (QTP or worse) from IDEA so next step is to see how it handles stand-alone execution...stay tuned

Since I want to support multiple tools/langauges in my framework I have decided to move all SQL from code to Store procedures for easier maintenance. Here is an overview picture of how it will look like when implemented:

VBScript: Sending e-mail

2009-05-05T09:03:00.003+02:00

Sending e-mail automaticly at various test contexts could be a useful feature (beware of "spam" if you are sending mail to a shared address...) so as a follow-up to this post: http://abouttesting.blogspot.com/2008/01/cdomessage.html I thought I would share an example on how to send an HTML-based e-mail using CDO:


'Sending E-mail sample

strEmailSubject = "Testing CDO"
strMessageFrom = "myemailfrom@mydomain.com"
strMessageTo = "myemailto@mydomain.com"
strHTMLBody = "<B>Bold</B>&nbsp;example"
strSMTPServer = "mail.mydomain.com"

SendEmail strEmailSubject, strMessageFrom, strMessageTo, strHTMLBody, strSMTPServer

Function SendEmail (strEmailSubject, strMessageFrom, strMessageTo, strHTMLBody, strSMTPServer)
Set objMessage = CreateObject("CDO.Message") 
objMessage.Subject = strEmailSubject
objMessage.From = strMessageFrom
objMessage.To = strMessageTo
'The line below shows how to send using HTML included directly in your script
objMessage.HTMLBody = strHTMLBody
Const cdoSendUsingPickup = 1 'Send message using the local SMTP service pickup directory. 
Const cdoSendUsingPort = 2 'Send the message using the network (SMTP over the network). 
Const cdoAnonymous = 0 'Do not authenticate
Const cdoBasic = 1 'basic (clear-text) authentication
Const cdoNTLM = 2 'NTLM
'==This section provides the configuration information for the remote SMTP server.
objMessage.Configuration.Fields.Item _
("http://schemas.microsoft.com/cdo/configuration/sendusing") = 2 
'Name or IP of Remote SMTP Server
objMessage.Configuration.Fields.Item _
("http://schemas.microsoft.com/cdo/configuration/smtpserver") = strSMTPServer
'Type of authentication, NONE, Basic (Base64 encoded), NTLM
objMessage.Configuration.Fields.Item _
("http://schemas.microsoft.com/cdo/configuration/smtpauthenticate") = cdoAnonymous
'Server port (typically 25)
objMessage.Configuration.Fields.Item _
("http://schemas.microsoft.com/cdo/configuration/smtpserverport") = 25 
'Use SSL for the connection (False or True)
objMessage.Configuration.Fields.Item _
("http://schemas.microsoft.com/cdo/configuration/smtpusessl") = False
'Connection Timeout in seconds (the maximum time CDO will try to establish a connection to the SMTP server)
objMessage.Configuration.Fields.Item _
("http://schemas.microsoft.com/cdo/configuration/smtpconnectiontimeout") = 60
objMessage.Configuration.Fields.Update
'==End remote SMTP server configuration section==
objMessage.Send
End Function

You can download the script here:
http://abouttesting.fileave.com/SendEmail.zip

Test framework - Multi-threading

2009-05-04T15:24:00.002+02:00

From http://abouttesting.blogspot.com/2008/06/test-framework-brief-description.html

Known limits:
Parallel execution is not implemented (you cannot use multiple test tool instances for a test run in order to decrease test execution time).

Check. Implemented today :-)

Solution: Call a Store procedure with test run id and test run option id (including test tool type) and it returns next compatible test case in the execution queue including Go/NoGo (if NoGo wait until Go critera is met).

Is this a necessary feature?

I have many test suites that are End-to-end based and spans over several test days. Since each day requires a lot of batch processing (included in test suite as well) you want to combine a lot of test scenarios in one test suite instead of having one test suite/test scenario. The largest test suite has now over 900 test cases (multi steps)...so anything that reduces overall execution time is very welcome :-)

Test data model part 2

2009-04-21T17:16:00.003+02:00

A snapshot document of my test framework data model is now available for download:

http://dl.dropbox.com/u/8538167/TestFiles/TEST_DATA_MODEL_090420.zip

What to use it for?

I think it mainly can serve as an inspiration source on how to (or how not to...) model your test framework as it is difficult to use it "as is" since it requires SQL logic (in code or in store procedures) and a bunch of GUIs (some has been published earlier).

When I read the document I realize that some things should be refactored (bit instead of varchar for True/False columns for example) but all design decisions made sense to me at the time of creation. The database is "hand-made" from scratch using SQL Server GUI to create and change tables. Columns and tables have been added from time to time ever since.

One of the reason why I choosed to use SQL Server instead of excel was that my AUT required of lot of parameters...if I do select distinct test_case_parameter_name from test_case_parameter the answer is 192...

Test framework - 'stop if too many errors feature'

2009-03-20T16:15:00.002+01:00

Another feature I implemented a few weeks ago is 'stop if too many errors'.

Since an e-mail is sent for each test case failure, the e-mail inbox could be flooded if for example, an unhandled test environment issue occurs or the AUT build is completely broken. So to prevent that I have a parameter for how many test case failures that are tolerated before stopping a test run. Not that complicated either but good to have (especially if the e-mails are sent to a shared address...)

:-)

Test framework - Stop feature

2009-03-13T16:19:00.002+01:00

I implemented a new feature in my automation framework today...

My test suites are usually very long scenarios and sometimes it takes hours to get to a test case which is by the end of a scenario. Imagine creating new test cases at the end, running the suite and a script failure occurs which in worst case require total rerun (that is also a major reason why I have a 'Resume' test feature)...

...so I thought a stop feature would be nice to have so I implemented a new state/status for a test case run status; 'Stop'

It work like this: Once a test run is created (more clues about my data model will be published here within a month or so, I hope) you set test case run status to Stop for a test case (via Report GUI) and the test execution will stop when the test case is up next in the test case execution queue (and sending an e-mail about it). Now, you can resume the test in debug mode and fix errors on the fly (remember to change status to 'No run' first). Or perhaps backup the AUT database before you continue?

That's it, not that complicated (I wonder why I haven't thought of it before)...

SQL: Copy/Paste a database...

2009-02-25T13:13:00.003+01:00

...could be a handy function.

Sometimes your test data gets corrupted or outdated i.e. useless in a quality context. If so make sure to have a set of backups with good valid test data. If the backup size is huge (deidentified production copies for example) it could be hours in performing backup/restore...so in order to skip the waiting part of the backup to finish you can do the backup/restore using one SQL script like this:


/* Copy a database to another on the same db server */

declare @databaseFrom varchar(100)
declare @databaseTo varchar(100)
declare @databaseCopyName varchar(500)
declare @randomNumber varchar(50)
declare @date varchar(50)
set @date = Replace(Replace(getDate(),' ','_'),':','_')

-- *** User input ************************
-- Database to copy
set @databaseFrom =    'MyFavoriteDataBase'
-- Target database 
set @databaseTo =    'OtherDataBase'
-- ***************************************

--Perform backup and restore (Do not forget to remove all connections on target database prior restore!)
set @randomNumber = Round((RAND()*1000000),0)
set @databaseCopyName = '\\myServer\myDrive$\myFolder\'+@databaseFrom+'_'+@date+'_'+@randomNumber

Print 'Database ' + @databaseFrom + ' will be saved as: ' + @databaseCopyName

declare @databaseCopyBackupName varchar(500)
Set @databaseCopyBackupName = @databaseFrom+'-Full Database Backup'

BACKUP DATABASE 
@databaseFrom
TO  DISK = 
@databaseCopyName
WITH NOFORMAT, NOINIT,  NAME = 
@databaseCopyBackupName,
SKIP, NOREWIND, NOUNLOAD,  STATS = 10

Print 'Database ' + @databaseFrom + ' were saved as: ' + @databaseCopyName

Print 'Starting restore to ' + @databaseTo

RESTORE DATABASE @databaseTo FROM  
DISK = @databaseCopyName
WITH  FILE = 1,  NOUNLOAD,  REPLACE,  STATS = 10

Print 'Restore of ' + @databaseCopyName + ' to ' + @databaseTo + ' is done!'

GO

Test data modell 1: Table overview and TEST_CASE

2009-02-03T13:47:00.005+01:00

I have been very busy the last months both at work and at home (we have switched from a house to an apartment)...

...but now I thought I should try to explain my test data modell which I use for administrating tests and test environments.

First out is an overview picture of all tables and perhaps the most important table, TEST_CASE including some brief information. I will gather all table information in an RTF-document which should be available for download here: http://dl.dropbox.com/u/8538167/TestFiles/TEST_DATA_MODEL.zip

QTP: Record & Playback part 2...

2008-12-02T09:16:00.001+01:00

...just changed my mind...will add some verification steps as well...

:-)

QTP: Record & Playback...

2008-12-02T08:53:00.002+01:00

...is in some cases very cost effective. A recent test took me < 10 minutes to record, adjust some objects with dynamic DP and add 140 records to datatable:

Search data
Click on some objects
Send page to printer
Click some more objects
Order another printout

I could have add some verifying logic as well but since this is a one-time test (bug investigation) I will check the printouts manually and I guess the overall ROI for this test is quite ok anyway.

Some blogs about testing/automation...

2008-11-18T21:41:00.003+01:00

...are listed Here. I don't know how the ranking system works in detail but it feels good to be on the list (I like lists...).

SQL: Script traceability

2008-10-24T10:22:00.002+02:00

A small snippet for automaticly in SQL Server get date and user name into columns like CREATED_DATE, CREATED_BY, LATEST_UPDATED_BY etc:


BEGIN TRAN
UPDATE [table]
SET 
LATEST_CHANGED_DATE = getDate(), 
LATEST_CHANGED_BY = SYSTEM_USER
WHERE [critera] = 'True'
COMMIT

In table:

LATEST_CHANGED_DATE LATEST_CHANGED_BY
2008-10-24 10:37:26.470 Stefan

SQL: Multiple insert

2008-10-02T15:34:00.003+02:00

I do some SQL scriptning for various testing purposes sometimes and I sure miss array functionality in SQL. There are however some workarounds...

One is to create temporary tables and fill it with data. It works excellent if you combine INSERT with SELECT FROM [TABLE] but what if you want to fill your table with hard-coded data? Here is one possible solution:


--Example of multiple insert

--Create temp table
CREATE TABLE #fpp([RID] int NOT NULL IDENTITY (1, 1) PRIMARY KEY, [MY_DATA] varchar(30))
go 

--Insert multiple data using union
insert into #fpp ([MY_DATA])
select 
(select 'data1') union
(select 'data2') union
(select 'data3') union
(select 'date4')
go

select * from #fpp

--Clean up
drop table #fpp

Java: Test Frameworks and the weather

2008-09-25T12:26:00.004+02:00

I have come to a conclusion to use existing test frameworks (JUnit...) in combination with my own when converting tests from VBScript into Java so I do not need to "re-invent the wheel". However I will reuse some features in my own framework which I have not found in existing ones.

The biggest advantage so far using Java instead of VBScript is that it makes it easier to collaborate with the Dev guys since we now "speaks the same langauge". So now we have a common build and testing project where we have set up a Hudson project where builds and tests runs several times a day continously. You get a dashboard view of the status for each release:

Did I mentioned I love dashboards...and being a Swede, also talking about the weather...

:-)

Java: Display Java code in HTML

2008-08-22T15:35:00.003+02:00

It was rather difficult to find a utility for Java code export to HTML but at last I found a free tool called Java Code Export which do a decent job but requires some manual steps after a conversion, why I am not 100% satisfied with this solution. Any tips are welcome in this subject.

For formating testing purposes I publish this tiny Java method for creating random numbers:

     /**

     *

     * @param lowerLimit

     * @param upperLimit

     * @return A random integer number between lowerLimit and upperLimit

     */

    static int createRandomNumberInt (int lowerLimit, int upperLimit)    {

        Random generator = new Random();

        return generator.nextInt (upperLimit-lowerLimit) + lowerLimit;

    }

Java: My first Class...

2008-08-13T14:00:00.002+02:00

...is ready and seems to work!

My programming experience has envolved some languages over the years and here is a summary:

1984-1990 Basic on Sinclair Spectrum (developing small simple games at home)
1990 Pascal on PC/Windows (small simple programs at high school)
1991 C on Unix and PC/Windows (simple programs at high school)
1991 Assembler (simple robot programs at high scool)
1994- JavaScript (web and test tool utilities at home and at work)
1998-2000 LabView (programs for hardware interaction at work)
2000- VBScript (automation and test tools utilities at work)
2002- SQL (various scripts at work including Store procedure programming)
2002-2006 Loadrunner "C" (performance scripts at work)

and now I can add

2008- Java (automation and test tools at work)

I have read a couple of Java books during my vacation and hopefully I understand a bit more about the OO concept than before as well as the Java syntax.

I have started off with a simple Class for test data utilities and I have rewritten some of my old VBScript functions into Java methods. And so far it works pretty well. One of the major benefits is that I now got an decent IDE in ItelliJ IDEA compared to using QTP's native IDE for VBScript.

My initial goal is to rework my test framework for Web Service testing as there are some powerful utilites available for Java (generate test classes from WSDL for example).

Selenium IDE: First impressions

2008-07-11T09:22:00.002+02:00

Installation of Selenium IDE went very smooth including installing FireFox 3.0 at first. Had to tweek some security settings in order to get proper access to our test environments but after that all went well accessing the AUT directly or via Selenium.

Did some basic AUT actions using Record/Replay in Selenium IDE and it worked pretty well considering it is a free tool. I had to adjust replay speed (the tests runs very fast at highest speed setting) in order for some objects to get recognized since I did not find any proper "wait...exists" command at first.

Unfortunate I could confirm that Selenium could not handle pop-up dialogs which it is a critical requirement for my AUT. I found some possible workarounds when googled on the subject but have not tried them yet (any tips are welcome!). So for the moment I put Selenium on hold.

Currently I am on vacation but my summer reading are books about Java and OO. Hopefully I soon have enough knowledge to start re-write some of my test utilites into Java using OO design. I've got IntelliJ IDEA installed which seems to be a very good Java IDE...

Test framework brief description

2008-06-25T09:40:00.002+02:00

Here is a brief description what happens when running tests using my test automation framework (and some future plans).

Data model:
Each test suite has one or more test scenarios which has one or more test cases.
When running test suites, a unique test run is created which has one or more test case execution queues.
All data is in a MS SQL Server database.

In automation GUI:
1. Select run mode, New or Resume (If resume select a test run which is not completed).
2. Select one or more test suites if New.
3. For each test suite select test level, test days duration, test environment and test tool if New.
4. For each test tool instance, select global run options like reporting, e-mail etc.
5. Press run button for each test tool instance which is not aleady running.
6. Test run is started for each test tool instance via Scheduled tasks.

For each test tool:
7a. Execute each test case in queue until all is executed or cancelled (could happen if a prevoius nested test case fails).

In test run report GUI:
7b. Follow test run progress during run-time (now also reporting at step basis).

In test case dashboard GUI:
7c. Latest test case result are shown during run-time for each version och configuration.

Known limits:
Parallel execution is not implemented (you cannot use multiple test tool instances for a test run in order to decrease test execution time).

Supported tools:
QTP
TestBatchRunner (home made tool for running java batch jobs)
TestAPIRunner (home made tool for Web Service, XML API and http requests)

Future plans:
Implement support for more tools (thinking of trying Selenium).
Learn Java and OO (natural language choice since our AUT is written in Java and allowing for more collaboration with Development).
Rewrite test framework using OO-concept in Java (currently in VBScript and JavaScript) if Selenium works OK.

Testing dashboard

2008-06-18T12:45:00.005+02:00

I like dashboards...both using them as well as developing them. If based on proper metrics they give a good picture of the current status in a single view. However, finding good metrics in testing is not a trivial task since counting test cases is not a good idea (especially for those of us who mainly do exploratory testing instead of executing scripted test cases).

I took the "Rapid Software Testing" course by James Bach about a year ago and he introduced me to "The Dashboard Concept" (see slide 138 in RST Slides). We had at the time a simular concept in an Excel sheet where colors were changed based on certain keywords but Excel has some limits when it comes to concurrent usage.

Now we are evaluting the use of my latest creation: AUTDashboard.zip

I have removed the Effort column from James' dashboard and I will probably add some extra columns for latest changed date and other things later on.

The dashboard aims at showing the test and quality status for an AUT, based on subjective conclusions for all test activities in a release/project.

If want to try it out, download it and place the files in C:\ if you don't want edit the html file. You also have to answer yes to all security warnings in order to get it to work.

Test automation report - source code

2008-06-13T15:11:00.004+02:00

Here you can find the source code for my node report (see previous post): wwwroot.zip

I have enclosed an excel file instead of my database but if you take a look at the sheets you get some clues how my test case data model is like.

Place the files under c:\Inetpub\wwwroot\ if you want to run immediatly else you have to alter some hardcoded filepaths in the html-files.

If you have local IIS you open the report by this URL:

http://localhost/Index.html?1

where the number after the ? is input parameter which test run report to show.

If you are successful in installation the page should look like this:

The report uses both VBScript and JavaScript (client side, that is why security warning will occur) and the components enclosed is free to use.

Good luck! I am off sailing this weekend so don't expect any support for a few days :-)

Test automation report

2008-06-04T14:13:00.004+02:00

Ever since my first QTP crash back in early 2000 I have disliked the QTP native report. The layout and content is OK but the fact that the result is not available until after the test run (and if QTP did not crash) is not satisfying according to my reuqirements in automation.

So about a year ago I began to log test result outside QTP so the it becomes available during run-time. When switching from Excel to SQL Server I did some web GUIs so it would be easier to follow the test progress...and now my latest report is looking quite good I think (it reminds a bit of QTP's native):

The concept is simply like this:

1. Read selected data from db
2. Render frames based on the data using JavaScript and VBScript (the pie chart requires Flashplayer)
3. Press update or refresh to get the latest results (if the test is running)

It is based on my test case data model and I use a JS Tree to generate test case nodes:

If you click on a test case all related data is shown in the main frame as well as clicking on test suite node (one report can have one or more test suites).

Does this look interesting? I will make an deidentified code copy and publish it later. The report does not require QTP and can be used with any tool which allows custom reporting. Must of the job is to implement a good custom reporting (prefeble to a database) from your test tool and have a good test case data model.

Test suite driver script

2008-05-29T13:16:00.004+02:00

As a follow-up to this week posts I thought I should share my automation test driver script. This contains my test case driver logic. All administration of the tests are handled via Web pages and the data is in an SQL Server database. The script is in the Script.mts file in a QTP Test and can be used in QTP (obvious) and for non-GUI tests via vbs standalone as shown in yesterdays post.


'**************************************
'***    test suite driver script    ***
'**************************************

'Get and set test run options for QTP
If intTestRunOptionID = "" Then intTestRunOptionID = 1

'Execute main
Main intTestRunOptionID

Sub Main (intTestRunOptionID)

    Set objTestRun = GetTestRunOptions (intTestRunOptionID)
   
    'Check Global run status and stop test if requested
    If objTestRun("TEST_RUN") = "True" Then
        'Start test and report
        ResetTestRun intTestRunOptionID, "TestStart"
        Else
        If objTestRun("TEST_RUN_TOOL") = "QuickTestPro" Then CloseAllBrowsers
        Exit Sub
    End If
   
    '*** Handle Result Records ****
    Select Case objTestRun("TEST_RESULT_OPTION")
        Case "New"
            'Create new result set
            intTestRunResultID = CreateAndUpdateNewResult (intTestRunOptionID, _
            "TestRun_" & Now(), objTestRun("REPORT_TO_NET") )
            'Create summary result
            CreateSummaryResult intTestRunResultID, intTestRunOptionID
            'Create result rows
            CreateResultRows intTestRunResultID, intTestRunOptionID
        Case "Resume"
            intTestRunResultID = objTestRun("TEST_RUN_RESULT_ID")
    End Select
   
    'Initialize
    blnAllTestRunsExecuted = False
   
    Do Until blnAllTestRunsExecuted
   
        'Get and set global test run data
        Set objTCGlobalParameter = GetGlobalTestParameters (intTestRunResultID)
        If objTCGlobalParameter("TEST_RUN_RESULT_SUMMARY_ID") = "" Then
            blnAllTestRunsExecuted = True
        End If
       
        If blnAllTestRunsExecuted Then
            objTCGlobalParameter("TEST_RUN_STATE") = "finished"
            Exit Do
            Else
            objTCGlobalParameter("TEST_RUN_STATE") = "started"
        End If
       
        'Initialize
        blnAllTestCasesExecuted = False
       
        '*** Open application and/or get version***
        If objTestRun("TEST_RUN_TOOL") = "QuickTestPro" Then
            objTCGlobalParameter("Version") = Trim(OpenApplication _
            (objTCGlobalParameter("TEST_ENVIRONMENT_ALIAS"), "TestHost", "MyAUT", "Start"))
            Else
            objTCGlobalParameter("Version") = GetAUTVersion _
            (objTCGlobalParameter("TEST_ENVIRONMENT_ALIAS"))
        End If
        objTCGlobalParameter("Release") = GetRelease (objTCGlobalParameter("Version"))       

        'Update summary with version
        UpdateSummary objTCGlobalParameter("TEST_RUN_RESULT_SUMMARY_ID"), "TEST_RUN_VERSION", _
        objTCGlobalParameter("Version")
       
        If objTestRun("EMAIL_OPTION") = "True" Then
            objTestRun("EMAIL_SUBJECT") = GetTestSuiteSummary (objTCGlobalParameter)
            objTestRun("HTML_BODY") = GetTestCaseResultRowsHTML(objTCGlobalParameter)
            SendEmail objTestRun
        End If
       
        '*** Execute Test Cases ***
        Do Until blnAllTestCasesExecuted
            'Check Global run status and stop test if requested
            Set objTestRun = GetTestRunOptions (intTestRunOptionID)
            If objTestRun("TEST_RUN") = "True" Then
                'Continue
                Else
                If objTestRun("TEST_RUN_TOOL") = "QuickTestPro" Then
                    CloseAllBrowsers
                End If
                objTCGlobalParameter("TEST_RUN_STATE") = "stopped"
                If objTestRun("EMAIL_OPTION") = "True" Then
                    objTestRun("EMAIL_SUBJECT") = GetTestSuiteSummary (objTCGlobalParameter)
                    objTestRun("HTML_BODY") = GetTestCaseResultRowsHTML(objTCGlobalParameter)
                    SendEmail objTestRun
                End If
                'Reset TestRun option
                ResetTestRun intTestRunOptionID, "TestStop"
                Exit Sub
            End If
            'Get and set test scenario data
            Set objCurrentTestProperties = GetCurrentTestProperties _
            (intTestRunResultID, objTCGlobalParameter("TEST_SUITE_ID"))
            If objCurrentTestProperties("TEST_RUN_RESULT_ROW_ID") = "" Then
                blnAllTestCasesExecuted = True
            End If
            If blnAllTestCasesExecuted Then
                Exit Do
            End If
            'Get reference test data
            Set objReferenceTestData = GetTestReferenceData _
            (objCurrentTestProperties("TEST_RUN_RESULT_ROW_ID"))           
            'Get test case parameters
            Set objTCParameter = GetTestCaseParameters _
            (objCurrentTestProperties("TEST_CASE_ID"))
           
            'Merge all parameters into one Dictionary
            Set objTCParameter = MergeDictionaries (objTCParameter, objTCGlobalParameter)
            Set objTCParameter = MergeDictionaries (objTestRun, objTCParameter)
            Set objTCParameter = MergeDictionaries (objTCParameter, objCurrentTestProperties)
            Set objTCParameter = MergeDictionaries (objTCParameter, objReferenceTestData)
   
            'Clean up
            Set objCurrentTestProperties = Nothing
            Set objReferenceTestData = Nothing
   
            'Report test case start
            UpdateTestCaseStart objTCParameter
            If objTCParameter("TEST_RUN_TOOL") = "QuickTestPro" Then
                'Initalize report dictionary
                Set dicMetaDescription = CreateObject("Scripting.Dictionary")
                dicMetaDescription("Status") = micDone 'Default value
                dicMetaDescription("PlainTextNodeName") = _
                "Test Case#" & objTCParameter("TEST_SCENARIO_ID") & "." & objTCParameter("TEST_CASE_ID") & " - " & _
                objTCParameter("TEST_CASE_DESCRIPTION")
                dicMetaDescription("StepHtmlInfo") = objTCParameter("TEST_CASE_CATEGORY_NAME") & _
                "<BR><BR>" & objTCParameter("TEST_CASE_DESCRIPTION")
                'Some backdoor settings
                dicMetaDescription("DllIconIndex") = 206
                dicMetaDescription("DllIconSelIndex") = 206
                dicMetaDescription("DllPAth") = _
                "C:\Program Files\Mercury Interactive\QuickTest Professional\bin\ContextManager.dll"
                intContext = Reporter.LogEvent("User", dicMetaDescription, Reporter.GetContext)
                'Set test case node as a parent
                Reporter.SetContext intContext
            End If
            'Log all input parameters
            If objTCParameter("TEST_RUN_TOOL") = "QuickTestPro" Then
                strTestCaseParameters = ""
                allKeys = ArraySort_asc(allKeys)'Sort
                For i = 0 to UBound(allKeys)
                    strTestCaseParameters =  strTestCaseParameters & "<TR><TD><B>" & _
                    allKeys(i) & "</B></TD><TD> " & objTCParameter(allKeys(i)) & "</TD></TR>"
                Next
                dicMetaDescription("PlainTextNodeName") = "Input parameters"
                dicMetaDescription("StepHtmlInfo") = "<DIV align=left><TABLE STYLE=""font-size: 8pt"">" & _
                strTestCaseParameters & "</TABLE></DIV>"
                Reporter.LogEvent "User", dicMetaDescription, Reporter.GetContext
            End If
   
            '**********************************************************
            '****** Execute test case and store output in object ******
            Set objTCData = ExecuteTestCase (objTCParameter)'**********
            '**********************************************************
            '**********************************************************

            'Save output values
            objTCData("TEST_RUN_RESULT_ROW_ID") = objTCParameter("TEST_RUN_RESULT_ROW_ID")
            UpdateTestCaseOutParameters objTCData

            'Log all output parameters and test result
            If objTCParameter("TEST_RUN_TOOL") = "QuickTestPro" Then
                allKeysOut = objTCData.Keys   'Get all the keys into an array
                allValuesOut = objTCData.Items   'Get all the items into an array
                strTestCaseParameters = ""
                allKeysOut = ArraySort_asc(allKeysOut)'Sort
                For i = 0 to UBound(allKeysOut)
                    If allValuesOut(i) <> "" Then'Log parameters with values only
                        strTestCaseParameters =  strTestCaseParameters & "<TR><TD><B>" & allKeysOut(i) & _
                        "</B></TD><TD> " & objTCData.Item(allKeysOut(i)) & "</TD></TR>"
                    End If
                Next
                'Set test case result
                Select Case objTCData("TEST_CASE_RESULT")
                    Case "Passed"
                        dicMetaDescription("Status") = micPass
                    Case "Failed"
                        dicMetaDescription("Status") = micFail
                    Case "Warning"
                        dicMetaDescription("Status") = micWarning
                    Case Else
                        dicMetaDescription("Status") = micDone
                End Select
                dicMetaDescription("PlainTextNodeName") = "Output parameters"
                dicMetaDescription("StepHtmlInfo") = "<DIV align=left><TABLE STYLE=""font-size: 8pt"">" & _
                strTestCaseParameters & "</TABLE></DIV>"
                Reporter.LogEvent "User", dicMetaDescription, Reporter.GetContext
            End If
   
            'Merge data
            Set objTCParameter = MergeDictionaries (objTCParameter, objTCData)
   
            'Report test case result
            UpdateTestCaseStop objTCParameter
            StoreTestCaseResult objTCParameter
        
            'In case of failure
            If objTCParameter("TEST_CASE_RESULT") = "Failed" Then
                'Set UI type
                If objTCParameter("TEST_CASE_GROUP_ID") = "5" Then
                    objTCParameter("TEST_INTERFACE") = "API"
                End If
                Set objError = CreateErrorFiles (objTCParameter)
                Set objTCParameter = MergeDictionaries (objTCParameter, objError)
                If objTestRun("EMAIL_OPTION") = "True" Then
                    objTestRun("EMAIL_SUBJECT") = "Test case ID:" & objTCParameter("TEST_CASE_ID") & " - " & _
                    objTCParameter("TEST_CASE_DESCRIPTION") & " failed"
                    objTestRun("HTML_BODY") = GetTestCaseErrorHTML(objTCParameter)
                    SendEmail objTestRun
                End If
                ElseIf objTCParameter("TEST_CASE_RESULT") = "Warning" Then
                    If objTestRun("TEST_RUN_TOOL") = "QuickTestPro" Then CloseAllBrowsers
                    objTCGlobalParameter("TEST_RUN_STATE") = "stopped due to Warning"
                If objTestRun("EMAIL_OPTION") = "True" Then
                    objTestRun("EMAIL_SUBJECT") = GetTestSuiteSummary (objTCGlobalParameter)
                    objTestRun("HTML_BODY") = GetTestCaseResultRowsHTML(objTCGlobalParameter)
                    SendEmail objTestRun
                End If
                'ExitTest
                Exit Sub               
            End If

            'Reset QTP report node
            If objTestRun("TEST_RUN_TOOL") = "QuickTestPro" Then Reporter.UnSetContext
   
            'Reset test data
            Set objTCData = Nothing
   
             'Reset test case data
            Set objTCParameter = Nothing
           
        Loop 'Get next test case
       
        'Close all browsers
        If objTestRun("TEST_RUN_TOOL") = "QuickTestPro" Then
            CloseAllBrowsers
        End If
       
        objTCGlobalParameter("TEST_RUN_STATE") = "finished"
        If objTestRun("EMAIL_OPTION") = "True" Then
            objTestRun("EMAIL_SUBJECT") = GetTestSuiteSummary (objTCGlobalParameter)
            objTestRun("HTML_BODY") = GetTestCaseResultRowsHTML (objTCGlobalParameter)
            SendEmail objTestRun
        End If
       
        'Update Test summary
        UpdateSummary objTCGlobalParameter("TEST_RUN_RESULT_SUMMARY_ID"), "TEST_RUN_STATUS", "Complete"
        UpdateSummary objTCGlobalParameter("TEST_RUN_RESULT_SUMMARY_ID"), "TEST_RUN_STOP", Cstr(Now())
   
        'Clean
        Set objTCGlobalParameter = Nothing
       
    Loop 'Get next test run
   
    'Reset TestRun option
    ResetTestRun intTestRunOptionID, "TestDone"

End Sub

'***********
'*** END ***
'***********

Sharing Code base with QTP

2008-05-28T12:43:00.002+02:00

As I mentioned before some test automation can be executed via a vbs stand-alone file instead of using QTP. How do you do that without the need of multiple code basis or branches?

The solution (the function getTextFileContent can be found in a previous post):


'************************************
'*** AUT test suite driver script ***
'*** vbs standalone version       ***
'*** by Stefan Thelenius May 2008 ***
'************************************

'Load common functions from library
Execute getTextFileContent("\\server\path\libraryFileName","Unicode")

'Set initial test run option parameter
strRunCode = "intTestRunOptionId = 2" & vbLF

'Load QTP test code
strRunCode = strRunCode & _
getTextFileContent("\\server\path\yourQTPTestName\Action1\Script.mts","utf-8")

'Execute test
Execute strRunCode

Pros:
Faster execution time
Licence free

Cons:
No run-time debugging
Not possible to use QTP features (Test object model...)
Need to implement parameter "TestTool" (If objParameter("TestTool") = "QTP" Then) for all QTP specific code lines, for example using the QTP native Reporter object.

I use it for Web Service testing and some URL bases web testing. Those test can executed via QTP as well using this concept (just change one parameter) so I usally run the test via QTP at first so debugging is possible and when stable I use my vbs-file instead.

QTP: Test execution performance

2008-05-27T11:38:00.003+02:00

One of my Web Service test suites has 174 test cases and it takes QTP 7 minutes to execute them all. At first it may seems to be a resonable amount of time, especially compared to simular GUI operations which would have taken hours in test execution.

When I execute the same test suite running as a vbs-file instead I'm down to 1 minute and 45 seconds!

What are the differences? The code is the same apart from that all QTP specific log functions like Reporter are replaced by custom logging to files (which should be slower since 1 folder and 2 xml-files are created for each test case).

My conclusion is that QTP has a lot of overhead features which affects test execution performance in a negative way, especially considering that must features are not that useful in the long run (at least not for me).

Ps. My QTP settings are Run mode "fast" and no QTP recovery scenario is involved

Handling of different AUT versions/configurations in automation

2008-05-23T11:48:00.003+02:00

My main AUT is a standard system for pension, insurance and fund administration. Since business logic is complex for those areas, the AUT is also a very complex system with lots of functionality which normally for a typical major finance company are distributed at several different systems.

The AUT has a release cycle of at least 4 major releases/year and some test object properties changes for each release. In addition the AUT has different customer configurations which in worst case results in slightly different ways to handle the business logic.

How to handle that situation in automation in a cost effective way?

Well, totally unaware of what best practise is in this area I have implemented the following solution:

1. One of my favorite developers did on my request a web service where you get the AUT release number.
2. Using the WS I begin each test suite execution to store the release number into a parameter which is passed to most functions via a Dictionary (see prevoius posts).
3. Each test case data set has a customer configuration id which is also becomes a "global" parameter.
4. Within each test case code I use the parameters to handle release/customer specific behaviour by using if or select statements (not nice but it works)

If objParameter("Release") <= "2.3.1" Then
'Do the old way
Else
'Do the new way
End If

5. When a release no longer is supported (after a year or so) I remove statements which is absolete.

In some rare cases I have to branch a test scenario data set (which is a set of nested test cases) if for example the test case execution order changes due to major change of business logic.

Fortunatly in most cases a generic behaviour/test properties works for all releases and customer configurations...which is main idea for a standard system :-)

One advantage with this approach is that I need only one set of "production" automation code (no code branching which I guess is the main alternative) which can be executed to different test environments containing different versions/configurations at any time.

Since we also handle AUT service updates between the major releases for each customer this is quite handy I think since it allows for backward automation compability as well as new automation features does not have to be implemented to several code basis.

"...test data independant"

2008-05-08T10:46:00.002+02:00

"...test data independant" (from post last week)

What do I mean by that ?

In automation, I try to avoid hard-coded test data which depends on whether the AUT database is in "start-position" or not.

Mostly I create test data during run-time using random values or getting random data from the AUT database which match my critera for the specific test case.

Consider a test case where you want to change an address for example. Here is an approach you could use (in pseudo-code]:

Select top 1 [myAddressColumns] from ADDRESS where [myCritera = True] order by newid() 'Note! Newid() MS SQL Server specificOptional
If blnNoAddressFound Then call createAddress 'In case of empty database call test case/method for create an address

Advantages:
Test case can be executed to any AUT database (empty or full)
You get test data variation using random values

Disadvantages:
It is more difficult to implement compare methods (but not impossible)

SQL - a tester's best friend?

2008-04-25T10:41:00.005+02:00

If you work with applications which has a database, good skills in SQL are required in order to be an efficiant tester (unless you sitting next to a friendly DBA).

With SQL (and other database operations) you can:

1. Find, change, create or delete test data
2. Do backup and restore of databases
3. Plus a lot of other useful things, for example you learn much about the application through the database/data model

If you cannot perform 1 and 2 by yourself you will encounter serious bottlenecks in your testing progress.

So, what you need are:

1. Learn basic SQL (select)
2. Get proper account settings in your test environment(s) database(s) - read only
3. Learn advanced SQL (join, update, insert, store procedures, backup/restore, temp tables etc)
4. Get proper account settings in your test environment(s) database(s) - SA or at least anything close

Beware, you must prove yourself worthy for the DBA to achieve no. 4

I went from stage 1 to 4 in a few months just by self-study articles and other people's SQL. I am not considering myself being a skilled DBA but for my testing purposes I am doing quite OK. Also in automation knowing SQL is priceless in order to get your automation test case dynamic and testdata independant. Further more you can build a lot of AUT specfic test tools which will improve your testing dramaticly.

Over and out...

ps.

Here is a list of my favourite database engines which I have worked with in my testing over the years (ranked from a user friendly perspective):

1. Microsoft SQL Server 2005
2. Oracle (don't remember version no)
3. DB2 (don't remember version no)

VBScript: Reading text files

2008-04-11T10:23:00.002+02:00

Until now I have used the FileSystemObject (FSO) when reading text files in automation but this week I wanted to read a utf-8 encoded file which did not work using FSO. I used my standard approach dealing with this kind of problem:

1. Collect information about the issue (mainly using Google)
2. Analyze information and design solution
3. Implement solution

So I found another COM-object which can be used reading files which seems to manage most of the Char Sets I normally deal with: Adodb.Stream

The result (getFileContent.vbs):


Function getTextFileContent (strFileName, strCharSet)
Const adTypeBinary = 1 'not used
Const adTypeText = 2
'Set default CharSet
If strCharSet = "" Then strCharSet = "ASCII"
' *** CharSets ***
'    Windows-1252
'    Windows-1257
'    UTF-16
'    UTF-8
'    UTF-7
'    ASCII
'    X-ANSI
'   iso-8859-2
Set objStreamFile = CreateObject("Adodb.Stream")
With objStreamFile
    .CharSet = strCharSet
    .Type= adTypeText
    .Open
    .LoadFromFile strFileName
    getTextFileContent = .readText
    .Close
End With
Set objStreamFile = Nothing
End Function

QTP: Debug class for Dictionary

2008-04-02T10:26:00.006+02:00

As I mentioned earlier, I use the Dictionary object a lot in my test framework. However it requires some extra efforts when debugging in QTP compared to ordinary variables since your Keys and Items are not visable by default in Debug viewer's Variables tab. You either manually add Keys to the Watch tab or add extra code where your Dictionary Keys and Items are stored in an array:

arrKeys = objDictionary.Keys
arrValues = objDictionary.Items

The last couple of days I have tried to learn a bit about Classes in VBScript and I noticed that a class public variables are available in QTP's Debug Variables tab in alphabetic order. That made me come up with this crazy idea to maybe replace the use of Dictionary object with a custom class instead...but Classes in VBScript seems a bit limited according to my requirements. However, for debugging purposes it could be nice to create a "Class copy object" of a Dictionary:

'Create a dictionary object and add some parameters
Set objDictionary = CreateObject("Scripting.Dictionary")
objDictionary.Add "A_Parameter","47"
objDictionary.Add "Z_Parameter",""
objDictionary.Add "C_Parameter","11"
'Create a class out of the dictionary and a new debug object
Execute CreateClass("DebugDictionary", objDictionary)
Set objDictionaryDebug = New DebugDictionary
'Display A_Parameter value
Msgbox objDictionaryDebug.C_Parameter
Set objDictionaryDebug = Nothing
Set objDictionary = Nothing


Function CreateClass (strClassName, objDictionary)
'* Create Class string from Dictionary object
'* By Stefan Thelenius 2008-04-02
'* Usage example: Execute CreateClass("MyClass", objDictionary)
'Get keys and values into arrays
arrKeys = objDictionary.Keys
arrValues = objDictionary.Items
For i = 0 to UBound(arrKeys)
    'Create Public variables
    strClassPublicVar = strClassPublicVar & "Public " & arrKeys(i) & vbLf
    'Set default values to variables
    strClassPublicInit = strClassPublicInit & arrKeys(i) & "=""" & arrValues(i) & """" & vbLf
Next
'Add number of Keys as Count Int
strClassPublicVar = strClassPublicVar & "Public Count" & vbLf
strClassPublicInit = strClassPublicInit & "Count=" & UBound(arrKeys)+1 & vbLf
'Set Class and initializer
strClassStart = "Class " & strClassName & vbLf
strClassEnd = "End Class"
strClassInitStart = "Private Sub Class_Initialize" & vbLf
strClassInitEnd = "End Sub" & vbLf
'Return class string
CreateClass = strClassStart & strClassPublicVar & strClassInitStart & strClassPublicInit & _
    strClassInitEnd & strClassEnd
End Function

Above code example is available for download here: http://abouttesting.fileave.com/DictionaryClass.vbs

When using QTP the debug view looks like this:

XML automation - Part V - Validate XML response

2008-03-25T09:42:00.011+01:00

This is the last part...but I will try to summarize the posts in an article later on...maybe I will add more features, for example how to handle attributes and multiple elements.

All the example code and files are available in compressed format here for download.

Please note that I have changed the test case parameter ConvertTempResult to #ConvertTempResult in order to mark up output parameters.

Here is the function for response validation:



Function validateXMLResponse (objDictionary)

Set xmlDoc = CreateObject("Msxml2.DOMDocument")

'loadXML

xmlDoc.loadXML(objDictionary.Item("XMLResponse"))

'Check XML syntax

If (xmlDoc.parseError.errorCode <> 0) Then

Set myErr = xmlDoc.parseError

objDictionary.Item("XMLResponseResult") = "Failed"

objDictionary.Item("XMLResponseResultDetails") = "ERROR: " & myErr.reason

Set validateXMLResponse = objDictionary

Exit Function

End If

Set root = xmlDoc.documentElement

'Put all test parameters in array

arrAllKeys = objDictionary.Keys

arrAllValues = objDictionary.Items

'Update values

For i = 0 to Ubound(arrAllKeys)

'Only process #-type parameters

If Left(arrAllKeys(i),1) = "#" Then

strParameterElementName = arrAllKeys(i)

strParameterElementName = Right(strParameterElementName,Len(strParameterElementName)-1)

strParameterElementValueExpected = arrAllValues(i)

'Get element and value

Set node = root.getElementsByTagName(strParameterElementName)

If node.Length > 0 Then

For n = 0 to node.Length-1

'Return actual value in [element]_Out

objDictionary.Item(strParameterElementName & "_Out") = node.Item(n).Text

strParameterElementValueActual = node.Item(n).Text

'Set result

If strParameterElementValueExpected = strParameterElementValueActual AND _

objDictionary.Item("XMLResponseResult") <> "Failed" Then

objDictionary.Item("XMLResponseResult") = "Passed"

Else

objDictionary.Item("XMLResponseResult") = "Failed"

objDictionary.Item("XMLResponseResultDetails") = objDictionary.Item("XMLResponseResultDetails") & _

strParameterElementName & " = " & strParameterElementValueActual & " , expected value " & _

strParameterElementValueExpected & " ;" 'use ; as separator

End If

Next

End If

Set node = Nothing

End If

Next

'Return

Set validateXMLResponse = objDictionary

'Clean up

Set root = Nothing

Set xmlDoc = Nothing

End Function

So finally we have these steps all together:


' Example of XML/Web Service automation in VBScript using WinHttp, XML DOM and ADO

' by Stefan Thelenius march 2008



'Set test case id and data source file

intTestCaseID = 1

strMyDataFile = "c:\MyTestData.xls"



'Get test case parameters

Set objMyTestParameters = getTestParametersIntoDictionary (strMyDataFile, _

"TestCase", "TestCaseID", intTestCaseID)



'Get global parameters

Set objMyGlobalParameters = getTestParametersIntoDictionary (strMyDataFile, _

"Global", "GlobalID", objMyTestParameters.Item("GlobalID"))



'Merge parameters

Set objMyTestParameters = mergeDictionaries (objMyGlobalParameters, objMyTestParameters)



'Create request from template

objMyTestParameters.Item("XMLRequest") = getXMLTemplate (objMyTestParameters.Item("XMLTemplateFile"), _

objMyTestParameters.Item("XMLTemplate"))



'Insert test parameters

objMyTestParameters.Item("XMLRequest") = setTestParametersInXMLRequest (objMyTestParameters)



'Send request

objMyTestParameters.Item("XMLResponse") = sendXMLRequest (objMyTestParameters)



'Validate response

Set objMyTestParameters = validateXMLResponse (objMyTestParameters)



'Display output

Msgbox objMyTestParameters.Item("XMLResponse")

Msgbox objMyTestParameters.Item("XMLResponseResult")

Msgbox objMyTestParameters.Item("XMLResponseResultDetails")

Msgbox objMyTestParameters.Item("ConvertTempResult_Out")

AdvancedQTP

2008-03-19T20:08:00.002+01:00

I am joining forces with the team behind AdvancedQTP as a moderator for some of their forums. The site is an excellent resource regarding QTP and vbScript and I strongly recommend it as first stop knowledge base since HP/Mercury's dito is hard to find these days...

XML automation - Part IV - Sending XML request

2008-03-11T13:57:00.006+01:00

This week it is time to send the Web Service request and recieve a response...I have also added some Global parameters in a new sheet in my Excel file so that we can handle test environment properties and so on:

A also added a function for merging Dictionary objects which I use sometimes in my framework. It might not be the best code design approach but I think the concept works well, create a number of dictionaries, merge them together and pass one dictionary to the test case including all test parameters.

Function mergeDictionaries (objDIC1, objDIC2)
If IsObject(objDIC2) Then
    allKeys = objDIC2.Keys
    allItems = objDIC2.Items
    For i = 0 To objDIC2.Count - 1 'Iterate through the array
        If objDIC1.Exists(allKeys(i)) Then
            'Value in objDIC2 is master
            objDIC1.Item(allKeys(i)) = allItems(i)
            Else
            objDIC1.Add allKeys(i),allItems(i)
        End If
    Next
End If
Set mergeDictionaries = objDIC1
End Function

And finally the funcion for sending the request


Function sendXMLRequest (objDictionary)
Dim WinHttpReq
Const HTTPREQUEST_SETCREDENTIALS_FOR_SERVER = 0
Set WinHttpReq = CreateObject("WinHttp.WinHttpRequest.5.1")
WinHttpReq.Open "POST", objDictionary.Item("PostURL"), False
WinHttpReq.setRequestHeader "Content-Type", "text/xml;charset=utf-8"
WinHttpReq.setRequestHeader "Accept", _
"text/xml, multipart/related, text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2"
WinHttpReq.setRequestHeader "Content-Transfer-Encoding", "binary"
WinHttpReq.setRequestHeader "Connection", "keep-alive"
WinHttpReq.setRequestHeader "SOAPAction", objDictionary.Item("SOAPAction")
If objDictionary.Item("UserName") <> "" Then    
    WinHttpReq.SetCredentials objDictionary.Item("UserName"), objDictionary.Item("Password"), _
    HTTPREQUEST_SETCREDENTIALS_FOR_SERVER
End If
'Add WS header and foot if SOAP
If objDictionary.Item("RequestType") = "SOAP" Then
    strHeader = "<?xml version=""1.0"" encoding=""utf-8""?>" &_
        "<soap:Envelope xmlns:xsi=""http://www.w3.org/2001/XMLSchema-instance""" &_
        " xmlns:xsd=""http://www.w3.org/2001/XMLSchema""" &_
        " xmlns:soap=""http://schemas.xmlsoap.org/soap/envelope/"">" &_
        "<soap:Body>"
    strFoot =  "</soap:Body></soap:Envelope>"
End If
WinHttpReq.Send(strHeader & objDictionary.Item("XMLRequest") & strFoot)
sendXMLRequest = WinHttpReq.ResponseText
End Function

So all together it should look like this:


intTestCaseID = 1
Set objMyTestParameters = getTestParametersIntoDictionary ("c:\MyTestData.xls", _
"TestCase", "TestCaseID", intTestCaseID)
Set objMyGlobalParameters = getTestParametersIntoDictionary ("c:\MyTestData.xls", _
"Global", "GlobalID", objMyTestParameters.Item("GlobalID"))
Set objMyTestParameters = mergeDictionaries (objMyGlobalParameters, objMyTestParameters)
objMyTestParameters.Item("XMLRequest") = getXMLTemplate (objMyTestParameters.Item("XMLTemplateFile"), _
objMyTestParameters.Item("XMLTemplate"))
objMyTestParameters.Item("XMLRequest") = setTestParametersInXMLRequest (objMyTestParameters)
objMyTestParameters.Item("XMLResponse") = sendXMLRequest (objMyTestParameters)
Msgbox objMyTestParameters.Item("XMLResponse")

Up next: Part V - Validate the response

XML automation - part III (test parameters)

2008-03-03T14:17:00.004+01:00

Now it is time to set some test parameters in our XML template (see previous post). In my test framework I handle test parameters in an SQL Server database but in my examples here I use good old Excel instead.

First, create and save an Excel file with your test parameters:

Second, new functions for reading parameters into Dictionary Object and set parameters in an xml:

strXMLTemplateFile = "c:\Templates.xml"
intTestCaseID = 1
Set objMyTestParameters = getTestParametersIntoDictionary ("c:\MyTestData.xls", "TestCase", "TestCaseID", intTestCaseID)
objMyTestParameters.Item("XMLRequest") = getXMLTemplate (strXMLTemplateFile, objMyTestParameters.Item("XMLTemplate"))
objMyTestParameters.Item("XMLRequest") = setTestParametersInXMLRequest (objMyTestParameters)
Msgbox objMyTestParameters.Item("XMLRequest")

Function getTestParametersIntoDictionary (strFileName, strSheetName, strKeyColumn, intKeyID)
'Initiate Dictionary
Set getTestParametersIntoDictionary = CreateObject("Scripting.Dictionary")
'Initiate Database connection
Set objDB = CreateObject("ADODB.Connection")
'Open connection
objDB.Open "Provider=Microsoft.Jet.OLEDB.4.0;Data Source="&strFileName&_
";Extended Properties=""Excel 8.0;HDR=Yes;IMEX=1"";"
strQuery="select * from [" & strSheetName & "$] where " & strKeyColumn & "=" & intKeyID
'Create a record set, execute query and store in dictionary
Set objRec = objDB.Execute(strQuery)
Do Until objRec.EOF
    For Each x In objRec.Fields
        'Convert NULL to empty string or clean from spaces
        strCurrentValue = x.Value
        If IsNull(strCurrentValue) Then
            strCurrentValue = ""
            Else
            strCurrentValue = Trim(strCurrentValue)
        End If
        'Add to dictionay
        getTestParametersIntoDictionary.Add Trim(x.Name), strCurrentValue
    Next
    objRec.MoveNext
Loop
'Clean up
objRec.Close
Set objRec = Nothing
objDB.Close
Set objDB = Nothing
End Function

Function setTestParametersInXMLRequest (objDictionary)
Set xmlDoc = CreateObject("Msxml2.DOMDocument")
'loadXML
xmlDoc.loadXML(objDictionary.Item("XMLRequest"))
'Check XML syntax
If (xmlDoc.parseError.errorCode <> 0) Then
   Set myErr = xmlDoc.parseError
   Msgbox "ERROR: " & myErr.reason
End If
Set root = xmlDoc.documentElement
'Put all test parameters in array
arrAllKeys = objDictionary.Keys
arrAllValues = objDictionary.Items
'Update values
For i = 0 to Ubound(arrAllKeys)
    'Only process $-type parameters and where values <> ""
    If Left(arrAllKeys(i),1) = "$" AND arrAllValues(i) <> "" Then
        strParameterElementName = arrAllKeys(i)
        strParameterElementName = Right(strParameterElementName,Len(strParameterElementName)-1)
        strParameterElementValue = arrAllValues(i)
        'Insert element and value
        Set node = root.getElementsByTagName(strParameterElementName)
        If node.Length > 0 Then
            For n = 0 to node.Length-1
                'Replace all
                node.Item(n).Text = strParameterElementValue
            Next
        End If
        Set node = Nothing
    End If
Next
'Return xml request
setTestParametersInXMLRequest = root.xml
'Clean up
Set root = Nothing
Set xmlDoc = Nothing
End Function

Up next: Part IV - Sending XML request

XML automation Part I and II re-make

2008-03-03T10:42:00.006+01:00

I found some public Web Services the other day (http://www.webservicex.net/) and thought using them in my examples would make it a bit more realistic...

So I have reworked my previous examples:

Test client

<html>
<!--
*********************************************************************************************************************
Purpose:              To send XML http request.
Assumptions:          Proper security settings in browser and DLL for WinHttp5.1 object
Author:                  Stefan Thelenius 2008-03-03.
'********************************************************************************************************************
-->
<head>
<title>Web Service Test Tool</title>
<script language="JavaScript">
self.resizeTo(screen.width,screen.height)
</script>
</head>
<body>
<style>
select {font-family: verdana; font-size: 8pt; font-weight: 500;}
input {font-family: verdana; font-size: 8pt; font-weight: 500;}
textarea {font-family: verdana; font-size: 8pt; font-weight: 500;}
</style>
<FONT FACE="verdana" SIZE=3>
<center>Web Service Test Tool</center>
<FONT FACE="verdana" SIZE=1>
<FORM NAME="form">
<SCRIPT LANGUAGE="vbscript">
document.write("<table width=""100%"" border=""0"" cellspacing=""0"" cellpadding=""0"">")
document.write("<tr>")
document.write("<td><left>")
document.write("<FONT FACE=""verdana"" SIZE=1>URL&nbsp")
document.write("<INPUT TYPE=""text"" NAME=""url"" SIZE=70 TITLE=""URL"">")
document.write("<FONT FACE=""verdana"" SIZE=1>&nbspSOAPAction&nbsp")
document.write("<INPUT TYPE=""text"" NAME=""soap"" SIZE=70 TITLE=""SOAP Action"">")
document.write("</td></tr>")
document.write("<tr>")
document.write("<td><left>")
document.write("<FONT FACE=""verdana"" SIZE=1>Username&nbsp")
document.write("<INPUT TYPE=""text"" NAME=""username"" SIZE=20 TITLE=""UserName"">")
document.write("<FONT FACE=""verdana"" SIZE=1>&nbsp&nbspPassword&nbsp")
document.write("<INPUT TYPE=""password"" NAME=""password"" SIZE=12>")
document.write("</left>")
document.write("<FONT FACE=""verdana"" SIZE=1>&nbsp&nbspResponse to Clipboard")
document.write("<input type=""checkbox"" name=""blnSave"">")
document.write("</td></tr>")
document.write("<br>")
document.write("<table width=""100%"" border=""0"" cellspacing=""0"" cellpadding=""0"">")
document.write("<tr>")
'Autofit request and response textarea
intWidth = screen.width / 13
intHeight = screen.height / 23
document.write("<td><FONT FACE=""verdana"" SIZE=1>Request<BR>")
document.write("<textarea name=""IN"" rows=""" & intHeight & """ cols=""" & intWidth & """>")
document.write("</textarea></td>")
document.write("<td><FONT FACE=""verdana"" SIZE=1>Response<BR>")
document.write("<textarea name=""OUT"" rows=""" & intHeight & """ cols=""" & intWidth & """>")
document.write("</textarea></td>")
document.write("</tr><tr><td>")
document.write("<INPUT TYPE=""button"" NAME=""cmdSend"" VALUE=""Send"" onclick=""SendRequest()"">")
document.write("&nbsp<INPUT TYPE=""button"" NAME=""cmdPretty"" VALUE=""Parse input"" onclick=""ParseInput()"">")
document.write("&nbsp<INPUT TYPE=""button"" NAME=""cmdClean"" VALUE=""Clean output"" onclick=""CleanResponse()"">")
document.write("</td><td id=""tid""><FONT FACE=""verdana"" SIZE=1>&nbsp</td>")
document.write("</tr></table></FORM>")
'Get current user as default
Dim objNet
Set objNet = CreateObject("WScript.NetWork")
form.username.Value = objNet.UserDomain & "\" & objNet.UserName
Set objNet = Nothing
Sub SendRequest
Dim WinHttpReq
Const HTTPREQUEST_SETCREDENTIALS_FOR_SERVER = 0
Set WinHttpReq = CreateObject("WinHttp.WinHttpRequest.5.1")
strStart = Timer
WinHttpReq.Open "POST", form.url.Value, False
WinHttpReq.setRequestHeader "Content-Type", "text/xml;charset=utf-8"
WinHttpReq.setRequestHeader "Accept", _
"text/xml, multipart/related, text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2"
WinHttpReq.setRequestHeader "Content-Transfer-Encoding", "binary"
WinHttpReq.setRequestHeader "Connection", "keep-alive"
WinHttpReq.setRequestHeader "SOAPAction", form.soap.Value
WinHttpReq.SetCredentials form.username.Value, form.password.Value, _
HTTPREQUEST_SETCREDENTIALS_FOR_SERVER
WinHttpReq.Send(form.IN.Value)
strStop = Timer
Set xmlDoc = CreateObject("Msxml2.DOMDocument")
'Parse output
xmlDoc.loadXML(WinHttpReq.ResponseText)
If (xmlDoc.parseError.errorCode <> 0) Then
   form.OUT.Value = WinHttpReq.ResponseText
   Else
   form.OUT.Value = xmlDoc.xml
End If
Set xmlDoc = Nothing
strTime = strStop - strStart
strTime = FormatNumber(strTime,2)
tid.Innertext = "Response time: " & strTime & " s"
Set WinHttpReq = Nothing
'Set clipboard = output
If form.blnSave.Value = True Then window.clipboardData.setData "Text",form.OUT.Value
End Sub
Sub ParseInput
Set xmlDoc = CreateObject("Msxml2.DOMDocument")
xmlDoc.loadXML(form.IN.Value)
If (xmlDoc.parseError.errorCode <> 0) Then
   Set myErr = xmlDoc.parseError
   Msgbox "ERROR: " & myErr.reason
   Set myErr = Nothing
   Set xmlDoc = Nothing
   Exit Sub
End If
form.IN.Value = xmlDoc.xml
Set xmlDoc = Nothing
End Sub
Sub CleanResponse
form.OUT.Value = ""
End Sub
</SCRIPT>
</body>
</html>

XML Templates (save into file c:\Templates.xml)

<?xml version="1.0" ?>
<Templates>
<myTemplate_01>
<ConvertTemp xmlns="http://www.webserviceX.NET/">
  <Temperature>10</Temperature>
  <FromUnit>degreeCelsius</FromUnit>
  <ToUnit>degreeFahrenheit</ToUnit>
</ConvertTemp>
</myTemplate_01>
<myTemplate_02>
<ConvertTemp xmlns="http://www.webserviceX.NET/">
  <Temperature>60</Temperature>
  <FromUnit>degreeFahrenheit</FromUnit>
  <ToUnit>degreeCelsius</ToUnit>
</ConvertTemp>
</myTemplate_02>
</Templates>

XML template function

strXMLTemplateFile = "c:\Templates.xml"
strMyTemplate = "myTemplate_01"
Msgbox getXMLTemplate (strXMLTemplateFile, strMyTemplate)
Function getXMLTemplate (strXMLTemplateFile, strMyTemplate)
Set xmlDoc = CreateObject("Msxml2.DOMDocument")
'loadXML
xmlDoc.load(strXMLTemplateFile)
'Check XML syntax
If (xmlDoc.parseError.errorCode <> 0) Then
   Set myErr = xmlDoc.parseError
   Msgbox "ERROR: " & myErr.reason
End If
Set root = xmlDoc.documentElement
'Select template and return
Set node = root.selectNodes("//" & strMyTemplate)
If node.Length > 0 Then
    getXMLTemplate = node.Item(0).childNodes.Item(0).xml
    Else
    getXMLTemplate = "No template found!"
End If
'Clean up
Set xmlDoc = Nothing
Set root = Nothing
Set node = Nothing
End Function

Code format...at last

2008-02-28T09:41:00.002+01:00

I appologize for the bad code format in my previous examples...but I have found a good site which can return well formated code: http://manoli.net/csharpformat/format.aspx

So from now on I hope my examples becomes more readable...

XML automation - part II (XML templates)

2008-02-25T07:47:00.008+01:00

When I automate XML requests I normally use templates for my requests.

Usage example:

<?xml version="1.0" ?>
<Templates>
<myTemplate_01>
<createNewSomething>
<anElement>somevalue</anElement>
<anotherElement>someothervalue</anotherElement>
</createNewSomething>
</myTemplate_01>
<myTemplate_02>
<createNewSomething>
<anElement>somevalue</anElement>
<anotherElement>someothervalue</anotherElement>
<optionalElement>yetanothervalue</optionalElement>
</createNewSomething>
</myTemplate_02>
</Templates>

So to use a template in an automated test case you need to pass a parameter which template to use

strXML = "<?xml version=""1.0"" ?><Templates><myTemplate_01>" &_
"<createNewSomething><anElement>somevalue</anElement>" &_
"<anotherElement>someothervalue</anotherElement>" &_
"</createNewSomething></myTemplate_01><myTemplate_02>" &_
"<createNewSomething><anElement>somevalue</anElement>" &_
"<anotherElement>someothervalue</anotherElement>" &_
"<optionalElement>yetanothervalue</optionalElement>" &_
"</createNewSomething></myTemplate_02></Templates>"
strMyTemplate = "myTemplate_01"
Msgbox getXMLTemplate (strXML, strMyTemplate)
Function getXMLTemplate (strXML, strMyTemplate)
Set xmlDoc = CreateObject("Msxml2.DOMDocument")
'loadXML
xmlDoc.loadXML(strXML)
'Check XML syntax
If (xmlDoc.parseError.errorCode <> 0) Then
  Set myErr = xmlDoc.parseError
  Msgbox "ERROR: " & myErr.reason
End If
Set root = xmlDoc.documentElement
'Select template and return
Set node = root.selectNodes("//" & strMyTemplate)
If node.Length > 0 Then
  getXMLTemplate = node.Item(0).childNodes.Item(0).xml
  Else
  getXMLTemplate = "No template found!"
End If
'Clean up
Set xmlDoc = Nothing
Set root = Nothing
Set node = Nothing
End Function

Up next: Part III - Insert test parameters into XML

XML automation - part I

2008-02-18T13:50:00.004+01:00

"...next mission is to build a test driver for our API which can be executed stand-alone in parallell with QTP"

Done!

I've recently got several requests on how to use the COM-object WinHttp, so I thought I would try to explain how I use the object in my test framework in a series of posts.

Part I - Building a simple client in order to send XML-request using WinHttp:

See html-source code below. I have removed some specific features (XML-templates and log handling) which I use internally.

Why this tool when there are several free tools out there? Well, it is a light-weight client (one html-file) which is easy to customize and it is a good way to get familiar with the WinHttp-object which you can use heavily in VBScript-based automation.

Up next: Part II - Using XML templates


<html>
<!-- 
*********************************************************************************************************************
Purpose:              To send XML http request.
Assumptions:          Proper security settings in browser and DLL for WinHttp5.1 object
Author:                  Stefan Thelenius 2008-02-18.
'********************************************************************************************************************
-->
<head>
<title>Web Service Test Tool</title>
<script language="JavaScript">
self.resizeTo(screen.width,screen.height)
</script>
</head>
<body>
<style>
select {font-family: verdana; font-size: 8pt; font-weight: 500;}
input {font-family: verdana; font-size: 8pt; font-weight: 500;}
textarea {font-family: verdana; font-size: 8pt; font-weight: 500;}
</style>
<FONT FACE="verdana" SIZE=3>
<center>Web Service Test Tool</center>
<FONT FACE="verdana" SIZE=1>
<FORM NAME="frmForm1">
<SCRIPT LANGUAGE="vbscript">
document.write("<table width=""100%"" border=""0"" cellspacing=""0"" cellpadding=""0"">")
document.write("<tr>")
document.write("<td><left>")
document.write("<FONT FACE=""verdana"" SIZE=1>URL&nbsp")
document.write("<INPUT TYPE=""text"" NAME=""url"" SIZE=70 TITLE=""URL"">")
document.write("<FONT FACE=""verdana"" SIZE=1>&nbsp&nbspUser&nbsp")
document.write("<INPUT TYPE=""text"" NAME=""username"" SIZE=20 TITLE=""UserName"">")
document.write("<FONT FACE=""verdana"" SIZE=1>&nbsp&nbspPassword&nbsp")
document.write("<INPUT TYPE=""password"" NAME=""password"" SIZE=12>")
document.write("</left>")
document.write("<FONT FACE=""verdana"" SIZE=1>&nbsp&nbspResponse to Clipboard<input type=""checkbox"" name=""blnSave"">")
document.write("</td></tr>")
document.write("<br>")
document.write("<table width=""100%"" border=""0"" cellspacing=""0"" cellpadding=""0"">")
document.write("<tr>")
'Autofit request and response textarea
intWidth = screen.width / 13
intHeight = screen.height / 23
document.write("<td><FONT FACE=""verdana"" SIZE=1>Request<BR>")
document.write("<textarea name=""IN"" rows=""" & intHeight & """ cols=""" & intWidth & """></textarea></td>")
document.write("<td><FONT FACE=""verdana"" SIZE=1>Response<BR>")
document.write("<textarea name=""OUT"" rows=""" & intHeight & """ cols=""" & intWidth & """></textarea></td>")
document.write("</tr><tr><td><INPUT TYPE=""button"" NAME=""cmdSend"" VALUE=""Send"" onclick=""SendRequest()"">")
document.write("&nbsp<INPUT TYPE=""button"" NAME=""cmdPretty"" VALUE=""Parse input"" onclick=""ParseInput()"">")
document.write("&nbsp<INPUT TYPE=""button"" NAME=""cmdClean"" VALUE=""Clean output"" onclick=""CleanResponse()"">")
document.write("</td><td id=""tid""><FONT FACE=""verdana"" SIZE=1>&nbsp</td>")
document.write("</tr></table></FORM>")
'Get current user as default
Dim objNet
Set objNet = CreateObject("WScript.NetWork")
frmForm1.username.Value = objNet.UserDomain & "\" & objNet.UserName
Set objNet = Nothing
Sub SendRequest
Dim WinHttpReq
Const HTTPREQUEST_SETCREDENTIALS_FOR_SERVER = 0
Set WinHttpReq = CreateObject("WinHttp.WinHttpRequest.5.1")
strStart = Timer
WinHttpReq.Open "POST", frmForm1.url.Value, False
WinHttpReq.setRequestHeader "Content-Type", "text/xml;charset=utf-8"
WinHttpReq.setRequestHeader "Accept", "text/xml, multipart/related, text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2"
WinHttpReq.setRequestHeader "Content-Transfer-Encoding", "binary"
WinHttpReq.setRequestHeader "Connection", "keep-alive"
WinHttpReq.SetCredentials frmForm1.username.Value, frmForm1.password.Value, HTTPREQUEST_SETCREDENTIALS_FOR_SERVER
WinHttpReq.Send(frmForm1.IN.Value)
strStop = Timer
Set xmlDoc = CreateObject("Msxml2.DOMDocument")
'Parse output
xmlDoc.loadXML(WinHttpReq.ResponseText)
If (xmlDoc.parseError.errorCode <> 0) Then
   frmForm1.OUT.Value = WinHttpReq.ResponseText
   Else
   frmForm1.OUT.Value = xmlDoc.xml
End If
Set xmlDoc = Nothing
strTime = strStop - strStart
strTime = FormatNumber(strTime,2)
tid.Innertext = "Response time: " & strTime & " s"
Set WinHttpReq = Nothing
'Set clipboard = output
If frmForm1.blnSave.Value = True Then window.clipboardData.setData "Text",frmForm1.OUT.Value
End Sub
Sub ParseInput
Set xmlDoc = CreateObject("Msxml2.DOMDocument")
xmlDoc.loadXML(frmForm1.IN.Value)
If (xmlDoc.parseError.errorCode <> 0) Then
   Set myErr = xmlDoc.parseError
   Msgbox "ERROR: " & myErr.reason
   Set myErr = Nothing
   Set xmlDoc = Nothing
   Exit Sub
End If
frmForm1.IN.Value = xmlDoc.xml
Set xmlDoc = Nothing
End Sub
Sub CleanResponse
frmForm1.OUT.Value = ""
End Sub
</SCRIPT>
</body>
</html>

Pairwise test case design part two

2008-02-13T16:17:00.004+01:00

I got a request to share the code for the "web shell", which I mentioned in earlier post.

I did a fast translation from Swedish to English and here is the result (please read PICT help file for proper format of the input file):

<html>

<head>
<title>Create PairWise Test Cases</title>
<SCRIPT LANGUAGE="vbscript">
Sub CreateTest
strInputFile = frmForm1.txtC.Value
strOutputFile = frmForm1.txtB.Value
frmForm1.Ptxt.Value = CreateTestCase(strInputFile, strOutputFile)
End Sub
Function CreateTestCase(strInputFile, strOutputFile)
'Set path to PICT program
strPath = "cmd /K C:\Program\Pict\pict.exe "
Dim fso
Set fso = CreateObject("Scripting.FileSystemObject")
'Copy input file to temp file
fso.CopyFile strInPutFile, "c:\temp.txt"
Dim oShell
Set oShell = CreateObject ("Wscript.shell")
'Run PICT
Set oExec = oShell.Exec(strPath & "c:\temp.txt > c:\Testfall.xls")
Set oExec = Nothing
Set oShell = Nothing
strCall = "FormatXLFile(""" & strOutputFile & """)"
kaka = setTimeout (strCall,10000,"VBScript")
Set fso = Nothing
CreateTestCase = "Creating file, please wait..."
End Function
Function FormatXLFile(strOutputFile)
Set ExcelObj = CreateObject("Excel.Application")
ExcelObj.Workbooks.Close
'Open file created by PICT
ExcelObj.Workbooks.Open "c:\Testfall.xls"
Set NewSheet = ExcelObj.Sheets.Item(1)
NewSheet.Activate
'Format
NewSheet.Rows(1).Font.Bold = True
NewSheet.Columns("A:AA").AutoFit
ExcelObj.ActiveWorkbook.Saved = True
NewSheet.Copy
'Save to output files
ExcelObj.ActiveWorkbook.SaveAs strOutputFile
ExcelObj.ActiveWorkbook.Saved = True
ExcelObj.Application.Quit
Set NewSheet = Nothing
Set ExcelObj = Nothing
Set ExcelObj = CreateObject("Excel.Application")
ExcelObj.Workbooks.Close
ExcelObj.Workbooks.Open strOutputFile
ExcelObj.Application.Visible = True
FormatXLFile = "Ready"
Set fso = CreateObject("Scripting.FileSystemObject")
'Delete temp files
fso.DeleteFile("c:\temp.txt")
fso.DeleteFile("c:\Testfall.xls")
Set fso = Nothing
frmForm1.Ptxt.Value = "File ready"
End Function

</SCRIPT>
</head>
<body>
<BODY BGCOLOR="white">
<FONT FACE="verdana" SIZE=1>
<FORM NAME="frmForm1">
<table width="300" border="0" cellspacing="0" cellpadding="0" align="left">
<FONT FACE="verdana" SIZE=1>
<tr>
<td><H3><Center>Create test cases<Center/></H3>
<FONT FACE="verdana" SIZE=1>
Input File syntax: [infilename.txt]<br />
Output File syntax: [outfilename.xls]<br />
</td>
</tr>
</table>
<br><br><br><br><br><br>
<table width="300" border="0" cellspacing="0" cellpadding="0" align="left">
<tr>
<td>
<Left>
<FONT FACE="verdana" SIZE=1>
Infile
</Left>
</td>
<td>
<INPUT TYPE="file" NAME="txtC" SIZE=40>
</td>
</tr>
<tr>
<td>
<Left>
<FONT FACE="verdana" SIZE=1>
Outfile
</Left>
</td>
<td>
<INPUT TYPE="file" NAME="txtB" SIZE=40>
</td>
</tr>
<tr>
<td>
<Center>
<FONT FACE="verdana" SIZE=1>
<INPUT TYPE="button" NAME="cmdCalc" VALUE="Create file" onclick="CreateTest">
</Center>
</td>
<td>
<INPUT TYPE="text" NAME="Ptxt" SIZE=40></td>
&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp
</td>
</tr>
</table>
</FORM>
</body>
</html>

Pairwise test case design

2008-02-13T08:25:00.003+01:00

I've found a good site about pairwise testing the other day:
http://www.pairwise.org/

About a year ago I implemented the use of tool which generates test cases accordingly to the pairwise technique: PICT

Since I prefer web-based UI rather than command-line dito I've created a web shell on top of PICT which makes it a little bit easier to use. In addition I use the Excel COM-object to make a nice format of the output file:

A day in my test-life...using my test tools

2008-01-30T13:53:00.000+01:00

A day in my test-life...using my test tools

08.00 Arrives at work, connects (dual monitors) and starts my laptop.
08.08 Logged in at last (crypted hard drives)!
08.09 Check emergency e-mail from production or test (if any)
08.10 Check status for all ten test environments (available in i single web page, VBScript-based)
08.11 Test environment maintaince
08.20 Check regression test dashboard (available in i single web page, VBScript-based) wheather to start some automated test suites or not
08.30 Check for bug fixes and new builds
08.31 Retesting bugs
09.15 Short test meeting regarding release status
09.30 Manual test session based on my product areas in the release (using my test tools for test data).
11.30 Lunch
12.15 Maintainance of test framework
14.00 Manual test session based on my product areas in the release (using my test tools for test data)
17.00 Going home

This is of course in case all things runs smoothly...

:-)

Test framework data model snapshot

2008-01-25T09:42:00.000+01:00

One of the most essential parts of my framework is the data model and the database. Here is a snapshot of some of the most important tables and their relationsship to each other:

Create Dictionary Object from a record set

2008-01-18T12:57:00.000+01:00

Here is a function for creating a Dictionary Object out of a record set:

CDO.Message

2008-01-15T21:22:00.000+01:00

I would recommend this site for learning about the CDO object which is perfect for creating and sending mail in automation.

For example when a test case fail I use CDO to send a mail with test case id and a screen shot of the error...

SQL Store Procedures

2007-12-19T11:11:00.000+01:00

Yes! I wrote my first SP yesterday and it works excellent. The SP returns test case and results statistics from test framework database and is called from our Intranet. Response time for the page is now 1 second which I think is reasonable...

My form for register regression test cases and their result is now ready and our test group has began using it. Since I am resusing my test framework of course all results are published in real-time at our Intranet (using the SP from above)...

My test framework...

2007-11-27T21:41:00.000+01:00

...is starting to payoff a bit I guess

I now use it in QTP, web-based test tools and in vbs-files. All test result is stored in a test database and Dashboard views are available via our Intranet. For example: We have scheduled batch jobs running nighly which actually is test cases so their result is stored as well.

I am also planning to have a simple form where manual regression test result can be registered (some sort of QC light). The main reason is to have better overview which parts were tested when and by whom. This does not include testing new functionality since we use ET and test sheets...

QTP: Done framework migration

2007-10-08T11:17:00.000+02:00

Done!

I Guess I will alter the data model some more but all data and scripts are converted...

...next mission is to build a test driver for our API which can be executed stand-alone in parallell with QTP

QTP: Migration part 4

2007-09-27T20:12:00.000+02:00

I started my test tool career about ten years ago with a custom tool in MS Access. Now ten years later I find this wonderful feature in MS Access 2003, create a web page with MS SQL Server underneath...

You can build a real handy GUI in just minutes...which I did (4 and counting...)

Almost done migration, have a few test scenarios to manually convert left...

QTP: Migration part 3

2007-09-19T21:26:00.000+02:00

Puh...

Done converting Test Case Functions with my new Dictionary Object concept, works fine so far...

Next...create web GUIs to handle the database

QTP: Migration part 2b

2007-09-13T07:56:00.000+02:00

Sorry...

You can see values of a Dictionary Object when debugging...

Add to Watch panel:

objMyDictionary.Item("MyParameter")

QTP: Migration part 2

2007-09-12T07:53:00.000+02:00

Done converting xls -> SQL Server...so far everything seems OK

Last week I 'discovered' the Dictionary Object...seems perfect when to pass/get parameters from a function (I don't use Actions). So now I am converting my test case functions using Dictionary Object instead of ordinary variables.

+ Easy to add parameters
+ Allows optional parameters
+ Allows multiple output values
+ Since my test parameters are now in a table (pair-wise) I can have a Dictionary Object with the entire recordset (select parameter_name, parameter_value where test_case_id = ...) which is easy to merge with my global test parameters (test environment etc) and have it in my call to a test case.

Set objTCOutputData = MyTestCase (objTCInputData)

- Values in a Dictionary Object are not visible when debugging :-(

QTP: Migration part 1

2007-09-08T17:21:00.000+02:00

I have always been a big fan of Excel, but last week when one of my test case files got corrupt after an update I decided it was to time to move forward...

...to have all my test data in MS SQL Server instead of xls-files.

So during this week I have been modifying my data model and created a database which will contain all my test case data. Currently I'm writing migration scripts which will transfer all my data from the xls-files to the new database...

QTP: Functions vs Actions

2007-09-01T08:48:00.000+02:00

I started my QTP career with Actions but are now only using Functions. Why?

- All variables are shown when debugging (action parameters don't)
- Easier to add parameters
- Faster to load test that has only one Action (the default one)
- No need to bother where reusable Actions are located.
- Possible to edit the functions without have to load other test containing reusable actions

Conclusion: Faster, Easier and more cost effective...

...but it is also a matter of automation taste...

Automation using QuickTest Pro

2007-08-21T08:01:00.000+02:00

QTP has a lot of features which looks nice at first but are not cost effective to use in the long run. I have created a framework with custom functions instead which is handy to work with. Here are the main features:

Custom reporting. Write the results to xls using ADO and use QTP report for detailed logging only. The results are presented at our Intranet in real-time. In case of test case failure, an e-mail is sent with a screenshot of the AUT.
DP instead of OR. Quicker to work with and easy maintainance.
Functions instead of Actions. Quicker to work with and easy maintainance.
Test cases in custom xls using ADO instead of the DataSheet.

More about this later on...

Part-time automation...

2007-06-28T08:42:00.000+02:00

Is it possible to have succes in automation without have at least one dedicated, full-time resource?

In most cases no, but in my case yes...

I probably spend 30% of my time in automation and the rest for sapient testing (see James Bach Blog). Of course during some periods the automation efforts have to be higher (during implementation, creating a framework among other things) but once settled (if you have a good framework) a relative small effort in maintanance, running tests and creating new tests could still be worth while...

In my case this is possible due to the following:

Previous experience in automation
An automation-friendly AUT (framework-based prefereble)

More about my automation framework later on...

Get random test data into Clipboard

2007-06-01T09:02:00.000+02:00

OK,

Then we know how to get random data from SQL Server. Would be nice to build a public web page around this where testers easily can get test data for immidiate pasting into the AUT.

Something like this maybe:

1. Specify data to get and from which test environment
2. Click on a button and display the test data and put automaticly in clipboard
3. Paste into the AUT

Solution:
Create a web form and include

vbscript

Sub GetTestData
'Set test environment and test data
strTestEnvironment = formname.TestEnvironment.Value'Form reference not used in this example
intTestDataIndex = formname.RandomTestData.SelectedIndex
If intTestDataIndex = 0 Then
strSQL = "SELECT top 1 testdata1 FROM table where testdatatype1 is not NULL ORDER BY NEWID()"
ElseIf intTestDataIndex = 1 Then
strSQL = "SELECT top 1 testdata2 FROM table where testdatatype2 is not NULL ORDER BY NEWID()"
End If
'****** Initialization ******
'Set test environment parameters
strDataBase = "DRIVER=SQL Server; DATABASE=[databasename];APP=[testtool];SERVER=[servername];Description=Testconnection"
'****** Data base call ******
Dim objDB
Set objDB = CreateObject("ADODB.Connection")
objDB.Open(strDataBase)
Set objRecordSet = objDB.Execute(strSQL)
Dim strResult
If objRecordSet.BOF = True And objRecordSet.EOF = True Then
strTestData = ""
Else
strResult = objRecordSet.GetString
strTestData = Left((Trim(strResult)),Len(Trim(strResult))-1)'Format data
End If
objDB.Close
Set objDB = Nothing
formname.TestData.Value = strTestData
End Sub

and finally

JavaScript

function copytestdata(input) {
GetTestData();
window.clipboardData.setData('Text',input.value);
}

Start the sequence by using the tag

INPUT TYPE="button" NAME="cmdGetTestData" VALUE="GetTestData" onclick="copytestdata(formname.TestData)"

Done!

Randomize test data

2007-05-22T08:14:00.000+02:00

Random test data could be a good thing in certain situations in software testing.

If your AUT (application under test) is running on MS SQL Server there is a smart function in

order by newid()

By using this the record set will sorted randomly.

To get one record randomly syntax will be

select top 1
mydata
from mytable
where
mydata = mytestdatarequirement
order by newid()

With this you can easily build small web apps. for testing using vbScript and/or JavaScript. More about that later...

In the beginning...

2007-05-18T13:30:00.000+02:00

Why a blog? Well, all testers that matters seems to have one so I better join the crowd. Will be blogging mainly about software testing.