One of my Web Service test suites has 174 test cases and it takes QTP 7 minutes to execute them all. At first it may seems to be a resonable amount of time, especially compared to simular GUI operations which would have taken hours in test execution.
When I execute the same test suite running as a vbs-file instead I'm down to 1 minute and 45 seconds!
What are the differences? The code is the same apart from that all QTP specific log functions like Reporter are replaced by custom logging to files (which should be slower since 1 folder and 2 xml-files are created for each test case).
My conclusion is that QTP has a lot of overhead features which affects test execution performance in a negative way, especially considering that must features are not that useful in the long run (at least not for me).
Ps. My QTP settings are Run mode "fast" and no QTP recovery scenario is involved
Augmented Experiential Testing
-
I am writing tools and collecting tools to implement a particular idea:
augmented experiential testing (AET). I talk about this in my classes, but
I want...
6 months ago
15 comments:
That's an interesting finding. Not that native VBS is faster, but that it is so much faster than QTP. I have written a few UI tests using native VBS and seen a speed improvement, but since so much time in my testing is spent waiting on the AUT, my performance improvement hasn't been as significant as what you reported.
More facts about this case:
Obvious the tests are run on the same machine and against the same test environment
The aggregated AUT response time is ca 25 s in both cases (I guess this is not that bad for 174 Web Service requests with a mixture of create, update, get and delete) :-)
Does your VBScript have a GUI to reconfigure when it runs? - No
Does your VBScript configure itself for runtime, with the help of a number of settings files? - No
There is a vast difference between running any script through a GUI and running it via the command line. Not all QTP's fault (no I don't work for HP).
Thank you for your comments, here are my replies:
"Does your VBScript have a GUI to reconfigure when it runs? - No"
Correct and in my example no GUI is involved, just a Web Service UI. My point is that QTP has poor performance executing VBScript in general. When testing GUI's that probably does not matter that much since GUI performance time often is in seconds rather than milliseconds (as for well designed Web Services). But if each VBScript step takes x % more time to execute in QTP that will make a difference in total execution time when running long tests.
"Does your VBScript configure itself for runtime, with the help of a number of settings files? - No"
Actually sort of yes...For each test case I reload all my test parameters (although from a database instead of files) during run-time.
"There is a vast difference between running any script through a GUI and running it via the command line. Not all QTP's fault (no I don't work for HP)."
Of course it is a difference, see my first reply above. Obvious QTP is not designed to execute VBScript fast as well to run tests 24x7. When running GUI tests, QTP is fantastic (thanks to Test Object Model) to handle the test objects but most other QTP features can easily be replaced by using standard COM-objects in order to improve performance as well reusabilty in other test contexts not involvning GUI tests.
Hi Stefan,
This is out of the exiting topic.
I have strarted to automate the webservices. I am learning now i have created a code like below..
**************
Set ssr0XML= XMLUtil.CreateXML()
ssr0XML.LoadFile "G:\work related\Automation\Webservices\XMLs\validate email.xml"
XMLstr = ssr0XML.ToString
WebService("ValidateEmailService").SendRequest "Isvalidate", ssr0XML
************************
when i execute the above code i get the error message
"The constructor to deserialize an object of type 'Mercury.QTP.WS.NET_Client.NetToolkitException' was not found"
i am not sure why is this error is shown ....
Re: gkt
Hi,
I cannot help you that much regarding QTP WS add-in since I don't use it. I have an alternative way to automate WS using standard COM-objects, XML templates and native vbs (you can find more info about it reading posts from March 08.
In my current automation project I am using Java-based framework when testing WS. I will probably do some posting about it later on.
Hi...
Can u suggest me any online site for the QTP .. where i can get help for the Webservices..... Please..
Thanks,
Girish
You could try SQA Forums or AdvancedQTP but I have not seen/met an expert on the WS add-in so far (that is one reason why I am not using the add-in besides it is buggy)...
Stefan,
I don't know whether you answer questions / queries. If you do, this one might interest you. I am running QTp in an environment where the network speed varies quite dramatically, However it seems to be slow/ very slow rather than the faster end. This seems to have an adverse affect on QTP where it does not record instructions. Have you ever come across this before? I am now aware that two other people have had the problem as well. Is it possible that it is related to network performance?
Hi,
"Have you ever come across this before?"
Fortunatly not. However I guess QTP requires best possible setup in terms of hardware and network speed.
/Stefan
From my search regarding QTP performance, macro has better performance tha QTp. Is that true? I try to reduce 30 data execution to leass than 10 secs but still it is 30 secs. Any help will be highly appreciated
how to reduce execution speed in QTP? Is QTP better than macro? or macro has low ececution time?
"how to reduce execution speed in QTP?"
Remove/reduce all wait statements and use conditional loops (Do Until...). Make sure that AUT response time is the performance bottleneck. Don't use Actions and QC.
"Is QTP better than macro?"
If have not measured macro performance (I guess you mean excel VBA) but my guess is yes. However QTP's test object model has no substitute as fas as I know so for GUI-testing QTP is my first choice and for non-GUI, vbs or Java.
This is your post:
When I execute the same test suite running as a vbs-file instead I'm down to 1 minute and 45 seconds!
Now Can you please explain me what you mean by running the test suite as a vbs file?? Do you mean opening QTP through vbs as test suite will always be run by QTP even if you are using vbs
"Now Can you please explain me what you mean by running the test suite as a vbs file??"
I am running the test by execute a vbs file containing same code (test driver and test cases). That works only on non-GUI tests like Web Service for example. QTP is not involved at all in the 1:45 example. See this post for more details about sharing code with QTP and native vbs: http://abouttesting.blogspot.com/2008/05/sharing-code-base-with-qtp.html
Post a Comment