PlanetJava
Custom Search

java-grinder-user
[Top] [All Lists]

What is TPS?

Subject: What is TPS?
Date: Tue, 19 May 2009 09:46:26 -0400
I'd like to introduce another TPS-related topic for discussion, prompted
by the recent discussion of "req/sec" vs "resp/sec", but different.
It's a lower-level question, about how Grinder exactly computes TPS.
The question is best illustrated by an example:
For simplicity, assume we're talking only about a single HTTP (e.g., Web
Server) and Grinder's regular HTTPClient, and that tests are written
such that only the calls to that server are Test.wrap()'ed (i.e., only
the HTTPRequest's etc. are wrap()'d, not some higher-level
composite/page method or anything like that).  Suppose also that the
Grinder script need to do some significant set-up work (~ "think time")
before each WS request.  To be specific, let's say it takes 0.9 sec of
think-time before every test call, and that every call to the server
takes 0.1 sec (on-wire, as measured by wrap()).  Suppose this test is
run for exactly 1 minute, in a single thread.
The question is, exactly what should "TPS" be?  Specifically, we can
distinguish Grinder-viewpoint (Console & logfiles) TPS, from
non-Grinder-viewpoint (network + server) viewpoint.  [If the network is
snappy (i.e., unless there's a WAN involved), "non-Grinder" ~= "server"
to first approximation, if we assume an infinitely fast network, which
we can do for the purposes of our simple example.  In other words, we
can think of "non-Grinder-viewpoint" as "server-viewpoint".]
Measured from Grinder's viewpoint, the answer would be 1 TPS (60 tests /
60 sec == 1 test/sec).  But measured from the non-Grinder viewpoint
(namely, the server sees 60 requests, and it services each request in
0.1 sec), the answer is 10 TPS (60 tests / (60*0.1) sec == test/sec).
If we did the same thing with 2 threads, the TPS would be 2 vs. 20
(assuming linear scaling); etc. Either one of these interpretations of
"TPS" is perfectly defensible, depending on what you're trying to
measure.
According to the Grinder Manual, it says the TPS reported/displayed in
the Console/logs is measured as: (number of tests that occurred per
sample interval) / (interval length).  In other words, that is the
Grinder-viewpoint.
But the non-Grinder viewpoint gives more optimistic perf numbers than
the Grinder viewpoint.  And, it's more "realistic", in one sense: it
takes Grinder out of the equation, and gives a better estimate of
pure-server-side (+ network) capability.  Which is an interesting thing
to measure.
So, what I'm suggesting is that BOTH Grinder-viewpoint TPS and
non-Grinder-viewpoint TPS are interesting, and it would be nice if
Grinder would report both numbers.  Perhaps the Grinder-viewpoint TPS
could be called "tests per sec", and the non-Grinder-viewpoint TPS could
be called "transaction per sec".
BTW, this same question (Grinder-viewpoint vs. non-Grinder-viewpoint)
cannot reasonably be asked of "mean-time" (latency) as opposed to TPS
(throughput), because that has a canonical answer: Grinder's timestamp
snapshots are taken inside wrap(), hence necessarily reflect the
non-Grinder viewpoint.  In the example stated, the non-Grinder viewpoint
yields a mean-time of 0.1 sec/test, while the Grinder viewpoint would
yield a mean-time of 1 sec/test -- clearly an unreasonable
interpretation (so Grinder doesn't report it, and shouldn't).
Unfortunately, the non-Grinder-viewpoint TPS cannot be derived from the
(non-Grinder-viewpoint) mean-time.  That is, Grinder support is needed
to get non-Grinder-viewpoint TPS (and Grinder does indeed have the info
available to report non-Grinder-viewpoint TPS).  In particular, one
can't simply invert (non-Grinder-viewpoint) mean-time to get
non-Grinder-viewpoint TPS (1 / 0.1 = 10, in the case of the example
above).  That works only in simple cases (1 thread, 1 server, linear
scaling).  But when you go to more complicated test scenarios (multiple
threads, multiple/cluster servers, non-linear response curve due to
load-induced gunk) simple mathematical tricks don't work.  You need to
use actual/observed measurements.
------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables 
unlimited royalty-free distribution of the report engine 
for externally facing server and web deployment. 
http://p.sf.net/sfu/businessobjects
msgmiddle
<Prev in Thread] Current Thread [Next in Thread>
Current Sitemap | © 2012 planetjava | Contact | Privacy Policy