Custom Search

[Top] [All Lists]

Re: What is TPS?

Subject: Re: What is TPS?
Date: Tue, 19 May 2009 13:20:00 -0400
We agree to disagree.
The TPS as reported by Grinder is not actually "per second", but rather
"per interval" (according to the Grinder documentation).  And, the
interval length is arbitrary.  So, continuing the example I used, if I
set the interval length to 100 sec, but run for only 60 sec (all other
parameters remaining the same), then Grinder would (presumably) report 0
TPS (because it didn't receive any reports).  Which isn't a reasonable
I do already, of course (and I'm sure everybody else does too), use
Grinder to drive the server to peak capacity.  And I agree that one
can't interpolate from small-scales test to large-scale ones.
But I disagree about whether the non-Grinder calculation is meaningful.
Certainly, the Grinder TPS calculation is a clear concept.  But the
non-Grinder calculation is well-defined (meaningful), and does give
interesting information.
In fact, I would "expect" that as server(+network) capacity is
approached, the Grinder number and non-Grinder number "should" converge
to one another (modulo round-off errors), with the non-Grinder number
always being larger than the Grinder number, due to Grinder
But it's exactly that expectation that I don't trust, that is, I'd like
to see the expectation verified by actual computation rather than mere
conjecture.  Perhaps it would help to think of it as a
Grinder-inefficiency metric, or cross-check on Grinder's TPS
calculations: The difference between non-Grinder TPS and Grinder TPS is
Grinder "think-time".  For simple tests, that think-time can be made
almost arbitrarily small.  But the script I've developed is pretty
sophisticated at this point, and I'm wondering if its complexity is
degrading the quality of my measurements.
-----Original Message-----
From: Philip Aston 
[mailto:[email protected]] 
Sent: Tuesday, May 19, 2009 12:37 PM
To: grinder-use
Subject: Re: [Grinder-use] What is TPS?
I disagree with this.
I don't think the "non-Grinder" calculation is meaningful, and what you
really want to know is what the peak TPS that the server can achieve.
There's nothing to stop you from starting more  worker processes to
drive the server up to its peak capacity, and measuring it directly.
TPS is a clear concept - it is a measurement of the actual test rate.
Using your numbers, the server received 60 requests in 60 seconds => it
was achieving 1 TPS. Simply because it turned around each request in 0.1
TPS does not mean it is therefore capable of doing 10 TPS. The server's
upper capacity depends on a number of factors, including the number of
CPUs/cores, and the common resources such as memory, backend network
calls, shared connection pools etc.  that parallel requests must access,
and it certainly can't be linearly derived based on the idle time at a
particular load.
- Phil
[email protected] wrote:
> I'd like to introduce another TPS-related topic for discussion,
> by the recent discussion of "req/sec" vs "resp/sec", but different.
> It's a lower-level question, about how Grinder exactly computes TPS.
> The question is best illustrated by an example:
> For simplicity, assume we're talking only about a single HTTP (e.g.,
> Server) and Grinder's regular HTTPClient, and that tests are written
> such that only the calls to that server are Test.wrap()'ed (i.e., only
> the HTTPRequest's etc. are wrap()'d, not some higher-level
> composite/page method or anything like that).  Suppose also that the
> Grinder script need to do some significant set-up work (~ "think
> before each WS request.  To be specific, let's say it takes 0.9 sec of
> think-time before every test call, and that every call to the server
> takes 0.1 sec (on-wire, as measured by wrap()).  Suppose this test is
> run for exactly 1 minute, in a single thread.
> The question is, exactly what should "TPS" be?  Specifically, we can
> distinguish Grinder-viewpoint (Console & logfiles) TPS, from
> non-Grinder-viewpoint (network + server) viewpoint.  [If the network
> snappy (i.e., unless there's a WAN involved), "non-Grinder" ~=
> to first approximation, if we assume an infinitely fast network, which
> we can do for the purposes of our simple example.  In other words, we
> can think of "non-Grinder-viewpoint" as "server-viewpoint".]
> Measured from Grinder's viewpoint, the answer would be 1 TPS (60 tests
> 60 sec == 1 test/sec).  But measured from the non-Grinder viewpoint
> (namely, the server sees 60 requests, and it services each request in
> 0.1 sec), the answer is 10 TPS (60 tests / (60*0.1) sec == test/sec).
> If we did the same thing with 2 threads, the TPS would be 2 vs. 20
> (assuming linear scaling); etc. Either one of these interpretations of
> "TPS" is perfectly defensible, depending on what you're trying to
> measure.
> According to the Grinder Manual, it says the TPS reported/displayed in
> the Console/logs is measured as: (number of tests that occurred per
> sample interval) / (interval length).  In other words, that is the
> Grinder-viewpoint.
> But the non-Grinder viewpoint gives more optimistic perf numbers than
> the Grinder viewpoint.  And, it's more "realistic", in one sense: it
> takes Grinder out of the equation, and gives a better estimate of
> pure-server-side (+ network) capability.  Which is an interesting
> to measure.
> So, what I'm suggesting is that BOTH Grinder-viewpoint TPS and
> non-Grinder-viewpoint TPS are interesting, and it would be nice if
> Grinder would report both numbers.  Perhaps the Grinder-viewpoint TPS
> could be called "tests per sec", and the non-Grinder-viewpoint TPS
> be called "transaction per sec".
> BTW, this same question (Grinder-viewpoint vs. non-Grinder-viewpoint)
> cannot reasonably be asked of "mean-time" (latency) as opposed to TPS
> (throughput), because that has a canonical answer: Grinder's timestamp
> snapshots are taken inside wrap(), hence necessarily reflect the
> non-Grinder viewpoint.  In the example stated, the non-Grinder
> yields a mean-time of 0.1 sec/test, while the Grinder viewpoint would
> yield a mean-time of 1 sec/test -- clearly an unreasonable
> interpretation (so Grinder doesn't report it, and shouldn't).
> Unfortunately, the non-Grinder-viewpoint TPS cannot be derived from
> (non-Grinder-viewpoint) mean-time.  That is, Grinder support is needed
> to get non-Grinder-viewpoint TPS (and Grinder does indeed have the
> available to report non-Grinder-viewpoint TPS).  In particular, one
> can't simply invert (non-Grinder-viewpoint) mean-time to get
> non-Grinder-viewpoint TPS (1 / 0.1 = 10, in the case of the example
> above).  That works only in simple cases (1 thread, 1 server, linear
> scaling).  But when you go to more complicated test scenarios
> threads, multiple/cluster servers, non-linear response curve due to
> load-induced gunk) simple mathematical tricks don't work.  You need to
> use actual/observed measurements.
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables 
unlimited royalty-free distribution of the report engine 
for externally facing server and web deployment.
grinder-use mailing list
[email protected]
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables 
unlimited royalty-free distribution of the report engine 
for externally facing server and web deployment.
<Prev in Thread] Current Thread [Next in Thread>
Current Sitemap | © 2012 planetjava | Contact | Privacy Policy