Custom Search

[Top] [All Lists]

Re: What is TPS?

Subject: Re: What is TPS?
Date: Tue, 19 May 2009 17:37:28 +0100
I disagree with this.
I don't think the "non-Grinder" calculation is meaningful, and what you
really want to know is what the peak TPS that the server can achieve.
There's nothing to stop you from starting more  worker processes to
drive the server up to its peak capacity, and measuring it directly.
TPS is a clear concept - it is a measurement of the actual test rate.
Using your numbers, the server received 60 requests in 60 seconds => it
was achieving 1 TPS. Simply because it turned around each request in 0.1
TPS does not mean it is therefore capable of doing 10 TPS. The server's
upper capacity depends on a number of factors, including the number of
CPUs/cores, and the common resources such as memory, backend network
calls, shared connection pools etc.  that parallel requests must access,
and it certainly can't be linearly derived based on the idle time at a
particular load.
- Phil
[email protected] wrote:
> I'd like to introduce another TPS-related topic for discussion, prompted
> by the recent discussion of "req/sec" vs "resp/sec", but different.
> It's a lower-level question, about how Grinder exactly computes TPS.
> The question is best illustrated by an example:
> For simplicity, assume we're talking only about a single HTTP (e.g., Web
> Server) and Grinder's regular HTTPClient, and that tests are written
> such that only the calls to that server are Test.wrap()'ed (i.e., only
> the HTTPRequest's etc. are wrap()'d, not some higher-level
> composite/page method or anything like that).  Suppose also that the
> Grinder script need to do some significant set-up work (~ "think time")
> before each WS request.  To be specific, let's say it takes 0.9 sec of
> think-time before every test call, and that every call to the server
> takes 0.1 sec (on-wire, as measured by wrap()).  Suppose this test is
> run for exactly 1 minute, in a single thread.
> The question is, exactly what should "TPS" be?  Specifically, we can
> distinguish Grinder-viewpoint (Console & logfiles) TPS, from
> non-Grinder-viewpoint (network + server) viewpoint.  [If the network is
> snappy (i.e., unless there's a WAN involved), "non-Grinder" ~= "server"
> to first approximation, if we assume an infinitely fast network, which
> we can do for the purposes of our simple example.  In other words, we
> can think of "non-Grinder-viewpoint" as "server-viewpoint".]
> Measured from Grinder's viewpoint, the answer would be 1 TPS (60 tests /
> 60 sec == 1 test/sec).  But measured from the non-Grinder viewpoint
> (namely, the server sees 60 requests, and it services each request in
> 0.1 sec), the answer is 10 TPS (60 tests / (60*0.1) sec == test/sec).
> If we did the same thing with 2 threads, the TPS would be 2 vs. 20
> (assuming linear scaling); etc. Either one of these interpretations of
> "TPS" is perfectly defensible, depending on what you're trying to
> measure.
> According to the Grinder Manual, it says the TPS reported/displayed in
> the Console/logs is measured as: (number of tests that occurred per
> sample interval) / (interval length).  In other words, that is the
> Grinder-viewpoint.
> But the non-Grinder viewpoint gives more optimistic perf numbers than
> the Grinder viewpoint.  And, it's more "realistic", in one sense: it
> takes Grinder out of the equation, and gives a better estimate of
> pure-server-side (+ network) capability.  Which is an interesting thing
> to measure.
> So, what I'm suggesting is that BOTH Grinder-viewpoint TPS and
> non-Grinder-viewpoint TPS are interesting, and it would be nice if
> Grinder would report both numbers.  Perhaps the Grinder-viewpoint TPS
> could be called "tests per sec", and the non-Grinder-viewpoint TPS could
> be called "transaction per sec".
> BTW, this same question (Grinder-viewpoint vs. non-Grinder-viewpoint)
> cannot reasonably be asked of "mean-time" (latency) as opposed to TPS
> (throughput), because that has a canonical answer: Grinder's timestamp
> snapshots are taken inside wrap(), hence necessarily reflect the
> non-Grinder viewpoint.  In the example stated, the non-Grinder viewpoint
> yields a mean-time of 0.1 sec/test, while the Grinder viewpoint would
> yield a mean-time of 1 sec/test -- clearly an unreasonable
> interpretation (so Grinder doesn't report it, and shouldn't).
> Unfortunately, the non-Grinder-viewpoint TPS cannot be derived from the
> (non-Grinder-viewpoint) mean-time.  That is, Grinder support is needed
> to get non-Grinder-viewpoint TPS (and Grinder does indeed have the info
> available to report non-Grinder-viewpoint TPS).  In particular, one
> can't simply invert (non-Grinder-viewpoint) mean-time to get
> non-Grinder-viewpoint TPS (1 / 0.1 = 10, in the case of the example
> above).  That works only in simple cases (1 thread, 1 server, linear
> scaling).  But when you go to more complicated test scenarios (multiple
> threads, multiple/cluster servers, non-linear response curve due to
> load-induced gunk) simple mathematical tricks don't work.  You need to
> use actual/observed measurements.
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables 
unlimited royalty-free distribution of the report engine 
for externally facing server and web deployment.
<Prev in Thread] Current Thread [Next in Thread>
Current Sitemap | © 2012 planetjava | Contact | Privacy Policy