-*-Mode: outline-*-

* Building httperf

Building httperf requires an ANSI C compiler and GNU make v3.76 (older
GNU make versions may work but have not been tested).  You can get GNU
make from ftp://prep.ai.mit.edu/pub/gnu/.

To build a httperf with debug support enabled, simply type "make".  To
build an optimized version, type "make HOW=OPTIM".

httperf has been tested under HP-UX 10.20, Red Hat Linux v5.0 (Alpha
and x86).  When attempting to build on any other platforms, it is
almost certainly necessary to edit the Makefile and it is also likely
that small changes need to be made to the various source files in
order to get a clean compile.

* Running httperf

IMPORTANT: It is crucial to run just one copy of httperf per client
machine.  httperf sucks up all available CPU time on a machine.  It is
therefore important not to run any other (CPU-intensive) tasks on a
client machine while httperf is running.  httperf is a CPU hog to
ensure that it can generate requests with high accuracy, so do not try
to change this without fully understanding what the issues are.

** Examples

The simplest way to invoke httperf is with a command line of the form:

 httperf --server wailua --port 6800

This command results in httperf attempting to make one request for URL
http://wailua:6800/.  After the reply is received, performance
statistics will be printed and the client exists (the statistics are
explained below).

A list of all available options can be obtained by specifying the
--help option (all option names can be abbreviated as long as they
remain unique).

A more realistic test case might be to issue 1000 HTTP requests
(calls) at a rate of 10 calls per second.  This can be achieved by
additionally specifying the --num-sessions and --rate parameters.
When specifying the --rate option, it's generally a good idea to also
specify a timeout value using the --timeout option.  In the example
below, a timeout of one second is specified (the ramification of this
option will be explained later):

 httperf --server wailua --port 6800 --num-sessions 100 --rate 10 --timeout 1

The performance statistics printed by httperf at the end of the test
might like this:

    Total: sessions 100 calls 100 replies 100 test-duration 9.905 s

    Session rate: 10.1 sess/s (99.1 ms/sess, <=1 concurrent sessions)
    Session time [ms]: min 4.6 avg 5.6 max 19.9 median 4.5 stddev 2.0
    Session time [ms]: connect 1.4

    Call rate: 10.1 call/s (99.1 ms/call)
    Call size [B]: 57.0

    Reply rate [replies/s]: min 10.0 avg 10.0 max 10.0 stddev 0.0 (1 samples)
    Reply time [ms]: response 4.1 transfer 0.0
    Reply size [B]: header 219.0 content 204.0 footer 0.0 (total 423.0)
    Reply status: 1xx=0 2xx=100 3xx=0 4xx=0 5xx=0

    CPU time [s]: user 2.71 system 7.08 (user 27.4% system 71.5% total 98.8%)
    Net I/O: 4.7 KB/s (0.0*10^6 bps)

    Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
    Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

There are six groups of statistics: overall results ("Total"),
connection related results (httperf calls them "Sessions"), results
relating to the issuing of HTTP requests ("Calls"), results relating
to the replies received from the server ("Reply"), miscellanous
results relating to the CPU time and networth bandwidth used, and,
finally, a summary of errors encountered ("Errors").  Let's discuss
each in turn:

** "Total" Results

The "Total" line summarizes how many sessions (TCP connections) were
initiated by the client, how many calls it sent out, how many replies
it received, and what the total test duration was.  The line below
shows that 100 sessions were initiated, 100 calls were performend and
100 replies were received.  It also shows that total test-duration was
9.905 seconds meaning that the average request rate was almost exactly
10 request per second.

    Total: sessions 100 calls 100 replies 100 test-duration 9.905 s

** "Session" Results

These results convey information related to the TCP connections that
are used to communicate with the web server.

Specifically, the line below show that new sessions (connections) were
initiated at a rate of 10.1 sessions per second.  This rate
corresponds to a period of 99.1 milli-seconds per session.  Finally,
the last number shows that at most one connection was open to the
server at any given time.

    Session rate: 10.1 sess/s (99.1 ms/sess, <=1 concurrent sessions)

The next line in the output gives lifetime statistics for successful
sessions.  The lifetime of a session is the time between a TCP
connection was initiated and the time the session was closed.  A
session is considered successful if it had at least one call that
resulted in a reply from the server.  The line shown below indicates
that the minimum ("min") session lifetime was 4.6 milliseconds, the
average ("avg") lifetime was 5.6 milliseconds, the maximum ("max") was
19.9 milliseconds, the median ("median") lifetime was 4.5
milliseconds, and that the standard deviation of the lifetimes was 2.0
milliseconds.

    Session time [ms]: min 4.6 avg 5.6 max 19.9 median 4.5 stddev 2.0

To compute the median time, httperf collects a histogram of session
lifetimes.  The granularity of this histogram is current 1
milliseconds and the maximum session lifetime that can be accommodated
with the histogram is 100 seconds (these numbers can be changed by
editing macros BIN_WIDTH and MAX_LIFETIME in stat/basic.c).  The
ramification of this is that the granularty of the median time is 1
milliseconds and that at least 50% of the lifetime samples must have a
lifetime of less than 100 seconds.

The final statistic in this section is the average time it took to
establish a TCP connection to the server (all successful TCP
connections establishments are counted, even connections for sessions
that may have failed eventually).  The line below shows that, on
average, it took 1.4 milliseconds to establish a connection.

    Session time [ms]: connect 1.4

** "Call" Results

The first line in the "Call"-related results give the rate at which
HTTP requests were issued and the period-length that the rate
corresponds to.  In the example below, the request rate was 10.1
requests per second, which corresponds to 99.1 milliseconds per call.

    Call rate: 10.1 call/s (99.1 ms/call)

As long as no persistent connections are employed, the "Call" results
are typically very similar or identical to the "Session" results.
However, when persistent connections are used, several calls can be
issued on a single session in which case the results would be
different.

The next line gives the average size of the HTTP request in bytes.  In
the line show below, the average request size was 57 bytes.

    Call size [B]: 57.0


** "Reply" Results

For simple measurements, the section with the "Reply" results is
probably the most interesting one.  The first line gives statistics on
the reply rate:

    Reply rate [replies/s]: min 10.0 avg 10.0 max 10.0 stddev 0.0 (1 samples)

The line above indicates that the minimum ("min"), average ("avg"),
and maximum ("max") reply rate was ten replies per second.  The
standard deviation, not surprisingly was, zero.  The last number shows
that only one reply rate sample was acquired.  The present version of
httperf collects one rate sample about once every five seconds.  To
obtain a meaningful standard deviation, it is recommended to run each
test long enough so at least thirty samples are obtained---this would
correspond to a test duration of at least 150 seconds, or two and a
half minutes.

The next line gives information on how long it took for the server to
respond and how long it took to receive the reply.  The line below
shows that it took 4.1 milliseconds between sending the first byte of
the request and receiving the first byte of the reply.  The time to
"transfer", or read, the reply was too short to be measured, so it
shows up as zero (as we'll see below, the entire reply fit into a
single TCP segment and that's why the transfer time was essentially
zero).

    Reply time [ms]: response 4.1 transfer 0.0

Next follow some statistics on the size of the reply---all numbers are
in reported bytes.  Specifically, the average length of reply headers,
the average length of the content, and the average length of reply
footers are given (HTTP/1.1 uses footers to realize the "chunked"
transfer encoding).  For convenience, the average total number of
bytes in the replies is also given.  In the example below, the average
header length ("header") was 219 bytes, the average content length
("content") was 204 bytes, and there were no footers ("footer"),
yielding a total reply length of 423 bytes on average.

    Reply size [B]: header 219.0 content 204.0 footer 0.0 (total 423.0)

The final piece in this section is a histogram on the status codes
received in the replies.  The example below shows that all 100 replies
were "successful" replies as they contained a status code of 200
(presumably):

    Reply status: 1xx=0 2xx=100 3xx=0 4xx=0 5xx=0


** Miscellaneous Results

This section starts with a summary of the CPU time the client
consumed.  The line below shows that 2.71 seconds were spent executing
in user mode ("user"), 7.08 seconds were spent executing in system
mode ("system") and that this corresponds to 27.4% user mode execution
and 71.5% system execution.  The total utilization was almost exactly
100%, which is expected given that httperf is a CPU hog:

    CPU time [s]: user 2.71 system 7.08 (user 27.4% system 71.5% total 98.8%)

Note that any time the total CPU utilization is significantly less
than 100%, some other processes must have been running on the client
machine while httperf was executing.  This makes it likely that the
results are "polluted" and the test should be rerun.

The next line gives the average network throughput in kilobytes per
second (where a kilobyte is 1024 bytes) and in megabits per second
(where a megabit is 10^6 bit).  The line below shows an average
network bandwidth of about 4.7 kilobyte per second.  The megabit per
second number is zero due to rounding errors.

    Net I/O: 4.7 KB/s (0.0*10^6 bps)

The network bandwidth is computed from the number of bytes sent and
received on TCP connections.  This means that it accounts for the
network payload only (i.e., it doesn't account for protocol headers)
and does not take into account retransmissions that may occur at the
TCP level.

** "Errors"

The final section contains statistics on the errors that occured
during the test.  The "total" figure shows the total number of
errors that occurs.  The two lines below show that in our example
run, no errors occured:

    Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
    Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

The meaning of each error is described below:

 client-timo:
	Each time a request is made to the server, a watchdog timer
	is started.  If no (partial) response is received by the time
	the watchdog timer expires, the httperf times out that request
	a increments this error counter.  This is the most common error
	when driving a server into overload.

 socket-timo
	The number of times a TCP connection failed with a
	socket-level time out (ETIMEDOUT).

 connrefused
	The number of times a TCP connection attempt failed with
	a "connection refused by server" error (ECONNREFUSED).

 connreset
	The number of times a TCP connection failed due to a reset
	(close) by the server.

 fd-unavail
	The number of times the httperf client was out of file
	descriptors.  Whenever this count is count is bigger than
	zero, the test results are meaning less because the client
	was overloaded (see discussion on setting --timeout below).

 addrunavail
	The number of times the client was out of TCP port numbers
	(EADDRNOTAVAIL).  This error should never occur.  If it
	does, the results should be ignored.

 ftab-full
	The number of times the system's file descriptor table
	is full.  Again, this error should never occur.  If it
	does, the results should be ignored.

 other
	The number of times other errors occurred.  Whenever this
	occurs, it is necessary to track down the actual error
	reason.  This can be done by compiling httperf with
	debug support and specifying option --debug 1.


** Selecting appropriate timeout values

Since the client machine has only a limited set of resource available,
it cannot sustain arbitrarily high HTTP request rates.  One limit is
that there are only roughly 60,000 TCP port numbers that have in use
at any given time.  Since, on HP-UX, it takes one minute for a TCP
connection to be fully closed (leave the TIME_WAIT state), the maximum
rate a client can sustain is about 1,000 requests per second.

The actual sustainable rate is typically lower than this because
before running out of TCP ports, a client is likely to run out of file
descriptors (one file descriptor is required per open TCP connection).
By default, HP-UX 10.20 allows 2048 file descriptors per process.
Without a watchdog timer, httperf could potentially quickly use up all
available file descriptors, at which point it could not induce any new
load on the server (this would primarily happen when the server is
overloaded).  To avoid this problem, httperf requires that the web
server must respond within the time specified by option --timeout.  If
it does not respond within that time, the client considers the session
to be "dead" and closes it (and increases the "client-timo" error
count).  The only exception to this rule is that after sending a
request, httperf allows the server to take some additional time before
it starts responding (to accommodate HTTP requests that take a long
time to complete on the server).  This additional time is called the
"server think time" and can be specified by option --think-timeout.
By default, this additional think time is one second so the server
would have to be able to respond within the time alloted by --timeout.

In practice, we found that with a --timeout value of 1 second, an HP
9000/735 machine running HP-UX 10.20 can sustain a rate of about 700
sessions per second before it starts to run out of filedescriptor (the
exact rate depends, of course, on a number of factors).  To achieve
web server loads bigger than that, it is necessary to employ several
independent machines, each running one copy of httperf.  A timeout of
one second effectively means that "slow" sessions will typically
timeout before TCP even gets a chance to retransmit (the initial
retransmission timeout is on the order of 3 seconds).  This is usually
OK, except that one should keep in mind that it has the effect of
truncating the session life time distribution.
