Benchmarks

Cherokee is really really fast. The speed at which any web server can serve requests for content is both directly tied to and limited by the I/O speed of underlying hardware and operating system. In this regard, a web server's performance is measured by the latency incurred after an I/O request to the underlying system has completed.

The primary objective of the Cherokee project is to reduce to zero the latency incurred between the time a dynamic or static system I/O request has completed and the time the resulting content is served to the requesting client. Admittedly, a very tough goal to reach.

In fact, it might be impossible. But achieving that which was once seen as impossible is what drives innovation. And it's innovation that drives the ongoing development of the Cherokee project, bringing us closer to the impossible with each new release.

To show the project's progression towards the ultimate goal, whenever a benchmark is performed it will be published right here. Older releases had impressive but not very thorough benchmarks. Whenever it is possible the conditions of the benchmark will be provided so that anyone can replicate the results. That is, after all, an essential basis of the Scientific method.

Cherokee 0.10.0

This benchmark was originally published at Alvaro's blog in a post under the title: The fastest and best designed among the modern web servers

Benchmarking software is not an easy thing to do. It may look trivial, but it requires a bast amount of time and effort when you want to do it right.

In this case, I've been working on a series of web server benchmarks. The target was to measure how the modern web servers perform in different scenarios: different contents, requests, load, and even different hardware.

Most of our previous benchmarks were based on the result of ab requesting a few hundred thousand times the very same file, and that isn't obviously measuring how the server would perform in a real production environment.

So, in this post I'm going to present the result of a more realistic benchmark. This time, instead of requesting a single file, there were 300 different files for the clients to request. Files were between 156 bytes and 14MB.

By the way, as you may have noticed, I said that I'd benchmark the "modern web server". That's an important point. I have chosen to benchmark: Cherokee, Lighttpd and Nginx. All the rest did not made it to be included; the two most notorious cases:

The always interesting but 15 years old Apache: There was a problem about it. I had to choose whether to use the pre-fork or threads based back-end. If I used pre-fork the performance would be -let's say- quite poor. But I couldn't run it with threads because I needed the same server to execute PHP (with mod_php) and that sort of set up is not thread-safe. So, by one reason or the other it wouldn't really fit in the test, unless I cheated by running it with threads for static content, and pre-fork for mod_php.
Non-FOSS servers: The big problem about these servers is that they don't allow to publish public performance benchmarks. So, there is nothing I can do here. We'll have to keep those number internal.

So, without any further delay, here you have the results: (X=concurrent clients, Y=reqs/s)

Note that the difference between Cherokee and the rest increases with the number of concurrent clients. That means that it'll be the best performing web server while your client number raises (when you need it the most, actually).

So, as you can see, Cherokee is the fastest and best designed among the modern web servers. But it's even better than that. The next benchmark will be focused on dynamic content delivery, and it will show how Cherokee outperforms all the rest by even a bigger difference. Stay tuned!

Edit (Sat Oct 18 17:16:23 CEST 2008): For the sake of correctness, here you have a couple of scripts (1, 2), and configuration files (1, 2, 3) that the benchmark has been based upon.

Cherokee 0.8.1 benchmark

This is just a quick benchmark made by Alvaro to have something to show on the release date. Better, more representative benchmarks should be added shortly.

Today, after releasing Cherokee 0.8.1 I decided it was time to test again how it was doing about performance. In the last months we have got a great new I/O cache layer and a couple of structural changes that were supposed to have a positive impact on the general server performance, although nobody checked whether those changes were as effective as we aimed them to be.

So, once again, here I am bringing good news. We have done it. Cherokee is actually the fastest web server among a set of the most common servers nowadays: Apache, Cherokee, Lighttpd and Nginx!!

The benchmark consisted on half a million requests of a 1.7KiB static file, with 20 concurrent clients, using a 1Gbit/s local network. The results (fastest to slowest) were:

Cherokee:

Server Software:        Cherokee/0.8.1
Server Hostname:        10.0.0.102
Server Port:            80

Document Path:          /index.html
Document Length:        1795 bytes

Concurrency Level:      20
Time taken for tests:   17.819725 seconds
Complete requests:      500000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    500000
Total transferred:      999007442 bytes
HTML transferred:       897506630 bytes
Requests per second:    28058.79 [#/sec] (mean)
Time per request:       0.713 [ms] (mean)
Time per request:       0.036 [ms] (mean, across all concurrent requests)
Transfer rate:          54747.93 [Kbytes/sec] received

Lighttpd:

Server Software:        lighttpd/1.4.19
Server Hostname:        10.0.0.102
Server Port:            80

Document Path:          /index.html
Document Length:        1795 bytes

Concurrency Level:      20
Time taken for tests:   21.248000 seconds
Complete requests:      500000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    470598
Total transferred:      991856958 bytes
HTML transferred:       897503590 bytes
Requests per second:    23531.63 [#/sec] (mean)
Time per request:       0.850 [ms] (mean)
Time per request:       0.042 [ms] (mean, across all concurrent requests)
Transfer rate:          45585.94 [Kbytes/sec] received

NginX:

Server Software:        nginx/0.5.33
Server Hostname:        10.0.0.102
Server Port:            80

Document Path:          /index.html
Document Length:        1795 bytes

Concurrency Level:      20
Time taken for tests:   23.741872 seconds
Complete requests:      500000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    500000
Total transferred:      1006000217 bytes
HTML transferred:       897500000 bytes
Requests per second:    21059.84 [#/sec] (mean)
Time per request:       0.950 [ms] (mean)
Time per request:       0.047 [ms] (mean, across all concurrent requests)
Transfer rate:          41379.30 [Kbytes/sec] received

Apache2.2:

Server Software:        Apache/2.2.8
Server Hostname:        10.0.0.102
Server Port:            80

Document Path:          /index.html
Document Length:        1795 bytes

Concurrency Level:      20
Time taken for tests:   35.438605 seconds
Complete requests:      500000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    495064
Total transferred:      1043777896 bytes
HTML transferred:       897500000 bytes
Requests per second:    14108.91 [#/sec] (mean)
Time per request:       1.418 [ms] (mean)
Time per request:       0.071 [ms] (mean, across all concurrent requests)
Transfer rate:          28762.81 [Kbytes/sec] received

For the record: I did my best configuring all the servers in the very same way. In all the cases I removed unnecessary rules that could have slowed down the server (checks for htpasswd files and so on). And all the binaries came from the Debian repository, except for Cherokee 0.8.1 that hasn't been packaged yet.

Anyway, this benchmark has been just a quick test. It is not certainly representing the result that these servers would have handling real traffic though. So, in the following days I will try to do a new a more accurate benchmark with static and dynamic content, compression, redirections, etc. I'm pretty sure the results will be even better.

Cherokee 0.6

This benchmark was performed by Miguel Angel Ajo Pelayo with Django running on Cherokee 0.6.0 and Apache 2.0.40. It shows Cherokee as a clear winner serving dynamic content. Specifically three times as much! And you haven't yet seen static results. The advantage is a lot bigger.

Cherokee + Apache + Lighttpd Benchmark

Software

Hardware

Background

I installed a fresh installation of Debian on the server hardware. Right after you login you will need to get sudo to perform root commands from your account:

Then add yourself to the /etc/sudoers file by running visudo and adding yourself in the user section. I just followed the root entry as this does not need to be a very secure server since it will not be running publicly. Now make sure you get back to your account and do:

Cherokee Setup Details

Apache Setup Details

I used the supplied highperformance.conf configuration file. I started the server with:

Lighttpd Setup Details

Benchmark

I will perform several different benchmarks on each webserver. This is to help gauge what type of performance each server can handle in the different conditions. Each test will have SSL turned on and turned off.

small static file test

large static file test

Results

I have included cherokee with both iocaching on and off. The out of the box setting is that iocache is turned on.

small static file test w/ keepalive

small static file test w/o keepalive

large static file test w/ keepalive

large static file test w/o keepalive

Conclusion

If you feel I need to supply any more detail about any part of this process, please let me know. I want this test to be as accurate as possible. I also would like to continue using this process to test future releases of each software and ultimately help improve them. I am a Cherokee developer, but if I notice anything that would help other webservers I am more than happy to let that community know.

Cherokee 0.5

Benchmark of Cherokee 0.5.4 under Windows, contributed to the main mailing list. Here is the link to the non abridged post.

Cherokee had one million static requests and one million scgi dynamic requests with two ab commands started at about the same time. More info follows: