Getting a feel for the performance of mod_wsgi

Disclaimer: there’s a lot that’s wrong with my testing setup and/or methodology. Given the kind of concurrency I’m playing around with, I run into a whole bunch of operating system and TCP/IP limits. Though I’ve tweaked those a little bit (for example # of open file descriptors, TCP window size), I haven’t been particularly scientific about it. Though I did various warm-ups and system reboots, I wasn’t particularly good at keeping consistent timing or consistently checking that there were no lingering connections in TIME_WAIT or whatever.

System under test:

• Server: Latest-gen macbook 2.4ghz intel core 2 duo
• Test client: Mac Pro tower dual core G5
• Switch: some crappy 1gbit whitelabel
• Stock Apache on Mac OS X 10.5 (prefork MPM, 64bit)
• Stock python 2.5 on Mac OS X 10.5
• mod_wsgi 2.3
• ApacheBench, Version 2.3, commands like ab -n 5000 -c 100 http://10.0.0.2/wsgitest

Out of the box:

• about 2000-3000 req/s for static file
• about 2000-2800 req/s for mod_wsgi in-process
• about 1500-2500 req/s when using mod_wsgi in daemon mode (with errors beyond concurrency of about 250, for various settings of p, t)
• concurrency=1000 makes ApacheBench start reporting lots of failures

Follows some MPM tuning, arriving to:

StartServers        100
MinSpareServers      10
MaxSpareServers     500
MaxClients          500
MaxRequestsPerChild   0

Results then become better (especially more stable):

• about 5000 req/s for static file
• With EnableMMAP off and EnableSendfile off, even concurrency=10 is already a problem for static file scenario, and req/s doesn’t go above 3000 req/s for concurrency>=5
• about 4300 req/s for mod_wsgi in process
• about 2700 req/s for mod_wsgi in daemon mode
• concurrency=1000 still makes ApacheBench start reporting lots of failures

Some more data:

                    hello.txt           wsgitest        wsgitest    wsgitest    wsgitest
                                        in-process      p=2,t=10    p=2,t=100   p=20,t=10
concurrency 10      req/s:  4784
                    ms/req:    0.21

concurrency 100     req/s:  5081        4394            3026        3154        2827
                    ms/req:    0.20        0.23            0.33        0.32        0.35

concurrency 200     req/s:  5307        4449            2688        2988        2711
                    ms/req:    0.19        0.24            0.37        0.34        0.36

concurrency 500     req/s:  4885        4137            2779        3019        2738
                    ms/req:    0.21        0.24            0.36        0.33        0.36

hello.txt is a 13 byte file containing "Hello World!\n"
wsgitest is a really simple wsgi script spitting out "Hello World!\n"
concurrency is the argument to ab -c
p is number of processes for mod_wsgi daemon mode
t is number of threads for mod_wsgi daemon mode
ms/req is the mean time per request across all concurrent requests as reported by ab

Tentative conclusions:

• With my hardware I have little chance of actually finding the limits of apache or mod_wsgi unless I spend a whole lot of time on much more carefully testing what I’m actually measuring
• running mod_wsgi in-process is probably a good idea if you tweak the MPM for it
• mod_wsgi on my laptop can probably easily serve over a billion requests/month after a little tuning
• mod_wsgi on my laptop can deal with 500 “concurrent” users without errors

…so, in other words, mod_wsgi is most likely “fast enough” and most likely will no be a bottleneck if I build something with it. Not exactly surprising. But, more importantly, now I have some baseline numbers for req/s performance on my system, so that I can run some “performance smoke tests” against software I write.