Benchmarking popular OS CMS - Sascha Endlicher‘s blog

Benchmarking popular OS CMS
Original PDF file
1. Our Motivation
One of our customers recently decided to migrate from PostNuke to
another CMS. His goal was to have a community driven CMS with portal
like features and a lot of interactivity for his audience. We suggested
and evaluated 4 alternative solutions: Xaraya, Drupal, Joomla and e107.For our evaluation we used httperf, a free tool of the Hewlett PackardDevelopment Company.
e107 is very easy to customize and the code is not bloatware, like allthe nukes are. The code is not based on nuke* code and from pastexperiences we were able to tell that it runs very fast on serverscompared to PostNuke or Joomla.
Joomlais another replacement candidate, but the code is spaghetti code, wefeel it is not as easy to customize and from or experience with othercustomers we know it has significant performance issues.
Drupaland Xaraya are even worse when it comes to performance. We have sitesrunning with Drupal - mainly union sites - but their performance sucks.
Some thoughts about e107:
+ very lightweight and fast
+ does not nearly need as many server resources as other systems
+ friendly community
+ very friendly user interface
+ can easily be modified
- 3rd party module market is not nearly as large as Joomla
- SEO for e107 would need to be implemented, but is relatively easy to do.
- has no integration for vbulletin
Some pros and cons for Joomla:
+ a lot of third party components allow for integration with vbulletin, gallery, etc
However, applications such as vbulletin are commercial and they also need a lot of server resources.
+ has an excellent third party component for SEO
- has a long history of security issues
- is very heavy on the server
- too commercialized module writer‘s community
A few pros and cons for Xaraya:
+ has very structurized code, the code widely supports many systems, but this causes a lot of unnecessary overhead on the server
+ offers some very sophisticated modules (i.e. the long awaited Amazon library - http://junglelib.com/)
- excellent content management system
- developer oriented
- few community features
Drupal on the other hand provides more interactivity, but the server load is terrible.
So, we decided it would be time to perform some actual benchmarks on the 5 CMS systems. We decided to use httperf, a free tool of the Hewlett Packard Development Company (http://www.hpl.hp.com/research/linux/httperf/)
Weset up httperf on a test VPS and benchmarked all 5 CMS systems on aremote Dual Core Dual Processor Xeon 3.0Ghz Server with 4GB RAM.
We used the web server on the host running at port 80. In this simple test, the same page was retrieved repeatedly.
There are, of course, at least two shortcomings in our test environment:
1)PostNuke is the only software that is filled with real world data. Allother CMS are benchmarked on their out of the box installation pages.
2)We only benchmarked the main page. For a real world experience it wouldbe desirable to benchmark the most frequent visited pages and comparethem.
2. Moderate Load Benchmarks
The rate at whichrequests were issued was initially 20 per second. The test involvedinitiating a total of 1 000 TCP connections and on each connection 1HTTP call was performed (a call consists of sending a request andreceiving a reply). The timeout option selects the number of secondsthat the client is willing to wait to hear back from the server. In ourcase it was set to 30 seconds. If this timeout expires, the toolconsiders the corresponding call to have failed.
We repeated alltests three times and took the medium test result for our comparison,with the exception of PostNuke. and e107. PostNuke was tested a totalof 8 times, as we could not believe the outcome initially. e107 wastested 4 times, as the e107 site blocked the requests during the firsttest, because e107 has a built-in flood protection.
We tested all sites with the following parameters and we allowed the server to calm down after each test run:
Timeout: 30 seconds
Number of concurrent requests: 20
Number of total connections: 1000.
So, if a site did not respond within 30 seconds, we concluded that the request timed out.
2.1. PostNuke
Maximum connect burst length: 11
Total: connections 1000 requests 529 replies 66 test-duration 100.897 s
Connection rate: 9.9 conn/s (100.9 ms/conn, <=625 concurrent connections)
Connection time [ms]: min 1255.8 avg 16905.8 max 29801.3 median 19422.5 stddev 9604.9
Connection time [ms]: connect 5020.4
Connection length [replies/conn]: 1.000
Request rate: 5.2 req/s (190.7 ms/req)
Request size [B]: 76.0
Reply rate [replies/s]: min 0.0 avg 0.7 max 3.0 stddev 1.0 (19 samples)
Reply time [ms]: response 16896.1 transfer 370.3
Reply size [B]: header 420.0 content 81315.0 footer 2.0 (total 81737.0)
Reply status: 1xx=0 2xx=66 3xx=0 4xx=0 5xx=0
CPU time [s]: user 1.67 system 45.59 (user 1.7% system 45.2% total 46.8%)
Net I/O: 52.6 KB/s (0.4*10^6 bps)
Errors: total 934 client-timo 934 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
2.2. Joomla
Maximum connect burst length: 1
Total: connections 1000 requests 1000 replies 1000 test-duration 56.244 s
Connection rate: 17.8 conn/s (56.2 ms/conn, <=198 concurrent connections)
Connection time [ms]: min 387.0 avg 5918.8 max 17308.0 median 6233.5 stddev 3427.5
Connection time [ms]: connect 1329.1
Connection length [replies/conn]: 1.000
Request rate: 17.8 req/s (56.2 ms/req)
Request size [B]: 83.0
Reply rate [replies/s]: min 13.5 avg 17.1 max 25.4 stddev 3.1 (11 samples)
Reply time [ms]: response 3448.4 transfer 1141.2
Reply size [B]: header 519.0 content 8621.0 footer 2.0 (total 9142.0)
Reply status: 1xx=0 2xx=1000 3xx=0 4xx=0 5xx=0
CPU time [s]: user 0.29 system 2.30 (user 0.5% system 4.1% total 4.6%)
Net I/O: 160.1 KB/s (1.3*10^6 bps)
Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
2.3. Xaraya
Maximum connect burst length: 16
Total: connections 1000 requests 781 replies 390 test-duration 98.585 s
Connection rate: 10.1 conn/s (98.6 ms/conn, <=612 concurrent connections)
Connection time [ms]: min 1136.6 avg 23984.7 max 50865.0 median 24306.5 stddev 12605.3
Connection time [ms]: connect 5243.6
Connection length [replies/conn]: 1.000
Request rate: 7.9 req/s (126.2 ms/req)
Request size [B]: 83.0
Reply rate [replies/s]: min 1.0 avg 3.9 max 8.4 stddev 2.2 (19 samples)
Reply time [ms]: response 20258.5 transfer 412.7
Reply size [B]: header 526.0 content 16521.0 footer 2.0 (total 17049.0)
Reply status: 1xx=0 2xx=390 3xx=0 4xx=0 5xx=0
CPU time [s]: user 1.54 system 39.18 (user 1.6% system 39.7% total 41.3%)
Net I/O: 66.5 KB/s (0.5*10^6 bps)
Errors: total 610 client-timo 610 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
2.4. e107
Maximum connect burst length: 1
Total: connections 1000 requests 1000 replies 1000 test-duration 50.075 s
Connection rate: 20.0 conn/s (50.1 ms/conn, <=8 concurrent connections)
Connection time [ms]: min 80.0 avg 153.9 max 362.5 median 143.5 stddev 31.4
Connection time [ms]: connect 0.7
Connection length [replies/conn]: 1.000
Request rate: 20.0 req/s (50.1 ms/req)
Request size [B]: 80.0
Reply rate [replies/s]: min 19.3 avg 20.0 max 20.4 stddev 0.3 (10 samples)
Reply time [ms]: response 153.1 transfer 0.0
Reply size [B]: header 346.0 content 6946.0 footer 0.0 (total 7292.0)
Reply status: 1xx=0 2xx=1000 3xx=0 4xx=0 5xx=0
CPU time [s]: user 8.28 system 25.93 (user 16.5% system 51.8% total 68.3%)
Net I/O: 143.8 KB/s (1.2*10^6 bps)
Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
2.5. Drupal
Maximum connect burst length: 85
Total: connections 1000 requests 1000 replies 1000 test-duration 58.164 s
Connection rate: 17.2 conn/s (58.2 ms/conn, <=235 concurrent connections)
Connection time [ms]: min 70.9 avg 6934.7 max 25599.4 median 6438.5 stddev 4465.9
Connection time [ms]: connect 1202.2
Connection length [replies/conn]: 1.000
Request rate: 17.2 req/s (58.2 ms/req)
Request size [B]: 83.0
Reply rate [replies/s]: min 10.8 avg 16.5 max 27.7 stddev 4.6 (11 samples)
Reply time [ms]: response 5667.6 transfer 64.8
Reply size [B]: header 579.0 content 4590.0 footer 2.0 (total 5171.0)
Reply status: 1xx=0 2xx=995 3xx=0 4xx=0 5xx=5
CPU time [s]: user 0.29 system 1.80 (user 0.5% system 3.1% total 3.6%)
Net I/O: 88.2 KB/s (0.7*10^6 bps)
Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
3. High Load Benchmarks
Therate at which requests were issued was 50 per second. The test involvedinitiating a total of 10 000 TCP connections and on each connection 1HTTP call was performed (a call consists of sending a request andreceiving a reply). The timeout option selects the number of secondsthat the client is willing to wait to hear back from the server. In ourcase it was set to 30 seconds. If this timeout expires, the toolconsiders the corresponding call to have failed.
3.1. PostNuke
Maximum connect burst length: 40
Total: connections 10000 requests 1323 replies 45 test-duration 250.822 s
Connection rate: 39.9 conn/s (25.1 ms/conn, <=1600 concurrent connections)
Connection time [ms]: min 2263.7 avg 18334.9 max 29988.5 median 22526.5 stddev 9464.3
Connection time [ms]: connect 8929.9
Connection length [replies/conn]: 1.000
Request rate: 5.3 req/s (189.6 ms/req)
Request size [B]: 76.0
Reply rate [replies/s]: min 0.0 avg 0.2 max 2.8 stddev 0.5 (49 samples)
Reply time [ms]: response 18003.9 transfer 329.8
Reply size [B]: header 450.0 content 82371.0 footer 2.0 (total 82823.0)
Reply status: 1xx=0 2xx=45 3xx=0 4xx=0 5xx=0
CPU time [s]: user 1.58 system 134.39 (user 0.6% system 53.6% total 54.2%)
Net I/O: 14.9 KB/s (0.1*10^6 bps)
Errors: total 9955 client-timo 9955 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
3.2. Joomla
Maximum connect burst length: 265
Total: connections 10000 requests 5190 replies 3920 test-duration 250.760 s
Connection rate: 39.9 conn/s (25.1 ms/conn, <=1565 concurrent connections)
Connection time [ms]: min 251.5 avg 20933.6 max 56767.0 median 19083.5 stddev 9344.5
Connection time [ms]: connect 9598.8
Connection length [replies/conn]: 1.000
Request rate: 20.7 req/s (48.3 ms/req)
Request size [B]: 83.0
Reply rate [replies/s]: min 0.2 avg 15.6 max 26.9 stddev 5.0 (49 samples)
Reply time [ms]: response 9718.7 transfer 1589.2
Reply size [B]: header 518.0 content 8530.0 footer 2.0 (total 9050.0)
Reply status: 1xx=0 2xx=3920 3xx=0 4xx=0 5xx=0
CPU time [s]: user 1.75 system 16.65 (user 0.7% system 6.6% total 7.3%)
Net I/O: 139.8 KB/s (1.1*10^6 bps)
Errors: total 6080 client-timo 6080 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
3.3. Xaraya
Maximum connect burst length: 60
Total: connections 10000 requests 2596 replies 593 test-duration 250.722 s
Connection rate: 39.9 conn/s (25.1 ms/conn, <=1666 concurrent connections)
Connection time [ms]: min 2535.2 avg 31292.4 max 50983.2 median 29611.5 stddev 11693.9
Connection time [ms]: connect 9433.3
Connection length [replies/conn]: 1.000
Request rate: 10.4 req/s (96.6 ms/req)
Request size [B]: 83.0
Reply rate [replies/s]: min 0.6 avg 2.4 max 9.0 stddev 1.7 (49 samples)
Reply time [ms]: response 23770.7 transfer 473.1
Reply size [B]: header 526.0 content 16531.0 footer 2.0 (total 17059.0)
Reply status: 1xx=0 2xx=593 3xx=0 4xx=0 5xx=0
CPU time [s]: user 1.42 system 116.69 (user 0.6% system 46.5% total 47.1%)
Net I/O: 40.2 KB/s (0.3*10^6 bps)
Errors: total 9407 client-timo 9407 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
3.4. e107
Maximum connect burst length: 218
Total: connections 10000 requests 6761 replies 5333 test-duration 239.428 s
Connection rate: 41.8 conn/s (23.9 ms/conn, <=1504 concurrent connections)
Connection time [ms]: min 153.2 avg 18901.1 max 58835.0 median 16706.5 stddev 9431.9
Connection time [ms]: connect 9109.6Connection length [replies/conn]: 1.000
Request rate: 28.2 req/s (35.4 ms/req)
Request size [B]: 80.0
Reply rate [replies/s]: min 2.0 avg 22.5 max 40.4 stddev 6.5 (47 samples)
Reply time [ms]: response 9831.2 transfer 0.0
Reply size [B]: header 344.0 content 6839.0 footer 0.0 (total 7183.0)
Reply status: 1xx=0 2xx=5333 3xx=0 4xx=0 5xx=0
CPU time [s]: user 0.82 system 12.47 (user 0.3% system 5.2% total 5.5%)
Net I/O: 158.5 KB/s (1.3*10^6 bps)
Errors: total 4667 client-timo 3778 socket-timo 0 connrefused 544 connreset 345
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
3.5. Drupal
Maximum connect burst length: 344
Total: connections 10000 requests 5033 replies 3889 test-duration 244.389 s
Connection rate: 40.9 conn/s (24.4 ms/conn, <=1664 concurrent connections)
Connection time [ms]: min 219.1 avg 23351.5 max 61186.9 median 21073.5 stddev 10799.6
Connection time [ms]: connect 10381.5
Connection length [replies/conn]: 1.000
Request rate: 20.6 req/s (48.6 ms/req)
Request size [B]: 83.0
Reply rate [replies/s]: min 5.9 avg 15.8 max 33.6 stddev 5.8 (47 samples)
Reply time [ms]: response 13293.2 transfer 85.3
Reply size [B]: header 578.0 content 4571.0 footer 2.0 (total 5151.0)
Reply status: 1xx=0 2xx=3846 3xx=0 4xx=0 5xx=43
CPU time [s]: user 0.38 system 4.81 (user 0.2% system 2.0% total 2.1%)
Net I/O: 81.7 KB/s (0.7*10^6 bps)
Errors: total 6111 client-timo 6111 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
4. Graphs
Thereis a lot more information that can be presented and gathered from thedata we collected during the benchmarks, but we decide to narrow ourfocus on two aspects: the ratio of total replies and the time it tookto complete the tests.
4.1. Graph 1: Ratio of total replies
Thisgraph shows the ratio of total replies in relation to the totalrequests. With a moderate load, Joomla, e107 and Drupal have noproblems to fulfill all requests. 100% of all requests are replied to.PostNuke has serious issues under moderate load. Xaraya is slightlybetter than PostNuke, but we need to keep in mind that Xaraya is an outof the box installation whereas the benchmarked PostNuke site is fullwith content collected over several years. Under heavy load there is anoticeable gap between PostNuke and Xaraya on one side and Joomla, e107and Drupal on the other side.

4.2. Graph 2: Average Time to complete benchmarks
Thisgraph shows the time it took to complete the tests. With a moderateload, Joomla, e107 and Drupal have no problems to fulfill all requestsin a timely manner. All 1000 requests are fulfilled withinapproximately 50 seconds. Both PostNuke and Xaraya take twice as longto complete the tests. Under heavy load, only e107 and Drupal arefaster in completing the benchmarks. e107 outperforms all other systems.

Please note:If you have performed similiar benchmarks with any of the givensystems, we‘d be very interested in seeing them. Please trackback tothis blog entry and/or leave a comment.
Posted bySascha EndlicherinGeek stuffat15:52|Comments (9)|Trackbacks (0)Mark as read
View as PDF:This entry |This month |Full blog