Archive for the ‘technical’ category

Benchmarking Solr

August 16, 2007

There was some discussion on the Vufind about moving from Tomcat to Jetty. I first wanted to see if it was possible to run this so I got the latest nightly build from Solr to see which packages were needed to run the server. I then grabbed the latest Jetty (6.1.5) since the version in Solr’s build was 6.1.3. I packaged the same files that were in Solr’s distribution and dropped Vufind’s schema and config file into Jetty and fired it up. Voila…it worked like a champ.

The thing I really wanted to know is if this Jetty version would perform in a similar fashion to Tomcat. What I did to test was set up two visualized servers on the same box. Each were set up with the exact same hardware (2GB RAM, 1 processor, bridged 1GB network, running Ubuntu 7.04 server). I also used the same Java tuning on both machines (“-server -Xmx1024m -Xms1024 -XX:+UseParallelGC -XX:+AggressiveOpts“). The only difference between the two was that one ran Tomcat and the other Jetty.

For the test, I indexed our library’s 1.8+ million catalog records on both machines which both chewed through the records in about 9 hours. To do the actual testing, I used JMeter to query both systems at the same time using a few scenarios that I thought might possibly be “real.”

In the first test, I sent 10 users with 100 queries for the book title “Flashman” to see what happened. I was pretty impressed with the results:

Server Samples Average Median Min Max
Jetty 1000 4ms 4ms 1ms 17ms
Tomcat 1000 3ms 4ms 0ms 28ms

You know, we might get a few more users than just 10 at a time, so I ramped it up to 100 users doing 10 queries. Again, there wasn’t much of a difference.

Server Samples Average Median Min Max
Jetty 1000 12ms 8ms 1ms 565ms
Tomcat 1000 9ms 7ms 1ms 530ms

Now to really ramp things up with 100 users doing 100 queries

Server Samples Average Median Min Max
Jetty 100000 9ms 6ms 1ms 2349ms
Tomcat 100000 9ms 6ms 1ms 1844ms

And, just for kicks, 1000 users with 10 queries

Server Samples Average Median Min Max
Jetty 10000 32ms 6ms 0ms 5643ms
Tomcat 10000 26ms 5ms 0ms 4295ms

With median results within a millisecond of each other, Andrew went ahead and swapped out Tomcat in favor for Jetty for its smaller footprint. I have to say that any time I’ve needed to do anything with JSP, I’ve opted to go with Tomcat. More because I know the name, but I think I’m going to keep Jetty on my list from now on! I want to take a closer look at their ANT and Eclipse Plugins!