Craig Kaes replies:
XCP is a highly configurable platform so despite a number of large load tests we’ve done on Linux, none would serve as a direct comparison to the 1M user Solaris test.  The closest one to that configuration that we did was a 200k user test spread over five XCP routers, each running on a dual proc xeon 3.6Ghz machine.  Additionally we used 16 connection manager processes spread of eight similar machines.  The two largest differences between this test and the Solaris test were 1) PostgreSQL 8.1 (Linux) vs. Oracle 10 (Solaris) and 2) the Linux test was run before a number of configuration and code optimizations were identified and implemented.  In the Linux test, the database was definitely the limiting factor where in the Solaris test, we were only using about 1/10 of the resources on the database server.

It would certainly be interesting to run a large scale Linux test similar to what we’ve already done on Solaris — especially with the optimizations in and with the knowledge we’ve gained.

What amazed even us was that as the scale of the test increased, the efficiency of Jabber XCP also increased while CPU utilization decreased significantly.

Do you mean exactly what you write or are there some “per user” missing?

——–

Craig Kaes Says:

November 17, 2006 at 4:06 pm
Yeah — cpu decreased per box — not overall. This decrease was linear which, together with the linear addition of the usersa shows overall linear scalability and also shows that we were too conservative in the number of users that we added as we added additional hardware

——–

Craig Kaes Says:

November 17, 2006 at 4:06 pm
To clarify - As we increased the number of processors, CPU utilization per box decreased, not overall CPU utilization. What this showed was that we could have added more users with each box that we added and that the growth was indeed linear. It would be absurd to assert exponential scalability

———-

Craig Kaes Says:
November 17, 2006 at 11:31 am

We used 2 4-core T2000s for our connection managers and 6 8-core T2000s for our routers. Because of the nature of the baseline and the constraints of the test, we had 3 of the 8 cores disabled on each of the
6 router boxes. It would be a delight to see what we could do if we used all 8 cores for those tests.

As for system tuning, we got access to a beta tool from Sun called CoolTuner (http://www.sun.com/download/products.xml?id=4501bae1). It added the following to the system file, but I’m not sure that any of it was actually needed. We were using Solaris 10 and the OS really never got in our way.

set ip:ip_squeue_bind=0
set ip:ip_squeue_fanout=1
set ipge:ipge_tx_syncq=1
set ipge:ipge_srv_fifo_depth=16000
set ipge:ipge_bcopy_thresh=512
set ipge:ipge_dvma_thresh=1
set pcie:pcie_aer_ce_mask=0×1 # Not necessary in T2000+ shipping June
2006
set rlim_fd_cur=260000
set rlim_fd_max=260000
set maxphys=1048576
set md:md_maxphys=1048576