In general, the out of the box ActiveMQ configuration works very well for the majority of usecases. One place where you will want to tweak things is in the area of supporting large numbers of concurrent connections.
The good news is that ActiveMQ is well capable and with the tweaks below you should be well on your way to supporting thousands of concurrent connections. Without further adieu, here’s what you need to do:
1) In your brokers
<!-- The transport connectors ActiveMQ will listen to --> <transportConnectors> <!-- use tcp port for network connectors only --> <transportConnector name="openwire" uri="tcp://localhost:61616"/> <!-- use nio port for producers/consumers --> <transportConnector name="openwire nio" uri="nio://localhost:62828?useQueueForAccept=false"/> </transportConnectors>
2) Again, in your brokers
<amq:destinationPolicy> <amq:policyMap> <amq:policyEntries> <amq:policyEntry queue=">" optimizedDispatch="true" lazyDispatch="false" producerFlowControl="false" memoryLimit="128 mb" strictOrderDispatch="true"> <amq:dispatchPolicy> <amq:strictOrderDispatchPolicy /> </amq:dispatchPolicy> <amq:messageGroupMapFactory> <amq:simpleMessageGroupMapFactory/> </amq:messageGroupMapFactory> <amq:subscriptionRecoveryPolicy> <amq:timedSubscriptionRecoveryPolicy recoverDuration="360000" /> </amq:subscriptionRecoveryPolicy> </amq:policyEntry> </amq:policyEntries> </amq:policyMap> </amq:destinationPolicy>
For more info on optimizedDispatch see Hiram‘s article here
3) At the OS level increase the number of file descriptors available to the broker
>ulimit -n 4096
4) Use a broker with a version > 5.3.0.3
5) Tweak your TCP/IP layer to support the connection load
On some operating systems you may need to tune the tcp/ip stack to be able to handle the incoming connection load. Below are the settings I’ve found worked well for Linux and Solaris.
Linux tcp tuning settings:
sudo /sbin/sysctl -w net.core.netdev_max_backlog=3000 sudo /sbin/sysctl -w net.ipv4.tcp_fin_timeout=15 sudo /sbin/sysctl -w net.core.somaxconn=3000
Note: The net.core.netdev_max_backlog controls the size of the incoming packet queue for upper-layer (java) processing.
Solaris tcp tuning settings:
# ndd -set /dev/tcp tcp_fin_wait_2_flush_interval 67500 # ndd -set /dev/tcp tcp_keepalive_interval 30000 # ndd -set /dev/tcp tcp_conn_req_max_q 8000
Lastly, when testing I’ve found the ActiveMQ jmx mbeans and jconsole to be indispensable in terms of monitoring thread counts and the number of concurrent connections. If all is configured properly each 16 incoming connections should result in one new thread in the broker VM.
These minor changes should allow you to support several thousand concurrent connections on conventional hardware. If you happen to try the settings above let me know how it goes.
Hi Dave,
Thanks for the explanation, very helpful.
One question though, in the following line:
What is the point of “useQueueForAccept” (couldn’t find anything on it), and do you absolutely need to specify a name for the connector?
Regards,
Maarten
Hi Maarten,
Glad it helped.
The useQueueForAccept controls wether the incoming socket is handled immediately or added to workqueue where its handled by a separate worker. In the case where
you have a large spike of incoming concurrent connections (e.g. server went down and clients are trying to reconnect) I found it worked better to handle each connection as it comes. Inevitably if the load is high enough some connection attempts will timeout, but if you are using the ActiveMQ failover transport – which you should, you can make this transparent to clients. In my testing I definitely got some connection setup failures when trying to connect >500 clients at the same time.
On the name for the connector, the name for the connector is optional. ActiveMQ will default it internally to the connector’s uri if its not set. The name is handy for debugging as I think its used when logging..
/Dave
Hi Dave
I am a bit confused regarding having both “openwire” and “openwire nio”
and also regarding having another port “62828″ (??)
do we really need both?
And also – what connection url should the clients use – “nio://localhost:62828?useQueueForAccept=false”
??
Please elaborate a bit more if possible
Thank you!
O.K.
PS:
And also, we are connecting from thousands of .NET clients using “NMS”… and there is no protocol option of “nio:” in NMS…
is it ok to use just “tcp:” then?
as far as I understand NIO is still TCP and from the client’s side it should be transparent whether its handled using nio or “old io” on the server side?
am I right or missing something important?
please help!!
thank you!
O.K.
I just wanted to try to express myself my clearly:
I am confused between “use tcp port for network connectors only”
and “use nio port for producers/consumers”
isn’t producers/consumers also a network connector?
or are there some other kinds of “network connectors”?
Thank you!
cheers,
O.K.
> I am confused between “use tcp port for network connectors only”
> and “use nio port for producers/consumers”
So on this I meant have your network connector connect to the standard tcp/transport rather than the nio transport.
> isn’t producers/consumers also a network connector?
No in activemq land, a network connector is a broker to broker connection. They connect two or more broker instances into a network of brokers. The subscription information is shared among all brokers in the network so if you have a producer on broker A and a consumer on broker B. The messages will be transparently transferred from broker A to the consumer on broker B over the network connector.
>do we really need both?
No, you can just enable the nio transport and it will handle all incoming connections using nio.
>And also – what connection url should the clients use – “nio://localhost:62828?useQueueForAccept=false”
Just the regular tcp:// transport url for the producers and consumers. The nio is configuring how the incoming connection is handled on the broker side.
> And also, we are connecting from thousands of .NET clients using “NMS”… and there is no protocol option of “nio:” in NMS…
> is it ok to use just “tcp:” then?
Yes absolutely.
>as far as I understand NIO is still TCP and from the client’s side it should be transparent whether its handled using nio or “old io” on the server side?
Yep thats right. Its not a client side concern. This is all on the server side. We are trying to minimize the number of broker threads created when you have a large incoming connection load.
Hope that helps
/Dave
Thanks a lot Dave!
It is all very clear now.
One last question:
re:
>4) Use a broker with a version > 5.3.0.3
I downloaded just a month ago and looked again now and also in the console
when it start and I don’t see “5.3.0.3″ anywhere… just “”5.3.0″…
where should I look to confirm that it is “….0.3″ (assuming you actually meant that)?
Also, I guess somewhat important – what Java version is recommended (we still use 1.5)
The reason I want to be very pedantic is that I am following the instructions from here and configured everything as suggested, but I still see about 2 threads per connection as opposed to 1 thread per 16 connections…
so does not seem that we got it quite right yet…
and also I am quite consistently getting a rather scary java crash
- have you ever seen anything like that?
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# SIGBUS (0xa) at pc=0xffffffff7ee00c10, pid=20145, tid=3611
#
# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_16-b02 mixed mode)
# Problematic frame:
# C [libc_psr.so.1+0xc10]
#
# An error report file with more information is saved as hs_err_pid20145.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
Please help!
Thank you!
cheers,
O.K.
Hi Oleg,
I used the fuse version for my testing so you could try <– this. JDK 1.5 should be fine.
In terms of the thread counts are you using queue’s or topics? I just realized I just set a destination policy on the queue, but if your using topics you may need to also setup a destination policy on the topic. Remember to make sure you enabled optimizedDispatch.
On the crash, would need to see the stack from the hs_err_pid, but you could also try mailing the activemq user list for that one. The folks on the list are usually very responsive.
/Dave
Thanks Dave!
We are using queues…. not topics…
Yes I think I will start posting a whole batch of questions to the mailing list forum very soon…
Now, interestingly when I reverted the settings (OptimizedDispatch etc)
back to the “vanilla” Out of the Box configuration, the crash error went away…
and here is what is pleasantly surprising.. I am actually able to have and sustain 5000 clients (each a producer and a consumer) with just plain TCP (not NIO) and those plain original settings (just with RAM values increased all over the place)
Are there any disadvantages of having it this way?
Or we actually should to try get working the recommendations that this post recommends…
Also, I should mention that we so far tried to have one IN queue and one OUT queue and have all of 5000 clients use them… and do a selector on “WHERE CustomeHeaderCLIENTID=’BLAHBLAH’”… but it seem that the consumers then becomes a serious bottleneck when the queue has a backlog (10-20K messages queued up)
Am I right in my understanding that this is what should be expected because the queue is not indexed in any way by this customer header we use for selector and therefor each consumer session does a FULL SCAN of sorts and potentially locks and blocks and when we have 5000 of this it becomes a bottleneck….
so we are going to change it now so that we have a dedicated queue for each consumer…
(and as far as I understand ActiveMQ now _can_ have 5000 queues easily)?
Are we on a right path here?
i.e. would you recommend one huge queue and 5000 clienst doing a “selector” on it or 5000 dedicated queues instead…
are there some “golden rules” regarding kind of considerations?
Thank you!
cheers,
O.K.
BTW, we are seriously contemplating getting a help from FUSE as we are a big PROGRESS customer already (for SonicMQ)… at least as an a consulting/training engagement (btw, by any chance maybe you or other ActiveMQ committers from FUSE feel like traveling to Australia? – that would be great if we got get one of the actual engineers to interrogate here
I actually talked to Gary Tully about the behaviour your seeing and it turns out there have been a number of recent upgrades to the way the dispatch threading works within the broker, so that explains why you can get so many connections now with the OOTB config.
The queue dispatch thread used to be per destination, but in recent 5.3.x versions its using a shared thread pool factory and respects the dedicated task runner configuration.
In terms of the selector/vs partitioned destination question, the latter (partitioning) will be better as the selectors are limited by the pagedInMessages window.
On the support question, thats very exciting. I would say use this form when your ready to go down that road
Cheer’s
/Dave
Hi Oleg Kiorsak,
I have a requirement where there are around 800 topic consumers and one publisher, Can you help send your configuration, i am more interested in the memory setting you had done to handle 5000 clients.
For my case messages are getting struck for some subscribers.
At least post some of the important tweaks you had done for ur scenario, i will change accordingly for topics.
Thanks & Regards,
Mahesh