Dave Stanley's Blog

In general, the out of the box ActiveMQ configuration works very well for the majority of usecases. One place where you will want to tweak things is in the area of supporting large numbers of concurrent connections.

The good news is that ActiveMQ is well capable and with the tweaks below you should be well on your way to supporting thousands of concurrent connections. Without further adieu, here’s what you need to do:

1) In your brokers /activemq.xml, enable the nio transport.

 
<!-- The transport connectors ActiveMQ will listen to -->
 <transportConnectors>
 
   <!-- use tcp port for network connectors only -->
   <transportConnector name="openwire" uri="tcp://localhost:61616"/>
 
   <!-- use nio port for producers/consumers -->
   <transportConnector name="openwire nio" uri="nio://localhost:62828?useQueueForAccept=false"/>
 
 </transportConnectors>

2) Again, in your brokers /activemq.xml, configure a destinationPolicy with optimizedDispatch=true, for example:

   <amq:destinationPolicy>
   <amq:policyMap>
    <amq:policyEntries>
     <amq:policyEntry queue=">"
       optimizedDispatch="true"
       lazyDispatch="false"
       producerFlowControl="false"
       memoryLimit="128 mb"
       strictOrderDispatch="true">
      <amq:dispatchPolicy>
       <amq:strictOrderDispatchPolicy />
      </amq:dispatchPolicy>
      <amq:messageGroupMapFactory>
       <amq:simpleMessageGroupMapFactory/>
      </amq:messageGroupMapFactory>
      <amq:subscriptionRecoveryPolicy>
       <amq:timedSubscriptionRecoveryPolicy recoverDuration="360000" />
      </amq:subscriptionRecoveryPolicy>
     </amq:policyEntry>
    </amq:policyEntries>
   </amq:policyMap>
  </amq:destinationPolicy>

For more info on optimizedDispatch see Hiram‘s article here

3) At the OS level increase the number of file descriptors available to the broker

>ulimit -n 4096

4) Use a broker with a version > 5.3.0.3

5) Tweak your TCP/IP layer to support the connection load

On some operating systems you may need to tune the tcp/ip stack to be able to handle the incoming connection load. Below are the settings I’ve found worked well for Linux and Solaris.

Linux tcp tuning settings:

sudo /sbin/sysctl -w net.core.netdev_max_backlog=3000
sudo /sbin/sysctl -w net.ipv4.tcp_fin_timeout=15
sudo /sbin/sysctl -w net.core.somaxconn=3000

Note: The net.core.netdev_max_backlog controls the size of the incoming packet queue for upper-layer (java) processing.

Solaris tcp tuning settings:

# ndd -set /dev/tcp tcp_fin_wait_2_flush_interval 67500
# ndd -set /dev/tcp tcp_keepalive_interval 30000
# ndd -set /dev/tcp tcp_conn_req_max_q 8000

Lastly, when testing I’ve found the ActiveMQ jmx mbeans and jconsole to be indispensable in terms of monitoring thread counts and the number of concurrent connections. If all is configured properly each 16 incoming connections should result in one new thread in the broker VM.

These minor changes should allow you to support several thousand concurrent connections on conventional hardware. If you happen to try the settings above let me know how it goes.

I’ve hit the error in the title a few times when using the maven bundle plugin to create osgi bundles.

[INFO] Cannot find lifecycle mapping for packaging: 'bundle'.
Component descriptor cannot be found in the component repository: org.apache.maven.lifecycle.mapping.LifecycleMappingbundle.

The fix it, set the ‘extensions’ element to true as shown below.

<plugin>
                <groupId>org.apache.felix</groupId>
                <artifactId>maven-bundle-plugin</artifactId>
                <version>2.0.1</version>
                <extensions>true</extensions>
                <configuration>
                    .... 
 
                </configuration>
</plugin>

Hope this saves you some digging!

Over on SOS Adrian has posted a set of slides that describe how to build multi-protocol, opensource, web services using JAX-WS.

The presentation gives a great overview of the architecture that allows Celtix Enterprise to deploy services that simulateneosly support multiple payloads (SOAP, XML, JSON) and transports (HTTP, JMS, AMQP). It also give some very helpful pointers on JAX-WS development in general – well worth checking it out.

Today I setup CXF, a new services framework project currently

under incubation at Apache. CXF is the merging of the Objectweb Celtix project and the Codehaus XFire project. Its still

early days for the project but if you check out the project goals you can get a good idea of the featureset that CXF will

eventually offer. Today I just wanted to get as far as running a couple of the demos.

Installing:

CXF has external dependencies on JDK 1.5 and Maven 2.x. As there is no official release yet you need the build the project

directly from the subversion repository. If you want to build the samples you will also need ant installed.

1. Install JDK 1.5 (approx 130MB)


The Sun download page has three different distributions, JDK with NetBeans IDE, JDK with JEE and just the straight 1.5 JDK. I

just opted for your basic 1.5 JDK.

2. Install Maven 2.0.4(approx 1.15MB)


The Maven install is just a zip/gz file. You just need to download the zip and unzip it on your system. Then you need to add

the /bin directory to your path.

>cd /x1/maven2.0.4
>wget http://www.apache.org/dist/maven/binaries/maven-2.0.4-bin.tar.gz
>tar -zxvf maven-2.0.4-bin.tar.gz
>export PATH=/x1/maven2.0.4/bin:$PATH

On windows download the .zip version from the link above and use Winzip to unpack the zip. Then set the path as shown here.

>set Path=\bin;%Path%

3. Install subversion windows or
unix (approx 11MB).

The windows link is to the subversion installer gui. This version of subversion uses BerkeleyDB 4.3. If you have any other applications on your system that might use a different version of BerkeleyDB (e.g Orbix), keep this in mind as you may see dll conflicts between the two versions. Once installed you should be able to run ‘svn ?’ from a command prompt and get a list of available svn commands.

4. If necessary install ant

Install is similar to maven, download and unzip to your system. Then setup the paths as shown:

>set ANT_HOME=c:\dev\ant
>set PATH=%PATH%;%ANT_HOME%\bin

>export ANT_HOME=/x1/ant
>export PATH=${PATH}:${ANT_HOME}/bin

Checking out the CXF source code:

To checkout the CXF source run :

>cd /x1/cxf
>svn co http://svn.apache.org/repos/asf/incubator/cxf/trunk

This is a good point to go and make yourself a cup of tea as this and the next step take some time.

Building the source code:

Once the source code is checked out, setup your environment so that java 1.5 is in your path. If you get this wrong you will see an error similar to this when you try and build with a JDK < JDK 1.5.

INFO] ------------------------------------------------------------------------
[ERROR] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Compilation failure

Failure executing javac, but could not parse the error:
javac: invalid target release: 1.5

I also set things up such that my /bin directory was in the path. To build the project run:

>cd /x1/cxf/trunk
>mvn install

The first time you build CXF, maven will download what seems like an endless amount of dependency jars to a local maven

repository on your machine. On windows this is below <> and on linux this was below <>. I have seen periodic download

failures where maven fails to download a dependency but re-running mvn install tended to work the second time around.

Another thing to watch out for on windows is that there's a fair bit of shuffling going on in the CXF codebase right now.
If the case of a file is renamed, svn on windows cannot detect this and this will leave you scratching your head as to why

the project won't build. If you end up with a compile error that doesn't make sense try a 'mvn clean' and then a 'mvn install' and see if that fixes it.

Building the distribution snapshot:

Once the build completes you need to cd into the /distribution directory and run 'maven install' a second time to build a

distributable version of cxf. This will produce a distribution with a samples directory which includes the 'hello_world' demo

that we want to run.

>cd /x1/cxf/trunk/distribution
>mvn install

Installing the distribution snapshot:

The distribution build produces two distribution files (cxf-2.0-incubator-M1-SNAPSHOT.tar.gz and cxf-2.0-incubator-M1-SNAPSHOT.zip). Take the appropriate distribution and unzip it in a separate directory to the checked out code, for example:

>cp /x1/cxf/trunk/distribution/target/cxf-2.0-incubator-M1-SNAPSHOT.tar.gz /x1/cxf-distro/
>cd /x1/cxf-distro/
>tar -zxvf cxf-2.0-incubator-M1-SNAPSHOT.tar.gz

Now you have something you can test with.

Running the hello world demo:

>cd /x1/cxf-distro/cxf-2.0-incubator-M1-SNAPSHOT/samples/hello_world
>ant
>ant -projecthelp
Buildfile: build.xml

Main targets:

 build           build demo client and server
 client          run demo client
 client-servlet  run demo client hitting servlet
 server          run demo server
Default target: build

Now we run the server and the client

>ant server&

server:
     [java] Starting Server
     [java] Server ready...

>ant client

client:
     [java] Invoking sayHi...
     [java] Server responded with: Bonjour

     [java] Invoking greetMe...
     [java] Server responded with: Hello dstanley

     [java] Invoking greetMe with invalid length string, expecting exception...

     [java] Invoking greetMeOneWay...
     [java] No response from server as method is OneWay

     [java] Invoking pingMe, expecting exception...
     [java] Expected exception: PingMeFault has occurred: PingMeFault raised by server
     [java] FaultDetail major:2
     [java] FaultDetail minor:1

Enabling Logging:

So after running the demo, it looked like everything worked but there was not much to see. The next step for me was to enable

some logging and get a peek at whats happening under the covers.

>cd /x1/cxf-distro/cxf-2.0-incubator-M1-SNAPSHOT/etc
>emacs logging.properties

See here for more details on configurable logging levels. I had to set both '.level' and 'java.util.logging.ConsoleHandler.level' in my logging.properties file to get logging output in my console.

.level= INFO
java.util.logging.ConsoleHandler.level = INFO 

I had to set the level to FINEST (very verbose!) to see the message going over the wire. Would be nice if this was logged at

INFO level.

Thats about it for this time. Hope this article helps. If you spot any problems feel free to point them out.

Prior to JDK 1.5, javac ignored classpath manifest.mf entries in jar files, for example:


- someclass.java references classes in child.jar

- parent.jar includes child.jar in its classpath manifest


Running:

javac -classpath parent.jar someclass.java

you receive “class not found” errors for the classes included in child.jar.

I can see how in some cases this was inconvenient as it meant that if a product (such as Orbix) splits its functionality over a lot of .jar files, then you have to list most/all of these .jar files in your CLASSPATH when compiling the project.

The Java 1.5 compiler is now aware of Manifest.mf files within jar files. This introduces some unexpected side effects as the javac 1.5 looks to recursively walk/unwind the jar stack. Depending on the number of jars in the manifest, this can increase compile times considerably. If you run into this problem the quickest solution looks to be to remove the jar containing the manifest from your CLASSPATH. Other JDK tools such as javap, javadoc, etc.. still ignore the Manifest.mf file.

Thanks to Ciaran McHale for the heads up on this one.

In JDK 1.4.x monitoring the JVMs memory usage was an arduous task requiring a special unsupported JVM. I’d heard JDK 1.5 had improved support but I haven’t had a reason to look at it until this week. It was a nice surprise to see how easy it is to enable monitoring:

1. Add -Dcom.sun.management.jmxremote to your java command line (and start your process).

2. Start [java 1.5]\jdk\bin\jconsole. Your done.

Example:

>java -version
java version "1.5.0_06"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
Java HotSpot(TM) Client VM (build 1.5.0_06-b05, mixed mode)

>java -Dcom.sun.management.jmxremote SoapDemo

>jps
2672 SoapDemo
3784 Jps

>jconsole 2672

In this example we check the jvm version is 1.5 or greater, enable the JMX agent, and launch jconsole with the correct proccess id (2672). The console monitors the process and allows you to track memory usage in all generations of the JVM Heap,
Thread State and you can also enable/disable verbose gc dynamically.

A more detailed overview of jconsole can be found here.