ActiveMQ Master/Slave KahaDB on OCFS2 shared file system

During my tests of shared storage clusters I wondered if ActiveMQ supports file locking on OCFS2 file system which I used on couple of occasions. While looking into it I came accross the following warning on the Apache project site:

OCFS2 Warning

Was testing using OCFS2 and both brokers thought they had the master lock - this is because OCFS2 only supports locking with fcntl and not lockf and flock, therefore mutex file locking from Java isn't supported.

From RedHat faq :

OCFS2: No cluster-aware flock or POSIX locks

GFS: fully supports Cluster-wide flocks and POSIX locks and is supported.

See this JIRA for more discussion: AMQ-4378

And true enough, when tested ActiveMQ-5.11.1 on one of the OCFS2 clusters, what happened is that both servers started as master due to lack of lock file on the share. Investigating further it came up that some good man created a patch that Apache never bothered to apply from some unknown reason. The case link AMQ-4694. The patch was attached to the ticket so thought I would try it and apply to latest 5.11.4 version.

Downloaded the source:

root@drbd01:~# aptitude install zip
root@drbd01:~# wget
root@drbd01:~# unzip
root@drbd01:~# cd activemq-parent-5.11.4/

and applied the patch:

root@drbd01:~/activemq-parent-5.11.4# vi ocfs2-locks.patch
root@drbd01:~/activemq-parent-5.11.4# patch -p0 -i ocfs2-locks.patch
patching file NOTICE
Hunk #1 succeeded at 45 with fuzz 2 (offset -4 lines).
patching file activemq-broker/pom.xml
patching file activemq-broker/src/main/java/org/apache/activemq/store/
patching file activemq-broker/src/main/java/org/apache/activemq/store/
Hunk #1 FAILED at 16.
1 out of 1 hunk FAILED -- saving rejects to file activemq-broker/src/main/java/org/apache/activemq/store/
patching file activemq-broker/src/main/java/org/apache/activemq/store/
patching file activemq-broker/src/main/java/org/apache/activemq/util/
patching file activemq-broker/src/main/java/org/apache/activemq/util/
patching file activemq-broker/src/main/java/org/apache/activemq/util/
patching file activemq-unit-tests/src/test/java/org/apache/activemq/util/
patching file assembly/pom.xml
Hunk #1 succeeded at 342 (offset 14 lines).
patching file assembly/src/main/descriptors/common-bin.xml
Hunk #1 succeeded at 191 (offset -9 lines).
patching file pom.xml
Hunk #1 succeeded at 881 with fuzz 1 (offset 44 lines).

Ok, so all good except it failed for the file. Easily fixed by applying that part of the patch manually:

root@drbd01:~/activemq-parent-5.11.4# vi activemq-broker/src/main/java/org/apache/activemq/store/
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * See the License for the specific language governing permissions and
 * limitations under the License.

import org.apache.activemq.util.LockFile;
import org.apache.activemq.util.LockResource;

 * An {@link AbstractResourceLocker} that utilizes a {@link LockFile}.
 * @org.apache.xbean.XBean element="shared-file-locker"
public class SharedFileLocker extends AbstractResourceLocker {

    protected LockResource newLockResource(File lockFileName) {
        return new LockFile(lockFileName, true);

Next installed Maven:

root@drbd01:~# wget
root@drbd01:~# tar -xzvf apache-maven-3.3.9-bin.tar.gz
root@drbd01:~# mv apache-maven-3.3.9 /usr/local/
root@drbd01:~# export PATH=/usr/local/apache-maven-3.3.9/bin:$PATH

Back to AMQ directory and build the project:

root@drbd01:~/activemq-parent-5.11.4# mvn -Dtest=false -DfailIfNoTests=false clean install -e -X

Install the built binaries:

root@drbd01:~# tar -xzvf /root/.m2/repository/org/apache/activemq/apache-activemq/5.11.4/apache-activemq-5.11.4-bin.tar.gz -C /opt/

and copy the setup from the previous AMQ version into the new one:

root@drbd01:/opt# cp -a apache-activemq-5.11.1/conf/{,jmx.access,jmx.password,jetty.xml,encompass-activemq.xml,star_encompasshost_com.jks} apache-activemq-5.11.4/conf/
root@drbd01:/opt# chown -R activemq\: apache-activemq-5.11.4
root@drbd01:/opt# rm activemq
root@drbd01:/opt# ln -s apache-activemq-5.11.4 activemq

The last bit is modifying the KahaDB locker to use the newly introduced shared-native-file-locker one:

root@drbd01:/opt# vi activemq/conf/activemq.xml
        <kahaDB directory="/share/activemq-data"
                  <shared-native-file-locker lockAcquireSleepInterval="5000"/>

After copying over the binary tarball to the other server, installing and performing the above steps, I started the server and the lock file was indeed created this time:

root@drbd01:/opt# ls -ltr /share/activemq-data/
total 160
drwxr-xr-x 3 activemq activemq     3896 Mar 17 15:27
-rw-r--r-- 1 root     root            5 Mar 17 15:30 test
-rw-r--r-- 1 activemq activemq        0 Mar 17 18:16 lock
-rw-r--r-- 1 activemq activemq 33030144 Mar 17 18:17 db-1.log
-rw-r--r-- 1 activemq activemq    12304 Mar 17 18:17 db.redo
-rw-r--r-- 1 activemq activemq    12288 Mar 17 18:17

and the other server can see it:

root@drbd02:/opt# start activemq && tail -f /opt/activemq/data/activemq.log | grep -i lock
2016-03-18 10:00:15,375 | INFO  | Database /share/activemq-data/lock is locked... waiting 5 seconds for the database to be unlocked. Reason: failed to acquire lock: [11] ( | | main
Written on March 18, 2016