SOLVED: Coherence 3.6 problem starting two nodes on the same machine

Thu, 10 May 2012 12:39:06 GMT

I am trying to configure Weblogic 10.3.5 with Coherence cluster and two nodes. Everything is running on my local machine. I configured Coherence with WLST and now trying to run both nodes. One node starts and second fails with following exception:

2012-05-10 12:57:37.681/6.175 Oracle Coherence GE 3.6.0.4 (thread=Environment.Background.Executor:Thread-1, member=n/a): Error while starting cluster: java.lang.NullPointerException at com.tangosol.coherence.component.net.TcpRing.ensureSelector(TcpRing.CDB:7) at com.tangosol.coherence.component.net.TcpRing.setServerSocketChannel(TcpRing.CDB:20) at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:510) at com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11) at com.tangosol.coherence.component.util.SafeCluster.startCluster(SafeCluster.CDB:3) at com.tangosol.coherence.component.util.SafeCluster.restartCluster(SafeCluster.CDB:7) at com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluster(SafeCluster.CDB:26) at com.tangosol.coherence.component.util.SafeCluster.start(SafeCluster.CDB:2) at com.tangosol.net.CacheFactory.ensureCluster(CacheFactory.java:998) at com.tangosol.net.DefaultConfigurableCacheFactory.ensureServiceInternal(DefaultConfigurableCacheFactory.java:923) at com.tangosol.net.DefaultConfigurableCacheFactory.ensureService(DefaultConfigurableCacheFactory.java:892) at com.oracle.coherence.environment.extensible.ExtensibleEnvironment.ensureService(ExtensibleEnvironment.java:497) at com.tangosol.net.DefaultConfigurableCacheFactory.ensureCache(DefaultConfigurableCacheFactory.java:874) at com.tangosol.net.DefaultConfigurableCacheFactory.configureCache(DefaultConfigurableCacheFactory.java:1231) at com.tangosol.net.DefaultConfigurableCacheFactory.ensureCache(DefaultConfigurableCacheFactory.java:290) at com.oracle.coherence.common.util.ObjectProxyFactory.ensureCache(ObjectProxyFactory.java:285) at com.oracle.coherence.common.util.ObjectProxyFactory.onDependenciesSatisfied(ObjectProxyFactory.java:307) at com.oracle.coherence.environment.extensible.dependencies.DependencyTracker.processLater(DependencyTracker.java:204) at com.oracle.coherence.environment.extensible.dependencies.DependencyTracker.processLater(DependencyTracker.java:46) at com.oracle.coherence.common.events.processing.AbstractAsynchronousEventProcessor$1.run(AbstractAsynchronousEventProcessor.java:53) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619)

I configured servers to run on the same port and enabled Unicast Port Auto Adjust, thinking that Coherence would be smart enough to figure out that port is in use.

But it is not that smart!

But Oracle Metalink has more information about it:

Cause

Coherence 3.6 has since Patch 2 been enhancing its network throughput by using two open UDP ports for each member. For this Coherence picks Unicast listener + 1 for the second port. It has been implemented as a part of COH-3722.

Note that Coherence 3.6.1 and onwards will not have the NullPointerException but a more meaningful error message.

Solution

Separate nodes which run on the same machine by at least two ports.

For example:

When you specify localport=6000 for one node, it will also use port 6001.

The next node you can specify to use localport=6002 will also use 6003

Again the next node you can specify to use localport=6004 will also use 6005

etc

I reconfigured Unicast ports accordingly and the problem is solved!

The case of missing JNDI branches

Sun, 20 Nov 2011 22:02:42 GMT

I’ve got a complaint from one of the developers: the JMS resources, defined in his local instance of Weblogic server, are defined correctly, but he can’t use them from the application he developing – the names are not present in JNDI tree. The problem is that in our environment the same scripts are used to create all Weblogic instances, from local development through production.

Quick look through Weblogic Console JNDI browser confirmed that the JMS resources are not in the tree. I took a look for exceptions or errors in the log files – everything looks ok, the logs are clean. Then I tried to change one of the parameters of the JMS resources and then Weblogic complained with the message saying that JMS resource is not yet started. And in the console it looks like it started ok.

Then I took a look at the Git logs of the recent changes for this type of the Weblogic domain and the only change was an introduction of the JDBCStores for JMS resources. But where are local development machines are pointed to? Is it local Oracle XE database? No, they are pointed to the shared database, used by other environment. This is clearly wrong! I don’t think that JDBCStore implementation is smart enough to recognise that connections are coming from different domains. So I removed JDBCStore usage from development domain and restarted it – voila, JMS resources are now in JNDI tree!

So Weblogic was misconfigured, but was silently ignoring the fact that it can not recognise data from JDBCStore and manifested it through JNDI tree. How weird!

Days of ... (weblogic)

SOLVED: Coherence 3.6 problem starting two nodes on the same machine

The case of missing JNDI branches