Days of ... (problem)

SOLVED: Coherence 3.6 problem starting two nodes on the same machine

Thu, 10 May 2012 12:39:06 GMT

I am trying to configure Weblogic 10.3.5 with Coherence cluster and two nodes. Everything is running on my local machine. I configured Coherence with WLST and now trying to run both nodes. One node starts and second fails with following exception:

2012-05-10 12:57:37.681/6.175 Oracle Coherence GE 3.6.0.4 (thread=Environment.Background.Executor:Thread-1, member=n/a): Error while starting cluster: java.lang.NullPointerException at com.tangosol.coherence.component.net.TcpRing.ensureSelector(TcpRing.CDB:7) at com.tangosol.coherence.component.net.TcpRing.setServerSocketChannel(TcpRing.CDB:20) at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:510) at com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11) at com.tangosol.coherence.component.util.SafeCluster.startCluster(SafeCluster.CDB:3) at com.tangosol.coherence.component.util.SafeCluster.restartCluster(SafeCluster.CDB:7) at com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluster(SafeCluster.CDB:26) at com.tangosol.coherence.component.util.SafeCluster.start(SafeCluster.CDB:2) at com.tangosol.net.CacheFactory.ensureCluster(CacheFactory.java:998) at com.tangosol.net.DefaultConfigurableCacheFactory.ensureServiceInternal(DefaultConfigurableCacheFactory.java:923) at com.tangosol.net.DefaultConfigurableCacheFactory.ensureService(DefaultConfigurableCacheFactory.java:892) at com.oracle.coherence.environment.extensible.ExtensibleEnvironment.ensureService(ExtensibleEnvironment.java:497) at com.tangosol.net.DefaultConfigurableCacheFactory.ensureCache(DefaultConfigurableCacheFactory.java:874) at com.tangosol.net.DefaultConfigurableCacheFactory.configureCache(DefaultConfigurableCacheFactory.java:1231) at com.tangosol.net.DefaultConfigurableCacheFactory.ensureCache(DefaultConfigurableCacheFactory.java:290) at com.oracle.coherence.common.util.ObjectProxyFactory.ensureCache(ObjectProxyFactory.java:285) at com.oracle.coherence.common.util.ObjectProxyFactory.onDependenciesSatisfied(ObjectProxyFactory.java:307) at com.oracle.coherence.environment.extensible.dependencies.DependencyTracker.processLater(DependencyTracker.java:204) at com.oracle.coherence.environment.extensible.dependencies.DependencyTracker.processLater(DependencyTracker.java:46) at com.oracle.coherence.common.events.processing.AbstractAsynchronousEventProcessor$1.run(AbstractAsynchronousEventProcessor.java:53) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619)

I configured servers to run on the same port and enabled Unicast Port Auto Adjust, thinking that Coherence would be smart enough to figure out that port is in use.

But it is not that smart!

But Oracle Metalink has more information about it:

Cause

Coherence 3.6 has since Patch 2 been enhancing its network throughput by using two open UDP ports for each member. For this Coherence picks Unicast listener + 1 for the second port. It has been implemented as a part of COH-3722.

Note that Coherence 3.6.1 and onwards will not have the NullPointerException but a more meaningful error message.

Solution

Separate nodes which run on the same machine by at least two ports.

For example:

When you specify localport=6000 for one node, it will also use port 6001.

The next node you can specify to use localport=6002 will also use 6003

Again the next node you can specify to use localport=6004 will also use 6005

etc

I reconfigured Unicast ports accordingly and the problem is solved!

The case of missing JNDI branches

Sun, 20 Nov 2011 22:02:42 GMT

I’ve got a complaint from one of the developers: the JMS resources, defined in his local instance of Weblogic server, are defined correctly, but he can’t use them from the application he developing – the names are not present in JNDI tree. The problem is that in our environment the same scripts are used to create all Weblogic instances, from local development through production.

Quick look through Weblogic Console JNDI browser confirmed that the JMS resources are not in the tree. I took a look for exceptions or errors in the log files – everything looks ok, the logs are clean. Then I tried to change one of the parameters of the JMS resources and then Weblogic complained with the message saying that JMS resource is not yet started. And in the console it looks like it started ok.

Then I took a look at the Git logs of the recent changes for this type of the Weblogic domain and the only change was an introduction of the JDBCStores for JMS resources. But where are local development machines are pointed to? Is it local Oracle XE database? No, they are pointed to the shared database, used by other environment. This is clearly wrong! I don’t think that JDBCStore implementation is smart enough to recognise that connections are coming from different domains. So I removed JDBCStore usage from development domain and restarted it – voila, JMS resources are now in JNDI tree!

So Weblogic was misconfigured, but was silently ignoring the fact that it can not recognise data from JDBCStore and manifested it through JNDI tree. How weird!

Problem testing Postfix

Thu, 10 Dec 2009 10:16:44 GMT

I am installing Postfix on my unmanaged VPS hosting machine and I am doing it first time ever. I followed the installation manual on Ubuntu site and everything looks all right except one tiny thing: I can’t send email using Telnet. Here is the response:

421 4.3.0 collect: Cannot write ./dfnBA9gfVO030273 (bfcommit, uid=0, gid=111): No such file or directory

I googled for an answer, but nothing useful came up so I tried to debug the problem myself. By looking into mail logs (/var/log/mail.log, found by examining /etc/syslog.conf) I found following strange response:

 start postfix/master[15758]: fatal: bind 0.0.0.0 port 25: Address already in use

That means only one thing: there is another process listening on this port. Hmm, what could it be? Sendmail, perhaps. I uninstalled it, according to guides I was using, but it was still running.

It’s time to stop it and start postfix!

sudo service sendmail stop



sudo service postfix restart

Problem solved!

VMWare DHCP service wont’start - SOLVED

Fri, 08 May 2009 12:43:22 GMT

I use VMWare to run a bunch of different Linux VMs on my home PC, which is running Vista SP1. Today I encountered strange thing: Linux VM won’t get an IP address from VMWare DHCP. I’ve opened VMWare Virtual Networks management console (as Administrator, of course, or it won’t let you make and save any changes) just to find out that DHCP service is not started. I tried to start it from there, but it failed to do so.

So I went to Vista Services console and tried to start “VMWare DHCP service from there”. Again, failed to start with no explanation, even cryptic one. So I went to Windows Events viewer and there, under Windows Logs-System, I found a lot of messages from VMNetDHCP:

The description for Event ID 2 from source VMnetDHCP cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

Can't open C:ProgramDataVMwarevmnetdhcp.conf: Access is denied.

/ The data is invalid

the message resource is present but the message is not found in the string/message table

Though the message says that the message is not found, it has enough information to fix the problem.

So, I’ve taken a look at the security properties for C:ProgramDataVMWare folder and found that SYSTEM account somehow is not in a list of permitted users. The only account, who was permitted to access that folder, was my Windows user account. So I added SYSTEM account with all permissions to this folder, waited when Windows propagated my change to all files and tried to start VMWare DHCP service again. It worked!

Then I simply restarted network within Linux VM (sudo /etc/init.d/networking restart) and got back to work.

JBOSS Seam 2.1.1GA on JBOSS AS 5.0.1GA: Table is not mapped

Thu, 07 May 2009 18:37:46 GMT

I created sample JBOSS Seam project in Eclipse and decided to generate entity code from existing database. It worked fine, but when I try to run it, it shows me that Hibernade is not able to map entity beans to the corresponding tables. The error message is like this:

org.hibernate.hql.ast.QuerySyntaxException: <your table> is not mapped

I spent around three hours trying to read all available information on it, but most of it just useless – it looks like people don’t know whats happening, so they recommend bizarre tricks. Like, for example, having empry seam.settings file in each folder, meh.

Finally, I found the solution in Seam Jira: https://jira.jboss.org/jira/browse/JBSEAM-3821

They rocemmend to to force Hibernate to use Seam’s EntityManager by changing two files:

1. components.xml

<persistence:entity-manager-factory name="bookingDatabase" installed="false"/>
    
    <persistence:managed-persistence-context name="em" auto-create="true"
       entity-manager-factory="#{bookingDatabase}" persistence-unit-jndi-name="java:/bookingEntityManagerFactory"/>

2. persistence.xml

<property name="jboss.entity.manager.factory.jndi.name" value="java:/bookingEntityManagerFactory"/>

Hope that Google will find that page and the proper solution would be much more easily found!