Creating OES SM instances on 64 bit systems

I've already opened a bug on this against OES 10gR3 CP5, but in case anyone else runs into it before it gets fixed I wanted to blog it too. (NOTE: CP5 is when official support was introduced for running OES on a 64 bit system with a 64 bit JVM)

When you run the ConfigTool on a 64 bit Linux host you may see an error like this:

Check completed succesfully.
Starting to make changes ...
Creating SSM instance...
2010-12-22 14:36:46,952 [Main Thread] ERROR com.bea.security.SsmConfigTool.AlesConfig - Instance wizard returned error: 255
Done creating SSM instance.
Loading ALES Policy using policyloader. Please wait ...
2010-12-22 14:36:46,956 [/software/oracle/middleware/ales32-ssm/wls-ssm/instance/irmssm/bin/policyloader.sh] ERROR com.bea.security.SsmConfigTool.Daemon - Error During execution of requestd command 
java.io.IOException: java.io.IOException: /software/oracle/middleware/ales32-ssm/wls-ssm/instance/irmssm/bin/policyloader.sh: not found

As you can see the first error is that the instancewizard failed. THAT problem is probably occurring on your system because when you applied the CP you chose a 64 bit JVM but the instancewizard.sh script has a hard coded JAVA_HOME pointing to a 32 bit JVM. When the instance wizard runs (under the covers) the JVM is trying to load a 64 bit JNI library that the CP installed.

The fix?

Edit instancewizard.sh and change JAVA_HOME to point to the right JVM. Like so:

JAVA_HOME="/usr/java/latest"

Incidentally I found the root cause by running instancewizard.sh with the -log= and -log_priority= flags. You can run ./instancewazard.sh -h to get the command line help.