I've already opened a bug on this against OES 10gR3 CP5, but in case anyone else runs into it before it gets fixed I wanted to blog it too. (NOTE: CP5 is when official support was introduced for running OES on a 64 bit system with a 64 bit JVM)
When you run the ConfigTool on a 64 bit Linux host you may see an error like this:
Check completed succesfully. Starting to make changes ... Creating SSM instance... 2010-12-22 14:36:46,952 [Main Thread] ERROR com.bea.security.SsmConfigTool.AlesConfig - Instance wizard returned error: 255 Done creating SSM instance. Loading ALES Policy using policyloader. Please wait ... 2010-12-22 14:36:46,956 [/software/oracle/middleware/ales32-ssm/wls-ssm/instance/irmssm/bin/policyloader.sh] ERROR com.bea.security.SsmConfigTool.Daemon - Error During execution of requestd command java.io.IOException: java.io.IOException: /software/oracle/middleware/ales32-ssm/wls-ssm/instance/irmssm/bin/policyloader.sh: not found
As you can see the first error is that the instancewizard failed. THAT problem is probably occurring on your system because when you applied the CP you chose a 64 bit JVM but the instancewizard.sh script has a hard coded JAVA_HOME pointing to a 32 bit JVM. When the instance wizard runs (under the covers) the JVM is trying to load a 64 bit JNI library that the CP installed.
Edit instancewizard.sh and change JAVA_HOME to point to the right JVM. Like so:
Incidentally I found the root cause by running instancewizard.sh with the -log= and -log_priority= flags. You can run ./instancewizard.sh -h to get the command line help.