How to Troubleshoot Concurrent Manager Start Up Issue Related to Service Manager

纵横四海

Applies to:
Oracle Application Object Library - Version: 11.5.9 to 12.0.4
Information in this document applies to any platform.

Goal
Concurrent manager does not start if the profile option "Concurrent:GSM Enabled" is set to "Y" at site level the following error occurs in ICM log file:
============================
Could not initialize the Service Manager FNDSM_<HOSTNAME>_<SID>.
============================

In this case only the Internal Manager is coming up and remaining managers do not come up as expected.

Setting the profile option "Concurrent:GSM Enabled" to "N" at site level allows the managers, except those detailed below, to start thus the issue is only with GSM services.
=====================
Output Post Processor
Workflow Agent Listener Service
Workflow Mailer Service
Workflow Document Web Services Service
=====================

For the above managers to be started the profile option "Concurrent:GSM Enabled" has to be set to "Y" at site level.

For a Service Manager to come up as expected the below criteria should be fulfilled :

1. There should be a Service Manager defined.
2. Entries in the FND_NODES should be correct.
3. Entries in tnsnames.ora and listener.ora should be correct on CP node.
4. Able to do a tnsping for all entries related to FNDSM in tnsnames.ora on CP node.
5. There should not be any errors in APPS Listener log file on CP node and it should be running.
6. Able to ping the hostname and the IP Address of the hostname.
7. FNDSM should not give the "segmentation fault" at the time of execution.

Solution
Verify all the steps that are necessary to start a Service Manager successfully.

1. There should be a Service Manager defined.

Run the following statement as the APPS user:

select CONCURRENT_QUEUE_NAME from FND_CONCURRENT_QUEUES where CONCURRENT_QUEUE_NAME like 'FNDSM%';

The above statement should return a value related to the CP node.

If the above does not return any value then please do the following:
Go to $FND_TOP/patch/115/sql

Connect SQLPLUS as APPS user and run the following script :


Reference Note 218893.1 :How to Create The Service Manager 'FNDSM' on Oracle Applications

(Note : The same "afdcm037.sql" is used for Release 12 also ).

This script will create libraries for FNDSM and create managers for preexisting nodes.

2. Verify the entries in the FND_NODES as correct.

Run the following statement as the APPS user:

select node_name "Node Name", node_mode "Mode", support_cp "C",
support_web "W", support_admin "A", support_forms "F"
from FND_NODES;

If the above statement returns any invalid node or IP address follow the steps provided in the following note to clean up and repopulate the table.

Note 260887.1 : Steps to Clean Nonexistent Nodes or IP Addresses from FND_NODES

3. Verify entries in tnsnames.ora on CP node.

The below entry should present in tnsnames.ora on CP node :

FNDSM_OracleTEST_VIS=
           (DESCRIPTION=
                      (ADDRESS=(PROTOCOL=tcp)(HOST=Oracletest.idc.Oracle.com)(PORT=1626))
                (CONNECT_DATA=
                      (SID=FNDSM)
                 )
            )

FNDSM_OracleTEST.Oracle.com_VIS=
          (DESCRIPTION=
                      (ADDRESS=(PROTOCOL=tcp)(HOST=Oracletest.idc.Oracle.com)(PORT=1626))
                 (CONNECT_DATA=
                       (SID=FNDSM)
                )
          )

Where OracleTEST is the "hostname", VIS is the "SID" and ".Oracle.com" is domain.

4. Check that tnsping is successful for all entries related to FNDSM in tnsnames.ora on CP node.

$ tnsping FNDSM_OracleTEST_VIS

TNS Ping Utility for Linux: Version 8.0.6.3.0 -

(c) Copyright 1997 Oracle Corporation. All rights reserved.

Attempting to contact (ADDRESS=(PROTOCOL=tcp)(HOST=Oracletest.idc.Oracle.com)(PORT=1626))
OK (50 msec)

$ tnsping FNDSM_OracleTEST.Oracle.com_VIS

TNS Ping Utility for Linux: Version 8.0.6.3.0 -

(c) Copyright 1997 Oracle Corporation. All rights reserved.

Attempting to contact (ADDRESS=(PROTOCOL=tcp)(HOST=Oracletest.idc.Oracle.com)(PORT=1626))
OK (0 msec)

5. Verify that no error exists in APPS Listener log file on CP node and it is running.

APPS Listener should be up and running for Service Manager to be up and running as expected. There should not be any error in the APPS Listener log file. If there will be any error in APPS Listener log file then the Service Manager will not start properly. Check the APPS Listener log file. You can find this APPS Listener log file in one directory up of $TNS_ADMIN named as "apps_<SID>.log".


6. Are you able to ping the host name/IP Address of the host.

Host name and the related IP address should be pingable to get a connection like below :

$ hostname
OracleTEST.Oracle.com
$ ping OracleTEST.Oracle.com
PING OracleTEST.Oracle.com (10.100.100.10) 56(84) bytes of data.
64 bytes from OracleTEST.Oracle.com (10.100.100.10): icmp_seq=0 ttl=64 time=0.029 ms
64 bytes from OracleTEST.Oracle.com (10.100.100.10): icmp_seq=1 ttl=64 time=0.028 ms

--- OracleTEST.Oracle.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.028/0.028/0.029/0.005 ms, pipe 2
$ ping 10.100.100.10
PING 10.100.100.10 (10.100.100.10) 56(84) bytes of data.
64 bytes from 10.100.100.10: icmp_seq=0 ttl=64 time=0.028 ms
64 bytes from 10.100.100.10: icmp_seq=1 ttl=64 time=0.028 ms

--- 10.100.100.10 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.028/0.028/0.028/0.000 ms, pipe 2

7. Verify FNDSM does not give the "segmentation fault" at the time of execution.

Go to $FND_TOP/bin

$ FNDSM <- Enter

It should not give any segmentation fault here. If it is giving any segmentation fault then it means that the executable is not relinked well. Relink the executable as below.

$ adrelink.sh force=y "fnd FNDSM"

For Linux Operating System Please follow the notes below to get correct the segmentation fault by FNDSM.

Note 430812.1 :After Linux Version Upgrade FNDSM Does Not Spawn

Note 423170.1 :Concurrent Manager Fails to Start Because of a 'Could not start Service Manager FNDSM_<server>_<SID>' Error
If the concurrent managers are still not coming up even with profile option "Concurrent:GSM Enabled" is set to "Y" at site level and having followed all the above steps, then please log a Service Request with ATG Component Tools Team under product 510, component CP .

References
Note 218893.1 - How to Create The Service Manager 'FNDSM' on Oracle Applications
Note 260887.1 - Steps to Clean Nonexistent Nodes or IP Addresses from FND_NODES
Note 423170.1 - Concurrent Manager Fails to Start Because of a 'Could not start Service Manager FNDSM__' Error
Note 430812.1 - After Linux Version Upgrade FNDSM Does Not Spawn
Note 555081.1 - After Doing a Clone All the Concurrent Managers Do Not Start


该贴已经同步到 纵横四海的微博