DBA Arena

There are reporting programs and data updating programs that need to run either periodically, or on an ad hoc basis. These programs, which run in the background while users continue to work on other tasks are run using the Concurrent Processing architecture. Concurrent Processing is an Oracle E-Business Suite feature that allows these non–interactive and potentially long-running functions to be executed efficiently alongside interactive operations on a specialized server, the Concurrent Processing Server.
Processes that run on the Concurrent Processing server are called Concurrent Requests.
When you submit such a request, either through HTML-based or Forms-based applications, a row is inserted into a database table specifying the program to be run. A Concurrent Manager then reads the applicable requests in the table, and starts the associated concurrent program.

Concurrent Manager:

• It is an executable that is registered as a program library within Oracle E-Business Suite, and which runs in its own operating system process.
• Runs operating system processes called target processes (often referred to as workers), each of which can start one concurrent program at a time.
• Can optionally run an immediate program (a concurrent program that runs in the concurrent manager's own operating system process) eg: PL/SQL programs.
• Can be allowed to run any concurrent program, or be specialized to run certain programs.
• Operates during the days and times defined by a work shift.

Important Concurrent Managers:

• Internal Concurrent Manager (ICM): It controls the behavior of all of the other managers, and because the ICM is the boss, it must be running before any other managers can be activated. The main functions of the ICM are to start up and shutdown the individual concurrent managers as defined by their workshifts, monitors for process failure and cleans up if a failure occurs.

• Conflict Resolution Manager (CRM): It ensures that incompatible concurrent requests do not run together. However, the ICM can be configured to take over the CRM's job to resolve incompatibilities.

• Standard Manager (SM): The SM functions to run any reports and batch jobs that have not been defined to run in any specific product manager.

Tuning the Concurrent Manager:

1. Tuning the Internal Concurrent Manager

2. Purging Concurrent Requests

3. Troubleshooting Oracle Apps performance problems

4. Adjusting the Concurrent Manager Cache Size

5. Analyzing the Oracle Apps Dictionary Tables
1. Tuning the Internal Concurrent Manager
The ICM performance is affected by the three important Oracle parameters PMON cycle, queue size, and sleep time.

• PMON cycle: This is the number of sleep cycles that the ICM waits between the time it checks for concurrent managers failures, which defaults to 20. You should change the PMON cycle to a number lower than 20 if your concurrent managers are having problems with abnormal terminations.

• Queue Size: The queue size is the number of PMON cycles that the ICM waits between checking for disabled or new concurrent managers. The default queue size of 1 PMON cycle should be used.

• Sleep Time: The sleep time parameter indicates the seconds that the ICM should wait between checking for requests that are waiting to run. The default sleep time is 60, but you can lower this number if you see you have a lot of request waiting (Pending/Normal). However, reducing this number to a very low value may cause excessive cpu utilization.

2. Purging Concurrent Requests
One important area of Concurrent Manager tuning is monitoring the space usage for the subsets within each concurrent manager. When the space in FND_CONCURRENT_PROCESSES and FND_CONCURRENT_REQUESTS exceed 50K, you can start to experience serious performance problems within your Oracle Applications. When you experience these space problems, a specific request called "Purge Concurrent Requests And/Or Manager Data" should be scheduled to run on a regular basis. This request can be configured to purge the request data from the FND tables as well as the log files and output files that accumulate on disk.

3. Adjusting the Concurrent Manager Cache Size
Concurrent manager performance can also be enhanced by increasing the manager cache size to be at least twice the number of target processes. The cache size specifies the number of requests that will be cached each time the concurrent manager reads from the FND_CONCURRENT_REQUESTS table. Increasing the cache size will boost the throughput of the managers by attempting to avoid sleep time.

4. Analyzing Oracle Apps Dictionary Tables for High Performance
It is also very important to run the request Gather Table Statistics on these tables:

• FND_CONCURRENT_PROCESSES

• FND_CONCURRENT_PROGRAMS

• FND_CONCURRENT_REQUESTS

• FND_CONCURRENT_QUEUES

Run the request "Analyze All Index Column Statistics" on the indexes of these tables. Since the APPLSYS user is the owner of these tables, so you can also just run the request Analyze Schema Statistics for APPLSYS.
To troubleshoot performance, a DBA can use various types of trace. A module trace, can be set by enabling the module's profile option Debug Trace from within the applications. Second, to enable trace for a specific request, log in as a user with the System Administrator responsibility. Navigate to Concurrent -> Program -> Define. Query for the request that you want to enable trace. At the bottom right of the screen you can check the box Enable Trace.

Concurrent Request Life Cycle

Concurrent Processing Database Tables

Various database tables are employed by the concurrent processing architecture:

Caution: Do not update these tables manually. You can (subject to your organization's archiving requirements) periodically run the "Purge Concurrent Requests and/or manager data" program to prevent these tables growing too large.

Concurrent Processing Operations

Because the Internal Concurrent Manager controls all the other managers, it must be running before any other manager can be activated. Once the ICM has been activated, it starts a Service Manager on each node that is enabled for concurrent processing. Acting as an agent of the ICM, the Service Manager starts the concurrent managers on its node, excluding any managers that have been deactivated, or that have no current work shift. The ICM can be activated and deactivated from the operating system prompt, or Oracle Applications Manager. It can also be deactivated (but not activated) from the Administer Concurrent Managers form.
When the ICM is initiated on UNIX, the $FND_TOP/bin/startmgr program is invoked. This calls $FND_TOP/bin/batchmgr, which then:
Starts a shell process
Starts the ICM process using the command FNDLIBR, with startup parameters FND, CPMGR, and FNDCPMBR
Creates log files (std.mgr and wnnn.mgr) in $APPLCSF/$APPLLOG
Normally, startmgr is run by the user account that owns the application software (for example, applmgr). This account must have write privileges to the log and out directories where the log and output files respectively are written.
The ICM starts up a Service Manager on each node that is enabled for concurrent processing, by instructing the node's Applications Listener (which is dedicated to Concurrent Processing) to spawn a process running the Service Manager executable (FNDSM). The Applications Listener must be configured to source the Oracle E-Business Suite environment file before FNDSM is spawned. Following startup, the Service Manager acts as an agent of the ICM to start and stop concurrent managers on that node, according to their defined work shifts.
Concurrent manager processes on a specific node can be seen by running the UNIX commands:
ps –ef | grep FNDLIBR
ps –ef | grep FNDSM

For every process that was successfully started at operating system level, the ICM inserts a row into FND_CONCURRENT_PROCESSES table. It then updates the RUNNING_PROCESSES column to reflect the actual running processes as shown in FND_CONCURRENT_QUEUES.

Viewing Concurrent Processing Output

The output from a concurrent processing job goes through several stages before being displayed to the user.
The Concurrent Processing server communicates with the database server via Oracle Net.
The log or output file associated with a concurrent request is passed back to the Report Review Agent, also known as the Web Review Agent.
The Report Review Agent passes a file containing the entire report to the Forms services.
The Forms services pass the report back to the user's browser one page at a time.

Viewing Concurrent Processing Output

Concurrent manager log files:

• O.S level:-
$INST_TOP/logs/appl/conc/log OR $APPLCSF/$APPLLOG
The following are the specific log file names.
Internal Concurrent Manager Log: *.mgr
Concurrent Manager Log: w.mgr
Transaction Manager: t.mgr
Conflict Resolution Manager: c.mgr
Request Log = l.req
Request Output = o.out

• SQL prompt:
Connect to sqlplus as apps user and run this query

SELECT 'ICM_LOG_NAME=' || fcp.logfile_name
FROM fnd_concurrent_processes fcp, fnd_concurrent_queues fcq
WHERE fcp.concurrent_queue_id = fcq.concurrent_queue_id
AND fcp.queue_application_id = fcq.application_id
AND fcq.manager_type = '0'
AND fcp.process_status_code = 'A';

Sample Output:
'ICM_LOG_NAME='||FCP.LOGFILE_NAME
-------------------------------------------------------------
ICM_LOG_NAME=/applprod/comndev/admin/log/DEV_moon1/DEV_0426.mgr

• Front end:-
Log in as a user with the System Administrator responsibility.
Navigate to Concurrent -> Manager -> Administer -> (Select a manager) Processes -> Manager Log.

Location of concurrent manager output files:

$INST_TOP/logs/appl/conc/out OR $APPLCSF/$APPLOUT

Check status of the concurrent manager:

There are 3 ways to find the status of the manager:
• O.S level
ps –ef | grep FNDLIBR

• Front end:
System Administrator > Concurrent > Manager > Administer

• SQL script
SQL> @$FND_TOP/sql/afimchk.sql

Start / Stop concurrent manager:

$ADMIN_SCRIPTS_HOME
adcmctl.sh start apps/apps_pwd (To start the concurrent manager)
adcmctl.sh stop apps/apps_pwd (To stop the concurrent manager)
Individual managers can also be started from front end by navigating to System Administrator > Concurrent > Manager > Administer

Submitting a concurrent request:

Log in as a user with the System Administrator responsibility. Navigate to Concurrent -> Requests -> Submit a New Request -> Single Request -> Name: Active Users -> Submit

Finding a concurrent request:

System Administrator -> Concurrent -> Requests

You can filter the search results with the following parameters:
• Request ID: A particular concurrent request.
• Name: All concurrent requests with this name.
• Date Submitted: Date at which the request was submitted.
• Date Completed: Date at which the request got completed.
• Status: Current status of the request.
• Phase: Current phase of the request.
• Requestor: Request submitted by.