Friday, July 22, 2011

Where can I find the most common log files for Middleware (including FMW, ADP, JVMD) monitoring?

While working on middleware (including OEM/FMW/ADP/JVMD) monitoring, when I run into an issue, I need to access logs. OEM logs can be found at various locations. Different components do logging in different files.
As per OEM documentation, there are various log files (documented at

The most common log files that I often access are:

OMS logs
OMS logs can be found in various directories and all contain information respective to that tier.

1. OMS logs are under
<EM_INSTANCE_BASE>/em/<OMS_NAME>/sysman/log
e.g. /u01/app/oracle/gc_inst/em/EMGC_OMS1/sysman/log

a. emoms.trc – this is most important trace file that contains oms actions and errors.
b. emoms.log - It has oms starting/stopping/any error when it generates an error.
c. emctl.log - emctl commands are logged in this file

2. Weblogic Server - Since em application run inside weblogic server. So these logs are often very important and contain useful info.
<EM_INSTANCE_BASE>/user_projects/domains/<domain_name>/servers/<SERVER_NAME>/logs/<SERVER_NAME>.out
e.g.
/u01/app/oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out

3. WebTier logs can be found under
<EM_INSTANCE_BASE>/<webtier_instance_name>/diagnostics/logs/OHS/<ohs_name>
e.g.
/u01/app/oracle/gc_inst/WebTierIH1/diagnostics/logs/OHS/ohs1

4. OPMN log
<EM_INSTANCE_BASE>/<webtier_instance_name>/diagnostics/logs/OPMN/<opmn_name>
e.g.
/u01/app/oracle/gc_inst/WebTierIH1/diagnostics/logs/OPMN/opmn1

EMAgent logs
EMAgent logs can be found in $AGEN_HOME/sysman/log directory

a. emagent.log - Log file used by the Agent process.  Contains all informational messages in local language.
b. emagent.nohup - Log file for the Agent monitor. This contains all actions the monitor has performed.
c. emagent.trc - Trace file used by the Agent process. Contains all the trace messages in English.
d. emagent_memdump_<time>.trc - Optional trace file, generated by an 'emctl status agent memory' command. Contains the overview of the memory usage of the Agent at that point in time.
e. emagentfetchlet.log - The fetchlet log files used by the EM Agent for certain data-gathering tasks
f. emagentfetchlet.trc - The fetchlet trace file used by the EM Agent for certain data-gathering tasks.
g. emagent_perl.trc - Trace file for the PERL scripts. This includes the PERL metrics and the discovery
h. emdctl.log - Agent control utility log file
i. emdctl.trc - Agent control utility trace file

ADP logs
When ADP is registered on same OEM UI
1. ADP Manager logs
Would be present under
$ADP_HOME/log/manager-log*.csv

2. ADP Agent logs
Would be present under target Application server directory
<APPSERVER_HOME>/AcseraAgent/* and
<APPSERVER_HOME>/AcseraLogs/*

In addition to these directories, adp agent also marks its foot print in application server specific log files for deploy/undeploy/start/stop/configuration files etc.

3. ADP UI logs
<MW_HOME>/sysman/ocamm/Log/*
e.g.
/u01/app/oracle/middleware/sysman/ocamm/Log/*

JVMD logs
When JVMD is installed with OEM
1. JVMD Manager logs
<EM_INSTANCE_BASE>/user_projects/domains/<domain_name>/servers/<SERVER_NAME>/logs/<SERVER_NAME>.out
e.g.
/u01/app/oracle/gc_inst/user_projects/domains/GCDomain/servers/EMAD4JMANAGER1/logs/EMAD4JMANAGER1.out

2. JVMD Agent logs
JVMD agent logs its output on stdout. So all its logs get logged in container output file e.g. if
jamagent is deployed on OMS weblogic server, you can find jamagent logs in
<EM_INSTANCE_BASE>/user_projects/domains/<domain_name>/servers/<SERVER_NAME>/logs/<SERVER_NAME>.out
e.g.
/u01/app/oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out

3. JVMD UI logs
Some logs goto <EM_INSTANCE_BASE>/em/<OMS_NAME>/sysman/log/emoms.log
e.g. /u01/app/oracle/gc_inst/em/EMGC_OMS1/sysman/log/emoms.log

and sometimes, you will find errors under
<EM_INSTANCE_BASE>/user_projects/domains/<domain_name>/servers/<SERVER_NAME>/logs/<SERVER_NAME>.out
e.g.
/u01/app/oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out

Wednesday, July 13, 2011

How to fix "java.lang.NoClassDefFoundError: oracle/jrf/PortabilityLayerException" ?

In EMGC 11 env., while monitoring FMW targets if you encounter following Error in emagent log:
2011-07-11 02:08:37,133 [nmefmgr_getJNIFetchlet] ERROR fetchlets.FabricFetchletWrapper logp.251 - Invocation Error in Fabric Fetchlet wrapper:
java.lang.reflect.InvocationTargetException
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:597)
  at oracle.sysman.emd.fetchlets.FabricFetchletWrapper.getMetric(FabricFetchletWrapper.java:109)
  at oracle.sysman.emd.fetchlets.FetchletWrapper.getMetric(FetchletWrapper.java:382)
...
Caused by: java.lang.NoClassDefFoundError: oracle/jrf/PortabilityLayerException
  at oracle.soa.management.internal.ejb.EJBLocatorImpl.lookupBean(EJBLocatorImpl.java:741)
  at oracle.soa.management.internal.ejb.EJBLocatorImpl.lookupFinderBean(EJBLocatorImpl.java:728)
  at oracle.soa.management.internal.ejb.EJBLocatorImpl.<init>(EJBLocatorImpl.java:165)
  at oracle.soa.management.facade.LocatorFactory.createLocator(LocatorFactory.java:35)
  at oracle.sysman.emd.fetchlets.FabricFetchlet.getLocator(FabricFetchlet.java:813)
  at oracle.sysman.emd.fetchlets.FabricFetchlet.getMetric(FabricFetchlet.java:194)
  ... 6 more
Caused by: java.lang.ClassNotFoundException: oracle.jrf.PortabilityLayerException
  at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
  at oracle.sysman.emd.fetchlets.thirdparty.TPJMXFetchletClassLoader.loadClass(TPJMXFetchletClassLoader.java:78)
..
Reason: This error is appearing as emagent requires "jrf-api.jar" to be present in its CLASSPATH.

A simple workaround to fix the issue is:
1. Stop the agent
2. Add following jarfile to CLASSPATH property in emd.properties of emagent
$MW_HOME/oracle_common/modules/oracle.jrf_11.1.1/jrf-api.jar
where MW_HOME is the Middleware Home of FMW target .
3. Restart the agent.

Sunday, July 03, 2011

EMGC11: JVMD Manager is deployed successfully, but UI shows manager as inactive

JVMD manager is deployed successfully.  JVMD Manager log shows that native library
is loaded. Manager get registered with EMDB. One can see that manager appears
on UI under JVM Dignostics->Setup->JVMs & Managers tab. But manager status is
shown as inactive. No data appears for any of the JVMs.

Cause: In case of JVMD, data interaction happens in two directions:
1. From JVMD manager to agent and vice versa via socket communication.
2. For real time data, EMUI calls JVMD manager via t3 calls.
Somewhere 2nd communication is broke.

Action: In case if OMS and JVMD manager are deployed on different machines
where there is a firewall between the two. Do check if t3 port is accessible
from JVMD manager machine to OMS machine. One can find related information under
emoms.log also.