How to configure the jvm diagnostics for Weblogic in EMGC11g

As I explained in my previous post, steps for configuring JVM Diagnostics with EMGC 11g are very much same. Only additional step for WebLogic Server is to discover WebLogic Domain and Servers and then deploy jamagent on discovered WLS targets.

To discover WebLogic Domain, goto Targets->Middleware-> Add 'Oracle Fusion Middleware/WebLogic Domain'. It will take you to WebLogic Server discovery page where you need provide details like:

·         WLS Admin Server hostname

·         Port

·         WLS username

·         WLS username passwd

·         Unique Domain Identifier

·         Agent - with which you would like to discover WLS domain. One can use remote agent for WLS discovery but with remote agent some of the WLS monitoring is limited.

One the WebLogic domain has discovered and all needed targets have been added.

One can go to WebLogic Server Target, goto Weblogic Domain HomePage, click on Domain menu->Select JVM Diagnostics->Deploy Agent option

With Deploy Agent page, one can deploy jamagent on selected WLS targets.

Note: Ensure that 'Select Monitoring' checkbox is selected, so that one agent is deployed, monitoring can be started.

How to configure JVM Diagnostics (AD4J) with EMGC11g

Note: Two one-offs have been released since EMGC 11g release.

Atleast one of these patches is MUST to apply in order for JVMD (AD4J) to work properly.

However, it is recommended to use the latest patch i.e.

p9901141_111010_Generic for the manager

p9930117_111010_Generic for the agent

I'm assuming that user is configuring JVMD for first time in EMGC. If customer has already configured JVMD previously then there would be additional steps to undeploy old ad4j manager and agent before applying the patch and some steps would be bit different.

1. Goto OMS home i.e. %cd $ORACLE_HOME/ad4j

2. %unzip ad4j.zip

3. Make sure that you run DeployAD4JManager.sh in a terminal that has no previous env. variable set.

   %sh DeployAD4JManager.sh

   This script would create a new managed server and would also deploy jammanager.ear

4. Once the AD4J manager is deployed, one can ensure that manager is up and running by looking at Manager status in Targets->Middleware->JVM diagnostics->Console Setup->JVM & Managers->Manager Status

5. Deploy agent either using automated deployment or manual deployment.

a. For automated deployment, ensure that WLS domain is discovered. Goto Targets->Middleware->WLS Domain-> JVM Diagnostics ->Deploy Agent Page. Enter relevant details like Weblogic Server admin username and password. Select relevant ad4j manager, provide details of host m/c where agent is deployed with which WLS domain was discovered. On submission, it creates a job that deploys agent on Target Weblogic Server.

This automated deployment feature works just for Weblogic Servers.

Once Job is successful, it creates a pool equivalent to domain name and agents deployed on target JVM's under that domain would be listed under that Pool. One can view JVMD data either by selecting Domain->JVM Diagnostics->Summary/Details or using Targets->Middleware->JVM diagnostics link at bottom.

b. For manual deployment, one can download jamagent.war from Targets->Middleware->JVM diagnostics->Setup->Download page and deploy it manually on target app server/JVM. Such JVM's by default show up under Default Pool. and in OOB install, Default Pool doesn't have monitoring enabled by default. One has to first select Poll Enabled for Default Pool, so that historical data can start appearing under this pool.

 

Ad4j (JVM Diagnostics) vs OCAMM (ADP)

Both AD4J (JVM Diagnostics) and OCAMM (ADP) are application performance mgmt solutions. Both of these focus on java-j2ee applications diagnostics.  Often we come across this question, how does one know which solution should be used where? What is the underlying technology? Though there is some common functionality overlap between these solutions. But in general, functionally and technically, both the tools are very different.

Similarities:

1. Both of these solutions focus on java-j2ee applications diagnostics.  
2. Both the solutions have their respective agents that need to be deployed on the target app server that one wants to monitor.
3. Both have similar architecture like manager, agent and UI. Where agent is deployed on target app server (JVM), monitors its activities, gathers data and sends it to respective manager. Manager in turn processed data and collect it in repository. UI displays the entire data.

Let me list some of the differences:

1.  Ad4j focuses primarily on exact response time and invocation count(cost/sample) metrics for service-layer constructs (methods/requests), but is limited in its ability to view arbitrary method metrics. However it does not have any higher level(application) context in relation to metadata constructs such as BPEL, WLI, or Portal. While CAMM focuses primarily on these higher-level constructs, but is limited in its ability to view arbitrary method metrics. Therefore, the two solutions actually compliment to each other .
2. Ad4j focuses on JVM level diagnostics. It shows top methods, top requests, top DB states and Top SQLs for a set of JVM(s). It has limitation in terms that it cannot relate these methods/requests to application at high level. While CAMM knows application components and models(like  WebCenter, Spring Beans, Composites, OSB, Web Services etc.) and can relate data with it.
3. On deploying ad4j agent, it starts showing real time data for a particular JVM like its real time thread system and memory system of a JVM.   When ADP is deployed to a Oracle WebLogic domain, IBM WebSphere cell, or an Oracle SOA Suite cluster, it automatically discovers information about this particular domain including all deployed applications, configuration, resources, and others. ADP displays this information in the Monitoring tab under Oracle Enterprise Manager.
4. Ad4j is a sampling tool. It samples the JVM metrics and monitors the JVM, gathers data and then predicts trends and analyzes data. ad4j does runtime Objects analysis. It uses two techniques for monitoring JVM, a. native diagnostic technique and  b. JVMTI calls within JVM. While CAMM is a BCI(Byte Code based instrumentation) based tool. CAMM utilizes model-driven service management, a unique model-driven approach to application service management that automatically discovers and models the structure of J2EE and composite applications. 
5. When deployed in production, ad4j claims less than 1% overhead while CAMM claims around 5% overhead. CAMM is also very useful in development environment.

In 11g, both the solutions have been integrated and in future plan is to integrate them with other products in Oracle middleware stack

ad4j/jam agent implementation methods for app server /standalone JVMs

ad4j agent can used to monitor any JVM. Be it an application server or a standalone JVM(Java applications).

This article explains the method to monitor standalone Java programs which do not have away of loading a servlet or another class. jamagent has "jamagent.jamrun" class that can be used for this purpose.

 For an application server, 11g jamagent.war can be deployed using one of the following two modes:

1. manual deployment of war file

2. automated deployment of war file on Target Weblogic Server

For an application server, 10g jamagent can be deployed using following mode:

1. manual deployment of war/ear file

For standalone JVM, 10g/11g, follow given steps:

1. Include the file jamagent.zip(jamagent.war in 11g) in the CLASSPATH

2. Change the java command line parameters to call the wrapper jamagent.jamrun as mentioned below.

Original command looks like:

%java $JVM_OPTIONS $TARGET_CLASS $TARGET_CLASS_PARAMS

Here JVM_OPTIONS are the JVM properties and options e.g. Xmx512M -Dweblogic.name

TARGET_CLASS is the program which being examined

TARGET_CLASS_PARAMS are parameters passed to the class

Modified command would be:

%java $JVM_OPTIONS jamagent.jamrun [$JAMAGENT_PARAMS_LIST] $TARGET_CLASS $TARGET_CLASS_PARAMS

ad4j/jam agent will use default parameters. If you want to change any of them, you can specify them as

name=value pairs. To specify a different manager/console and port you will specify

jamconshost=<hostname where manager/console is running>, jamconsport=<listen port of manager/console>

e.g. For Weblogic server,

Original command is:

% $JAVA_HOME/bin/java ${JAVA_VM} ${MEM_ARGS} ${JAVA_OPTIONS} \

-Dweblogic.Name=${SERVER_NAME} \

-Dweblogic.ProductionModeEnabled=${PRODUCTION_MODE} \

-Djava.security.policy="${WL_HOME}/server/lib/weblogic.policy" \

weblogic.Server

The class being called is weblogic.Server. There are no parameters.

Modified command would be:

We are going to insert out class and parameters before that, so it will look like :

%$JAVA_HOME/bin/java ${JAVA_VM} ${MEM_ARGS} ${JAVA_OPTIONS} \

-Dweblogic.Name=${SERVER_NAME} \

-Dweblogic.ProductionModeEnabled=${PRODUCTION_MODE} \

-Djava.security.policy="${WL_HOME}/server/lib/weblogic.policy" \

jamagent.jamrun weblogic.Server

If you want to change some JAM parameters you can specify them as

%$JAVA_HOME/bin/java ${JAVA_VM} ${MEM_ARGS} ${JAVA_OPTIONS} \

-Dweblogic.Name=${SERVER_NAME} \

-Dweblogic.ProductionModeEnabled=${PRODUCTION_MODE} \

-Djava.security.policy="${WL_HOME}/server/lib/weblogic.policy" \

jamagent.jamrun jamconshost=abc jam_jvm_id=5001 weblogic.Server

Ensure that JAM Agent Libraries/Classes is added to the Classpath

Also ensure that jam/jamagent/WEB-INF/classes directory is added to the CLASSPATH.

It is best to do this just before the call to java.

Here we find a couple of lines above the call to Java.

# Start WebLogic server

CLASSPATH="${WEBLOGIC_CLASSPATH}${CLASSPATHSEP}${CLASSPATH}"

We change it to

# Start WebLogic server

CLASSPATH="${WEBLOGIC_CLASSPATH}${CLASSPATHSEP}${CLASSPATH}${CLASSPATHS

EP}/u01/app/em/jamagent/deploy/jamagent.zip"

Target Parameters

What about the -Dweblogic and other properties : You need to insert jamagent.jamrun just before

the calling program and it's parameters. This should happen after all the JVM options and

properties have been specified.

JAM Agent Input Parameters Description

jamconshost - em_console_host The host where JAM manager/console is running

jamconsport - 3600 The port ad4j manager/console is listening on

jamconsretr - 300 Seconds to wait before retrying broken console connection

jamtimeout - 900 Seconds to timeout for long operations

jamloglevel - 3 Level of logging. Valid values are 1-5.

jamjvmid - Server Port Unique Identifier for JVM, Default of 5555 for unrecognized

Servers

jammaxbackoff - 10 Amount of time to wait for long operations

jamdelaystartup - 0 Number of seconds to delay target program

jamisdaemon - false jamisdaemon=true will prevent the agent from exiting

Note: this parameter jamisdaemon=true should be set to true for jamagent to not to exit.

Oracle Composite Application Monitor and Modeler (OCAMM - ADP) with EMGC11g

Started with name Acsera Manager , renamed as ClearApp, later named as OCAMM (Oracle Composite Application Monitor and Modeler) and now it's known as ADP (Application Dependency and Performance).

ADP (OCAMM) is a solution in Application Service Management(ASM) area. It understands the Java, J2EE, SOA, OSB and Portal applications. ADP(OCAMM) can:

- discover components

- analyze components in depth

- capture complex relationships between services, components & run-time environments (i.e. JVMs)

-dynamically adjust to relationship changes

- take into account required application services (i.e. SOA, BPEL, OSB, portal, web services etc.), facilities (i.e. performance monitoring, root-cause analysis, etc.) and platforms (i.e. WebLogic, Websphere, OC4J etc.)

ADP is supported on Windows, Linux, Solaris, HPUX and AIX platforms.

ADP needs database for repository. It supports Oracle and My SQL as its repository.

ADP has been integrated with EMGC 11g.

ADP in EMGC 11g has following components:

1. UI - ADP Pages to display ADP data
2. Manager - ADP Manager is the core analytical engine of the ADP ASM system. In real-time, ADP Manager performs complex mathematical modeling and statistical calculations with summarized data from all ADP Java Agents.
3. Java Agent - ADP Java Agents are the data collectors of the ADP ASM system. ADP Java Agents are deployed to all managed application servers to perform a series of tasks including collecting performance managements, tracking contextual relationships, and summarizing data in real-time while introducing as little overhead as possible. At the expiration of the predefined aggregation interval, these agents forward the summarized data to ADP for additional analysis. For various Java EE platforms such as Oracle SOA Suite, Oracle WebLogic, and IBM WebSphere, ADP leverages their deployment infrastructures to quickly deploy the ADP Java Agents to all application servers.
4.   Database - ADP stores its analyzed data and application models in the Management Repository. There is a separate schema for ADP in the Management Repository and this schema is external to the ADP installation.

One can learn more about ADP(OCAMM) at http://download.oracle.com/docs/cd/E11857_01/install.111/e17558/intro.htm

What is Cross Tier (xtier) functionality in JVM Diagnostics (AD4J)?

Using JVM Diagnostics(AD4J), one can diagnose an issue across 3 tiers i.e. Java application<-> JVM <-> Database. User can relate an application activity to database activity. As of now, only oracle database is supported.

If cross tier is configured correctly, in real time user can see DB Wait link for active JVM threads that are busy in db activity. While in monitoring data, data appears in Top DB States and Top SQLs. One can use filters on them. By looking at Database pages, one can see JVM activity inside db.

Cross tier configuration requires: 

• Java application and JVM with jamagent running and registered with console
• Database to be monitored needs to be registered with console
• Dbagent needs to be running on machine where database is running as same user who is running oracle database.

To ensure that cross tier is configured correctly:

1.            Navigate to Targets->Middleware->JVM Diagnostics->Setup->Databases page and ensure that database to be monitored is registered correctly. Verify following:

•             OS User field should specify the user who installed the database

•             DB User field should specify the DB application user credentials (Should have access to GV$ and V$ views)

•             DB User (Explain Plan) field should specify DB system user credentials

2.            DB Agent should be running on the machine on which DB is installed with correct manager IP and port

./dbagent jamconshost=<ManagerIP> jamconsport=<ManagerPort>

Note: On aix, either dbagent needs to run as root user or set the suid to root for dbagent binary.

3.            To verify cross tier functionality, go to Targets->Middleware->JVM Diagnostics. Select JVM on left Panel and click JVM->Threads->Real Time Analysis page for the JVM if you encounter a thread that is in the DB Wait state, it should be a hyperlink so you can follow the thread into the Database pages and see its database session information including the SQL that it executes.

In addition, if monitoring is started, data would appear in Top DBStates and Top SQLs table and charts.

Real User Experience Insight (RUEI) - Moniforce

RUEI stands for Real User Experience Insight. RUEI can help organizations to understand the real end user experiences, identify issues and fix them before they cause losses.

RUEI works on Network Protocol Analysis (NPA) technology. It doesn't require any modification, changes or instrumentation in application. Because of it's passive monitoring approach, it's an ideal solution in production env.

RUEI is based on 3 layered architecture:
1. Data Collector
2. Data Processor
3. Data Presenter

1. Data collector
is responsible for collecting raw data and pass it as input to Data processor.  Data can be collected either via SPAN port or Network Tap.

2. Data Processor
is responsible for processing the raw data into proper OLAP datasets. It gets all URLs (webpages) and categorizes them based on application. These applications are defined and configured with RUEI.

3. Data Presenter
It's a J2EE based web application. That analyses and reports the data. Reports can be created on following criteria:
a. Time Period (year, month, week, day, hour, min)
b. client (country, provider, provider network, city, client IP)
c. server (IP address)
d. URL ( site, website, name, POST & GET arguments)
e. Page (application, group, page)
f. Customer name

RUEI 6.5 has been integrated with JVM Diagnostics(AD4J), Application Dependency and Performance (ADP/OCAMM) and Request Monitoring.

Oracle Enterprise Manager 11g Grid Control Middleware Management Packs

Oracle Enterprise Manager 11g Grid Control comes with various management packs to manage different Middleware components and their performance. It includes:

a. Oracle WebLogic Server Management Pack [http://www.oracle.com/technetwork/oem/grid-control/overview/ds-wls-mgmt-132529.pdf]
This pack helps in
– spanning application performance management,
- Configuration management,
- Service level management and
- Operations automation
– improves performance and availability of Java applications and web services, improves service by avoiding downtime, and reduces cost by automating manual and error-prone operations.

b. Oracle SOA Management Pack [http://www.oracle.com/technetwork/oem/grid-control/overview/ds-soa-mgmt-pack-1-128301.pdf]
i) Oracle BPEL Process Manager
ii) Oracle Service Bus
iii) Oracle SOA Suite 11g
SOA Management Pack helps in
-Tracking and monitoring end-to-end business transactions across tiers
-Monitor the performance of SOA implementation technologies such as the Oracle SOA Suite and Oracle Service Bus
-Integrated web service testing and synthetic transaction monitoring
-Integrated authoring, attachment, and monitoring of security policies
-Collection and analysis of SOA configuration information

c. WebCenter Suite Management Pack [http://www.oracle.com/technetwork/oem/grid-control/overview/ds-camm-owebcentersuite-134498.pdf]
This pack helps in
-Monitoring WebCenter and WebLogic Portal applications via dashboards and drill-downs to root cause analysis
-Correlating WebCenter Services with underlying ADF Task Flows as well as underlying components
-Low overhead for 24/7 production monitoring
-No coding (or source code) needed
-Easy-to-read map of the WebCenter and WebLogic Portal application model-driven relationship views
-Automatic APM configuration determines what to monitor
-Auto-correlation ties every metric to high-end portal, transaction, and business functions
-Dynamic, auto-customized service-level dashboard allows any IT staff to manage performance

d. FMW Performance Management [http://www.oracle.com/technetwork/oem/grid-control/ds-as-dp-130384.pdf]
i) JVM diagnostics (AD4J)
ii)Application Dependency and Performance (OCAMM)
iii) Request Monitoring

e. Oracle Identity Management Pack [http://download.oracle.com/technology/products/oem/identity_management_pack_gc11.pdf]
i) Oracle Access Manager (OAM)
ii) Oracle Identity Federation Server (OIF)
iii)Oracle Identity Manager (OIM)
iv) Oracle Internet Directory (OID)
v) Oracle Virtual Directory (OVD)
vi) Directory Integration Platform Server
This pack has following features:
- Single-Step Discovery for both Oracle Identity Management 10g and Oracle Identity Management 11g components - Performance Monitoring: Proactively monitor your Oracle Identity Management environment from both systems & end-user perspectives.
- Configuration Management
- Service Level Management

f. Oracle Coherence Management Pack [http://www.oracle.com/technetwork/middleware/coherence/ds-coherence-mgmt-1-130356.pdf]
This pack has following features:
-Complete cluster visibility
-Monitor cache performance, stability and load distribution on nodes
-Provision new cluster or nodes
-Start/stop nodes or entire cluster
-Support for event tracking, notifications, charts and historical metric trends
-Monitors the entire Coherence infrastructure including monitoring WebLogic Server, Coherence Cluster, and the database as a single system
-Real time propagation of configuration changes to a node, a cache, or an entire cluster
-Monitor caches used by WebLogic applications

Oracle EM11g JVM Diagnostics (AD4J), associating JVM Diagnostics data with EM target

Note: These instructions are valid only for EMGC 11g release

Manual deployment of jamagent on a JVM(an app server) is very straight forward like in previous releases, user can download the jamagent.war and deploy it on target JVM or application server.

In EMGC11g, if user has a scenario where user wants to deploy jamagent on already discovered Weblogic EM target and associate JVM Diagnostics with this target but due to security restriction or any other reason user cannot use the automated deployment feature of JVM Diagnostics.

In such cases, one can execute given steps:
a. Create a JVM Pool of same name as that of WLS domain target name except that every '/' should be replaced by '__' (two underscores).
o e.g. if WLS domain Target name is "/sample_EMGC_DOMAIN/EMGC_DOMAIN"
o JVM Pool name will become " __sample_EMGC_DOMAIN__EMGC_DOMAIN"
o Select "Poll enabled" checkbox, if you want to start monitoring for this pool

b. Download jamagent.war from Targets->Middleware->JVM Diagnostics->Setup->Download Page
o In WEB-INF/web.xml of jamagent.war, value of the following 3 tags needs to be updated and then jamagent.war has to be bundled again.
o oracle.ad4j.groupidprop –The name of EM WLS target on which deployment has to be done. This is shown on Ad4j agent Deployment page under column “Target Name”
o oracle.ad4j.jvmidprop - The type of EM WLS target on which deployment has to be done. This is shown on Ad4j Deployment page under column “Target Type”. Currently the supported value is always “weblogic_j2eeserver”
o jampool – The name of the domain for which we are doing deployment. It can be found on the deployment page after text “List of targets under domain name”. But "/" replaced by "__" (forward slash replaced by double underscore)
Deploy re-bundled jamagent.war on Target WLS manually.

c. Goto Target-> Middleware Page, select the WebLogic Domain/WebLogic Server Target Menu->JVM Diagnostics menu items. You should be able to view JVM Diagnostics data for these targets now.

Oracle Enterprise Manager 11g JVM Diagnostics (AD4J)

Started with name JAM (Java Activity Monitoring), renamed as JADE (Java Application Diagnostic Expert), later named as AD4J (Application Diagnostics For Java) and now it's known as JVM Diagnostics.

Oracle's JVM Diagnostics improves the availability, performance and provides in-depth diagnostics for Java applications with low overhead. This comes with Oracle Enterprise Manager Grid Control 11g. Following are the highlights of this feature:

• Agent is easy to deploy and no application server restart is required
• Immediate visibility into JVM and its stack. No application instrumentation required is to provide in-depth method level visibility into all active transactions happening in JVM.
• Identifies the slowest requests, slowest methods, Database states, SQLs and the bottleneck resources such as CPU, I/O, memory, slow running or hanging threads and Database.
• Detects memory leaks and allows administrators to perform memory leak analysis in production environment without stopping the application. Administrators can take multiple heap snapshots over a period of time and compare them to find the Java objects leaking memory.
• It allows tracing Java requests to the associated database sessions. It highlights the slowest SQL queries and help administrators to tune SQL and the database to improve the performance application. It facilitates smooth communication between the database administrators and application administrators by isolating the problems to the database or Java code.

It can be downloaded from http://www.oracle.com/technetwork/oem/grid-control/downloads/index.html