EM12c: How to fix 'Job Purge' repository scheduler job when it shows as down, though its up (EM12cR2 upgrade issue)

Issue: After upgrading EM from BP1 to EM12cR2, EM reports 'Job Purge' repository scheduler job as down though it's actually up.

One can see the status of this Job, on the Repository page (Setup -> Manage Cloud Control -> Repository), in the Repository Scheduler Jobs Status section, the "Job Purge" job is shown as down with a red arrow and blanks for both Next Scheduled Run and Last Scheduled Run. All the other jobs look good and are running successfully.

Action:
Please run the following SQL as SYSMAN to correct the problem.

UPDATE MGMT_PERFORMANCE_NAMES SET display_name = NULL,
   dbms_jobname = NULL, is_dbmsjob = 'N', is_deleted = 'Y'
  WHERE UPPER(DBMS_JOBNAME) LIKE 'MGMT_JOB_ENGINE.APPLY_PURGE_POLICIES%';
COMMIT;

Now it should look like as given below:

Note: One can also refer to discussion at this Oracle forum or this blog

EM12c: How to discover a standalone OHS (Oracle HTTP Server)?

OHS(Oracle HTTP Server) can be discovered and monitored in EM12c. If an OHS is part of weblogic domain, it gets discovered during Weblogic Farm/domain discovery and monitoring automatically starts for that OHS. Details of metrics collected for an OHS that is part of weblogic domain can be found here.

However, if one has a standalone OHS. Its discovery and monitoring is bit different. Steps for discovering standalone OHS are given later in the article. One has to be aware that even after discovery of standalone OHS, all one gets is availability status metric for OHS. No other metrics are collected for standalone OHS.

Here are the steps to discover standalone OHS.

1. Go to Setup> Add Targets > Add Targets Manually
2. select Add Non-Host Targets by Specifying Target Monitoring Properties,
3. select Target Type as Oracle HTTP Server and the Monitoring Agent,
4. on the next page you will need to add the properties for OHS

Target Name - For example "Oracle_WT1"
Machine Name - fully qualified host name like "abc.xyz.com"
Oracle Home Path - /u01/Oracle/Middleware/Oracle_WT1
EM Target Type - oracle_apache
Port Number - 7777  (based on the opmnctl status -l  output ), 7777 is the default http port for OHS
Version of Apache Server - 2.2

There are many other mandatory fields, which are relevant only when OHS is registered with weblogic and can be ignored for standalone OHS discovery. So one can provide dummy values for them(e.g. "na"). These values are not used. Example is
given in following screenshot.

EM12c: Some interesting new features in Middleware Management with EM 12c R2(12.1.0.2) ?

With EM12c R2, FMW plugin version 12.1.0.3 has been released. With this release lot of new features have been introduced under Middleware Management and Application Performance Management areas. One can find complete list of features on MW mgmt and APM links.

Out of all features, I find following features somewhat more interesting:
1.    Business Application - A new target has been added called as Business Application target. This target combines capabilities of RUEI+BTM+JVM+ADP. The BA is associated with a system target, RUEI applications and Business Transactions.
The BA target enables a user to see and understands the health of the business application. The target represents a holistic view of the monitored application. It has its own dashboard.
Business Application has
•    Business Application Target
•    Business Application Dashboard
•    Business Applications List Page
•    Drill-down from Request Instance Diagnostics to RUEI Session Diagnostics
•    BTM Transaction to EMCC Target Association
•    Business Application Creation Wizard
•    Experience Monitoring in the Business Application Dashboard

For more details on Business Applications, click here.
Click here for demo.

2.    MWaaS (MiddleWare as a Service) – EM12cR2 cloud management has been enhanced for providing MiddleWare services along with IaaS and DBaaS. It allows administrators to create, monitor and manage a private or public MWaaS environment. MWaaS has a complete runtime env. comprised of all services necessary to deploy and run an enterprise-class java-j2ee application, including services such as application hosting, persistence store, application integration and APIs that enable programmatic access to additional computing services that might be required by an application.

MiddleWare as a Service has
•    Resource Pool
•    MWaaS Quotas
•    MWaaS Request Settings
•    Service Templates
•    Provision and De-provision Service Instance
•    Manage Application Lifecycle
•    Service Instance Scaling
•    Manage Data Sources

For more details on MWaaS, click here.

3.    Exalogic management enhancements – With EM12cR2 and Ops center, now one can see Exalogic Schematic views and manage Exalogic boxes well. These enhancements are:
•    Integrated hardware and software schematics
•    Hardware-software topology views
•    Hardware targets monitoring (Compute, ZFA appliance, Infiniband Fabric, ILOM)
•    OTD monitoring
•    Support for virtual and non-virtual configurations
•    Monitoring of Exalogic vServer guest VMs
•    Health checks
•    Trusted partition/vCPU licensing report
Click here for demo.

4.    Service Target Dashboard - With this new feature, administrators can quickly create "out of box" overviews of the health of related services. They can create templates that contain a group of services selected from the defined services in Enterprise Manager. Any Enterprise Manager user can use these defined dashboards to quickly correlate the health of services, the status of key metrics, incidents and related status of the service level agreements tied to the service.
For more details, see this demo .

EM12c: Enterprise Manager Cloud Control 12c Release 2 (12.1.0.2) Now Available!

Oracle Enterprise Manager Cloud Control 12c Release 2 (12.1.0.2) is now available on OTN on ALL platforms. Highlight of this release is simultaneous release of EM12c R2 on all platforms. This is primarily a stability release which incorporates many of issues and feedback reported by early adopters. In addition, this release contains many new features and enhancements in areas across the board.
New Capabilities and Features

• Enhanced management capabilities for enterprise private clouds:
  • Introduces new capabilities to allow customers to build and manage a Java Platform-as-a-Service (PaaS) cloud based on Oracle Weblogic Server. The new capabilities include guided set up of PaaS Cloud, self-service provisioning, automatic scale out and metering and chargeback.
  • Enhanced lifecycle management capabilities for Oracle WebLogic Server environments: Combining in-context multiple domain, patching and configuration file synchronizations.
  • Integrated Hardware-Software management for Oracle Exalogic Elastic Cloud through features such as rack schematics visualization and integrated monitoring of all hardware and software components.
• The latest management capabilities for business-critical applications include:
  • Business Application Management: A new Business Application (BA) target type and dashboard with flexible definitions provides a logical view of an application’s business transactions, end-user experiences and the cloud infrastructure the monitored application is running on.
  • Enhanced User Experience Reporting: Oracle Real User Experience Insight has been enhanced to provide reporting capabilities on client-side issues for applications running in the cloud and has been more tightly coupled with Oracle Business Transaction Management to help ensure that real-time user experience and transaction tracing data is provided to users in context.
• Several key improvements address ease of administration, reporting and extensibility for massively scalable cloud environments including dynamic groups, self-updateable monitoring templates, bulk operations against many events, etc.

New and Revised Plug-Ins:
Several plug-Ins have been updated as a part of this release resulting in either new versions or revisions. Revised plug-ins contain only bug-fixes and while new plug-ins incorporate both bug fixes as well as new functionality.

Plug-In Name	Version
Enterprise Manager for Oracle Database	12.1.0.2 (revision)
Enterprise Manager for Oracle Fusion Middleware	12.1.0.3 (new)
Enterprise Manager for Chargeback and Capacity Planning	12.1.0.3 (new)
Enterprise Manager for Oracle Fusion Applications	12.1.0.3 (new)
Enterprise Manager for Oracle Virtualization	12.1.0.3 (new)
Enterprise Manager for Oracle Exadata	12.1.0.3 (new)
Enterprise Manager for Oracle Cloud	12.1.0.4 (new)

Installation and Upgrade:

• All major platforms have been released simultaneously (Linux 32 / 64 bit, Solaris (SPARC), Solaris x86-64, IBM AIX 64-bit, and Windows x86-64 (64-bit) )
• Enterprise Manager 12.1.0.2 is a complete release that includes both the EM OMS and Agent versions of 12.1.0.2.
• Installation options available with EM 12.1.0.2: User can do fresh Install or an upgrade from versions EM 10.2.0.5, 11.1, or 12.1.0.2 ( Bundle Patch 1 not mandatory).
• Upgrading to EM 12.1.0.2 from EM 12.1.0.1 is not a patch application (similar to Bundle Patch 1) but is achieved through a 1-system upgrade.

Documentation:

• Oracle Enterprise Manager Cloud Control Introduction Document provides a broad overview of capabilities and highlights"What's New" in EM 12.1.0.2.
• All updated Oracle Enterprise Manager documentation can be found on OTN

Customer Webcast - EM 12c Installation and Upgrade: This webcast is for customers who are interested in learning how to successfully deploy or upgrade to EM 12.1.0.2.

• Customer Webcast - Installation and Upgrade - September 21
  (registration and info on OTN starting September 12)

Enterprise Manager 12c R2 Resources:

• OTN Download Page
• Upgrade Guide

EM12c: How to reset the OEM Session timeout (Login timeout)?

As per EM 12c, there are following OMS properties that can influence session time out:

1. oracle.sysman.eml.maxInactiveTime

This property is used for the UI to time out a logged in user session. Default value is 45 min.

emctl utility can be used to reset this parameter.

e.g.

$ ./emctl get property -name oracle.sysman.eml.maxInactiveTime

Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0 

Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.

SYSMAN password:

Value for property oracle.sysman.eml.maxInactiveTime for oms host.abc.com:4890_Management_Service is 45

$ ./emctl set property -name oracle.sysman.eml.maxInactiveTime -value 60

Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0 

Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.

SYSMAN password:

Property oracle.sysman.eml.maxInactiveTime for oms host.abc.com:4890_Management_Service has been set to value 60

OMS restart is required to reflect the new property value

After resetting the oracle.sysman.eml.maxInactiveTime, do stop and start oms.

$ ./emctl stop oms

$ ./emctl start oms

2. oracle.adf.view.rich.poll.timeout

The ADF framework uses this property for UI pages that use 'polling'. Default value is 10 min.

Resetting of this property also requires OMS restart.

EM12c: Synthetic Monitoring vs Real End User Monitoring

EM12c has Service Level Management features. Services can be managed via Synthetic Monitoring or via Real User Transaction Monitoring.

Synthetic User Experience Monitoring is achieved via Beacons based Service tests

• It ensures the performance of specific business process or user
• Can be used to determine location-specific performance problems by monitoring from multiple geographic locations
• It ensures the performance of applications based on various supported protocols e.g. DNS, FTP, HTTP(s), Ping, ICMP Ping, IMAP, JDBC SQL Timing, LDAP, NNTP, Oracle SQL Timing, POP, Port Checker (includes SSL), SMTP, SOAP, TNS Ping, Web Transaction, WEBDAV, ATS transaction, CalDAV and Custom Script.
• Beacons actively monitor the infrastructure.

Real User Experience Monitoring is achieved via RUEI (Real User Experience Insight)

• It catches the performance problems for any user of HTTP applications
• Catches unexpected performance issues or unique user interactions
• Diagnoses problems quickly with “replay”
• Determines exact usage of applications
• RUEI passively monitors the end users.

Both monitoring techniques are complementary in nature, as synthetic monitoring will not execute real use cases (e.g. order process) or detect issues in a use-case it was not configured for, while RUEI monitors and detects issues when a specific use-case is executed by a user.

EM12c - How to take oms logs in debug mode?

Run below command on OMS to activate debug log and collect emoms.log/trc.

Fetch the current setting of logging by running below command
%<middleware_home>/oms/bin/emctl get property -sysman_pwd  <repos user psswd> -name log4j.logger.oracle.sysman
Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0 
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Value for property log4j.logger.oracle.sysman for oms <hostname>:<port>_Management_Service is null


Then set the log level to debug
%<middleware_home>/oms/bin/emctl set property -sysman_pwd  <repos user psswd> -name log4j.logger.oracle.sysman -value DEBUG -module logging
Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0 
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Property log4j.logger.oracle.sysman for oms <hostname>:<port>_Management_Service has been set to value DEBUG
OMS restart is not required to reflect the new property value

After enabling OMS java debug log, visit required EM pages
Take the OMS logs i.e. EMGC_OMS1.out, emoms.log and emoms.trc etc.

EM12c: How to discover FMW targets automatically using auto discover feature?

EM 12c cloud control has a concept of auto discovery of targets

It involves two steps:

1. Discovering the potential target

2. Promoting the target and assigning it to an EM agent

Users can enable this automatic discovery of potential targets on a host.

To enable auto discovery, one needs to ensure that EM agent has the discovery part of plugin deployed on the host.

After auto discovery is configured to run on the managed host, discovery takes place periodically and sends potential targets to the Management repository.

Users can review these targets from the discovery results UI and then promote the target, so taht EM can monitor and manage these.

Ensure that agent has been deployed on the managed host.

1. Select Setup menu-> Add Target-> Configure Auto Discovery.

2. Set the schedule(in days) at which the discovery job will be run. This schedule will be applied to all selected hosts. By default the job will run every 24 hours.

3. Click Add Host. Select the host machine where you want to discover targets.

4. Select the host in the table, then click Edit Parameters to specify the Middleware Home(s) to search for targets.

 (Note: The Middleware Home is the top-level directory for all Oracle Fusion Middleware products, created when Oracle WebLogic Server is installed. e.g. /u01/app/middleware)

5. Click OK to enable the automatic discovery.

6. Once automatic discovery has been enabled, one should verify Setup -> Add Target->Auto Discovery Results regularly to view the list of discovered targets.

7. Select the Non-Host Targets tab to view the discovered Oracle Fusion Middleware targets.

8. Select a target e.g. select Oracle WebLogic Domain target type then click Promote.

For promotion of WLS domain target, provide following parameters:

        The hostname for the WebLogic Administration Server

        The WebLogic Administration Server port.

        The WebLogic Administration Server user name and password.

        A Unique Domain Identifier for domain

        EM Agent that will be used to discover the FMW targets.

9. Click Continue. EM will discover all FMW targets within the WLS domain.

10. Click Close in the Finding Targets dialog to automatically assign EM Agents to the discovered targets.

11. Click Add Targets to assign EM Agents as listed in the Assign Agents page.

12. Click Close. The Results page displays the targets and Agent assignments.

13. Click OK when finished.

Wait for sometime before these targets are visible and monitored. All the agents used for monitoring the targets must be up.

Note: The automatic discovery feature is not supported for Oracle WebLogic Server versions 7.x or 8.x.

EM12c: How to fix java.lang.UnsatisfiedLinkError: /tmp/libjamagent.so.1: failed to map segment from shared object: Operation not permitted error while deploying Jamagent?

Problem: While deploying jamagent, following error is seen in jamagent log:
12/07/04 07:58:34 JAM Agent: jaminit.init()
12/07/04 07:58:34 JAM Agent: jaminit.init() Instantiating jamagent
12/07/04 07:58:34 OS is Linux
12/07/04 07:58:34 arch is i386
12/07/04 07:58:34 JAM Agent loadNative() Wrote to [/tmp/libjamagent.so.1]
12/07/04 07:58:34 >>java.library.path: /u01/usr/apps/abc/10.1.3/appsutil/jdk/jre/lib/i386/server:/u01/usr/apps/abc/10.1.3/appsutil/jdk/jre/lib/i386:/u01/usr/apps/abc/10.1.3/appsutil/jdk/
12/07/04 07:58:34 JAM Agent: loadNative Exception loading [/tmp/libjamagent.so.1] /tmp/libjamagent.so.1: /tmp/libjamagent.so.1: failed to map segment from shared object: Operation not permitted
12/07/04 07:58:34 java.lang.UnsatisfiedLinkError: /tmp/libjamagent.so.1: /tmp/libjamagent.so.1: failed to map segment from shared object: Operation not permitted
12/07/04 07:58:34       at java.lang.ClassLoader$NativeLibrary.load(Native Method)
12/07/04 07:58:34       at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1803)
12/07/04 07:58:34       at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1699)
12/07/04 07:58:34       at java.lang.Runtime.load0(Runtime.java:770)
12/07/04 07:58:34       at java.lang.System.load(System.java:1003)
12/07/04 07:58:34       at jamagent.jamagent.loadNative(jamagent.java:688)
12/07/04 07:58:34       at jamagent.jamagent.<init>(jamagent.java:995)
12/07/04 07:58:34       at jamagent.jaminit.init(jaminit.java:52)

Possibility is that error is encountered when /tmp directory is mounted with -noexec option.

Solution: Make /tmp directory executable using following command:
mount -o remount,exec /tmp

Retry deploying jamagent.

EM12c: How to recover EM(OMS) to its original wallet and cert after messing up with 3rd party wallet and certificate?

Problem: I was trying to secure OMS using 3rd party wallet. Created a certificate request. Secured OMS and console using this wallet and certificate successfully. But then while trying to start oms, it did not come up and always showing javax.net.ssl.SSLHandshakeException in emctl.log:

ERROR oms.StatusOMSCmd processStatusOMS.309 - Received fatal alert: handshake_failure

javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

        at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)

        at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:136)

        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1720)

        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:954)

        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1138)

        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:632)

        at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)

        at java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:109)

        at HTTPClient.HTTPConnection.sendRequest(HTTPConnection.java:3366)

        at HTTPClient.HTTPConnection.handleRequest(HTTPConnection.java:3281)

        at HTTPClient.HTTPConnection$9.run(HTTPConnection.java:3032)

        at HTTPClient.HTTPConnection$9.run(HTTPConnection.java:3023)

        at HTTPClient.HttpClientConfiguration.doAction(HttpClientConfiguration.java:666)

        at HTTPClient.HTTPConnection.doAction(HTTPConnection.java:5401)

        at HTTPClient.HTTPConnection.setupRequest(HTTPConnection.java:3023)

        at HTTPClient.HTTPConnection.Get(HTTPConnection.java:878)

        at oracle.sysman.emctl.oms.StatusOMSCmd.pingOMSCheck(StatusOMSCmd.java:343)

        at oracle.sysman.emctl.oms.StatusOMSCmd.processStatusOMS(StatusOMSCmd.java:237)

        at oracle.sysman.emctl.wls.OMSController.statusOMS(OMSController.java:1222)

        at oracle.sysman.emctl.wls.OMSController.startOMS(OMSController.java:943)

        at oracle.sysman.emctl.wls.OMSController.main(OMSController.java:329)

2012-06-06 13:20:28,196 [main] INFO  wls.OMSController statusOMS.1354 - statusOMS finished with result: 9

I did figure out that there is a potential issue with certificate/wallet that I generated. It’s not valid. So I wanted to bring oms back to its original configuration with its wallet and certificate.

Solution: Using following commands, I was able to bring up the OMS without any issues.

%emctl stop oms

%emctl secure oms

%emctl secure console -self_signed

%emctl stop oms -all

%emctl start oms