EM12c: How to setup MWaaS? - Part I

In next few articles, we will learn to setup MWaaS.

In this article, we will just learn about the concept of MWaaS and its pre-requisites.

Oracle Cloud Management Pack for Oracle Middleware delivers capabilities spanning the entire Middleware cloud lifecycle. 

It lets Cloud administrators identify pooled resources, configure role-based access, define the service catalog, and the related chargeback plans. 

It allows Cloud users to request middleware services, and consume them on-demand. 

It also allows for users to scale-up and down their platforms to adapt to changes in workload. 

Deploy/undeploy/redeploy applications, create data sources and monitor these applications. 

Finally, it lets both parties to understand the costs of the service delivered, and establish accountability for consumption of resources. 

 

Prerequisite for MWaaS:

1.       EM12cR2 along with EM repository 

2.       Plugin Requirements

-Enterprise Manager for Oracle Virtualization 12.1.0.3+

-Enterprise Manager for Oracle Cloud Application 12.1.0.4+

-Enterprise Manager for Oracle Fusion Middleware 12.1.0.3+

-Enterprise Manager for Chargeback and Capacity Planning 12.1.0.3+

3.       Target machines:One

Note: WLS Provisioning would be done using OOB WLS profile. 

4.       For WLS provisioning using OOB profile, we require Fusion Middleware (WLS 10.3.6) with JRF enabled and that can be done using following either Webcenter or SOA installers. 

-WLS 10.3.6 generic installer http://www.oracle.com/technetwork/middleware/weblogic/downloads/wls-main-097127.html

-WebCenter portal 11.1.1.6.0 (PS5) installer http://www.oracle.com/technetwork/middleware/webcenter/portal/downloads/index.html

                                                   or

-SOA 11.1.1.6.0 (PS5) installer http://www.oracle.com/technetwork/middleware/soasuite/downloads/index.html

5.       On target machine, we need to have Middleware Home installed. It can be installed using wls 10.3.6 generic installer and webcenter/soa installer as mentioned in step 4. This Middleware Home should be a software only install. No weblogic instance should be created on this MW home.

 For more details, pls refer http://docs.oracle.com/cd/E24628_01/doc.121/e28814/cloud_mw_setup.htm#CHDHCCIA

EM12c: How to fix BI Publisher 11.1.1.6.0 report error that shows invalid username/password error?

Issue: With EM12cR2, 11.1.1.6.0 BI Publisher is installed and integrated successfully. However, on accessing any report via BI Publisher as sysman user. It shows following error:

Parameter name: targettype weblogic.common.resourcepool.ResourceDeadException: 0: Could not connect to 'oracle.jdbc.OracleDriver'. The returned message is: ORA-01017: invalid username/password; logon denied It is likely that the login or password is not valid. It is also possible that something else is invalid in the configuration or that the database is not available.

Following error is seen in BIP managed server's log file(BIP.out):
2013-Jan-31 12:26:28EMReposAuthenticatorHelper: Following SQLException occurred.
SQLException code=28009 error msg: ORA-28009: connection as SYS should be as SYSDBA or SYSOPER

Problem verification:
When trying to connect EM repository as mgmt_view user via sqlplus. It works fine.

%$ORACLE_HOME/bin/sqlplus mgmt_view/<passwd>@<emrep>
SQL*Plus: Release 11.2.0.3.0 Production on Fri Feb 1 10:06:48 2013
Copyright (c) 1982, 2011, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options


Apparently, at some point along with other EM repos users, mgmt_view user password got modified using sqlplus. Which is the not supported way of modifying mgmt_view user passwd.

Action:
On OMS machine, execute the following command:
% $ORACLE_HOME/bin/emctl config oms -change_view_user_pwd -sysman_pwd <sysman_pwd> -user_pwd <user_pwd>
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Successfully changed password. You must restart the BI Publisher Enterprise Managed Server in order for BI Publisher to function correctly.


Restart the BIP server using weblogic console and restart oms using emctl commands:
% $ORACLE_HOME/bin/emctl stop oms

% $ORACLE_HOME/bin/emctl start oms

EM12c: How to install and integrate BI Publisher with OEM?

Note: With Enterprise Manager 12c Cloud Control Release 2 (12.1.0.2.0), Only BI EE 11.1.1.6.0, is supported while With Enterprise Manager 12c Cloud Control Release 1 (12.1.0.1.0), Only BI EE 11.1.1.5.0, is supported for use .
[BI Publisher 11.1.1.5.0 or 11.1.1.6.0 can be downloaded from http://www.oracle.com/technetwork/middleware/bi-enterprise-edition/downloads/bus-intelligence-11g-165436.html]

BI Publisher installation and integration with EM is a two step process.
1. Do a software only install of BI Publisher
Install BIP using runInstaller binary and ensure to do software only install.

2. Integrate BIP with EM 12c
a. goto GCDomain dir. Take backup of this dir.
b. Goto $OMS_HOME i.e. $MIDDLEWARE_HOME/oms/bin directory
c. run configureBIP
Provide 'sys', 'Admin Server user' and 'Node Manager' password.

On demand demo/screenwatch of entire install and integration process can be accessed from here

EM12c: How to create a J2EE Application Component in software library?

To create a J2EE Application component in Software library for provisioning purpose, follow given steps:

1. Select Enterprise menu->Provisioning and Patching->Software Library.

2. Assuming you have already create a folder. Select that folder. In this case,  'myJ2EEApps' folder has been already created. On Software library page. select Create Entity-> Component

3. Select Java EE Application and click Continue.

4. On the Create Java EE Application Wizard: Describe page,
enter the Name, Description, and click Next.

5. On next page, select Software library upload location and
Select archive file from local or an agent machine.

Wait till archive file gets uploaded.

6. On next Review page, review all details and click on Save and Upload button.

7. It will create the J2EE app entity and will submit a job to upload the archive in software library.

8. Once the J2EE App is created, It will look like as given below.

Now uploaded J2EE app can be used to deploy/redeploy during provisioning.

Note: For more details, pls refer http://docs.oracle.com/cd/E24628_01/em.121/e27046/prov_java_ee.htm#CBBHEDHI

EM12c: JVMD Monitoring Sizing Tips

JVMD is a sampling tool that takes samples from JVM on a fixed frequency and then analyzes the collected data.

Following are some simple parameters that one needs to watch and can tune for JVMD monitoring.

1. Size of JVMD monitoring data depends on number of JVMs being monitored, sampling frequency for each JVM, number of active threads in a JVM, stack size of each of JVM thread etc.

2. Some info w.r.t. to JVMD sizing has been shared on http://docs.oracle.com/cd/E24628_01/install.121/e24089/sizing.htm#EMADV11866

3. JVMD polling and sampling frequency can be tuned from JVM pool setting page as given below

4. JVMD console/Engine parameters page can be used for following settings:

5. Along with these settings, it would be good idea to regularly monitor MGMT_AD4J_TS  tablespace growth for sometime like a month or so. After a month, if no new JVMs are added, tablespace size should become more constant and
predictable.

Note: These steps are also applicable for EM11g.

EM12c: How to create OS script based Metric Extension and deploy it on a target?

To create OS shell script based metric extension (ME), follow given steps:

1. Select Enterprise->Monitoring->Metric Extensions

2. Provide values for the metric extension properties in Create New wizard first page.

3. Provide shell scripting binary like /bin/sh, Shell Script name, Delimiter and Starts With expression as given below and Select Add button to add the shell script.

4. In this example, very simple script has been specified to explain the steps.

5. In next step of wizard, one needs to specify the output columns. As per given sample script, two columns Uptime and Users have been created here.

6. Select Credentials in next step to be used for this metric extension.

7. To test this metric extension, select Test Targets

8. Click on Run Test button to test the metric extension on Test target. You should see test output in Test Results panel.

9. Before submitting, review final definition for proposed metric extension.

10. Metric Extension is successfully created.

11. Once the metric extension is created, next step would be to save it as Deployable Draft as given below.

12. Once the status of metric extension shows as Deployable Draft. Select the metric extension and Deploy it to target as shown below.

13. You should see that the Metric Extension is deployed successfully.

14. After 15 min., Go to the target where this metric extension has been deployed. Select Monitoring->All Metrics. Yous should see following:

EM12c: How to use Weblogic Domain Provisioning Profile with Gold Image?

Assumption: User has EM12cR2 with FMW 12.1.0.3 plugin

A WebLogic Domain Provisioning Profile consists of the Middleware Home, binaries and the domain configuration. One can create a WLS domain provisioning profile, save it in the Software Library, and then use the saved profile as the source for creating new WebLogic domains. This will ensure that future WebLogic installations follow a standard, consistent configuration.
Lets see an example how to create WLS domain provisioning profile.

Assumption:
1. Both source as well destination Host and source WLS domain are monitored target on EMCC.
2. You have the host credentials for both source machine as well as destination host.
3. Source Weblogic Domain should be JRF enabled.

Steps to create WLS domain provisioning profile:
1. Select Enterprise menu->Provisioning and Patching->Software Library.

2. From the Actions menu->click Create Folder to create a folder in which the profile you are creating is to be stored as shown below:

3. Select the newly created folder->Actions menu-> Create Entity-> Component

4. A Create Component window appears. From the Select Subtype drop-down list, select the "WebLogic Domain Provisioning Profile" component and click Continue.

5. In the Create WebLogic Domain Provisioning Profile wizard: Describe page, provide details as given below:

6. Click Next. The Create WebLogic Domain Provisioning Profile: Configure page appears. Do ensure that checkbox for 'Include the binaries from Middleware Home in the profile to be created.' is selected. Now using search icon, select the source Weblogic Domain, from which you would like to create the gold image.
Provide the host credentials in the lower panel as shown here:

7. Click Next. In the Review page, review the profile information and click Save and Upload.

8. The <Job Name> is displayed at the top of the page. Navigate to the Job Activity page and check the job status.
Once it has been completed, navigate to the Software Library page and confirm if all the components of the profile (WebLogic Domain, MWHome, and Profile) have a Ready Status. You can also verify if the profile has been successfully created by navigating to the Middleware Provisioning page. From the Enterprise menu, select Provisioning and Patching, then select Middleware Provisioning and check if the profile is displayed there. You can now use this profile as the source for future WebLogic Domain installations.

Steps to clone a WLS domain from a WLS domain provisioning profile:
1.  Select Enterprise menu->Provisioning and Patching-> Middleware Provisioning.

2. Select profile that was created in earlier steps from the Profiles section->click Provision. ( or select the Middleware Provisioning deployment procedure and click Launch)

3. In the Source page of Provisioning wizard, the selected component and its location is displayed. Click Next.

4. In the Middleware Provisioning: Destinations page, specify the destination hosts on which the WebLogic Domain is to be cloned. Also provide host credentials. For details look at given image:

In the lower panel "Destination Locations" section, one can specify the location of the Middleware Home on the Host machine. The Create a New Middleware Home option is selected by default.

In the Middleware Home Directory field, one enter the full path to the directory in which the Middleware Home is to be created.

In the JDK Home Location field, enter the absolute path to the JDK directory to be used on the destination Host. This field can be edited only if the JDK Home in the source domain is in an external location. If the JDK Home in the source domain is internal and installed in the Middleware Home, this field cannot be edited.

In the Working Directory field, specify the directory on the destination Host on which the cloning related files are temporarily stored.

5. Click Next. Now the Middleware Provisioning: Domain Configuration page appears. This page contains a set of links to several pages where one can enter the properties that are most likely to be reconfigured like domain name, listen addresses for the administration server and managed servers, Node Manager/Machine configuration and JDBC data sources. By default, the source domain configuration settings are used as default values for the destination domain.

In the Server Startup Mode, select JRF startup clssses.

For more details on configuration, go to step 15-23 on http://docs.oracle.com/cd/E24628_01/em.121/e27046/prov_fmw.htm#CHDIJCGE

6. In the Schedule page, select default values. Click Next.

7. On the Review page, review the details. click Submit.

8. In the Procedure Activity page, view the status of the execution of the job and steps in the deployment procedure. Click the Status link for each step to view the details of the execution of each step. One can click Debug to set the logging level to Debug and click Stop to stop the procedure execution.

9. Once the procedure is successful, goto Targets->Middleware and see the newly created wls domain target.

EM12c: How to deploy JVMD agent on weblogic 12c manually?

Assumption: User has EM12cR2 with FMW 12.1.0.3 plugin

Simple option would be to deploy jamagent via Diagnostics Agent Deployment wizard. In case when deployment job fails for some reason then jamagent can be deployed manually. Instructions are given below.

1. Goto Setup Menu->Application Performance Management->Select JVM Diagnostics Engine->Click on Configure button as shown here

2. Select 'JVMs and Pools' tab on 'JVM Diagnostics Setup' page

3. Click on Download button

4. It will show you a Dialogue box that will list options to download JVMD Agent/DB Agent/JVMD Engine. Select JVMD Agent.

5. For a weblogic server, do provide the WLS server EM target name as Target Name and WLS Domain name as Pool Name. Click on Download button.

6. Download jamagent.war

7. Take this jamagent.war and deploy it via weblogic console UI.

Note: One can find more info on http://docs.oracle.com/cd/E24628_01/install.121/e24215/jvmd_using.htm#BCGCHEAG

EM12c: How to change the EM Agent Port?

The EMAgent uses a particular port to receive requests from the OMS. This port number is defined by default when one installs the EMAgent on a target host.
If one needs to modify this port at some point, one can use the following steps.

To change the EMAgent port:

1. Stop the EMAgent.
$ $AGENT_HOME/bin/emctl stop agent

2. Modify the $AGENT_HOME/sysman/config/emd.properties file in the EMAgent home directory:
    For example:
    EMD_URL=http://abc.xyz.com:3872/emd/main

    Modify the <port number> in the EMD_URL property so the EMAgent uses a new unused port on the target host.
    EMD_URL=http://abc.xyz.com:3875/emd/main

3. Start the EMAgent.
$ $AGENT_HOME/bin/emctl start agent

Note:
With this port change the target name of agent will not change on OMS side, it will still show the original port. But it will work using the new port now as given below

$ ./emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 2 
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 12.1.0.2.0
OMS Version       : 12.1.0.2.0
Protocol Version  : 12.1.0.1.0
Agent Home        : /home/oracle/agent12c/agent_inst
Agent Binaries    : /home/user01/agent12c/core/12.1.0.2.0
Agent Process ID  : 5732
Parent Process ID : 5643
Agent URL         : https://abc.xyz.com:3875/emd/main/
Repository URL    : https://oms.xyz.com:4901/empbs/upload
Started at        : 2012-11-09 10:02:47
Started by user   : oracle
Last Reload       : (none)
Last successful upload                       : 2012-11-09 11:35:21
Last attempted upload                        : 2012-11-09 11:35:21
Total Megabytes of XML files uploaded so far : 0.2
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 64.97%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2012-11-09 11:37:11
Last successful heartbeat to OMS             : 2012-11-09 11:37:11
Next scheduled heartbeat to OMS              : 2012-11-09 11:38:11

---------------------------------------------------------------
Agent is Running and Ready

EM12c: Managing the Manager (MTM) with EM12cR2

With release of EM12cR2, it has MTM pages to manage the Enterprise Manager more effectively. 

These MTM pages contain monitoring and diagnostic info for the EM itself. Using these pages, one can

-view the health of EM and its various components like OMS, repository, services etc

-track the performance (notification, upload etc.)

-view diagnostics info

-manage agents

Health Overview Page

Select Setup ->Manage Cloud Control-> Health Overview

All major areas of Enterprise Manager are shown here. It has:

Overview: Provides key information for active Management Services such as the Management Agents, the WebLogic Administration Server, total number of monitored targets, number of administrators, and server load balancer (SLB) upload and console URLs, provided SLB is configured. If configured, the SLB upload and console URLs are also displayed.

Repository Details: Provides physical information about the Management Repository and the host on which the database is located. You can drill down into the database home page for more information and carry out administrative operations.

Job System Status: Displays key operational parameters of the Enterprise Manager Job service. For detailed information, you can click on the status icon to drill down into the Enterprise Manager Job Service home page.

Console Activity: Displays the overall load on the Enterprise Manager console through the average number of requests per minute and the average time required to process those requests.

Alerts: Provides details on the metric errors recorded and when an alert was triggered. In-context links to Incident Manager are also provided.

Performance Charts: Upload Backlog and Upload Rate, Backoff Requests, Notification backlog. One can drill down into any chart to view detailed metric information.

One can also see diagnostics metrics.

Repository Page

Select Setup ->Manage Cloud Control->Repository

The Repository page provides you with an overview of the status and performance of the Repository DBMS Jobs that handle part of EM’s maintenance and monitoring functionality. These DBMS jobs run within the Management Repository and require no user input. Charts showing the key Repository Details and Backlog in Repository Collection are provided. The Scheduler Status region provides the status of the scheduler and the number of Job Queue Processes.

From the Repository Details region, one can expand Management Service Repository Sessions details to view the exact number of repository connections per individual Enterprise Manager sub-component such as the event system, console, job system, or connector framework.

The Repository Scheduler Jobs Status region provides details of the DBMS Jobs regarding their status, throughput, processing time, the next scheduled run and the last scheduled run.

Advanced Queuing (AQ) provides database-integrated message queuing functionality that allows application messages to be stored persistently and propagated between queues on different machines and databases. The Management Services AQ Status region list the details of DEQUEUE and ENQUEUE status for messages specific to the Management Services AQ.

Agents Page

Select Setup -> Manage Cloud Control-> Agents.

The Agents page lists all Management Agents within your monitored environment. This page also includes misconfigured, blocked and upgradable and non- upgradable Agents.

From here one can navigate to any agent's homepage.

Management Services

Select Setup -> Manage Cloud Control->Management Services

The Management Services page displays the list of Management Services, their status, incidents, the loader throughput , CPU usage, and the JVM memory usage metrics. In addition, the Management Services displayed can be filtered by Normal Mode, Console Only, PBS only and Standby Management Services.

Viewing Enterprise Manager Topology and Charts

 To view EM topology

Select Setup-> Manage Cloud Control ->Health Overview

Select OMS and Repository menu ->Members-> Topology

The Enterprise Manager Topology page provides a graphical representation of the Enterprise Manager infrastructure components and their association. Each node in the hierarchy displays key information about the member type, the host on which it resides, and the number of incidents, if any. The incident icons on each of the nodes expand to display a global view of current status for each node in the hierarchy.

To view EM charts

Select Setup-> Manage Cloud Control ->Health Overview

Select OMS and Repository menu ->Monitoring -> Charts

The Enterprise Manager Charts page displays eight charts representing key areas that together indicate the overall health of Enterprise Manager. These are Overall Files Pending Load -Agent, Job Step Backlog, Job Step Throughput (per second), Request Processing Time (ms), Database Time (centiseconds per second), CPU Utilization (%), Pages Paged-in (per second), Pages Paged-out (per second). Data can be viewed for the Last 24 hours, last 7 days or last 31 days.

For more details, refer http://docs.oracle.com/cd/E24628_01/doc.121/e24473/mtm.htm

EM12c: How to change the password for Weblogic Admin server and Nodemanager when existing password is known and register it with emctl commands?

[Caution: If you have already configured BI Publisher, make sure to change this password with BIP as well, else it can break BIP reports.]

One can change admin server and nodemanager password from weblogic console but to ensure that emctl acknowledges this password, there are additional steps.

1. Goto admin server console of EM_GC(GCDomain) domain
https://<hostname.doaminname>:<port>/console

2. Login to Admin server console as user weblogic and its password

3. Go to Security Realms->myrealm->Users and Groups->weblogic->Passwords

4. Provide the new password and save it

5. To change the nodemanager password, follow given steps:

In case of multiple OMS Setup, steps a and b need to be executed only on the OMS server where Admin server is running.

    a.Go to GCDomain->Security(In the right section) ->Advanced

    b.Enter the new password in the "NodeManager Password" and "Confirm NodeManager Password" fields

6.Click on 'Activate Changes' in the left section

7.Stop the OMS
$<OMS_HOME>/bin/emctl stop oms -all

Stop the Agent on OMS Host
$<AGENT_HOME>/bin/emctl stop agent

8. Modify the boot.properties file of the Admin Server
($DOMAIN_HOME/servers/EMGC_ADMINSERVER/security/boot.properties):
In the above file you will see encrypted values for username and password properties. Change the password value to the new password set(clear text).
After modifying the file, it should look like:
==============================================
username={AES}pyu0QhTe6+Xpk+OjEw7vy55OudlxwDXrftY7Pbm95qA=
password=<new_passwd_in_cleartext>
==============================================

9. Modify the boot.properties file of the Managed Server
($DOMAIN_HOME/servers/EMGC_OMS1/data/nodemanager/boot.properties):
(In case of multiple OMS Setup this step need to be performed on each OMS server)
In the above file you will see encrypted values for username and password properties. Change the password value to the new password set(clear text).
After modifying the file, it should look like:
================================================
TrustKeyStore=DemoTrust
username={AES}pyu0QhTe6+Xpk+OjEw7vy55OudlxwDXrftY7Pbm95qA=
password=<new_password_in_clear_text>
================================================

10.Edit the nm_password.properties file under
<gc_inst>/user_projects/domains/GCDomain/config/nodemanager/nm_password.properties
(In case of multiple OMS Setup, this step need to be performed on each OMS server)

Change
================================================
hashed=Qg8bEYV0uA7xJiPclZO0YY1e5rk\=
================================================
to
================================================
password=<new_nodemanager_password>
username=nodemanager
================================================

11.Run following command to save the new passwords to EM Credential store
$<OMS_ORACLE_HOME>/bin/emctl secure create_admin_creds_wallet -admin_pwd <weblogic_pwd> -nodemgr_pwd <node_manager_pwd>


12.Start the OMS
$<OMS_ORACLE_HOME>/bin/emctl start oms