Oracle Grid HA Set up - 10.2.0.5.0

Install RAC 11.1.0.6.0 database first (2 nodes rac1 and rac2)

This is a 4 node Grid configuration -99% HA Grid Architecture
-------------------------------------------------------------
Node1 - Rac1 - Oracle Database 11G 11.0.1.6
Node1 - Rac2 - Oracle Database 11G 11.0.1.6
Node3 - OMS1 - 10.2.0.5.0
Node4 - OMS2 - 10.2.0.5.0

Overview of Installation and Configuration
-----------------------------------------

Node OMS1

1.OMS1 Install 10.2.0.1 Grid on Linux - Software only
2.Upgrade to 10.2.0.5 software only
3.Upgrade the agent 10.2.0.5
4.Configure the Grid using ConfigureFC.pl perl script
5.Drop and Recreate the Grid repository using RepManager
6.Start the OMS process and agent.Check everything is functional
7.Stop the entire system and take a cold backup
8.start the OMS and Agent

Node OMS2
9.Install the grid 10.2.0.1 (using additional management service option)
10.Upgrade the grid 10.2.0.1 and agent
11.export the oms configuration files from OMS1 and import into OMS2 (please see below)
12.setup EMCLI
13.Configure the secure ,Agent registration,webcache ports in LBR

14.secure the OMS and agent on both nodes (OMS1 and OMS2)
15.Check the agenet upload status is fine with LBR , each OMS server is able to loopback to LBR server
16.Stop the entire system and have a backup.


issues encountered

emctl start oms - fails with the following error
-----------------------------------------------
Connection to the repository failed.Verify that the repository connection information provided is correct.

Check the emoms.log file for any errors.It could be ORA-01017 , invalid username/password.

Solution:-

alter user sysman identified by sysman123
emctl stop agent
opmnctl stopall

Change the SYSMAN password in the file emoms.properties
Backup the file emoms.properties
$ cd $ORACLE_HOME/sysman/config
$ cp emoms.properties emoms.properties.orig
Edit the file emoms.properties and change the following lines
oracle.sysman.eml.mntr.emdRepPwd=sysman123
oracle.sysman.eml.mntr.emdRepPwdEncrypted=FALSE

When you restart the OMS, the FALSE setting will change to TRUE and the password will become encrypted in the file


OMS Hearbeat unknown
--------------------
Could be a bug.The agent is not to able to upload into OMS using the HTTP protocol.
So secure the agent first and then the agent

./emctl secure oms

./emctl secure agent


Failover testing for the OMS service
-----------------------------------


[oracle@rac1 config]$ /u01/oracle/product/oms/oms10g/opmn/bin/opmnctl status

Processes in Instance: EnterpriseManager0.rac1.ushasuji.com
-------------------+--------------------+---------+---------
ias-component | process-type | pid | status
-------------------+--------------------+---------+---------
DSA | DSA | N/A | Down
HTTP_Server | HTTP_Server | 31109 | Alive
LogLoader | logloaderd | N/A | Down
dcm-daemon | dcm-daemon | N/A | Down
OC4J | home | 31110 | Alive
OC4J | OC4J_EMPROV | 31111 | Alive
OC4J | OC4J_EM | 31112 | Alive
WebCache | WebCache | 31157 | Alive
WebCache | WebCacheAdmin | 31117 | Alive

On Node2
--------

SQL> select username,machine from v$session where username is not null;

USERNAME MACHINE
------------------------------ ----------------------------------------------------------------
SYSMAN rac1.ushasuji.com
SYSMAN rac1.ushasuji.com
SYSMAN rac1.ushasuji.com
SYSMAN rac1.ushasuji.com
SYSMAN rac1.ushasuji.com
SYSMAN rac1.ushasuji.com
SYS rac2
SYSMAN rac1.ushasuji.com
SYSMAN rac1.ushasuji.com
SYSMAN rac1.ushasuji.com
SYSMAN rac1.ushasuji.com
SYSMAN rac1.ushasuji.com
SYS rac2
SYS rac1
SYS rac2
SYSMAN rac1.ushasuji.com
SYSMAN rac1.ushasuji.com

17 rows selected.

SQL> select host_name from v$instance;

HOST_NAME
----------------------------------------------------------------
rac2

SQL> shutdown abort
ORACLE instance shut down.

On Node 1 - Start the database and you see the Grid process dynamically switch (for rac1 and rac2)
to node1 from node2
-------------------

[oracle@rac1 config]$ ps -ef|grep pmon
oracle 12144 10322 0 00:31 pts/6 00:00:00 grep pmon
[oracle@rac1 config]$


SQL> /

USERNAME MACHINE
------------------------------ ----------------------------------------------------------------
SYS rac1
SYS rac1

SQL> /

USERNAME MACHINE
------------------------------ ----------------------------------------------------------------
SYSMAN rac2.ushasuji.com
SYSMAN rac1.ushasuji.com
SYSMAN rac2.ushasuji.com
SYSMAN rac1.ushasuji.com
SYS rac1
SYSMAN rac1.ushasuji.com
SYS rac1
SYS rac1

SQL> /

USERNAME MACHINE
------------------------------ ----------------------------------------------------------------
SYSMAN rac2.ushasuji.com
SYSMAN rac2.ushasuji.com
SYSMAN rac1.ushasuji.com
SYSMAN rac2.ushasuji.com
SYSMAN rac2.ushasuji.com
SYSMAN rac1.ushasuji.com
SYSMAN rac2.ushasuji.com
SYSMAN rac1.ushasuji.com
SYSMAN rac2.ushasuji.com
SYSMAN rac1.ushasuji.com
SYSMAN rac2.ushasuji.com
SYSMAN rac1.ushasuji.com
SYSMAN rac2.ushasuji.com
SYS rac1
SYSMAN rac1.ushasuji.com
SYSMAN rac2.ushasuji.com
SYS rac1
SYS rac1

18 rows selected.


SQL> !hostname
rac1


Again on node 2
-------------
start the instance 2

USERNAME MACHINE
------------------------------ ----------------------------------------------------------------
SYS rac2
SYS rac1
SYS rac2


USERNAME MACHINE
------------------------------ ----------------------------------------------------------------
SYSMAN rac1.ushasuji.com
SYSMAN rac1.ushasuji.com
SYSMAN rac1.ushasuji.com
SYSMAN rac1.ushasuji.com
SYSMAN rac1.ushasuji.com
SYSMAN rac1.ushasuji.com
SYSMAN rac1.ushasuji.com
SYSMAN rac1.ushasuji.com
SYSMAN rac1.ushasuji.com
SYSMAN rac1.ushasuji.com
SYSMAN rac1.ushasuji.com
SYSMAN rac1.ushasuji.com
SYS rac1
SYS rac1
SYSMAN rac1.ushasuji.com
SYSMAN rac1.ushasuji.com
SYS rac1
SYS rac1

18 rows selected.

Now the process are evenly distributed on nodes

USERNAME MACHINE
------------------------------ ----------------------------------------------------------------
SYSMAN rac2.ushasuji.com
SYSMAN rac2.ushasuji.com
SYSMAN rac2.ushasuji.com
SYSMAN rac2.ushasuji.com
SYSMAN rac2.ushasuji.com
SYSMAN rac2.ushasuji.com
SYSMAN rac2.ushasuji.com
SYSMAN rac2.ushasuji.com
SYS rac2
SYS rac1
SYS rac2
SYSMAN rac2.ushasuji.com
SYSMAN rac2.ushasuji.com

13 rows selected.


Export the OMS configuration on rac1 node
-------------------------------------------

Enter Enterprise Manager Root (SYSMAN) Password : ExportConfig started... Exporting emoms properties... Exporting emomslogging properties... Exporting secure properties... Export has determined that the OMS is not fronted by an SLB. The local hostname was NOT exported. The exported data can be imported on any host but resecuring of all agents will be required. Please see the EM Advanced Configuration Guide for more details. Exporting port properties... Exporting emkey... Backup has been written to file: export_oms/opf_20090615_073839.bka The export file contains sensitive data. Please ensure that it is kept secure. ExportConfig completed successfully! [oracle@rac1 bin]$ cd export_oms/ [oracle@rac1 export_oms]$ ls opf_20090615_073839.bka [oracle@rac1 export_oms]$ ls -ltr
total 40 -rw-r----- 1 oracle oinstall 38060 Jun 15 07:38 opf_20090615_073839.bka [oracle@rac1 export_oms]$ scp opf_20090615_073839.bka rac2:/u01/oracle/product/oms/oms10g/bin/export_oms
opf_20090615_073839.bka 100% 37KB 37.2KB/s 00:00 [oracle@rac1 export_oms]$




import this backup oms file on to rac2 node


Enter Agent Registration Password: Pre-import OMS backup successfully stored to /u01/oracle/product/oms/oms10g/sysman/backup/opf_20090615_050013.bka Configuring emoms.properties... Configuring emomslogging.properties... Copying emkey to system... Emkey verified. Resecuring the OMS... OMS resecured!


We will be adding DataGuard service to this HA Grid environment soon.