Introduction
We have successfully implemented
Active/Passive Topology design for OBIEE 11g application.
Software/OS
version used:
OBIEE 11g 111.1.1.5.0 (single instance
database 11.2.0.3.0) running on top of Oracle 11g 11.2.0.3.0 CRS (aka
integrated with GNS/DHCP/dynamic SCAN configuration). Linux : RHEL 5.6 and SAN
storage used.
Technical
briefing:
1. Used 2 shared storage devices : 1 for ASM(Vote/OCR)
and 1 for OBIEE fusion middleware
2. 2 Virtual IP used – 1 for database listener and 1 for
OBIEE application.
3. 2 hosts used – in first host (say OMIPRI) database
will be actively running and OBIEE will be in standby mode. In 2nd host(OMISEC),
OBIEE will be active and Database will be in standby mode.
So if OMIPRI crashes, Database will move to OIMSEC
or if OMISEC crashes, OBIEE will migrate to OMIPRI. So in net , at any point in
time, OBIEE application will be active.
All the 3 (A,B and C) managed via CRS.
4.OBIEEVIP, Database VIP, GNS, SCAN addresses resolved
via DNS. This configuration supports both HA and scaling. When both nodes
up and running, HA is 100% and it drops to 0% HA when there is only one node is active at any point in time.
When
both nodes are active: Database running on OIMPRI and OBIEE running on OIMSEC
----------------------------------------------------------------------------------
[root@oimpri
~]# ps -ef|grep tns
oracle 5707
1 0 09:31 ? 00:00:00
/dbfiles/app/oracle/product/11.2.0/db/bin/tnslsnr LISTENER -inherit
oracle 6047
1 0 09:32 ? 00:00:00
/u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN2 -inherit
oracle 6076
1 0 09:32 ? 00:00:00
/u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN3 -inherit
root 9999 26162 0 11:22 pts/4 00:00:00 grep tns
[root@oimpri
~]# ps -ef|grep pmon
oracle 4892
1 0 09:29 ? 00:00:00 asm_pmon_+ASM1
oracle 5987
1 0 09:32 ? 00:00:00 ora_pmon_oimap
root 10001 26162 0 11:22 pts/4 00:00:00 grep pmon
[root@oimpri
~]#
Name Type Target State
-----------------------------------------------------------------------------------
ora.scan2.vip Other ONLINE ONLINE on oimpri
ora.oimsec.vip Other ONLINE ONLINE on oimsec
ora.LISTENER_SCAN3.lsnr Other ONLINE ONLINE on oimpri
ora.oc4j Other ONLINE ONLINE on oimpri
ora.gsd Other OFFLINE OFFLINE
ora.asdb.cfcdb Database Instance ONLINE
OFFLINE
ora.oimsec.ASM2.asm Other ONLINE ONLINE on oimsec
ora.asdblsnr.cfcdblsnr Database Listener ONLINE
ONLINE on oimpri
ora.obieevip.cfcvip Virtual IP ONLINE ONLINE on oimsec
ora.LISTENER_SCAN2.lsnr Other ONLINE ONLINE on oimpri
ora.obieeserver1.cfcas Opmn Instance ONLINE ONLINE on oimsec
ora.oimap.db Other ONLINE ONLINE on oimpri
ora.obieeserver.cfcas WebLogic Domain ONLINE
ONLINE on oimsec
ora.oimsec.gsd Other OFFLINE
OFFLINE
ora.LISTENER_SCAN1.lsnr Other ONLINE ONLINE on oimsec
ora.dbvip.cfcvip Virtual IP ONLINE ONLINE on oimpri
ora.gns.vip Other ONLINE ONLINE on oimpri
ora.oimpri.ons Other ONLINE ONLINE on oimpri
ora.asm Other ONLINE ONLINE on oimpri
ora.gns Other ONLINE ONLINE on oimpri
ora.DATA.dg Other ONLINE ONLINE on oimpri
ora.net1.network Other ONLINE ONLINE on oimpri
ora.cvu Other ONLINE ONLINE on oimpri
ora.oimsec.ons Other ONLINE OFFLINE
ora.scan3.vip Other ONLINE ONLINE on oimpri
ora.scan1.vip Other ONLINE ONLINE on oimsec
ora.dbdisk.cfcdisk Shared Disk ONLINE ONLINE on oimpri
ora.registry.acfs Other ONLINE ONLINE on oimpri
ora.ons Other ONLINE ONLINE on oimpri
ora.obieedisk.cfcdisk Shared Disk ONLINE ONLINE on oimsec
ora.oimpri.ASM1.asm Other ONLINE ONLINE on oimpri
ora.oimpri.vip Other ONLINE ONLINE on oimpri
ora.oimpri.gsd Other OFFLINE OFFLINE
1* select username,machine,count(*) from
v$session where username is not null group by username,machine
SQL>
/
USERNAME MACHINE
COUNT(*)
------------------------------
---------------------------------------------------------------- ----------
SYS oimpri
5
DEV_MDS oimsec
16
SQL>
After
the Database node (OIMPRI) crash: Database switched to OIMSEC. Now both
database and OBIEE running on the same node: OIMSEC
-------------------------------------------------------------------------------------------------
On
OIMSEC Node After the Database Failover:
[oracle@oimsec
bin]$ ps -ef|grep tns
oracle 775
1 0 08:13 ? 00:00:00
/u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
oracle 1058
1 0 08:13 ? 00:00:00
/dbfiles/app/oracle/product/11.2.0/db/bin/tnslsnr LISTENER -inherit
oracle 16112
1 0 07:45 ? 00:00:00
/u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN3 -inherit
oracle 16121
1 0 07:45 ? 00:00:00
/u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN2 -inherit
[oracle@oimsec
bin]$ ps -ef|grep pmon
oracle 1279
1 0 08:13 ? 00:00:00 ora_pmon_oimap
oracle 4742
1 0 07:04 ? 00:00:00 asm_pmon_+ASM2
Name Type Target State
-----------------------------------------------------------------------------------
ora.scan2.vip Other ONLINE ONLINE on oimsec
ora.oimsec.vip Other ONLINE ONLINE on oimsec
ora.LISTENER_SCAN3.lsnr Other ONLINE ONLINE on oimsec
ora.oc4j Other ONLINE ONLINE on oimsec
ora.gsd Other OFFLINE OFFLINE
ora.asdb.cfcdb Database Instance ONLINE
ONLINE on oimsec
ora.oimsec.ASM2.asm Other ONLINE ONLINE on oimsec
ora.asdblsnr.cfcdblsnr Database Listener ONLINE
ONLINE on oimsec
ora.obieevip.cfcvip Virtual IP ONLINE ONLINE on oimsec
ora.LISTENER_SCAN2.lsnr Other ONLINE
ONLINE on oimsec
ora.obieeserver1.cfcas Opmn Instance ONLINE ONLINE on oimsec
ora.oimap.db Other ONLINE OFFLINE
ora.obieeserver.cfcas WebLogic Domain ONLINE
ONLINE on oimsec
ora.oimsec.gsd Other OFFLINE OFFLINE
ora.LISTENER_SCAN1.lsnr Other ONLINE ONLINE on oimsec
ora.dbvip.cfcvip Virtual IP ONLINE ONLINE on oimsec
ora.gns.vip Other ONLINE ONLINE on oimsec
ora.asm Other ONLINE ONLINE on oimsec
ora.gns Other ONLINE ONLINE on oimsec
ora.DATA.dg Other ONLINE ONLINE on oimsec
ora.net1.network Other ONLINE ONLINE on oimsec
ora.cvu Other ONLINE ONLINE on oimsec
ora.oimsec.ons Other ONLINE ONLINE on oimsec
ora.scan3.vip Other ONLINE ONLINE on oimsec
ora.scan1.vip Other ONLINE ONLINE on oimsec
ora.dbdisk.cfcdisk Shared Disk ONLINE ONLINE on oimsec
ora.registry.acfs Other ONLINE ONLINE on oimsec
ora.ons Other ONLINE ONLINE on oimsec
ora.obieedisk.cfcdisk Shared Disk ONLINE ONLINE on oimsec
ora.oimpri.vip Other ONLINE ONLINE on oimsec
1* select username,machine,count(*) from
v$session where username is not null group by username,machine
SQL>
/
USERNAME MACHINE
COUNT(*)
------------------------------
---------------------------------------------------------------- ----------
SYS oimpri
5
DEV_MDS oimsec
4
After
the OBIEE node (OIMSEC) crash: OBIEE and Database switched to OIMPRI. OMIPRI is
already up without any servicing running.
-------------------------------------------------------------------------------------------------
On
oimpri Node After the Database Failover:
[oracle@oimpri
bin]$ ps -ef|grep tns
oracle 775
1 0 08:13 ? 00:00:00
/u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
oracle 1058
1 0 08:13 ? 00:00:00
/dbfiles/app/oracle/product/11.2.0/db/bin/tnslsnr LISTENER -inherit
oracle 16112
1 0 07:45 ? 00:00:00
/u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN3 -inherit
oracle 16121
1 0 07:45 ? 00:00:00
/u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN2 -inherit
[oracle@oimpri
bin]$ ps -ef|grep pmon
oracle 1279
1 0 08:13 ? 00:00:00 ora_pmon_oimap
oracle 4742
1 0 07:04 ? 00:00:00 asm_pmon_+ASM2
Name Type Target State
-----------------------------------------------------------------------------------
ora.scan2.vip Other ONLINE ONLINE on oimpri
ora.oimpri.vip Other ONLINE ONLINE on oimpri
ora.LISTENER_SCAN3.lsnr Other ONLINE ONLINE on oimpri
ora.oc4j Other ONLINE ONLINE on oimpri
ora.gsd Other OFFLINE OFFLINE
ora.asdb.cfcdb
Database Instance ONLINE
ONLINE on oimpri
ora.oimpri.ASM2.asm Other ONLINE ONLINE on oimpri
ora.asdblsnr.cfcdblsnr Database Listener ONLINE
ONLINE on oimpri
ora.obieevip.cfcvip Virtual IP ONLINE ONLINE on oimpri
ora.LISTENER_SCAN2.lsnr Other ONLINE ONLINE on oimpri
ora.obieeserver1.cfcas Opmn Instance ONLINE ONLINE on oimpri
ora.oimap.db Other ONLINE OFFLINE
ora.obieeserver.cfcas WebLogic Domain ONLINE
ONLINE on oimpri
ora.oimpri.gsd Other OFFLINE OFFLINE
ora.LISTENER_SCAN1.lsnr Other ONLINE ONLINE on oimpri
ora.dbvip.cfcvip Virtual IP ONLINE ONLINE on oimpri
ora.gns.vip Other ONLINE ONLINE on oimpri
ora.asm Other ONLINE ONLINE on oimpri
ora.gns Other ONLINE ONLINE on oimpri
ora.DATA.dg Other ONLINE ONLINE on oimpri
ora.net1.network Other ONLINE ONLINE on oimpri
ora.cvu Other ONLINE ONLINE on oimpri
ora.oimpri.ons Other ONLINE ONLINE on oimpri
ora.scan3.vip Other ONLINE ONLINE on oimpri
ora.scan1.vip Other ONLINE ONLINE on oimpri
ora.dbdisk.cfcdisk Shared Disk ONLINE ONLINE on oimpri
ora.registry.acfs Other ONLINE ONLINE on oimpri
ora.ons Other ONLINE ONLINE on oimpri
ora.obieedisk.cfcdisk Shared Disk ONLINE
ONLINE on oimpri
ora.oimpri.vip Other ONLINE ONLINE on oimpri
1* select username,machine,count(*) from
v$session where username is not null group by username,machine
SQL>
/
USERNAME MACHINE
COUNT(*)
------------------------------
---------------------------------------------------------------- ----------
SYS oimpri 5
DEV_MDS oimpri
7
Maintenance
o OMISEC is completed, so that OBIEE resources manually switched to OIMSEC
---------------------------------------------------------------------------------------
[root@oimpri
~]# ps -ef|grep tns
oracle 5707
1 0 09:31 ? 00:00:00
/dbfiles/app/oracle/product/11.2.0/db/bin/tnslsnr LISTENER -inherit
oracle 6047
1 0 09:32 ? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr
LISTENER_SCAN2 -inherit
oracle 6076
1 0 09:32 ? 00:00:00
/u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN3 -inherit
root 9999 26162 0 11:22 pts/4 00:00:00 grep tns
[root@oimpri
~]# ps -ef|grep pmon
oracle 4892
1 0 09:29 ? 00:00:00 asm_pmon_+ASM1
oracle 5987
1 0 09:32 ? 00:00:00 ora_pmon_oimap
root 10001 26162 0 11:22 pts/4 00:00:00 grep pmon
[root@oimpri
~]#
Name Type Target
State
-----------------------------------------------------------------------------------
ora.scan2.vip Other ONLINE ONLINE on oimpri
ora.oimsec.vip Other ONLINE ONLINE on oimsec
ora.LISTENER_SCAN3.lsnr Other ONLINE ONLINE on oimpri
ora.oc4j Other ONLINE ONLINE on oimpri
ora.gsd Other OFFLINE OFFLINE
ora.asdb.cfcdb Database Instance ONLINE
OFFLINE
ora.oimsec.ASM2.asm Other ONLINE ONLINE on oimsec
ora.asdblsnr.cfcdblsnr Database Listener ONLINE
ONLINE on oimpri
ora.obieevip.cfcvip Virtual IP ONLINE ONLINE on oimsec
ora.LISTENER_SCAN2.lsnr Other ONLINE ONLINE on oimpri
ora.obieeserver1.cfcas Opmn Instance ONLINE ONLINE on oimsec
ora.oimap.db Other ONLINE ONLINE on oimpri
ora.obieeserver.cfcas WebLogic Domain ONLINE
ONLINE on oimsec
ora.oimsec.gsd Other OFFLINE OFFLINE
ora.LISTENER_SCAN1.lsnr Other ONLINE ONLINE on oimsec
ora.dbvip.cfcvip Virtual IP ONLINE
ONLINE on oimpri
ora.gns.vip Other ONLINE ONLINE on oimpri
ora.oimpri.ons Other ONLINE ONLINE on oimpri
ora.asm Other ONLINE ONLINE on oimpri
ora.gns Other ONLINE ONLINE on oimpri
ora.DATA.dg Other ONLINE ONLINE on oimpri
ora.net1.network Other ONLINE ONLINE on oimpri
ora.cvu Other ONLINE ONLINE on oimpri
ora.oimsec.ons Other ONLINE OFFLINE
ora.scan3.vip Other ONLINE ONLINE on oimpri
ora.scan1.vip Other ONLINE ONLINE on oimsec
ora.dbdisk.cfcdisk Shared Disk ONLINE ONLINE on oimpri
ora.registry.acfs Other ONLINE ONLINE on oimpri
ora.ons Other ONLINE ONLINE on oimpri
ora.obieedisk.cfcdisk Shared Disk ONLINE ONLINE on oimsec
ora.oimpri.ASM1.asm Other ONLINE ONLINE on oimpri
ora.oimpri.vip Other ONLINE ONLINE on oimpri
ora.oimpri.gsd Other OFFLINE OFFLINE
1* select username,machine,count(*) from
v$session where username is not null group by username,machine
SQL>
/
USERNAME MACHINE COUNT(*)
------------------------------
---------------------------------------------------------------- ----------
SYS oimpri
5
DEV_MDS oimsec
3
All
resource dependency trees
-------------------------------------------------
ora.scan2.vip
ora.oimsec.vip
ora.LISTENER_SCAN3.lsnr
ora.oc4j
ora.gsd
ora.asdb.cfcdb
|
+->ora.asdblsnr.cfcdblsnr
|
|
|
+->ora.dbvip.cfcvip
|
|
|
+->ora.dbdisk.cfcdisk
|
+->ora.dbdisk.cfcdisk
ora.oimsec.ASM2.asm
ora.LISTENER_SCAN2.lsnr
ora.obieeserver1.cfcas
|
+->ora.obieevip.cfcvip
|
+->ora.obieedisk.cfcdisk
ora.oimap.db
ora.obieeserver.cfcas
|
+->ora.obieevip.cfcvip
|
+->ora.obieedisk.cfcdisk
ora.oimsec.gsd
ora.LISTENER_SCAN1.lsnr
ora.gns.vip
ora.asm
ora.gns
ora.DATA.dg
ora.net1.network
ora.cvu
ora.oimsec.ons
ora.scan3.vip
ora.scan1.vip
ora.registry.acfs
ora.ons
ora.oimpri.vip
Cluster
Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1
ONLINE ONLINE oimpri
ora.LISTENER_SCAN2.lsnr
1
ONLINE ONLINE oimsec
ora.LISTENER_SCAN3.lsnr
1
ONLINE ONLINE oimsec
ora.asdb.cfcdb
1
ONLINE ONLINE oimsec
ora.asdblsnr.cfcdblsnr
1
ONLINE ONLINE oimsec
ora.cvu
1
ONLINE ONLINE oimsec
ora.dbdisk.cfcdisk
1
ONLINE ONLINE oimsec
ora.dbvip.cfcvip
1
ONLINE ONLINE oimsec
ora.gns
1
ONLINE ONLINE oimsec
ora.gns.vip
1
ONLINE ONLINE oimsec
ora.obieedisk.cfcdisk
1
ONLINE ONLINE oimsec
ora.obieeserver.cfcas
1
ONLINE ONLINE oimsec
ora.obieeserver1.cfcas
1
ONLINE OFFLINE
ora.obieevip.cfcvip
1
ONLINE ONLINE oimsec
ora.oc4j
1
ONLINE ONLINE oimsec
ora.oimap.db
1 ONLINE
OFFLINE
ora.oimpri.vip
1
ONLINE ONLINE oimpri
ora.oimsec.vip
1
ONLINE ONLINE oimsec
ora.scan1.vip
1
ONLINE ONLINE oimpri
ora.scan2.vip
1
ONLINE ONLINE oimsec
ora.scan3.vip
1
ONLINE ONLINE oimsec
Database
cluster resource ora.asdb.cfcdb dependency tree
--------------------------------------------------------
Basic information
------------------------+------------------------
Name | ora.asdb.cfcdb
Type | Database Instance
Target state |
ONLINE
Resource state |
ONLINE on oimpri
Restart count |
0
Failure count |
0
Hosting members |
oimpri, oimsec
------------------------+------------------------
Common CRS parameters
------------------------+------------------------
Auto start | Yes
Check interval | 120 sec
Failover delay | 30 sec
Failure interval | 700 sec
Failure threshold | 5
Restart attempts | 4
Script timeout | 300 sec
Start timeout | 300 sec
Stop timeout | 300 sec
------------------------+------------------------
Resource specific parameters
------------------------+------------------------
Database component | dbinstance
ORACLE_HOME |
/dbfiles/app/oracle/product/11.2.0/db
ORACLE_SID | oimap
pfile | Database default
------------------------+------------------------
Resource dependency tree(s)
-------------------------------------------------
ora.asdb.cfcdb
|
+->ora.asdblsnr.cfcdblsnr
|
|
|
+->ora.dbvip.cfcvip
|
|
|
+->ora.dbdisk.cfcdisk
|
+->ora.dbdisk.cfcdisk
Database
cluster resource ora.obieeserver.cfcas dependency tree
--------------------------------------------------------
Basic information
------------------------+------------------------
Name |
ora.obieeserver.cfcas
Type | WebLogic Domain
Target state |
ONLINE
Resource state |
ONLINE on oimsec
Restart count |
0
Failure count |
0
Hosting members |
oimpri, oimsec
------------------------+------------------------
Common CRS parameters
------------------------+------------------------
Auto start | Yes
Check interval | 50 sec
Failover delay | 20 sec
Failure interval | 350 sec
Failure threshold | 5
Restart attempts | 4
Script timeout | 600 sec
Start timeout | 600 sec
Stop timeout | 600 sec
------------------------+------------------------
Resource specific parameters
------------------------+------------------------
Component home |
(WebLogic domain) |
/fmwfiles/app/oracle/obieehome/user_projects/domains/bifoundation_domain
------------------------+------------------------
Components
(Managed servers)
------------------------+------------------------
Name | Monitor
------------------------+------------------------
AdminServer | default
bi_server1 | default
------------------------+------------------------
Resource dependency tree(s)
-------------------------------------------------
ora.obieeserver.cfcas
|
+->ora.obieevip.cfcvip
|
+->ora.obieedisk.cfcdisk
[root@oimsec
bin]#
Advantages :
1.Automatic recovery of
Database and OBIEE resources from Node failures.
2.Lost Cost : License
3.Can be effectively used
in environments like where cost is a constraint but scaling can be compromised.
So this setup is best suited for small to Mid-level customers.
4.Planned Maintenance: HW
can be upgraded while the application is up and running, like Memory upgrades –
both nodes.
5.High availability with 2
Tier setup (scaling) at reduced cost.
Disadvantages:
1.May not possible to
completely switch over or fail over the ASCRS resources when there
some dependencies active.
So either manual
intervention or some sort of automatic scripts to be in place in such scenario.
2.OBIEE sessions may need
to refreshed or reinitiated after the Failover or switch over.
Oracle is aware of these
complexities and still supports this architecture for low to mid- level
customers (Cost).
Please remember it is
possible to build (oracle supported) 100% HA (both DB and OBIEE) +
Scaling (Vertical or Horizontal) in place for OBIEE (or any type of
Fusion Middleware application) component – at the expense of COST.
3. At present ASCRS is
supported only in Unix platforms.
Conclusion:
In Our Real Practical DR Scenario
:
|
Switch Time
|
Database Node Crash - Database
automatically migrated to OBIEE - OIMSEC Node
|
1 Min 20 Sec
|
OBIEE Node Crash - OBIEE
automatically migrated to Database Node - OIMPRI Node
|
3 Min 10 Sec
|
Manual Failover the Database to
OBIEE(OIMSEC)Node - Maintenance
|
1Min 40 Sec
|
Manual Failover the OBIEE to
Database (OIMPRI) Node - Maintenance
|
2 Min 50 Sec
|
