Oracle Home Cloning For RAC

A short note on the steps for cloning Oracle Home for RAC:

1. On all Nodes: Copy existing ORACLE_HOME binaries to a new OH location
$cp -pr /opt/oracle/product/10.2.0.2 /opt/oracle/product/10.2.0.4

2. Add entries for the new Oracle_HOME into the inventory

• CD to the new OH: cd /opt/oracle/product/10.2.0.4/clone/bin
• perl clone.pl ORACLE_HOME=/opt/oracle/product/10.2.0.4 ORACLE_HOME_NAME=Ora10g_10204 "CLUSTER_NODES=alps,everest" "LOCAL_NODE=alps"
  
• Execute /opt/oracle/product/10.2.0.4/root.sh script as root.
• Apply the 10.2.0.4 patchset unto the newly created OH

APEX Configuration - 11g

APEX is included with the default Oracle11g installation. You can check if the APEX component is included in the installation by verifying the existence of apex directory under $OH and also by checking DBA_REGISTRY:

SQL> Select comp_name, version, status from dba_registry where comp_name
2 like '%Express%';

COMP_NAME VERSION STATUS
------------------------------ ------------------------------ -----------
Oracle Expression Filter 11.1.0.6.0 VALID
Oracle Application Express 3.0.1.00.08 VALID

Configuring APEX

1. Login as SYSDBA and run the script $OH/apex/apxconf

SQL> @apxconf

PORT
----------
8080

Enter values below for the XDB HTTP listener port and the password for the Application Express ADMIN user.
Default values are in brackets [ ].
Press Enter to accept the default value.


Enter a password for the ADMIN user [] chandra
Enter a port for the XDB HTTP listener [ 8080]
...changing HTTP Port

PL/SQL procedure successfully completed.


PL/SQL procedure successfully completed.


Session altered.

...changing password for ADMIN

PL/SQL procedure successfully completed.


Commit complete.

SQL>

2. Set the port number:

SQL> exec dbms_xdb.sethttpport(8080);

PL/SQL procedure successfully completed.

3. Unlock the ANONYMOUS ACCOUNT:

4. Logon to APEX using http://hostname:8080/apex/apex_admin



Use ADMIN as the username and the password which we used at step one above.



5. Create a WORKSPACE










With this we will have a WORKSPACE to start creating our own applications.

ORA-27102: out of memory

Though we may have sufficient free physical memory, we often see this error message indicative of low memory availability. But the actual cause of this error is typically caused by the incorrect value set for shmall kernel parameter.

1. Let' set the value for shmall to a very low value.
Current Value:

[root@chandra ~]# sysctl -a |grep shmall
kernel.shmall = 1000000
[root@chandra ~]#

Setting the value to a new/low value:

[root@chandra ~]# sysctl -w kernel.shmall=10000
kernel.shmall = 10000
[root@chandra ~]# sysctl -a |grep shmall
kernel.shmall = 10000
[root@chandra ~]#

2. An attempt to startup the database fails with ORA-27102

SQL> startup
ORA-27102: out of memory
Linux Error: 28: No space left on device
SQL>

It failed with "out of memory" error message, though, we have 500M free space

[root@chandra ~]# free
total used free shared buffers cached
Mem: 2066192 1488748 577444 0 18496 1134776
-/+ buffers/cache: 335476 1730716
Swap: 8225172 0 8225172
[root@chandra ~]#

3. Let's set the shmall back to value to a high value:

[root@chandra ~]# sysctl -w kernel.shmall=1000000
kernel.shmall = 1000000
[root@chandra ~]#

4. An attempt to startup the database just works, even though nothing from the memory utilization or availability perspective changed!

Oracle 10g CRS upgrade to 11g CRS

To upgrade Oracle 10g CRS to 11g CRS, we have two options -

• Perform a rolling upgrade (this requires that your current CRS version >=10.2.0.3 (or 10.2.0.2 with Bundle patch). This option allows us to upgrade the CRS without a complete unavailabilty of downtime
  
• Upgrade the CRS on all the nodes at the same time with complete downtime.

I will go over the steps for upgrading the CRS to 11g using the first option mentioned above.

On Node 1:

1. Stop the CRS: Either use $CRS_HOME/bin/crsctl stop crs as root or run the following from the staging as root:

/Staging_Area/11.1.0.6/clusterware/upgrade/preupdate.sh –crshome $CRS_HOME –crsuser oracle

2. Invoke runInstaller from the staging area:



3. As you notice, the option to specify the Home and the destination directory is disabled since OUI has detected the existence of a Cluster.


4. Select the local node (in my case it is alps01), selecting both the nodes requires the CRS to be down on both the nodes - meaning it would require a complete downtime - and would no more be a rolling upgrade:



5. Not shutting down the CRS on the local node would cause the following error:

6. This information would appear in the install logs - pretty detailed and informative :-)

7. Once you shutdown the CRS on the local node and when re-attempted, it should be fine (no errors or warnings):


8. On clicking next, the summary screen would appear. Note that we are upgrading the CRS only the local node (alps01):

9. Install progress screen:

10. At the end of the upgrade, the Installer would prompted us to run the rootupgrade script as root:


11. Output of the rootupgrade script:

12. Check the status of CRS versions:


On Node 2:
13. Shutdown the CRS on node 2. Note that while performing a rolling upgrade, we have to invoke the OUI from the second node and this can't be done from node 1.

14. Invoke the OUI from node 2 and select only the local node and deselect the remote or first node:

15. Cluster Verification screen....looks good.

17. Prompt to run rootupgrade script after the upgrade on Node 2:

18. Output of rootupgrade script on Node 2:

[root@everest02 upgrade]# /opt/oracle/product/crs/install/rootupgrade
Checking to see if Oracle CRS stack is already up...
WARNING: directory '/opt/oracle/product' is not owned by root
WARNING: directory '/opt/oracle' is not owned by root

Oracle Cluster Registry configuration upgraded successfully
Adding daemons to inittab
Attempting to start Oracle Clusterware stack
Waiting for Cluster Synchronization Services daemon to start
Waiting for Cluster Synchronization Services daemon to start
Waiting for Cluster Synchronization Services daemon to start
Cluster Synchronization Services daemon has started
Event Manager daemon has started
Cluster Ready Services daemon has started
Oracle CRS stack is running under init(1M)
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node :
node 2: everest02 everest02-priv everest02
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
clscfg -upgrade completed successfully
CRS stack on this node, is successfully upgraded to 11.1.0.6.0
Checking the existence of nodeapps on this node
Creating '/opt/oracle/product/crs/install/paramfile.crs' with data used for CRS configuration
Setting CRS configuration values in /opt/oracle/product/crs/install/paramfile.crs

[root@everest02 upgrade]#

19. Check or verify the version of CRS:

Configuring SSH

Configure SSH between two nodes

While logged in as oracle perform the following on each of the nodes of the RAC:

1. cd $HOME
2. mkdir ~/.ssh
3. chmod 700 ~/.ssh
4. /usr/bin/ssh-keygen -t rsa
5. /usr/bin/ssh-keygen -t dsa

On Node 1:

1. cd $HOME/.ssh
2. cat id_rsa.pub >> authorized_keys
3. cat id_dsa.pub >> authorized_keys
4. Copy the authorized_keys file to the node 2. scp authorized_keys node2:/opt/oracle/.ssh

On Node 2:

1. cd $HOME/.ssh
2. cat id_rsa.pub >> authorized_keys
3. cat id_dsa.pub >> authorized_keys
4. scp authorized_keys node1:/opt/oracle/.ssh

• Now perform a ssh between all the nodes including the node-priv hostnames as well. Check to make sure that ssh is configured well without prompting for the password (on both the nodes):
  

$ ssh coke.pinnacle.com date
Sun Aug 12 08:41:42 CDT 2007
$ ssh pepsi.pinnacle.com date
Sun Aug 12 08:42:23 CDT 2007
$ ssh coke-priv date
Sun Aug 12 08:42:45 CDT 2007
$ ssh pepsi-priv date
Sun Aug 12 08:43:22 CDT 2007

Firewire Modules not loading or not compatible with the kernel version

The Firewire modules are currently not available for all the kernel versions of Red Hat 4.0, and I really doubt if Oracle is going to make available the Firewire modules for them. For example, my installed kernel version is 2.6.9-42.0.0.0.1.EL, but there are no firewire modules available for this version of kernel and the only closest module available is oracle-firewire-modules-2.6.9-42.0.3.EL-1286-1.i686.rpm. v Obviously, the versions are not compatible and the modules wouldn't load, even if you install and when attempted you are likely to see:

# rpm -ivhi --force oracle-firewire-modules-2.6.9-42.0.3.EL-1286-1.i686.rpm

Preparing... ########################################### [100%] 1:oracle-firewire-modules########################################### [100%]

# modprobe ohci1394 ; modprobe sbp2 FATAL: Module ohci1394 not found. FATAL: Module sbp2 not found. #

There are possibly two options to fix this.

Option 1: The first one is to load the kernel installed by the firewire module, if you are not very particular about the kernel version. Once the firewire modules are installed, the /etc/grub.conf is populated with the Firewire kernel and which looks some thing like this:

#cat /etc/grub.conf
# grub.conf generated by anaconda
# # Note that you do not have to rerun grub after making changes to this file
# NOTICE: You do not have a /boot partition. This means that
# all kernel and initrd paths are relative to /, eg.
# root (hd0,1)
# kernel /boot/vmlinuz-version ro root=/dev/sda2
# initrd /boot/initrd-version.img
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,1)/boot/grub/splash.xpm.gz

hiddenmenu
title Enterprise (2.6.9-42.0.0.0.1.ELhugemem)
root (hd0,1)
kernel /boot/vmlinuz-2.6.9-42.0.0.0.1.ELhugemem ro root=LABEL=/ rhgb quiet
initrd /boot/initrd-2.6.9-42.0.0.0.1.ELhugemem.img
title Enterprise-smp (2.6.9-42.0.0.0.1.ELsmp)
root (hd0,1)
kernel /boot/vmlinuz-2.6.9-42.0.0.0.1.ELsmp ro root=LABEL=/ rhgb quiet
initrd /boot/initrd-2.6.9-42.0.0.0.1.ELsmp.img
title Enterprise-up (2.6.9-42.0.0.0.1.EL) root (hd0,1) kernel /boot/vmlinuz-2.6.9-42.0.0.0.1.EL ro root=LABEL=/ rhgb quiet initrd /boot/initrd- 2.6.9-42.0.0.0.1.EL.img
title Other
rootnoverify (hd0,0)
chainloader +1

=========================================================

The entries in orange color are the ones which have been populated by the Firewire modules/kernel. To load your OS using this kernel, you simply have to update the grub.conf to use that kernel to load by modifying the default=0 to default=2. That should all, when you boot next time using the Firewire modules/kernel, you should be able to load the sbp2, ohci1394 and ieee1394 modules without any problems.

Option 2:

1. When you install oracle-firewire-modules-2.6.9-42.0.3.EL-1286-1.i686.rpm rpm it will install the modules under /lib/modules/2.6.9-42.0.3.EL/kernel/drivers/ieee1394, but your existing kernel would be as indicated by `uname -r`. In my case it was 2.6.9-42.0.0.0.1.EL.
2. Make a directory ieee1394 under /lib/modules/2.6.9-42.0.0.0.1.EL/kernel/drivers
3. copy the modules which have been installed under /lib/modules/2.6.9-42.0.3.EL/kernel/drivers/ieee1394 to /lib/modules/2.6.9- 42.0.0.0.1.EL/kernel/drivers/ieee1394.
4. Then run the following commands:
   

#depmod -a ieee1394.ko;modprobe ieee1394
#depmod -a ohci1394.ko;modprobe ohci1394
#depmod -a pcilynx.ko;modprobe pcilynx
#depmod -a raw1394.ko;modprobe raw1394
==============================================

Update the etc/modprobe.conf. For example, the entry in the /etc/modprobe.conf file would be: options sbp2 exclusive_login=0

Now you should be able to load the Firewire modules without changing your kernel.

==================================================================

Checking the integrity of the downloaded firewire module:

$md5sum oracle-firewire-modules-2.6.9-42.0.3.EL-1286-1.i686.rpm
7fe3c83291ffab284f6f941b717dacb1 oracle-firewire-modules-2.6.9-42.0.3.EL-1286-1.i686.rpm

Verify the above code (7fe3c83291ffab284f6f941b717dacb1) with that
indicated in the source website.

Export/Import Pump - Copying Schemas

Let’s say that we have a Schema in Database A, which needs to be copied over to Database B. With the traditional export/import, we have to precreate the user account in the Database B, before we can import the schema. But with oracle10g’s Pump utility, it is just one simple step:

Here is an example. In this example, the schema CHANDRA exists in database A and we attempt to copy over the same schema as CHANDRA3 in a remote database (rhes01.myoracle.com).

$impdp system/******* schemas=chandra network_link=rhes01.myoracle.com remap_schema=chandra:chandra3

Import: Release 10.2.0.3.0 - Production on Saturday, 05 May, 2007 18:54:21

Copyright (c) 2003, 2005, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - Production

With the Partitioning, OLAP and Data Mining options

Starting “SYSTEM”.”SYS_IMPORT_SCHEMA_01″: system/******** schemas=chandra network_link=rhes01.myoracle.com remap_schema=chandra:chandra3

Estimate in progress using BLOCKS method…

Processing object type SCHEMA_EXPORT/TABLE/TABLE_DATA

Total estimation using BLOCKS method: 0 KB

Processing object type SCHEMA_EXPORT/USER

Processing object type SCHEMA_EXPORT/SYSTEM_GRANT

Processing object type SCHEMA_EXPORT/ROLE_GRANT

Processing object type SCHEMA_EXPORT/DEFAULT_ROLE

Processing object type SCHEMA_EXPORT/PRE_SCHEMA/PROCACT_SCHEMA

Processing object type SCHEMA_EXPORT/TABLE/TABLE

Job “SYSTEM”.”SYS_IMPORT_SCHEMA_01″ successfully completed at 18:54:51

******************************************************************************************
This creates the user CHANDRA3 as well in the remote/target database.

Don’t forget to provide different filenames when PARALLEL is used with Export Dump.

If you use PARALLEL option with Export Dump and fail to provide file names equal to the number of PARALLEL, then it would perform the export in serial - meaning writes to a single file.

For example, if you have something like this: $ expdp dumpfile=expdat.dmp directory=dump_dir parallel=4
It would create just one file with the name expdat.dmp. Instead you can specify some thing like:

$ expdp dumpfile=expdat.%U.dmp directory=dump_dir parallel=4

This would create 4 files in parallel with names (expdat.01.dmp, expdat.02.dmp, expdat.03.dmp..).

*********************************************************************************