Yet Another Oracle DBA Blog

Converting time values in trace files into wall-clock time

2010-04-12T10:18:00.000-07:00

There may be cases where you would like to know the exact wall-clock time when a particular wait-event occurred or the SQL was executed by looking at the trace file.
Oracle captures the wall-clock time in the trace files and represents it in the following format:

AIT #2: nam='SQL*Net message from client' ela= 16445143 driver id=1650815232 #bytes=1 p3=0 obj#=-1 tim=1271092519750890

The value, we are interested in is the last one (tim=1271092519750890).

The tim value can be converted into wall-clock time by using the following:

Take the first 10 digits, and then:

$date -d @1271092519 +"%m-%d-%Y %T"

Example: tim=1271092519750890, first 10 characters - 1271092519

$date -d @1271092519 +"%m-%d-%Y %T"

04-12-2010 12:15:19

This could be useful in situations where you would like to find out when a user session completed execution, went back to the client to do some other processing and when it came back to database to execute subsequent SQLs.

BTW, there seems to be a difference in how this value is represented in 11g compared previous versions. I could not get the accurate wall-clock time for a tim value from 10g database.

ORA-27504 while starting 11gR1 database with 11gR2 Grid Infrastructure

2010-02-28T11:37:00.001-08:00

I have Oracle11g R2 (11.2.0.1) Grid infrastructure on a two node RAC. Installed 11.1.0.7 binaries and when attempting to create a database, encountered the following errors:

ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:skgxnqtsz failed with

status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: SKGXN not av
clsssinit ret = 21
interconnect information is not available from OCR

On researching, found the following solution to fix those errors and was able to create the 11.1.0.7 database with 11gR2 grid infrastructure:

As root:

[root@eletrac01 bin]# ./olsnodes -t -n

eletrac01 1 Unpinned
eletrac02 2 Unpinned

[root@eletrac01 bin]# ./crsctl pin css -n eletrac01 eletrac02
CRS-4664: Node eletrac01 successfully pinned.
CRS-4664: Node eletrac02 successfully pinned.
[root@eletrac01 bin]#

[root@eletrac01 bin]# ./olsnodes -t -n
eletrac01 1 Pinned
eletrac02 2 Pinned
[root@eletrac01 bin]#

Failed Switchover - Oracle11g

2010-01-16T19:34:00.000-08:00

I recently had a situation where we had a failed switchover process. Was able to switch the Primary to Standby successfully, but encountered some issues while switching over standby to primary - therefore ended up having two standbys! The easiest alternative to get out of the situation would be to ACTIVATE one of the standbys and then rebuild the primary - but it's not always desirable (size, efforts, unprotected primary etc..).

We even attempted the oracle provided command to cancel the switchover, but encountered the following (On original primary):

SQL> ALTER DATABASE PREPARE TO SWITCHOVER CANCEL;
ALTER DATABASE PREPARE TO SWITCHOVER CANCEL
*
ERROR at line 1:
ORA-00600: internal error code, arguments: [adbdrv_switch], [384], [], [], [],
[], [], [], [], [], [], []

Here is I ended up doing:

All of these have been performed on original primary:
1. Startup Mount;
SQL> startup force nomount;
ORACLE instance started.

Total System Global Area 238530560 bytes
Fixed Size 1335724 bytes
Variable Size 150998612 bytes
Database Buffers 83886080 bytes
Redo Buffers

2. Recreate the controlfile using the trace (got from standby)
SQL> @scripts/ccf.sql

Control file created.

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01589: must use RESETLOGS or NORESETLOGS option for database open

SQL> alter database open resetlogs;

Database altered.

SQL>
3. That's it…that did it!

SQL> Select open_mode, database_role, switchover_status from V$database;

OPEN_MODE DATABASE_ROLE SWITCHOVER_STATUS
-------------------- ---------------- --------------------
READ WRITE PRIMARY TO STANDBY

SQL>

4. Here are the entries from standby - didn't have to do anything - it continued through RESETLOGS!

RFS[18]: New Archival REDO Branch(resetlogs_id): 708469623 Prior: 708454293
RFS[18]: Archival Activation ID: 0x6fece72a Current: 0x0
RFS[18]: Effect of primary database OPEN RESETLOGS
RFS[18]: Incarnation entry added for Branch(resetlogs_id): 708469623 (ETSI01)
Sat Jan 16 21:08:09 2010
Setting recovery target incarnation to 4
Sat Jan 16 21:08:09 2010
Archived Log entry 33 added for thread 1 sequence 1 ID 0x6fece72a dest 1:
Sat Jan 16 21:08:09 2010
RFS[19]: Assigned to RFS process 5022

Segment Comparison (structure) in Oracle11g

2009-12-15T07:19:00.001-08:00

There are bunch third party products in the market which do a pretty good job at finding out the structual differences between objects or schemas. Toad comes to my mind which does an outstanding job...but it costs $$$! With Oracle11g, it's quite easy to figure out differences between two objects. Please note that use of this feature needs Change Management pack license. This is available with 11.1.0.7, but not documented in 11gR1 docs, but does have a mentioned in 11gR2 docs. The API or the package is DBMS_METADATA. Here is how it works:

Let's say we have two schemas in different databases and would like to compare the table structures:

SQL> Select dbms_metadata_diff.compare_alter('TABLE','APP_DATA','APP_DATA_TEMP', 'CHANDRA','CHANDRA',null,'BDBE01') DIFF from dual;

DIFF
--------------------------------------------------------------------------------
ALTER TABLE "CHANDRA"."APP_DATA" ADD ("CREATE_DATE" DATE)
ALTER TABLE "CHANDRA"."APP_DATA" ADD ("UPDATE_DATE" DATE)
ALTER TABLE "CHANDRA"."APP_DATA" RENAME TO "APP_DATA_TEMP"

Explanation for the argument used in the function call: Object_type, table_name, table_name, schema_owner, schema_owner, db_link_for_source (if null, then local), db_link_for_target.

If the order of table names, source and target are changed:

SQL> Select dbms_metadata_diff.compare_alter('TABLE','APP_DATA_TEMP','APP_DATA', 'CHANDRA','CHANDRA','BDBE01',null) DIFF from dual;

DIFF
--------------------------------------------------------------------------------
ALTER TABLE "CHANDRA"."APP_DATA_TEMP" DROP ("CREATE_DATE")
ALTER TABLE "CHANDRA"."APP_DATA_TEMP" DROP ("UPDATE_DATE")
ALTER TABLE "CHANDRA"."APP_DATA_TEMP" RENAME TO "APP_DATA"

Interesting, it even has an ALTER to rename the table to make both same in all respects.

I don't see an option to include or exclude various segment attributes such as storage, tablespace etc...may be because the intent of this API seems to be different. Thought this oracle provided utility would come in handy to quickly find out the structural differences.

Oracle11gR2: Deferred Segment Creation and export behavior!

2009-09-14T12:23:00.000-07:00

I guess by now most of us know what Oracle11gR2's feature - Deferred Segment Creation really means. This post is about the impact this feature going to have while exporting a schema or tables using the traditional export. By default, any new tables created would have SEGMENT CREATION DEFERRED option set (unless you are using CTAS). This default behavior could be turned off by setting DEFERRED_SEGMENT_CREATION to false. The impact with this is - if you happen to export any of the tables, which do not have any rows (meaning empty tables without segments) - they will NOT be exported. This is only true with the legacy/old export and not with the export pump utility(10g and above). Something we have to remember if we can't get rid of the old habit of using old export! Of course, export is deprecated.

Oracle11gR2 Table / Tablespace Compression

2009-09-14T10:31:00.000-07:00

As far as I remember, oracle introduced the basic compression feature with Oracle9i, but what's new with 11g is the ability to compress data generated as a result of OLTP transactions (almost all DMLs) as well - previously it was only available for bulk operations. Another neat feature of 11g is the ability to evaluate whether not a table is a right candidate for compression , if yes what would be compression benefits in regards space utilization. This is done using DBMS_COMPRESSION (using Compression Advisor). I wanted to use this procedure to see how accurately it could give us the space savings and also see what would be impact on performance, in general.

I am using a 18million record table for this example.

1. Space usage for the uncompressed table:

SQL> select segment_name, bytes, a.blocks, compression, compress_for from
2 dba_segments a, dba_tables b
3 where a.segment_name = b.table_name and a.segment_name='T1';

SEGMENT_NAME BYTES BLOCKS COMPRESS COMPRESS_FOR
------------------------------ ---------- ---------- -------- ------------
T1 1476395008 180224 DISABLED

2. Execute the DBMS_COMPRESSION procedure to get information on space savings:

(Note that this procedure example works only in 11gR2, the function names and the arguments are different in 11gR1.)

SQL> declare
2 lv_cmp_ratio number;
3 lv_comptype_str varchar2(300);
4 lv_BLKCNT_CMP number;
5 lv_BLKCNT_UNCMP number;
6 lv_ROW_CMP number;
7 lv_ROW_UNCMP number;
8 begin
9 dbms_compression.GET_COMPRESSION_RATIO(
10 SCRATCHTBSNAME=>'USERS',
11 OWNNAME=>'CHANDRA',
12 TABNAME=>'T1',
13 PARTNAME =>null,
14 COMPTYPE =>2, ---2 means OLTP
15 BLKCNT_CMP=>lv_BLKCNT_CMP,
16 BLKCNT_UNCMP=>lv_BLKCNT_UNCMP,
17 ROW_CMP =>lv_ROW_CMP,
18 ROW_UNCMP =>lv_ROW_UNCMP,
19 CMP_RATIO=>lv_cmp_ratio,
20 COMPTYPE_STR=>lv_COMPTYPE_STR);
21 dbms_output.put_line('====================================================');
22 dbms_output.put_line('1. Compression Ratio :'||lv_cmp_ratio);
23 dbms_output.put_line('2. Block Count :'||lv_blkcnt_cmp);
24 dbms_output.put_line('3. Compression Type :'||lv_comptype_str);
25 dbms_output.put_line('4. Blk Count Compressed :'||lv_BLKCNT_CMP);
26 dbms_output.put_line('5. Blk Count Un-compressed:'||lv_BLKCNT_UNCMP);
27 dbms_output.put_line('6. Row Count Compressed :'||lv_row_cmp);
28 dbms_output.put_line('4. Row Count Un-Compressed:'||lv_row_uncmp);
29 dbms_output.put_line('====================================================');
30 end;
31 /
====================================================
1. Compression Ratio :2.38995215311004784688995215311004784689
2. Block Count :836
3. Compression Type :"Compress For OLTP"
4. Blk Count Compressed :836
5. Blk Count Un-compressed:1998
6. Row Count Compressed :256
4. Row Count Un-Compressed:107
====================================================

PL/SQL procedure successfully completed.

According to the above information, if we look at the compressed (836 blocks) and uncompressed blocks(1998) counts (4 and 5 above), we should at least 50% gain!

3. Create a new tablespace with COMPRESS option. Unless I am missing something obvious, I couldn't get the correct syntax in both 11gR1 and R2 documentation. BTW, when I executed the same SQL on 11gR1 - I got - ORA-14464: Compression Type not specified

SQL> create tablespace USER_DATA_COMPRESSED default compress for OLTP datafile '/u01/oracle/oradata/DET01/user_data_compressed.dbf' size 10M autoextend on;

Tablespace created.

4. Create copy of T1 table (our example table) in COMPRESSED tablespace:

SQL> create table chandra.t1_compressed tablespace USER_DATA_COMPRESSED
2 as Select * from chandra.t1;

Table created.

5. Let's see how much space does the compressed table take when compared to the uncompressed/original table:

SQL> select segment_name, bytes, a.blocks, compression, compress_for from dba_segments a,
2 dba_tables b
3 where a.segment_name = b.table_name and a.segment_name in ('T1','T1_COMPRESSED');

SEGMENT_NAME BYTES BLOCKS COMPRESS COMPRESS_FOR
------------------------------ ---------- ---------- -------- ------------
T1_COMPRESSED 612368384 74752 ENABLED OLTP
T1 1476395008 180224 DISABLED

As theDBMS_COMPRESSION procedure indicated, we did see less than 50% space utilization with the OLTP compression, so far so good. How about performance? I performed such generic tests, involving SELECTS and other DMLs and here are the results:

Any activity involving Full Table Scan, the performance was 4 times faster on compressed tables compared with uncompressed table.
Inserts and Updates took close to twice longer on compressed table compared with uncompressed table and deletes about 10% more.

When we only talk about other benefits of compression - they would be lesser memory utilization, redo generation etc.. comes to my mind. Of course, at the cost of higher CPU consumption.

Oracle11gR2 ASM New Feature - Intelligent Data Placement

2009-09-11T22:24:00.001-07:00

Intelligent Data Placement, a new feature of Oracle11gR2 ASM is aimed at optimizing data access. This is achieved by giving us the capability to place the data on those sectors of the hard drive where the I/O performance is high/efficient. These sectors are the outer sectors. Before going into the details of the ASM's feature, let's attempt to understand briefly how and why the outer sector's i/o performance is better compared to the inner sectors.

The above mentioned efficiencies come from a technique used in hard drives called "Zoned Bit Recording". This method or technique used to increase capacity and data access speeds on hard disks by improving the utilization of the larger, outer tracks of the disk. Outer tracks contain more sectors per track than the one before. This makes more efficient use of the larger tracks on the outside of the disk. The data transfer rate is higher when reading outside cylinders than reading the inside ones, since outer cylinders contain more data.

Coming back to the original topic - ASM now classifies reads/writes into two - COLD and HOT. As you guessed, HOT read/writes are those coming from the blocks which are classified as HOT - meaning placed in the outer sectors. COLD is the default - meaning data placed in inner sectors. Data blocks of most often accessed segments are candidates for placement in HOT region. This is done at the datafile level in an ASM disk.

You could use SQL*Plus or DBConsole to make these changes.

It's just matter of picking the appropriate radio-button depending on the needs/access patterns to switch between HOT and COLD regions.

We know what the feature means - but I wanted to put the feature to test to really see the benefit in regards to I/O rates/speeds using some general test cases. Here is how the I/O rates or performance was for a set of tests (comprising a mix of read/writes(deletes/inserts/updates) with the files being in the default zone (COLD). Then ran the same tests while the datafile was placed in the HOT region. Here are the test results. Note that when you move a file from one region to another, you have to initiate a re-balance to make the change effective.

With Intelligent Data Placement:

ALTER DISKGROUP ASMDISK01 MODIFY FILE '+ASMDISK01/BDBE01/DATAFILE/APP_DATA.256.697380955' ATTRIBUTES (HOT)

I executed the above mentioned test cases multiple times to make sure I see or notice consistent (and thus reliable and dependable) I/O performance. During my tests I saw that there was consistently an increase of about 10% to 15% in "Average Throughput" and about (-)3% to +5% in the "Average Response Times". Of course, we may have to perform more realistic tests with application specific I/O patterns.

I think we can say that with this new feature, Oracle (with ASM) will be able to successfully exploit the Zoned Bit Recording technique of hard drives for optimizing I/O performance. Again, not sure if this really going to make a huge impact given the large cache and disk sizes.

Oracle11g R2 - impressive and intelligent CRS installer (OUI)!

2009-09-09T08:28:00.000-07:00

I was really impressed with the enhancements made to the installer, apart from the change in the look and feel. Prior to Oracle11gR2, installer would allow us to place the redundant copies of OCR/Voting devices/files unto the same file system (of course it doesn't make any sense though). But with 11gR2, it prevents us from placing the mirrors unto the same file system. There is no IGNORE option as well...either you have to place it on different file systems/locations or use a single copy. So the installer has "best practices" built into it.

Here is how I first attempt to use - placing all the three unto the same file system:

And here is the error message:

I ended up using a single copy for now:

BTW, the installer allows us to configure SSH, test SSH, option to use ASM or shared file system for OCR/Vote etc... The best of all, it now provides a script to fix any of the fix-able requirements such as kernel parameters etc..

ORA-03001 - while trying to rename tables owned by others - workaround

2009-09-08T18:45:00.000-07:00

There could be situations where you may like to rename the tables owned by others, but do not have access to the password (of the schema owner) or don't want to change the password (using "by values" etc...). You could perform almost all the activities by using "ALTER SESSION SET CURRENT_SCHEMA=;", but rename table wouldn't work and following error message would be encountered:

chandra@DOE01:11g> show user
USER is "CHANDRA"
chandra@DOE01:11g> alter session set current_schema=CP_ADMIN;

Session altered.

chandra@DOE01:11g> rename APP_DATA to APP_DATA_BAK;
rename APP_DATA to APP_DATA_BAK
*
ERROR at line 1:
ORA-03001: unimplemented feature

chandra@DOE01:11g>

To get around this, we could create a simple procedure similar to the following:

create or replace procedure rename_others_tab (orig_tab_name in varchar2, new_tab_name in varchar2)
as
lv_sql_str varchar2(100);
lv_exists number(1);
begin
select count(*) into lv_exists from user_tables where table_name=orig_tab_name;
if lv_exists = 1 then
lv_sql_str := 'rename '||orig_tab_name||' to '||new_tab_name;
dbms_output.put_line (lv_sql_str);
execute immediate lv_sql_str;
else
dbms_output.put_line('ERROR:'||orig_tab_name||' does not exist!');
end if;
end;
/

Note that you would be creating the above procedure while your current_schema is set to the owner of the table.

Then:

chandra@DOE01:11g> exec rename_others_tab('APP_DATA','APP_DATA_BAK');

PL/SQL procedure successfully completed.

chandra@DOE01:11g>

The table name would be renamed....Of course, use the above procedure (work-around) with caution and when ABSOLUTELY needed.

Thought would be of use to others....

ORA-01620 while creating a RAC standby for non-RAC primary

2009-08-07T18:21:00.000-07:00

Today I was trying to test some thing and had to create a RAC standby for a Non-RAC (single-instance) Primary database. I know, it doesn't make sense to have a RAC standby for a single-instance primary, but that's not the point. First instance was mounted without any problems, but when attempted to mount the second standby instance got the following error message:

SQL> alter database mount standby database;
alter database mount standby database
*
ERROR at line 1:
ORA-01620: no public threads are available for mounting

I couldn't find quick notes on the web to get around this issue and therefore had to find my own for this unique situation (having a RAC standby for SI-Primary). Here is what I did on Primary:

SQL> ALTER DATABASE ADD LOGFILE THREAD 2 group 13 size 5M;

Database altered.

SQL> ALTER DATABASE ADD LOGFILE THREAD 2 group 14 size 5M;

Database altered.

SQL> alter database enable public thread 2;

Database altered.

Then created new copy of standby controlfile and shipped it over to standby and attempted to mount both the instances, it worked fine! Of course, had the primary been a RAC, we wouldn't have seen this in first place :-) BTW, this is on Oracle11g 11.1.0.7

Tablespace Point-in-time Recovery (TSPITR) using RMAN

2009-08-04T09:52:00.000-07:00

Scenario: A logical corruption occurred due to an user error and have to quickly get back to the state just before the time of the error. Possible options: Database point-in-time recovery, but the downside is the entire database has to be unavailable and other transactions after the PITR would be lost. The other option would be Tablespace Point-in-time-Recovery (TSPITR) - the obvious choice here. I had known that RMAN could be used to perform TSPITR, but I realized just today how powerful, simple, automated, error-free and fast it could be to perform TSPITR using RMAN - it's really amazing. The intent of this post is just to high-light and appreciate the usefulness of RMAN to perform TSPITR.

Assuming we have the latest backup, just executing the following statement would do everything needed to get the entire tablespace to a point-in-time in the past. Believe me, you don't have to do anything other than just this statement!

recover tablespace APP_DATA until time
"to_date('2009-08-04 12:15:00’,’YYYY-MM-DD HH24:MI:SS')"
auxiliary destination ’/opt/oracle/temp’;

Once the above statement is executed, RMAN does the following for us:

Creates auxiliary instance (including the pfile etc..)
Mounts the auxiliary instance
Makes the candidate tablespace into OFFLINE
Restores the basic tablespace UNDO, SYTEM,SYSAUX and then the required tablespace
Applies archives (completes recovery)
Opens the database
Performs an export of the objects residing in the tablespace
Shutdown aux instance
Import the objects into the target database
Remove aux instance and cleanup

That's it....you now have all the objects back in the tablespace!.

You can view the output of the RMAN session by downloading this file.. which is quite self-explanatory and informative too!

ORA-27090: Message 27090 not found on 10.2.0.4

2009-07-30T17:38:00.000-07:00

Lately we have been noticing ORA-27090 on our 10.2.0.4 databases. Here is the complete error message, as appeared in the alert log file. This is observed on Suse Linux, Red hat and OEL 5.3:

===============================================================
Errors in file /tmp/test_dbw0_4400.trc:
ORA-27090: Message 27090 not found; product=RDBMS; facility=ORA
Additional information: 3
Additional information: 128
Additional information: 1100 #--This is the value for the parameter aio-max-nr
===============================================================
Looks like this error condition is only encountered or reported in databases using 10.2.0.4 oracle version. On research found that this has something to do with the kernel setting fs.aio-max-nr. aio-max-nr sets the systemwide maximum number of AIO requests. Apparently, when the number of requests reach this number the error is reported. I guess the question would be, how can we determine where we are at currently in relation to the max limit, right? This is answered by aio-nr, which maintains a cumulative/running total of number of aio events used.

I was able to consistently reproduce the error condition by playing around with the settings for aio-max-nr parameter. As such, no apparent problems are noticed on the database side, but did did notice that the standby databases would fail to continue with the managed recovery process.

Here is how it looked like before starting up an 10.2.0.4 instance:

#/sbin/sysctl -a |grep aio
fs.aio-max-nr = 1100 #---I reset it to this value from default.
fs.aio-nr = 0

When the instance was started, the value looked like this:
#/sbin/sysctl -a |grep aio
fs.aio-max-nr = 1100 #---I reset it to this value from default.
fs.aio-nr = 1060

As soon as it hit the ceiling, the above mentioned error was reported in the alert log. When I raised the value, the error message disappeared.

As soon as the instances are shutdown the value for fs.aio-nr gets reset.

Hope this information would come in handy.

How to check if CRS is set to auto-start on reboots

2009-07-15T08:36:00.001-07:00

Sometime back, some one asked me a question - "how to confirm that CRS is set to restart on reboot?"

CRS can be configured to either to restart or not to restart on reboots. Disable restarts would be typically useful for troubleshooting CRS reboots or validating any system level changes.

To enable restarts (default):
$CRS_HOME/bin/crsctl enable crs

To disable restarts of crs on reboots:
$CRS_HOME/bin/crsctl disable crs

Note that the above needs access to root.

Whenever the above commands are run, it modifies the following file (on linux), which indicates whether or not the CRS is set to auto-restart on reboots:

/etc/oracle/scls_scr/node_name/root/crsstart

It contains either "enable" or "disable" keywords.

ORA-29702 - Starting RAC instance in non-rac mode

2009-06-20T21:28:00.000-07:00

Assume the following situation: Due to some OS or network related problem you are not able to start-up the CRS (this does happen!) before it impacts the database availability and subsequently your business. Your primary goal now is to make the database available as soon as possible irrespective of its mode (RAC or Non-RAC). The reason being you would like to reduce the impact to your business. If you attempt to startup the instance without CRS, you get the following error:

SQL> startup;
ORA-29702: error occurred in Cluster Group Service operation
SQL>

This error message indicates that it can't start the instance since CRS is not available. Here is the quick work-around to startup the instance. The option is to remove RAC option from your binaries and start the instance. Once you have resolved the CRS related issues, you can turn-on the RAC option back (of course, your instance needs to be down while to turn-on or turn-off the options). Here is how we turn-off:

cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk rac_off ioracle

Set cluster_database=false

You should now be able to startup the instance without CRS stack being up and running.

SQL> Select * from v$option where parameter like 'Real%';

PARAMETER VALUE
---------------------------------------- ----------
Real Application Clusters FALSE
Real Application Testing TRUE

Turn-on the RAC option with:

make -f ins_rdbms.mk rac_on ioracle

Alternatively, you can install a non-rac oracle binaries to quickly start the instance.

By the way, you turn off other options in similar way such as Partitioning (part_on/part_off), DB Vault (dv_on/dv_off) etc..

Oracle11g - Killing session in RAC (in remote instance)

2009-06-16T07:22:00.000-07:00

Prior to Oracle11g, whenever you want to kill a session connected to a non-local (remote) instance, you had to make an explicit connection to that particular instance and then attempt to kill.

Example:

SQL> Select instance_number from v$instance;

INSTANCE_NUMBER
---------------
1 ---> I am connected to instance 1 (Local Instance)

1 row selected.

SQL> Select inst_id, sid, serial# from gv$session where username='CHANDRA';

INST_ID SID SERIAL#
---------- ---------- ----------
2 125 7526 ---> CHANDRA is connected to Inst# 2 (Non-local/remote Instance)

1 row selected.

With 11g, you could kill the session which is connected to instance 2, while you are connected to instance 1:

SQL> Alter system kill session '125,7526,@2' immediate; #--@2 indicates remote instance.

System altered.

SQL> Select inst_id, sid, serial# from gv$session where username='CHANDRA';

no rows selected

This really makes life easy - especially you are using SQL*Plus and not any other front-end tool.

Errors while installing Oracle9i in Redhat 5.3 (libstdc++-libc6.1-1.so.2)

2009-05-25T19:55:00.000-07:00

Today, I was trying to install Oracle9i on Redhat 5.3 (2.6.18 kernel), got the following error when runInstaller was involved:

$ Initializing Java Virtual Machine from /tmp/OraInstall2009-05-09_06-44-18AM/jre/bin/java. Please wait...
/tmp/OraInstall2009-05-09_06-44-18AM/jre/bin/i386/native_threads/java: error while loading shared libraries: libstdc++-libc6.1-1.so.2: cannot open shared object file: No such file or directory

When check for libstdc++- library, I found the following to be installed by default on RH 5.3:

[root@alps tmp]# rpm -qa |grep libstdc++-
libstdc++-4.1.2-44.el5
libstdc++-devel-4.1.2-44.el5
compat-libstdc++-296-2.96-138
compat-libstdc++-33-3.2.3-61
[root@alps tmp]#

Looks like it is looking for libstdc++-lib6.1-1.

When I did the following the initial error message disappeared:

# cd /usr/lib
# ln -s libstdc++-3-libc6.2-2-2.10.0.so libstdc++-libc6.1-1.so.2

Then I got the following error:

$ ./runInstaller
$ Initializing Java Virtual Machine from /tmp/OraInstall2009-05-09_06-50-11AM/jre/bin/java. Please wait...
Error occurred during initialization of VM
Unable to load native library: /tmp/OraInstall2009-05-09_06-50-11AM/jre/lib/i386/libjava.so: symbol __libc_wait, version GLIBC_2.0 not defined in file libc.so.6 with link time reference

When applied the oracle patch: 3006854 - it fixed it:

[root@alps 3006854]# sh rhel3_pre_install.sh
Applying patch...
Ensuring permissions are correctly set...
Done.
Patch successfully applied
[root@alps 3006854]#

Adaptive/Intelligent Cursor Sharing in 11g - turning-off

2009-05-19T18:30:00.000-07:00

Here is how the cursor sharing works at a high level with bind variables, especially in conjunction with Histograms (prior to oracle 11g):

Optimizer peeks value for the bind on initial parse
Initial value of the bind determines the plan
Same plan is used/shared regardless of future bind values
One plan not always appropriate and optimal for all bind values and results in following problems or side-effects:

- Performance problems
- Instability issues
- Unhappy end users

Adaptive Cursor Sharing address this problem. it shares execution plans ONLY when bind values are "equivalent" - means if it doesn't result in performance degradation. This is the default behavior in Oracle11g.

More information on adaptive cursor sharing can be found at: http://optimizermagic.blogspot.com/2009/04/update-on-adaptive-cursor-sharing.html - The intention of this post is not to talk about this feature - but how to turn-off this default behavior. I can't really think of a reason why anyone would like to turn this feature off, but it is possible, if at all we need it for some reason. This can be achieved by using NO_BIND_AWARE hint.

I believe this hint is meant to be an alternative for using the hidden parameter - _optim_peek_user_binds=false which is used to turn-off bind peeking.

Here is the example:

The default behavior:

SQL> variable lv_id number;

SQL> exec :lv_id :=100;

PL/SQL procedure successfully completed.

SQL> Select count(*) from sh.sales where prod_id = :lv_id;

COUNT(*)
----------
0

----if the bind values are peeked, we would see records in V$SQL_CS_STATISTICS:

SQL> Select address, child_number, peeked, executions from v$sql_cs_statistics;

ADDRESS CHILD_NUMBER P EXECUTIONS
-------- ------------ - ----------
34C54884 0 Y 1

Now, let's turn off bind peeking.
======================

SQL> alter system flush shared_pool;

System altered.

SQL> exec :lv_id :=1000;

PL/SQL procedure successfully completed.

SQL> Select /*+ NO_BIND_AWARE */ count(*) from sh.sales where prod_id = :lv_id;

COUNT(*)
----------
0
----With the above hint, the sql is not bind aware, therefore we don't see any records in v$SQL_CS_STATISTICS view.

SQL> Select address, child_number, peeked, executions from v$sql_cs_statistics;

no rows selected

SQL>

I guess, this is a better and elegant way to turn-off bind-peeking at the SQL level instead of turning-off bind-peeking at the instance level using the hidden parameter - _optim_peek_user_binds. Again, this parameter is typically used to prevent the side/ill-effects of bind-peeking and the same is now (with 11g) obviated with the introduction of Adaptive Cursor Sharing feature in 11g.

BTW, this hint is available in 11.1.0.7.

Oracle11g CRS new command

2009-05-18T17:56:00.000-07:00

Just found out that the functionality of the "crsctl" command has been expanded in Oracle11g. Prior to Oracle11g, we did not have the ability to check the status of the cluster on other (non-local) nodes in the cluster using "crsctl" - but oracle11g makes it now possible:

[root@racnode01 bin]# ./crsctl check cluster
racnode01 ONLINE
racnode02 ONLINE
[root@racnode01 bin]#

Of course, we could very well use "crsstat" command to determine the status of CRS on non-local nodes - but not straight-forward and simple.

Oracle Support Bug Status Codes...

2009-04-22T12:15:00.000-07:00

When you have an SR open with Oracle needing a bug fix, most of the times the status would be indicated in codes. Here is the list of codes and what it means...thought would come in handy:

10 - Description Phase: Development is requesting more information.
16 - Support bug screening: Bug is being reviewed by our Bug Diagnostics group.
11 - Code Bug (Response/Resolution): Bug is being worked by Development.
30 - Additional Information Requested: Bug is being worked by Support and/or more information was requested by Development.
37 - To Filer for Review/Merge Required: Bug has been fixed but the patch will be merged into the next patchset.
80 - Development to Q/A: Bug is being regression tested for future release.
81 - Q/A to Dev/Patch or Workaround Avble: Patch released via Metalink.
90 - Closed, Verified by Filer: Bug has been fixed and is closed.
91 - Closed, Could Not Reproduce: Bug is closed as not reproducible.
92 - Closed, Not a Bug: Bug is closed as not a bug (not reproducible or setup issue).
93 - Closed, Not Verified by Filer: Bug has been fixed and is closed.
95 - Closed, Vendor OS Problem: Bug is closed as an OS problem.
96 - Closed, Duplicate Bug: Bug is closed as a duplicate bug.

ORA-38856 - while opening a cloned RAC database

2009-04-17T04:54:00.000-07:00

I was attempting to open a cloned RAC database and got the following error:

SYS@NETFRD1> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-38856: cannot mark instance UNNAMED_INSTANCE_2 (redo thread 2) as enabled

The fix was to use the undocumented parameter - _no_recovery_through_resetlogs=TRUE to open the database. I guess this is one of those scenarios where Oracle suggests using this undocumented/unsupported parameter. Here is what I did to get pass the problem:

1. Shutdown;
2. Startup mount (with the parameter _no_recovery_through_resetlogs=TRUE set in init.ora)
3. Open the database with RESETLOGS;

SYS@NETFRD1> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-38856: cannot mark instance UNNAMED_INSTANCE_2 (redo thread 2) as enabled

SYS@NETFRD1> shutdown immediate;
ORA-01109: database not open

Database dismounted.
ORACLE instance shut down.
SYS@NETFRD1> !vi /tmp/init$ORACLE_SID.ora

SYS@NETFRD1> startup mount pfile=/tmp/init$ORACLE_SID.ora
ORACLE instance started.

Total System Global Area 1577058304 bytes
Fixed Size 2084232 bytes
Variable Size 385876600 bytes
Database Buffers 1174405120 bytes
Redo Buffers 14692352 bytes
Database mounted.
SYS@NETFRD1> alter database open resetlogs;

Database altered.

ORA-39006, ORA-39065,ORA-25306,ORA-39079 while exporting using EXPDP

2009-02-16T08:03:00.000-08:00

Encountered the following error messages which attempting to do an export of a database using Data Pump utility with 10.2.0.4:

ORA-39006: internal error
ORA-39065: unexpected master process exception in DISPATCH
ORA-39079: unable to enqueue message DG,KUPC$S_3_20090213124234,MCP, ,1,Y
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 86
ORA-06512: at "SYS.KUPC$QUE_INT", line 924
ORA-25306: Cannot connect to buffered queue's owner instance

ORA-39097: Data Pump job encountered unexpected error -39079
ORA-39065: unexpected master process exception in KUPC$QUEUE_INT.PUT_STATUS
ORA-39079: unable to enqueue message DG,KUPC$S_3_20090213124234,MCP, ,1,Y
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 86
ORA-06512: at "SYS.KUPC$QUE_INT", line 924
ORA-25306: Cannot connect to buffered queue's

This seems to be a bug but couldn't find any published information in metalink. Setting the value for the parameters aq_tm_processes to 4 from a value of 0 worked. There are several other databases with the default setting of 0, but they don't seem to have problems with expdp. Hopefully, this will have a fix in 10.2.0.5.

Hope this helps others who encounter similar errors.

Bypassing/ignoring or skipping ORA-1555 errors

2009-02-08T20:18:00.000-08:00

I was researching a corruption issue using various Metalink link notes and stumbled upon a note/bug which provides a way to ignore or skip ORA-1555 errors for queries. Obviously, this will result in getting inconsistent data/result-set. As of now, I am not sure as to what type situations will have a practical use of this feature. This is available in Oracle 11g 11.1.0.7. This is done using a hint - SCN_ASCENDING.

Example:

In Session 1:

chandra@AEDB01:11g> Select Distinct phone from acs_emp;

PHONE
--------------------
972-500-8999

1 row selected.

In Session 2:

chandra@AEDB01:11g> Select /*+ scn_ascending */ * from acs_emp;

While this session is in progress, I have updated the phone column:

Back in Session 1:

chandra@AEDB01:11g> update acs_emp set phone='888-888-8888';

100000 rows updated.

chandra@AEDB01:11g> commit;

Commit complete.

Back to Session 2 where our select was in progress: You will notice that we got data which was NOT consistent with the data which existed at the point in time when the query was initiated. The SCN got changed in the middle - meaning inconsistent / questionable data!. Without the above mentioned HINT, it wouldn't have been possible to see such inconsistent result set. Hopefully, this hint is a well-intentioned one.

650831 972-500-8999
650831 972-500-8999
650831 972-500-8999
650831 972-500-8999
650831 972-500-8999
650831 972-500-8999
650831 972-500-8999
650831 972-500-8999
650831 972-500-8999
650831 972-500-8999
650831 972-500-8999
652088 888-888-8888
652088 888-888-8888
652088 888-888-8888
652088 888-888-8888
652088 888-888-8888
652088 888-888-8888
652088 888-888-8888
652088 888-888-8888
652088 888-888-8888
652088 888-888-8888

java.lang.UnsatisfiedLinkError: libawt.so: libXp.so.6 during 10g Install on OEL 5.0

2008-09-03T11:01:00.000-07:00

Today, while I attempting to install Oracle10gR2 (10.2.0.1) on a new server with Oracle Enterprise Linux 5.0, got the following error when OUI was invoked.

java.lang.UnsatisfiedLinkError: libawt.so: libXp.so.6: cannot open shared object file: No such file or directory

Later, found that the package libXp-1.0.0-8.i386.rpm doesn't get installed by default with OEL 5.0. Once the package was installed, the OUI was good to go!

[root@rhvm01 RPMs]# rpm -ivh libXp-1.0.0-8.i386.rpm
warning: libXp-1.0.0-8.i386.rpm: Header V3 DSA signature: NOKEY, key ID 37017186
Preparing... ########################################### [100%]
1:libXp ########################################### [100%]
[root@rhvm01 RPMs]#

Oracle Home Cloning For RAC

2008-05-19T07:39:00.001-07:00

A short note on the steps for cloning Oracle Home for RAC:

1. On all Nodes: Copy existing ORACLE_HOME binaries to a new OH location
$cp -pr /opt/oracle/product/10.2.0.2 /opt/oracle/product/10.2.0.4

2. Add entries for the new Oracle_HOME into the inventory

CD to the new OH: cd /opt/oracle/product/10.2.0.4/clone/bin
perl clone.pl ORACLE_HOME=/opt/oracle/product/10.2.0.4 ORACLE_HOME_NAME=Ora10g_10204 "CLUSTER_NODES=alps,everest" "LOCAL_NODE=alps"
Execute /opt/oracle/product/10.2.0.4/root.sh script as root.
Apply the 10.2.0.4 patchset unto the newly created OH

APEX Configuration - 11g

2008-04-27T11:48:00.000-07:00

APEX is included with the default Oracle11g installation. You can check if the APEX component is included in the installation by verifying the existence of apex directory under $OH and also by checking DBA_REGISTRY:

SQL> Select comp_name, version, status from dba_registry where comp_name
2 like '%Express%';

COMP_NAME VERSION STATUS
------------------------------ ------------------------------ -----------
Oracle Expression Filter 11.1.0.6.0 VALID
Oracle Application Express 3.0.1.00.08 VALID

Configuring APEX

1. Login as SYSDBA and run the script $OH/apex/apxconf

SQL> @apxconf

PORT
----------
8080

Enter values below for the XDB HTTP listener port and the password for the Application Express ADMIN user.
Default values are in brackets [ ].
Press Enter to accept the default value.

Enter a password for the ADMIN user [] chandra
Enter a port for the XDB HTTP listener [ 8080]
...changing HTTP Port

PL/SQL procedure successfully completed.

PL/SQL procedure successfully completed.

Session altered.

...changing password for ADMIN

PL/SQL procedure successfully completed.

Commit complete.

SQL>

2. Set the port number:

SQL> exec dbms_xdb.sethttpport(8080);

PL/SQL procedure successfully completed.

3. Unlock the ANONYMOUS ACCOUNT:

4. Logon to APEX using http://hostname:8080/apex/apex_admin

Use ADMIN as the username and the password which we used at step one above.

5. Create a WORKSPACE

With this we will have a WORKSPACE to start creating our own applications.

ORA-27102: out of memory

2008-03-08T15:47:00.000-08:00

Though we may have sufficient free physical memory, we often see this error message indicative of low memory availability. But the actual cause of this error is typically caused by the incorrect value set for shmall kernel parameter.

1. Let' set the value for shmall to a very low value.
Current Value:

[root@chandra ~]# sysctl -a |grep shmall
kernel.shmall = 1000000
[root@chandra ~]#

Setting the value to a new/low value:

[root@chandra ~]# sysctl -w kernel.shmall=10000
kernel.shmall = 10000
[root@chandra ~]# sysctl -a |grep shmall
kernel.shmall = 10000
[root@chandra ~]#

2. An attempt to startup the database fails with ORA-27102

SQL> startup
ORA-27102: out of memory
Linux Error: 28: No space left on device
SQL>

It failed with "out of memory" error message, though, we have 500M free space

[root@chandra ~]# free
total used free shared buffers cached
Mem: 2066192 1488748 577444 0 18496 1134776
-/+ buffers/cache: 335476 1730716
Swap: 8225172 0 8225172
[root@chandra ~]#

3. Let's set the shmall back to value to a high value:

[root@chandra ~]# sysctl -w kernel.shmall=1000000
kernel.shmall = 1000000
[root@chandra ~]#

4. An attempt to startup the database just works, even though nothing from the memory utilization or availability perspective changed!

Oracle 10g CRS upgrade to 11g CRS

2008-02-09T10:19:00.000-08:00

To upgrade Oracle 10g CRS to 11g CRS, we have two options -

Perform a rolling upgrade (this requires that your current CRS version >=10.2.0.3 (or 10.2.0.2 with Bundle patch). This option allows us to upgrade the CRS without a complete unavailabilty of downtime
Upgrade the CRS on all the nodes at the same time with complete downtime.

I will go over the steps for upgrading the CRS to 11g using the first option mentioned above.

On Node 1:

1. Stop the CRS: Either use $CRS_HOME/bin/crsctl stop crs as root or run the following from the staging as root:

/Staging_Area/11.1.0.6/clusterware/upgrade/preupdate.sh –crshome $CRS_HOME –crsuser oracle

2. Invoke runInstaller from the staging area:

3. As you notice, the option to specify the Home and the destination directory is disabled since OUI has detected the existence of a Cluster.

4. Select the local node (in my case it is alps01), selecting both the nodes requires the CRS to be down on both the nodes - meaning it would require a complete downtime - and would no more be a rolling upgrade:

5. Not shutting down the CRS on the local node would cause the following error:

6. This information would appear in the install logs - pretty detailed and informative :-)

7. Once you shutdown the CRS on the local node and when re-attempted, it should be fine (no errors or warnings):

8. On clicking next, the summary screen would appear. Note that we are upgrading the CRS only the local node (alps01):

9. Install progress screen:

10. At the end of the upgrade, the Installer would prompted us to run the rootupgrade script as root:

11. Output of the rootupgrade script:

12. Check the status of CRS versions:

On Node 2:
13. Shutdown the CRS on node 2. Note that while performing a rolling upgrade, we have to invoke the OUI from the second node and this can't be done from node 1.

14. Invoke the OUI from node 2 and select only the local node and deselect the remote or first node:

15. Cluster Verification screen....looks good.

17. Prompt to run rootupgrade script after the upgrade on Node 2:

18. Output of rootupgrade script on Node 2:

[root@everest02 upgrade]# /opt/oracle/product/crs/install/rootupgrade
Checking to see if Oracle CRS stack is already up...
WARNING: directory '/opt/oracle/product' is not owned by root
WARNING: directory '/opt/oracle' is not owned by root

Oracle Cluster Registry configuration upgraded successfully
Adding daemons to inittab
Attempting to start Oracle Clusterware stack
Waiting for Cluster Synchronization Services daemon to start
Waiting for Cluster Synchronization Services daemon to start
Waiting for Cluster Synchronization Services daemon to start
Cluster Synchronization Services daemon has started
Event Manager daemon has started
Cluster Ready Services daemon has started
Oracle CRS stack is running under init(1M)
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node :
node 2: everest02 everest02-priv everest02
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
clscfg -upgrade completed successfully
CRS stack on this node, is successfully upgraded to 11.1.0.6.0
Checking the existence of nodeapps on this node
Creating '/opt/oracle/product/crs/install/paramfile.crs' with data used for CRS configuration
Setting CRS configuration values in /opt/oracle/product/crs/install/paramfile.crs

[root@everest02 upgrade]#

19. Check or verify the version of CRS:

Configuring SSH

2007-09-03T08:09:00.000-07:00

Configure SSH between two nodes

While logged in as oracle perform the following on each of the nodes of the RAC:

cd $HOME
mkdir ~/.ssh
chmod 700 ~/.ssh
/usr/bin/ssh-keygen -t rsa
/usr/bin/ssh-keygen -t dsa

On Node 1:

cd $HOME/.ssh
cat id_rsa.pub >> authorized_keys
cat id_dsa.pub >> authorized_keys
Copy the authorized_keys file to the node 2. scp authorized_keys node2:/opt/oracle/.ssh

On Node 2:

cd $HOME/.ssh
cat id_rsa.pub >> authorized_keys
cat id_dsa.pub >> authorized_keys
scp authorized_keys node1:/opt/oracle/.ssh

Now perform a ssh between all the nodes including the node-priv hostnames as well. Check to make sure that ssh is configured well without prompting for the password (on both the nodes):

Firewire Modules not loading or not compatible with the kernel version

2007-08-15T08:49:00.000-07:00

The Firewire modules are currently not available for all the kernel versions of Red Hat 4.0, and I really doubt if Oracle is going to make available the Firewire modules for them. For example, my installed kernel version is 2.6.9-42.0.0.0.1.EL, but there are no firewire modules available for this version of kernel and the only closest module available is oracle-firewire-modules-2.6.9-42.0.3.EL-1286-1.i686.rpm. v Obviously, the versions are not compatible and the modules wouldn't load, even if you install and when attempted you are likely to see:

# rpm -ivhi --force oracle-firewire-modules-2.6.9-42.0.3.EL-1286-1.i686.rpm

Preparing... ########################################### [100%] 1:oracle-firewire-modules########################################### [100%]

# modprobe ohci1394 ; modprobe sbp2 FATAL: Module ohci1394 not found. FATAL: Module sbp2 not found. #

There are possibly two options to fix this.

Option 1: The first one is to load the kernel installed by the firewire module, if you are not very particular about the kernel version. Once the firewire modules are installed, the /etc/grub.conf is populated with the Firewire kernel and which looks some thing like this:

#cat /etc/grub.conf
# grub.conf generated by anaconda
# # Note that you do not have to rerun grub after making changes to this file
# NOTICE: You do not have a /boot partition. This means that
# all kernel and initrd paths are relative to /, eg.
# root (hd0,1)
# kernel /boot/vmlinuz-version ro root=/dev/sda2
# initrd /boot/initrd-version.img
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,1)/boot/grub/splash.xpm.gz

hiddenmenu
title Enterprise (2.6.9-42.0.0.0.1.ELhugemem)
root (hd0,1)
kernel /boot/vmlinuz-2.6.9-42.0.0.0.1.ELhugemem ro root=LABEL=/ rhgb quiet
initrd /boot/initrd-2.6.9-42.0.0.0.1.ELhugemem.img
title Enterprise-smp (2.6.9-42.0.0.0.1.ELsmp)
root (hd0,1)
kernel /boot/vmlinuz-2.6.9-42.0.0.0.1.ELsmp ro root=LABEL=/ rhgb quiet
initrd /boot/initrd-2.6.9-42.0.0.0.1.ELsmp.img
title Enterprise-up (2.6.9-42.0.0.0.1.EL) root (hd0,1) kernel /boot/vmlinuz-2.6.9-42.0.0.0.1.EL ro root=LABEL=/ rhgb quiet initrd /boot/initrd- 2.6.9-42.0.0.0.1.EL.img
title Other
rootnoverify (hd0,0)
chainloader +1

=========================================================

The entries in orange color are the ones which have been populated by the Firewire modules/kernel. To load your OS using this kernel, you simply have to update the grub.conf to use that kernel to load by modifying the default=0 to default=2. That should all, when you boot next time using the Firewire modules/kernel, you should be able to load the sbp2, ohci1394 and ieee1394 modules without any problems.

Option 2:

When you install oracle-firewire-modules-2.6.9-42.0.3.EL-1286-1.i686.rpm rpm it will install the modules under /lib/modules/2.6.9-42.0.3.EL/kernel/drivers/ieee1394, but your existing kernel would be as indicated by `uname -r`. In my case it was 2.6.9-42.0.0.0.1.EL.
Make a directory ieee1394 under /lib/modules/2.6.9-42.0.0.0.1.EL/kernel/drivers
copy the modules which have been installed under /lib/modules/2.6.9-42.0.3.EL/kernel/drivers/ieee1394 to /lib/modules/2.6.9- 42.0.0.0.1.EL/kernel/drivers/ieee1394.
Then run the following commands:

#depmod -a ieee1394.ko;modprobe ieee1394
#depmod -a ohci1394.ko;modprobe ohci1394
#depmod -a pcilynx.ko;modprobe pcilynx
#depmod -a raw1394.ko;modprobe raw1394
==============================================

Update the etc/modprobe.conf. For example, the entry in the /etc/modprobe.conf file would be: options sbp2 exclusive_login=0

Now you should be able to load the Firewire modules without changing your kernel.


==================================================================

Checking the integrity of the downloaded firewire module:

$md5sum oracle-firewire-modules-2.6.9-42.0.3.EL-1286-1.i686.rpm
7fe3c83291ffab284f6f941b717dacb1 oracle-firewire-modules-2.6.9-42.0.3.EL-1286-1.i686.rpm

Verify the above code (7fe3c83291ffab284f6f941b717dacb1) with that
indicated in the source website.

Export/Import Pump - Copying Schemas

2007-08-13T15:38:00.000-07:00

Let’s say that we have a Schema in Database A, which needs to be copied over to Database B. With the traditional export/import, we have to precreate the user account in the Database B, before we can import the schema. But with oracle10g’s Pump utility, it is just one simple step:

Here is an example. In this example, the schema CHANDRA exists in database A and we attempt to copy over the same schema as CHANDRA3 in a remote database (rhes01.myoracle.com).

$impdp system/******* schemas=chandra network_link=rhes01.myoracle.com remap_schema=chandra:chandra3

Import: Release 10.2.0.3.0 - Production on Saturday, 05 May, 2007 18:54:21

Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - Production

With the Partitioning, OLAP and Data Mining options

Starting “SYSTEM”.”SYS_IMPORT_SCHEMA_01″: system/******** schemas=chandra network_link=rhes01.myoracle.com remap_schema=chandra:chandra3

Estimate in progress using BLOCKS method…

Processing object type SCHEMA_EXPORT/TABLE/TABLE_DATA

Total estimation using BLOCKS method: 0 KB

Processing object type SCHEMA_EXPORT/USER

Processing object type SCHEMA_EXPORT/SYSTEM_GRANT

Processing object type SCHEMA_EXPORT/ROLE_GRANT

Processing object type SCHEMA_EXPORT/DEFAULT_ROLE

Processing object type SCHEMA_EXPORT/PRE_SCHEMA/PROCACT_SCHEMA

Processing object type SCHEMA_EXPORT/TABLE/TABLE

Job “SYSTEM”.”SYS_IMPORT_SCHEMA_01″ successfully completed at 18:54:51

******************************************************************************************
This creates the user CHANDRA3 as well in the remote/target database.

Don’t forget to provide different filenames when PARALLEL is used with Export Dump.

If you use PARALLEL option with Export Dump and fail to provide file names equal to the number of PARALLEL, then it would perform the export in serial - meaning writes to a single file.

For example, if you have something like this: $ expdp dumpfile=expdat.dmp directory=dump_dir parallel=4
It would create just one file with the name expdat.dmp. Instead you can specify some thing like:

$ expdp dumpfile=expdat.%U.dmp directory=dump_dir parallel=4

This would create 4 files in parallel with names (expdat.01.dmp, expdat.02.dmp, expdat.03.dmp..).

*********************************************************************************

Oracle 11g RAC Install on Red Hat 5.0 (Linux) using NFS

2007-08-13T10:53:00.000-07:00

Here is the procedure I followed to install Oracle 11g RAC on RHEL 5.0 and thought would share this information with everyone who is interested in checking out Oracle11g RAC on Linux. Most of the pre-installation steps should work for Oracle10g as well. This configuration is not for production systems and would be good for testing purposes only.

I have used my two existing two Intel based systems for the purpose, and in all it took about 8 hours to get the whole thing working (including the OS install – as a third bootable Operating System – the first one being Windows XP the second being RHEL 4.0 with 10gR2 RAC).....much better than the initial installation experiences with Oracle10g RAC.

Please note that I am not using Firewire or a third server here for provisioning the shared storage, but NFS - the details follow.

Hardware details:

Node 1: pepsi.pinnacle.com (Intel Pentium 4 2.80Ghz, 2G RAM with 80GB and 160GB hard drives, 2 NICs)
Node 2: coke.pinnacle.com (Intel Pentium 4 2.80Ghz, 2G RAM with 160GB, 2 NICs)

RHEL 5.0 Installation:

I have performed a complete/full installation – basically all/everything...so that all the RPMs are included therein. Kernel version used - 2.6.18-8.el5xen.

Installation Steps:

Here are high-level installation steps:

Check OS/Packages requirements
Configure public and private network.
Create oracle user and dba group.
Setup SSH for oracle user account between the two nodes.
Set up shared storage using NFS (details follow)
Oracle11g CRS Installation
Oracle11g RDBMS Software
Oracle11g RAC Database creation.

The details.......

1. Configure Operating System for Oracle11g RAC Install (Both nodes)

The following RPMs are required/recommended for Oracle11g RAC installation on RHEL 5.0.

binutils-2.17.50.0.6-2.el5
compat-libstdc++-33-3.2.3-61
elfutils-libelf-0.97-5 – elfutils-libelf-devel-0.125-3.el5 was installed by default
elfutils-libelf-devel-0.125
glibc-2.5-12
glibc-common-2.5-12
glibc-devel-2.5-12
gcc-4.1.1-52
gcc-c++-4.1.1-52
libaio-0.3.106
libaio-devel-0.3.106 – not installed by default
libgcc-4.1.1-52
libstdc++-4.1.1
libstdc++-devel-3.4.3-22.1 – libstdc++-devel-4.1.1-52.el5 was installed by default.
make-3.81-1.1
sysstat-7.0.0 – not installed.
UnixODBC-2.2.11 – not installed.
UnixODBC-devel-2.2.11 – not installed.

All the above RPMs were installed and available by default (may be because I have done a complete install). Installed the following 4 missing RPMs from RHEL 5.0 CDs (using rpm -ivh ).

libaio-devel-0.3.106
sysstat-7.0.0
UnixODBC-2.2.11
UnixODBC-devel-2.2.11

Kernel Parameters:

Add the following lines to /etc/sysctl.conf

kernel.shmall = 2097152
kernel.shmmax = 2147483648
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default = 4194304
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 262144

Run sysctl -p to read the values as indicated in /etc/sysctl.conf file.

CVUQDISK Package:

Install the cvuqdisk package – located in the CRS software distribution (/clusterware/rpm):

[root@coke rpm]# export CVUQDISK_GRP=dba
[root@coke rpm]# rpm -ivh cvuqdisk-1.0.1-1.rpm
Preparing... ########################################### [100%]
1:cvuqdisk ########################################### [100%]
[root@coke rpm]# pwd
/share/Oracle-Linux-Software/11g/clusterware/rpm
[root@coke rpm]#

Here is the information from the /etc/hosts file. The contents of this file are same on both the nodes.

$ cat /etc/hosts
#----Entries for Node1
192.168.1.160 pepsi.pinnacle.com pepsi
192.168.1.151 pepsi-vip.pinnacle.com pepsi-vip
172.1.1.160 pepsi-priv.pinnacle.com pepsi-priv
#---Entries for Node2
192.168.1.150 coke.pinnacle.com coke
192.168.1.161 coke-vip.pinnacle.com coke-vip
172.1.1.150 coke-priv.pinnacle.com coke-priv

Node 1:
[root@coke ~]# ifconfig -a eth0
eth0 Link encap:Ethernet HWaddr 00:16:76:4B:79:F1
inet addr:192.168.1.150 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::216:76ff:fe4b:79f1/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3507 errors:0 dropped:0 overruns:0 frame:0
TX packets:1675 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:665921 (650.3 KiB) TX bytes:188926 (184.4 KiB)
[root@coke ~]# ifconfig -a eth1
eth1 Link encap:Ethernet HWaddr 00:09:5B:E1:CD:DD
inet addr:172.1.1.150 Bcast:172.1.255.255 Mask:255.255.0.0
inet6 addr: fe80::209:5bff:fee1:cddd/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:310738 errors:0 dropped:0 overruns:0 frame:0
TX packets:254557 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:141179047 (134.6 MiB) TX bytes:345107330 (329.1 MiB)

Node 2:
[root@pepsi ~]# ifconfig -a eth0
eth0 Link encap:Ethernet HWaddr 00:16:76:38:D8:C5
inet addr:192.168.1.160 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::216:76ff:fe38:d8c5/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:419 errors:0 dropped:0 overruns:0 frame:0
TX packets:244 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:80282 (78.4 KiB) TX bytes:27401 (26.7 KiB)
[root@pepsi ~]# ifconfig -a eth1
eth1 Link encap:Ethernet HWaddr 00:09:5B:BB:D8:F1
inet addr:172.1.1.160 Bcast:172.1.255.255 Mask:255.255.0.0
inet6 addr: fe80::209:5bff:febb:d8f1/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:361934 errors:0 dropped:0 overruns:0 frame:0
TX packets:326328 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:366491792 (349.5 MiB) TX bytes:148740977 (141.8 MiB)
Interrupt:17 Base address:0xe000

2. Configure SSH between two nodes

While logged in as oracle perform the following on each of the nodes of the RAC:

cd $HOME
mkdir ~/.ssh
chmod 700 ~/.ssh
/usr/bin/ssh-keygen -t rsa
/usr/bin/ssh-keygen -t dsa

On Node 1:

cd $HOME/.ssh
cat id_rsa.pub >> authorized_keys
cat id_dsa.pub >> authorized_keys
Copy the authorized_keys file to the node 2. scp authorized_keys node2:/opt/oracle/.ssh

On Node 2:

cd $HOME/.ssh
cat id_rsa.pub >> authorized_keys
cat id_dsa.pub >> authorized_keys
scp authorized_keys node1:/opt/oracle/.ssh

Now perform a ssh between all the nodes including the node-priv hostnames as well. Check to make sure that ssh is configured well without prompting for the password (on both the nodes):

$ ssh coke.pinnacle.com date
Sun Aug 12 08:41:42 CDT 2007
$ ssh pepsi.pinnacle.com date
Sun Aug 12 08:42:23 CDT 2007
$ ssh coke-priv date
Sun Aug 12 08:42:45 CDT 2007
$ ssh pepsi-priv date
Sun Aug 12 08:43:22 CDT 2007

3. Configuring the Shared Storage

This is the one of the critical components required for RAC. There are couple of other storage options available for setting up Test environment at home, such as Firewire, NAS and/or NFS. I have installed Oracle RAC with both Oracle9i and Oracle10g using Firewire and this time around I wanted to see if I can use NFS instead. Please note that this is only for testing and experimentation only and will not be suitable for any kind of production use. Usually, we would be needing an separate server/node which can host the NFS file systems, but I have, instead of using a third server, I have configured one of the nodes to host the NFS file systems. Here is the procedure:

On Node 1 (coke):

1. Created three partitions on the local hard drive with a size of 10G each.

[root@coke ~]# fdisk /dev/sda
The number of cylinders for this disk is set to 9726.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Command (m for help): p
Disk /dev/sda: 80.0 GB, 80000000000 bytes
255 heads, 63 sectors/track, 9726 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 2550 20482843+ c W95 FAT32 (LBA)
Command (m for help): n
Command action
e extended
p primary partition (1-4)
e
Partition number (1-4): 2
First cylinder (2551-9726, default 2551):
Using default value 2551
Last cylinder or +size or +sizeM or +sizeK (2551-9726, default 9726):
Using default value 9726
Command (m for help): p
Disk /dev/sda: 80.0 GB, 80000000000 bytes
255 heads, 63 sectors/track, 9726 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 2550 20482843+ c W95 FAT32 (LBA)
/dev/sda2 2551 9726 57641220 5 Extended
Command (m for help): n
Command action
l logical (5 or over)
p primary partition (1-4)
l
First cylinder (2551-9726, default 2551):
Using default value 2551
Last cylinder or +size or +sizeM or +sizeK (2551-9726, default 9726): +10G
Command (m for help): n
Command action
l logical (5 or over)
p primary partition (1-4)
l
First cylinder (3768-9726, default 3768):
Using default value 3768
Last cylinder or +size or +sizeM or +sizeK (3768-9726, default 9726): +10G
Command (m for help): n
Command action
l logical (5 or over)
p primary partition (1-4)
l
First cylinder (4985-9726, default 4985):
Using default value 4985
Last cylinder or +size or +sizeM or +sizeK (4985-9726, default 9726): +10G
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Device Boot Start End Blocks Id System
/dev/sda1 * 1 2550 20482843+ c W95 FAT32 (LBA)
/dev/sda2 2551 9726 57641220 5 Extended
/dev/sda5 2551 3767 9775521 83 Linux
/dev/sda6 3768 4984 9775521 83 Linux
/dev/sda7 4985 6201 9775521 83 Linux

[root@coke ~]# parted /dev/sda print
Model: ATA WDC WD800JD-75MS (scsi)
Disk /dev/sda: 80.0GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number Start End Size Type File system Flags
1 32.3kB 21.0GB 21.0GB primary fat32 boot, lba
2 21.0GB 80.0GB 59.0GB extended
5 21.0GB 31.0GB 10.0GB logical
6 31.0GB 41.0GB 10.0GB logical
7 41.0GB 51.0GB 10.0GB logical
Information: Don't forget to update /etc/fstab, if necessary.

Create file systems

[root@coke ~]# mkfs.ext3 /dev/sda5
[root@coke ~]# mkfs.ext3 /dev/sda6
[root@coke ~]# mkfs.ext3 /dev/sda7

Create local mount points

[root@coke ~]# mkdir /nfs1 /nfs2 /nfs3

Label the partitions/file systems

[root@coke ~]# e2label /dev/sda5 nfs1
[root@coke ~]# e2label /dev/sda6 nfs2
[root@coke ~]# e2label /dev/sda7 nfs3

Add the following lines to the /etc/fstab file on node 1(coke)

LABEL=nfs1 /nfs1 ext3 defaults 1 1
LABEL=nfs2 /nfs1 ext3 defaults 1 1
LABEL=nfs3 /nfs1 ext3 defaults 1 1

Mount the files systems:

[root@coke ~]# mount /nfs2
[root@coke ~]# mount /nfs2
[root@coke ~]# mount /nfs3
[root@coke ~]# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda3 47611656 2938484 42215632 7% /
tmpfs 966124 0 966124 0% /dev/shm
/dev/sda5 9621848 152688 8980384 2% /nfs1
/dev/sda6 9621848 152688 8980384 2% /nfs2
/dev/sda7 9621848 152688 8980384 2% /nfs3

Add the following entries in the /etc/exports file:

/nfs1 *(rw,sync,no_wdelay,insecure_locks,no_root_squash)
/nfs2 *(rw,sync,no_wdelay,insecure_locks,no_root_squash)
/nfs3 *(rw,sync,no_wdelay,insecure_locks,no_root_squash)

Configure the NFS services:

[root@coke ~]# chkconfig nfs on
[root@coke ~]# service nfs restart
Shutting down NFS mountd: [ OK ]
Shutting down NFS daemon: [ OK ]
Shutting down NFS quotas: [ OK ]
Shutting down NFS services: [ OK ]
Starting NFS services: [ OK ]
Starting NFS quotas: [ OK ]
Starting NFS daemon: [ OK ]
Starting NFS mountd: [ OK ]
[root@coke ~]#

With the above, we have /nfs* file systems which are available for both the nodes. Peform the following steps on both the nodes:

Create mount points to mount the nfs file systems

#mkdir /u01 /u02 /u03 on two nodes

2. Add the following lines to the /etc/fstab file:

coke-priv:/nfs1 /u01 nfs
rw,bg,hard,nointr,tcp,vers=3,timeo=300,rsize=32768,wsize=32768,actimeo=0 0 0
coke-priv:/nfs2 /u02 nfs
rw,bg,hard,nointr,tcp,vers=3,timeo=300,rsize=32768,wsize=32768,actimeo=0 0 0
coke-priv:/nfs3 /u03 nfs
rw,bg,hard,nointr,tcp,vers=3,timeo=300,rsize=32768,wsize=32768,actimeo=0 0 0

3. Mount the nfs filesystem locally:

#mount /u01
#mount /u02
#mount /u03

4. Now we have all the three file systems shared among the two nodes and ready for use.

Here is how it will look like on Node 1:

$ df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda3 47611656 7029084 38125032 16% /
tmpfs 966124 270188 695936 28% /dev/shm
/dev/sda5 9621848 1925276 7207796 22% /nfs1
/dev/sda6 9621848 152700 8980372 2% /nfs2
/dev/sda7 9621848 152696 8980376 2% /nfs3
172.1.1.150:/nfs1 9621856 1925280 7207808 22% /u01
172.1.1.150:/nfs2 9621856 152704 8980384 2% /u02
172.1.1.150:/nfs3 9621856 152704 8980384 2% /u03

On Node 2:

$ df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda5 44639012 7245820 35089072 18% /
tmpfs 477484 266084 211400 56% /dev/shm
coke-priv:/nfs1 9621856 1925280 7207808 22% /u01
coke-priv:/nfs2 9621856 152704 8980384 2% /u02
coke-priv:/nfs3 9621856 152704 8980384 2% /u03

● Change the owner of the file systems /u01, /u02 and /u03 to oracle:dba.

4. CRS Installation

Set up the environment variables for oracle (such ORACLE_BASE, PATH etc..)

With this we have all the basic pre-installations tasks taken care of and are ready to invoke the Oracle11g OUI from the CRS software distribution, but before that, let's run the CVU:

$ ./runcluvfy.sh stage -pre crsinst -n coke,pepsi

Invoke the Installer:

Running orainstRoot.sh and root.sh scripts on both the nodes:

[root@coke ~]# /opt/oracle/oraInventory/orainstRoot.sh
Changing permissions of /opt/oracle/oraInventory to 770.
Changing groupname of /opt/oracle/oraInventory to dba.
The execution of the script is complete
[root@coke ~]# /opt/oracle/product/11.1.0/crs/root.sh
WARNING: directory '/opt/oracle/product/11.1.0' is not owned by root
WARNING: directory '/opt/oracle/product' is not owned by root
WARNING: directory '/opt/oracle' is not owned by root
Checking to see if Oracle CRS stack is already configured
/etc/oracle does not exist. Creating it now.
Setting the permissions on OCR backup directory
Setting up Network socket directories
Oracle Cluster Registry configuration upgraded successfully
The directory '/opt/oracle/product/11.1.0' is not owned by root. Changing owner to root
The directory '/opt/oracle/product' is not owned by root. Changing owner to root
The directory '/opt/oracle' is not owned by root. Changing owner to root
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node :
node 1: coke coke-priv coke
node 2: pepsi pepsi-priv pepsi
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Now formatting voting device: /u01/oradb/ocr/vote
Format of 1 voting devices complete.
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
Cluster Synchronization Services is active on these nodes.
coke
Cluster Synchronization Services is inactive on these nodes.
pepsi
Local node checking complete. Run root.sh on remaining nodes to start CRS daemons.
[root@coke ~]# /opt/oracle/product/11.1.0/crs/root.sh
[root@pepsi ~]# /opt/oracle/oraInventory/orainstRoot.sh
Changing permissions of /opt/oracle/oraInventory to 770.
Changing groupname of /opt/oracle/oraInventory to dba.
The execution of the script is complete
[root@pepsi ~]# /opt/oracle/product/11.1.0/crs/root.sh
WARNING: directory '/opt/oracle/product/11.1.0' is not owned by root
WARNING: directory '/opt/oracle/product' is not owned by root
WARNING: directory '/opt/oracle' is not owned by root
Checking to see if Oracle CRS stack is already configured
/etc/oracle does not exist. Creating it now.
Setting the permissions on OCR backup directory
Setting up Network socket directories
Oracle Cluster Registry configuration upgraded successfully
The directory '/opt/oracle/product/11.1.0' is not owned by root. Changing owner
to root
The directory '/opt/oracle/product' is not owned by root. Changing owner to root
The directory '/opt/oracle' is not owned by root. Changing owner to root
clscfg: EXISTING configuration version 4 detected.
clscfg: version 4 is 11 Release 1.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node :
node 1: coke coke-priv coke
node 2: pepsi pepsi-priv pepsi
clscfg: Arguments check out successfully.
NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
Cluster Synchronization Services is active on these nodes.
coke
pepsi
Cluster Synchronization Services is active on all the nodes.
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
Running vipca(silent) for configuring nodeapps
Creating VIP application resource on (2) nodes...
Creating GSD application resource on (2) nodes...
Creating ONS application resource on (2) nodes...
Starting VIP application resource on (2) nodes...
Starting GSD application resource on (2) nodes...
Starting ONS application resource on (2) nodes...
Done.
[root@pepsi ~]#

$ crsctl check crs
Cluster Synchronization Services appears healthy
Cluster Ready Services appears healthy
Event Manager appears healthy
$
We have a up and running Oracle 11g CRS.

Note that you have to always bring up Node 1 first and then Node 2, since Node 1 is hosting the NFS file system required by Node 2.

More to follow......

Thanks to Tim Hall (oracle-base) for sharing the ideas/thoughts on NFS share/mount options.

Convert Non-Partitioned to Partitioned Table - using Online Re-definition feature (10g)

2007-08-13T06:22:00.000-07:00

Here is a working example to convert a non-partitioned table to a partitioned table. With this example, we will not see how to convert a table, but will expose to the usage of Online Redefinition feature. There is one neat (apart from others) enhancement to the online redefinition feature, which we are going to talk about.

We have a non-partitioned table T1 with about 2 million records
It has a primary key, grants to various roles and public synonyms (dependencies)
Our goal is to partition the table using the primary key column (col1 number(100))

Some restrictions of Online Redefinition:

Cannot belong to SYS or SYSTEM Schema.
The table to be redefined online should not be using User-defined data types
Should not be a clustered table.
Should not have MV Logs or MVs defined

You cannot move a table from one schema to another using Online Redefinition feature.
1. Check to make sure that the table can use the online redefintion feature:

SQL> EXEC DBMS_REDEFINITION.CAN_REDEF_TABLE(’CHANDRA’,'T1′);