Tuesday, September 8, 2009

ORA-03001 - while trying to rename tables owned by others - workaround

There could be situations where you may like to rename the tables owned by others, but do not have access to the password (of the schema owner) or don't want to change the password (using "by values" etc...). You could perform almost all the activities by using "ALTER SESSION SET CURRENT_SCHEMA=;", but rename table wouldn't work and following error message would be encountered:

chandra@DOE01:11g> show user
USER is "CHANDRA"
chandra@DOE01:11g> alter session set current_schema=CP_ADMIN;

Session altered.

chandra@DOE01:11g> rename APP_DATA to APP_DATA_BAK;
rename APP_DATA to APP_DATA_BAK
*
ERROR at line 1:
ORA-03001: unimplemented feature


chandra@DOE01:11g>

To get around this, we could create a simple procedure similar to the following:

create or replace procedure rename_others_tab (orig_tab_name in varchar2, new_tab_name in varchar2)
as
lv_sql_str varchar2(100);
lv_exists number(1);
begin
select count(*) into lv_exists from user_tables where table_name=orig_tab_name;
if lv_exists = 1 then
lv_sql_str := 'rename '||orig_tab_name||' to '||new_tab_name;
dbms_output.put_line (lv_sql_str);
execute immediate lv_sql_str;
else
dbms_output.put_line('ERROR:'||orig_tab_name||' does not exist!');
end if;
end;
/


Note that you would be creating the above procedure while your current_schema is set to the owner of the table.

Then:

chandra@DOE01:11g> exec rename_others_tab('APP_DATA','APP_DATA_BAK');

PL/SQL procedure successfully completed.

chandra@DOE01:11g>

The table name would be renamed....Of course, use the above procedure (work-around) with caution and when ABSOLUTELY needed.

Thought would be of use to others....

Friday, August 7, 2009

ORA-01620 while creating a RAC standby for non-RAC primary

Today I was trying to test some thing and had to create a RAC standby for a Non-RAC (single-instance) Primary database. I know, it doesn't make sense to have a RAC standby for a single-instance primary, but that's not the point. First instance was mounted without any problems, but when attempted to mount the second standby instance got the following error message:

SQL> alter database mount standby database;
alter database mount standby database
*
ERROR at line 1:
ORA-01620: no public threads are available for mounting


I couldn't find quick notes on the web to get around this issue and therefore had to find my own for this unique situation (having a RAC standby for SI-Primary). Here is what I did on Primary:

SQL> ALTER DATABASE ADD LOGFILE THREAD 2 group 13 size 5M;

Database altered.

SQL> ALTER DATABASE ADD LOGFILE THREAD 2 group 14 size 5M;

Database altered.

SQL> alter database enable public thread 2;

Database altered.


Then created new copy of standby controlfile and shipped it over to standby and attempted to mount both the instances, it worked fine! Of course, had the primary been a RAC, we wouldn't have seen this in first place :-) BTW, this is on Oracle11g 11.1.0.7

Tuesday, August 4, 2009

Tablespace Point-in-time Recovery (TSPITR) using RMAN

Scenario: A logical corruption occurred due to an user error and have to quickly get back to the state just before the time of the error. Possible options: Database point-in-time recovery, but the downside is the entire database has to be unavailable and other transactions after the PITR would be lost. The other option would be Tablespace Point-in-time-Recovery (TSPITR) - the obvious choice here. I had known that RMAN could be used to perform TSPITR, but I realized just today how powerful, simple, automated, error-free and fast it could be to perform TSPITR using RMAN - it's really amazing. The intent of this post is just to high-light and appreciate the usefulness of RMAN to perform TSPITR.

Assuming we have the latest backup, just executing the following statement would do everything needed to get the entire tablespace to a point-in-time in the past. Believe me, you don't have to do anything other than just this statement!


recover tablespace APP_DATA until time
"to_date('2009-08-04 12:15:00’,’YYYY-MM-DD HH24:MI:SS')"
auxiliary destination ’/opt/oracle/temp’;

Once the above statement is executed, RMAN does the following for us:

  • Creates auxiliary instance (including the pfile etc..)
  • Mounts the auxiliary instance
  • Makes the candidate tablespace into OFFLINE
  • Restores the basic tablespace UNDO, SYTEM,SYSAUX and then the required tablespace
  • Applies archives (completes recovery)
  • Opens the database
  • Performs an export of the objects residing in the tablespace
  • Shutdown aux instance
  • Import the objects into the target database
  • Remove aux instance and cleanup
That's it....you now have all the objects back in the tablespace!.

You can view the output of the RMAN session by downloading this file.. which is quite self-explanatory and informative too!

Thursday, July 30, 2009

ORA-27090: Message 27090 not found on 10.2.0.4

Lately we have been noticing ORA-27090 on our 10.2.0.4 databases. Here is the complete error message, as appeared in the alert log file. This is observed on Suse Linux, Red hat and OEL 5.3:

===============================================================
Errors in file /tmp/test_dbw0_4400.trc:
ORA-27090: Message 27090 not found; product=RDBMS; facility=ORA
Additional information: 3
Additional information: 128
Additional information: 1100 #--This is the value for the parameter aio-max-nr
===============================================================
Looks like this error condition is only encountered or reported in databases using 10.2.0.4 oracle version. On research found that this has something to do with the kernel setting fs.aio-max-nr. aio-max-nr sets the systemwide maximum number of AIO requests. Apparently, when the number of requests reach this number the error is reported. I guess the question would be, how can we determine where we are at currently in relation to the max limit, right? This is answered by aio-nr, which maintains a cumulative/running total of number of aio events used.

I was able to consistently reproduce the error condition by playing around with the settings for aio-max-nr parameter. As such, no apparent problems are noticed on the database side, but did did notice that the standby databases would fail to continue with the managed recovery process.

Here is how it looked like before starting up an 10.2.0.4 instance:

#/sbin/sysctl -a |grep aio
fs.aio-max-nr = 1100 #---I reset it to this value from default.
fs.aio-nr = 0

When the instance was started, the value looked like this:
#/sbin/sysctl -a |grep aio
fs.aio-max-nr = 1100 #---I reset it to this value from default.
fs.aio-nr = 1060

As soon as it hit the ceiling, the above mentioned error was reported in the alert log. When I raised the value, the error message disappeared.

As soon as the instances are shutdown the value for fs.aio-nr gets reset.

Hope this information would come in handy.

Wednesday, July 15, 2009

How to check if CRS is set to auto-start on reboots

Sometime back, some one asked me a question - "how to confirm that CRS is set to restart on reboot?"

CRS can be configured to either to restart or not to restart on reboots. Disable restarts would be typically useful for troubleshooting CRS reboots or validating any system level changes.

To enable restarts (default):
$CRS_HOME/bin/crsctl enable crs

To disable restarts of crs on reboots:
$CRS_HOME/bin/crsctl disable crs

Note that the above needs access to root.

Whenever the above commands are run, it modifies the following file (on linux), which indicates whether or not the CRS is set to auto-restart on reboots:

/etc/oracle/scls_scr/node_name/root/crsstart

It contains either "enable" or "disable" keywords.

Saturday, June 20, 2009

ORA-29702 - Starting RAC instance in non-rac mode

Assume the following situation: Due to some OS or network related problem you are not able to start-up the CRS (this does happen!) before it impacts the database availability and subsequently your business. Your primary goal now is to make the database available as soon as possible irrespective of its mode (RAC or Non-RAC). The reason being you would like to reduce the impact to your business. If you attempt to startup the instance without CRS, you get the following error:

SQL> startup;
ORA-29702: error occurred in Cluster Group Service operation
SQL>

This error message indicates that it can't start the instance since CRS is not available. Here is the quick work-around to startup the instance. The option is to remove RAC option from your binaries and start the instance. Once you have resolved the CRS related issues, you can turn-on the RAC option back (of course, your instance needs to be down while to turn-on or turn-off the options). Here is how we turn-off:

cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk rac_off ioracle

Set cluster_database=false

You should now be able to startup the instance without CRS stack being up and running.

SQL> Select * from v$option where parameter like 'Real%';

PARAMETER VALUE
---------------------------------------- ----------
Real Application Clusters FALSE
Real Application Testing TRUE

Turn-on the RAC option with:

make -f ins_rdbms.mk rac_on ioracle

Alternatively, you can install a non-rac oracle binaries to quickly start the instance.


By the way, you turn off other options in similar way such as Partitioning (part_on/part_off), DB Vault (dv_on/dv_off) etc..

Tuesday, June 16, 2009

Oracle11g - Killing session in RAC (in remote instance)

Prior to Oracle11g, whenever you want to kill a session connected to a non-local (remote) instance, you had to make an explicit connection to that particular instance and then attempt to kill.

Example:

SQL> Select instance_number from v$instance;

INSTANCE_NUMBER
---------------
1 ---> I am connected to instance 1 (Local Instance)

1 row selected.

SQL> Select inst_id, sid, serial# from gv$session where username='CHANDRA';

INST_ID SID SERIAL#
---------- ---------- ----------
2 125 7526 ---> CHANDRA is connected to Inst# 2 (Non-local/remote Instance)

1 row selected.

With 11g, you could kill the session which is connected to instance 2, while you are connected to instance 1:

SQL> Alter system kill session '125,7526,@2' immediate; #--@2 indicates remote instance.

System altered.

SQL> Select inst_id, sid, serial# from gv$session where username='CHANDRA';

no rows selected

This really makes life easy - especially you are using SQL*Plus and not any other front-end tool.