Lately we have been noticing ORA-27090 on our 10.2.0.4 databases. Here is the complete error message, as appeared in the alert log file. This is observed on Suse Linux, Red hat and OEL 5.3:
===============================================================
Errors in file /tmp/test_dbw0_4400.trc:
ORA-27090: Message 27090 not found; product=RDBMS; facility=ORA
Additional information: 3
Additional information: 128
Additional information: 1100 #--This is the value for the parameter aio-max-nr
===============================================================
Looks like this error condition is only encountered or reported in databases using 10.2.0.4 oracle version. On research found that this has something to do with the kernel setting fs.aio-max-nr. aio-max-nr sets the systemwide maximum number of AIO requests. Apparently, when the number of requests reach this number the error is reported. I guess the question would be, how can we determine where we are at currently in relation to the max limit, right? This is answered by aio-nr, which maintains a cumulative/running total of number of aio events used.
I was able to consistently reproduce the error condition by playing around with the settings for aio-max-nr parameter. As such, no apparent problems are noticed on the database side, but did did notice that the standby databases would fail to continue with the managed recovery process.
Here is how it looked like before starting up an 10.2.0.4 instance:
#/sbin/sysctl -a |grep aio
fs.aio-max-nr = 1100 #---I reset it to this value from default.
fs.aio-nr = 0
When the instance was started, the value looked like this:
#/sbin/sysctl -a |grep aio
fs.aio-max-nr = 1100 #---I reset it to this value from default.
fs.aio-nr = 1060
As soon as it hit the ceiling, the above mentioned error was reported in the alert log. When I raised the value, the error message disappeared.
As soon as the instances are shutdown the value for fs.aio-nr gets reset.
Hope this information would come in handy.
Thursday, July 30, 2009
Wednesday, July 15, 2009
How to check if CRS is set to auto-start on reboots
Sometime back, some one asked me a question - "how to confirm that CRS is set to restart on reboot?"
CRS can be configured to either to restart or not to restart on reboots. Disable restarts would be typically useful for troubleshooting CRS reboots or validating any system level changes.
To enable restarts (default):
$CRS_HOME/bin/crsctl enable crs
To disable restarts of crs on reboots:
$CRS_HOME/bin/crsctl disable crs
Note that the above needs access to root.
Whenever the above commands are run, it modifies the following file (on linux), which indicates whether or not the CRS is set to auto-restart on reboots:
/etc/oracle/scls_scr/node_name/root/crsstart
It contains either "enable" or "disable" keywords.
CRS can be configured to either to restart or not to restart on reboots. Disable restarts would be typically useful for troubleshooting CRS reboots or validating any system level changes.
To enable restarts (default):
$CRS_HOME/bin/crsctl enable crs
To disable restarts of crs on reboots:
$CRS_HOME/bin/crsctl disable crs
Note that the above needs access to root.
Whenever the above commands are run, it modifies the following file (on linux), which indicates whether or not the CRS is set to auto-restart on reboots:
/etc/oracle/scls_scr/node_name
It contains either "enable" or "disable" keywords.
Subscribe to:
Posts (Atom)