DBA Sensation

February 20, 2009

Detailed steps for removing a node from 10gR2 3-nodes RAC

Filed under: [RAC] — Tags: , , , , — zhefeng @ 11:36 am

Background:

we have to remove a node from Oracle 10gR2 RAC.

OS: Redhat EL4

Node name:

db01

db02

db05

Storage: ASM (instance name +ASM)

The most important 3 steps that need to be followed are;

A.         Remove the instance using DBCA.

B.         Remove the node from the cluster.

C.         Reconfigure the OS and remaining hardware.

Here is a breakdown of the above steps.

A. Remove the instance using DBCA.

————————————–

1. Verify that you have a good backup of the OCR (Oracle Configuration  Repository) using ocrconfig -showbackup or dd command.

Run the backup manually with dd for OCR and Voting disk:

[oracle@db01 ocr_voting]$ pwd

/databases/oracle/backup/ocr_voting

[oracle@db01 ocr_voting]$ dd if=/dev/raw/raw1 of=before_del_3rdnode.ocr

[oracle@db01 ocr_voting]$ dd if=/dev/raw/raw2 of=before_del_3rdnode.vote

2. Run DBCA from one of the nodes you are going to keep (db01).  Leave the database up and also leave the departing instance up and running.

3. Choose “Instance Management”

4. Choose “Delete an instance”

5. On the next screen, select the cluster database (p10c) from which you will delete an instance.  Supply the system privilege username and password.

6. On the next screen, a list of cluster database instances will appear.  Highlight the instance you would like to delete then click next.

7.         If you have services configured, reassign the services.  Modify the

services so that each service can run on one of the remaining

instances.  Set “not used” for each service regarding the instance

that is to be deleted.  Click Finish.

8.         If your database is in archive log mode you may encounter the

following errors (10gR2 doesn’t have this issue):

ORA-350

ORA-312

This may occur because the DBCA cannot drop the current log, as

it needs archiving.  This issue is fixed in the 10.1.0.3

patchset. But previous to this patchset you should click the

ignore button and when the DBCA completes, manually archive

the logs for the deleted instance and dropt the log group.

SQL>  alter system archive log all;

SQL>  alter database drop logfile group 2;

9.         Verify that the dropped instance’s redo thread has been removed by

querying v$log.  If for any reason the redo thread is not disabled

then disable the thread.

SQL> alter database disable thread 2;

10. Verify that the instance was removed from the OCR (Oracle Configuration Repository) with the following commands:

srvctl config database -d <db_name>

[oracle@vanpgprodb01 dbca]$ srvctl config database -d p10c

db01 p10c1 /databases/oracle/db

db02 p10c2 /databases/oracle/db

cd <CRS_HOME>/bin/crs_stat

11. If this node had an ASM instance and the node will no longer be a part of the cluster you will now need to remove the ASM instance with:

srvctl stop asm -n <nodename>

[oracle@db01 dbca]$ srvctl stop asm -n db05

srvctl remove asm -n <nodename>

[oracle@db01 dbca]$ srvctl remove asm -n db05

Verify that asm is removed with:

srvctl config asm -n <nodename>

[oracle@db01 dbca]$ srvctl config asm -n db05  –the output is nothing instead of show you the ASM instance name and asm home

Delete the ASM folders on deleting node:

rm -r $ORACLE_BASE/admin/+ASM
[root@db05 admin]# rm -rf +ASM

rm -f $ORACLE_HOME/dbs/*ASM*

[root@db05 dbs]# rm -rf *ASM*

Remove the ASM library on node3:

[root@db05 db]# /etc/init.d/oracleasm stop

Unmounting ASMlib driver filesystem: [  OK  ]

Unloading module “oracleasm”: [  OK  ]

[root@db05 db]# rpm -qa | grep oracleasm

oracleasmlib-2.0.2-1

oracleasm-2.6.9-55.ELsmp-2.0.3-1

oracleasm-support-2.0.3-1

[root@db05 db]# rpm -ev oracleasm-support-2.0.3-1 oracleasm-2.6.9-55.ELsmp-2.0.3-1 oracleasmlib-2.0.2-1

warning: /etc/sysconfig/oracleasm saved as /etc/sysconfig/oracleasm.rpmsave

[root@db05 db]# rm -f /etc/sysconfig/oracleasm.rpmsave

[root@db05 db]# rm -f /etc/rc.d/init.d/oracleasm

[root@db05 db]# rm -f /etc/rc0.d/*oracleasm*

[root@db05 db]# rm -f /etc/rc1.d/*oracleasm*

[root@db05 db]# rm -f /etc/rc2.d/*oracleasm*

[root@db05 db]# rm -f /etc/rc3.d/*oracleasm*

[root@db05 db]# rm -f /etc/rc4.d/*oracleasm*

[root@db05 db]# rm -f /etc/rc5.d/*oracleasm*

[root@db05 db]# rm -f /etc/rc6.d/*oracleasm*

B.         Remove the Node from the Cluster

—————————————-

Once the instance has been deleted, the process of removing the node from the cluster is a manual process. This is accomplished by running scripts on the deleted node to remove the CRS install, as well as scripts on the remaining nodes to update the node list.  The following steps assume that the node to be removed is still functioning.

1. Remove the listener and nodeapp on deleting node:

1). To delete node first stop and remove the nodeapps on the node you are removing.  Assuming that you have removed the ASM instance as the root user on a remaining node;

# srvctl stop nodeapps -n <nodename>

[oracle@db01 dbs]$ srvctl stop nodeapps -n db05

[oracle@db01 dbs]$ crs_stat -t

Name           Type           Target    State     Host

————————————————————

ora.p10c.db    application    ONLINE    ONLINE    db02

ora….c1.inst application    ONLINE    ONLINE    db01

ora….c2.inst application    ONLINE    ONLINE    db02

ora….SM1.asm application    ONLINE    ONLINE   db01

ora….01.lsnr application    ONLINE    ONLINE    db01

ora….b01.gsd application    ONLINE    ONLINE    db01

ora….b01.ons application    ONLINE    ONLINE    db01

ora….b01.vip application    ONLINE    ONLINE    db01

ora….SM2.asm application    ONLINE    ONLINE    db02

ora….02.lsnr application    ONLINE    ONLINE    db02

ora….b02.gsd application    ONLINE    ONLINE    db02

ora….b02.ons application    ONLINE    ONLINE    db02
ora….b02.vip application    ONLINE    ONLINE    db02

ora….05.lsnr application    OFFLINE   OFFLINE

ora….b05.gsd application    OFFLINE   OFFLINE

ora….b05.ons application    OFFLINE   OFFLINE

ora….b05.vip application    OFFLINE   OFFLINE

2). Run netca.  Choose “Cluster Configuration”.

3). Only select the node you are removing and click next.

4). Choose “Listener Configuration” and click next.

5). To delete the listener: Choose “Delete” and delete any listeners configured on the node you are removing.

6).  Run <CRS_HOME>/bin/crs_stat.  Make sure that all database resources are running on nodes that are going to be kept.  For example:

NAME=ora.<db_name>.db

TYPE=application

TARGET=ONLINE

STATE=ONLINE on <node2>

[oracle@db05 db]$ crs_stat -t

Name           Type           Target    State     Host

————————————————————

ora.p10c.db    application    ONLINE    ONLINE    db02

ora….c1.inst application    ONLINE    ONLINE    db01

ora….c2.inst application    ONLINE    ONLINE    db02

ora….SM1.asm application    ONLINE    ONLINE    db01

ora….01.lsnr application    ONLINE    ONLINE   db01

ora….b01.gsd application    ONLINE    ONLINE   db01

ora….b01.ons application    ONLINE    ONLINE   db01

ora….b01.vip application    ONLINE    ONLINE    db01

ora….SM2.asm application    ONLINE    ONLINE   db02

ora….02.lsnr application    ONLINE    ONLINE    db02

ora….b02.gsd application    ONLINE    ONLINE    db02

ora….b02.ons application    ONLINE    ONLINE    db02

ora….b02.vip application    ONLINE    ONLINE    db02

ora….b05.gsd application    OFFLINE   OFFLINE

ora….b05.ons application    OFFLINE   OFFLINE

ora….b05.vip application    OFFLINE   OFFLINE

Ensure that this resource is not running on a node that will be removed (this step hasn’t been done in our case since crs stats shows everything correct).  Use <CRS_HOME>/bin/crs_relocate to perform this.

Example:  crs_relocate ora.<db_name>.db

7).  As the root user, remove the nodeapps on the node you are removing.

# srvctl remove nodeapps -n <nodename>

[root@db05 db]# srvctl remove nodeapps -n db05

Please confirm that you intend to remove the node-level applications on node db05 (y/[n]) y

[root@db05 db]# crs_stat -t

Name           Type           Target    State     Host

————————————————————

ora.p10c.db    application    ONLINE    ONLINE    db02

ora….c1.inst application    ONLINE    ONLINE    db01

ora….c2.inst application    ONLINE    ONLINE    db02

ora….SM1.asm application    ONLINE    ONLINE    db01

ora….01.lsnr application    ONLINE    ONLINE    db01

ora….b01.gsd application    ONLINE    ONLINE    db01

ora….b01.ons application    ONLINE    ONLINE    db01

ora….b01.vip application    ONLINE    ONLINE    db01

ora….SM2.asm application    ONLINE    ONLINE    db02

ora….02.lsnr application    ONLINE    ONLINE    db02

ora….b02.gsd application    ONLINE    ONLINE    db02

ora….b02.ons application    ONLINE    ONLINE    db02

ora….b02.vip application    ONLINE    ONLINE    db02

2. Remove the Oracle Database Software from the Node to be Deleted

1). On node3 make sure you have correct ORACLE_HOME

[oracle@db05 db]$ echo $ORACLE_HOME

/databases/oracle/db

2). Update Node List for Oracle Database Software – (Remove node3):

[oracle@db05 bin]$ export DISPLAY=10.50.133.143:0

[oracle@db05 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME CLUSTER_NODES=”” -local

Starting Oracle Universal Installer…

No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.

The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /databases/oracle/oraInventory

‘UpdateNodeList’ was successful.

Note: Although the OUI does not launch an installer GUI, the DISPLAY  environment variable still needs to be set!

3). De-install Oracle Database Software

Next, run the OUI from the node to be deleted (linux3) to de-install the Oracle Database software. Make certain that you choose the home to be removed and not just the products under that home.

4). Update Node List for Remaining Nodes in the Cluster (on any of remaining nodes)

[oracle@db01 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME “CLUSTER_NODES={vanpgprodb01,vanpgprodb02}”

Starting Oracle Universal Installer…

No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.

The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /databases/oracle/oraInventory

‘UpdateNodeList’ was successful.

3. Remove the Node to be Deleted from Oracle Clusterware

1). Remove Node-Specific Interface Configuration

[oracle@db05 db]$ export ORA_CRS_HOME=/databases/oracle/crs

[oracle@db05 db]$ grep ‘^remoteport’ $ORA_CRS_HOME/opmn/conf/ons.config

remoteport=6200

[oracle@db05 bin]$ ./racgons remove_config vanpgprodb05:6200

racgons: Existing key value on vanpgprodb05 = 4948.

WARNING: db05:6200 does not exist.

[oracle@db05 bin]$ ./racgons remove_config db05:4948

racgons: Existing key value on vanpgprodb05 = 4948.

racgons: db05:4948 removed from OCR.

[oracle@db05 bin]$ ./oifcfg delif -node db05

PROC-4: The cluster registry key to be operated on does not exist.

PRIF-11: cluster registry error

Note: this error has been approved ok.

4. Disable Oracle Clusterware Applications

Running this script will stop the CRS stack and delete the ocr.loc  file on the node to be removed. The nosharedvar option assumes the ocr.loc file is not on a shared file sytem.

While logged into node3 as the root user account, run the following:

[root@vanpgprodb05 install]# ./rootdelete.sh local nosharedvar nosharedhome

CRS-0210: Could not find resource ‘ora.db05.LISTENER_DB05.lsnr’.

CRS-0210: Could not find resource ‘ora.db05.ons’.

CRS-0210: Could not find resource ‘ora.db05.vip’.

CRS-0210: Could not find resource ‘ora.db05.gsd’.

Shutting down Oracle Cluster Ready Services (CRS):

Feb 17 15:24:19.153 | INF | daemon shutting down

Stopping resources. This could take several minutes.

Successfully stopped CRS resources.

Stopping CSSD.

Shutting down CSS daemon.

Shutdown request successfully issued.

Shutdown has begun. The daemons should exit soon.

Checking to see if Oracle CRS stack is down…

Checking to see if Oracle CRS stack is down…

Oracle CRS stack is not running.

Oracle CRS stack is down now.

Removing script for Oracle Cluster Ready services

Updating ocr file for downgrade

Cleaning up SCR settings in ‘/etc/oracle/scls_scr’

5. Delete Node from Cluster and Update OCR

Upon successful completion of the rootdelete.sh script, run the rootdeletenode.sh script to delete the node (linux3) from the Oracle cluster and to update the Oracle Cluster Registry (OCR). This script should be run from a pre-existing / available node in the cluster (node1) as the root user account:

Before executing rootdeletenode.sh, we need to know the node number associated with the node name to be deleted from the cluster. To determine the node number, run the following command as the oracle user account from node1:

[oracle@db01 bin]$ pwd

/databases/oracle/crs/bin

[oracle@db01 bin]$ olsnodes -n

db01    1

db02    2

db05    3

Note: notice the node # from result, we need to use it for removing node.

[root@db01 install]# pwd

/databases/oracle/crs/install

[root@db01 install]# ./rootdeletenode.sh db05,3

CRS-0210: Could not find resource ‘ora.db05.LISTENER_DB05.lsnr’.

CRS-0210: Could not find resource ‘ora.db05.ons’.

CRS-0210: Could not find resource ‘ora.db05.vip’.

CRS-0210: Could not find resource ‘ora.db05.gsd’.

CRS-0210: Could not find resource ora.db05.vip.

CRS nodeapps are deleted successfully

clscfg: EXISTING configuration version 3 detected.

clscfg: version 3 is 10G Release 2.

Successfully deleted 14 values from OCR.

Key SYSTEM.css.interfaces.nodevanpgprodb05 marked for deletion is not there. Ignoring.

Successfully deleted 5 keys from OCR.

Node deletion operation successful.

‘db05,3’ deleted successfully

[root@db01 install]# ../bin/olsnodes -n

db01    1

db02    2

6. Update Node List for Oracle Clusterware Software – (Remove node3)

From Node3 as Oracle user:

[oracle@db05 bin]$ pwd

/databases/oracle/crs/oui/bin

[oracle@db05 bin]$ export ORA_CRS_HOME=/databases/oracle/crs

[oracle@db05 bin]$ export DISPLAY=10.50.133.143:0

[oracle@db05 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=$ORA_CRS_HOME CLUSTER_NODES=”” -local CRS=true

Starting Oracle Universal Installer…

No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.

The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /databases/oracle/oraInventory

‘UpdateNodeList’ was successful.

7. De-install Oracle Clusterware Software

[oracle@db05 bin]$ pwd

/databases/oracle/crs/oui/bin

[oracle@db05 bin]$ ./runInstaller

After delete the crs software, the directory will be deleted as well.

8. Update Node List for Remaining Nodes in the Cluster

[oracle@db01 bin]$ export ORA_CRS_HOME=/databases/oracle/crs

[oracle@db01 bin]$ export DISPLAY=10.50.133.143:0

[oracle@db01 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=$ORA_CRS_HOME “CLUSTER_NODES={db01,db02}” CRS=true

Starting Oracle Universal Installer…

No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.

The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /databases/oracle/oraInventory

‘UpdateNodeList’ was successful.

C.         Reconfigure the OS and remaining hardware.

————————————————-

1. Check the tnsnames.ora on the rest of nodes if exists.

2. Delete oracle_home and crs_home

3. Next, as root, from the deleted node, verify that all init scripts and soft links are removed:

For Linux:

rm -f /etc/init.d/init.cssd

rm -f /etc/init.d/init.crs

rm -f /etc/init.d/init.crsd

rm -f /etc/init.d/init.evmd

rm -f /etc/rc2.d/K96init.crs

rm -f /etc/rc2.d/S96init.crs

rm -f /etc/rc3.d/K96init.crs

rm -f /etc/rc3.d/S96init.crs

rm -f /etc/rc5.d/K96init.crs

rm -f /etc/rc5.d/S96init.crs

rm -Rf /etc/oracle

rm -rf /etc/ora.tab

Reference:

1. Removing a Node from an Oracle RAC 10g Release 2 Cluster on Linux – (CentOS 4.5 / iSCSI)

by Jeff Hunter http://www.idevelopment.info/data/Oracle/DBA_tips/Oracle10gRAC/CLUSTER_23.shtml

2. How to delete a node from 3 node RAC in 10GR2

http://www.oraclefaq.net/2007/06/21/how-to-delete-a-node-from-3-node-rac-in-10gr2/

3. 10 Adding and Deleting Nodes and Instances on UNIX-Based Systems

Oracle® Database Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide 10g Release 2 (10.2) Part Number B14197-09

http://download.oracle.com/docs/cd/B19306_01/rac.102/b14197/adddelunix.htm#BEIJAJHH

4. Removing a Node from a 10g RAC Cluster  Doc ID: 269320.1

https://metalink2.oracle.com/metalink/plsql/f?p=130:14:4196773229713543167::::p14_database_id,p14_docid,p14_show_header,p14_show_help,p14_black_frame,p14_font:NOT,269320.1,1,1,1,helvetica

Advertisements

February 9, 2009

error for dynamic sql on sql server

Filed under: 3. MS SQL Server — zhefeng @ 12:55 pm

i made a Stored procedure for dropping tables older than 6 months, it was running ok by failed recently and the error is:
Procedure expects parameter ‘@statement’ of type ‘ntext/nchar/nvarchar’

Here is the STP itself:
USE [OLAPPERF]
GO
/****** Object: StoredProcedure [dbo].[stp_removetbls] Script Date: 02/09/2009 12:52:04 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO

ALTER PROCEDURE [dbo].[stp_removetbls]
— Add the parameters for the stored procedure here
AS
BEGIN
SET NOCOUNT ON

DECLARE @table_name varchar( 128 )

DECLARE cur_tablenames CURSOR FOR select name from olapperf.sys.all_objects where type = ‘U’ and create_date<dateadd(month,-6,getdate())

DECLARE @sqlstring varchar(1000)

OPEN cur_tablenames
FETCH NEXT FROM cur_tablenames INTO @table_name
WHILE @@FETCH_STATUS = 0
BEGIN

SET @sqlstring = ‘drop table ‘+ @table_name

exec sp_executesql @sqlstring

FETCH NEXT FROM cur_tablenames INTO @table_name
END

CLOSE cur_tablenames
DEALLOCATE cur_tablenames
END

After searching the solution i found it is very easy, like here said(http://dotnetjunkies.com/WebLog/richard.dudley/archive/2004/09/29/27169.aspx):
If you are building dynamic SQL, you should declare your parameter as one of the accepted types (e.g., declare @sql nvarchar(4000)). If you are using a direct statement as your query, you need to preface it with the letter N, as seen below.

incorrect:
execute sp_executesql ‘select * from pubs.dbo.employee where job_lvl = @level’,
‘@level tinyint’,
@level = 35

correct:
execute sp_executesql N’select * from pubs.dbo.employee where job_lvl = @level’,
N’@level tinyint’,
@level = 35

So what i did is just change my sqlstring definition from varchar(1000) to nvarchar(1000), done! Everything works again.

February 2, 2009

Change IP address for oracle RAC public and VIP interfaces

Filed under: [RAC] — Tags: — zhefeng @ 4:33 pm

My company is doing massive re-ip project. So our Oracle RAC ip address has to be changed as well. Fortunately, we don’t need to change hostname, otherwise the story will be more complicated.

1. Current Environment
1). Machine IP:
Node1: vmrac01
Node2: vmrac02

## eth1-Public
10.50.96.101 vmrac01 vmrac01.test.com
10.50.96.102 vmrac02 vmrac02.test.com
## eth0-Private
192.168.199.1 vmracprv01 vmracprv01.test.com
192.168.199.2 vmracprv02 vmracprv02.test.com
## VIP
10.50.96.103 vmracvip01 vmracvip01.test.com
10.50.96.104 vmracvip02 vmracvip02.test.com

2). cluster information:
cluster name — vm10cls
database name — v10c
Instance 1 — v10c1
Instance 2 — v10c2
Node1 — vmrac01
Node2 — vmrac02

2. New IP changing map(different subnet mask too):
10.50.96.101/255.255.255.0 vmrac01 –> 10.50.99.41/255.255.252.0
10.50.96.102/255.255.255.0 vmrac02 –> 10.50.99.42/255.255.252.0
10.50.96.103/255.255.255.0 vmracvip01 –> 10.50.99.43/255.255.252.0
10.50.96.104/255.255.255.0 vmracvip02 –> 10.50.99.44/255.255.252.0

3. steps 1 — change RAC IP settings
1). bring service down, make sure everything was offline except css daemon
bash-3.1$ srvctl stop database -d v10c
bash-3.1$ srvctl stop nodeapps -n vmrac01
bash-3.1$ srvctl stop nodeapps -n vmrac02
bash-3.1$ crs_stat -t
Name Type Target State Host
————————————————————
ora.v10c.db application OFFLINE OFFLINE
ora….c1.inst application OFFLINE OFFLINE
ora….c2.inst application OFFLINE OFFLINE
ora….SM1.asm application OFFLINE OFFLINE
ora….01.lsnr application OFFLINE OFFLINE
ora….c01.gsd application OFFLINE OFFLINE
ora….c01.ons application OFFLINE OFFLINE
ora….c01.vip application OFFLINE OFFLINE
ora….SM2.asm application OFFLINE OFFLINE
ora….02.lsnr application OFFLINE OFFLINE
ora….c02.gsd application OFFLINE OFFLINE
ora….c02.ons application OFFLINE OFFLINE
ora….c02.vip application OFFLINE OFFLINE

2). backup OCR and Voting disks
bash-3.1$ ocrcheck|grep -i file
Device/File Name : /dev/raw/raw1
bash-3.1$ crsctl query css votedisk
0. 0 /dev/raw/raw2
located 1 votedisk(s).

#dd if=/dev/raw/raw1 of=/database/temp/ocr_vote_bk/ocr.bak
#dd if=/dev/raw/raw2 of=/database/temp/ocr_vote_bk/vote.bak

3). get current config:
bash-3.1$ oifcfg getif
eth0 10.50.96.0 global public –current network for public
eth1 192.168.199.0 global cluster_interconnect –we are not going to change this

4). delete current public ip:
bash-3.1$ oifcfg delif -global eth0

5). change to new network:
bash-3.1$ oifcfg setif -global eth0/10.50.99.0:public

6). change vip address:
a. check current one
bash-3.1$ srvctl config nodeapps -n vmrac01 -a
VIP exists.: /vmracvip01/10.50.96.103/255.255.255.0/eth0
bash-3.1$ srvctl config nodeapps -n vmrac02 -a
VIP exists.: /vmracvip02/10.50.96.104/255.255.255.0/eth0
b. Modify VIP component (has to be the css owner, “root” usually)
#srvctl modify nodeapps -n vmrac01 -A 10.50.99.43/255.255.252.0/eth0
#srvctl modify nodeapps -n vmrac02 -A 10.50.99.44/255.255.252.0/eth0
c. double verify the changes
bash-3.1$ srvctl config nodeapps -n vmrac01 -a
VIP exists.: /vmracvip01/10.50.99.43/255.255.252.0/eth0
bash-3.1$ srvctl config nodeapps -n vmrac02 -a
VIP exists.: /vmracvip02/10.50.99.44/255.255.252.0/eth0

7). change the hosts file(on both nodes):
## eth1-Public
10.50.99.41 vmrac01 vmrac01.test.com
10.50.99.42 vmrac02 vmrac02.test.com
## eth0-Private
192.168.199.1 vmracprv01 vmracprv01.test.com
192.168.199.2 vmracprv02 vmracprv02.test.com
## VIP
10.50.99.43 vmracvip01 vmracvip01.test.com
10.50.99.44 vmracvip02 vmracvip02.test.com

8). if the listener is using any IP address, it also needs to be changed.

4. Steps 2 — change OS IP settings
1). change IP
[root@vmrac01]# vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=static
IPADDR=10.50.99.41
NETMASK=255.255.252.0
HWADDR=00:50:56:BD:05:14
ONBOOT=yes

[root@vmrac02]# vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=static
IPADDR=10.50.99.42
NETMASK=255.255.252.0
HWADDR=00:50:56:BD:3E:08
ONBOOT=yes

2). change the default gateway on both nodes (if needed, here since they are in same vlan so i didn’t change them)
[root@vmrac01 ~]# cat /etc/sysconfig/network
NETWORKING_IPV6=yes
HOSTNAME=vmrac01
NETWORKING=yes
NISDOMAIN=nis
GATEWAY=10.50.96.1 <– here is the default gateway to be changed

3) Change the IP Address’es on the known_hosts ssh config files for oracle user
$ su – oracle
$ cd .ssh
$ cp known_hosts known_hosts.bak
$ modify the old IP’s to the new IP’s

4). restart network (on both node)
#service network restart

5). restart crs daemon (on both node)
#crsctl stop crs
#crsctl start crs

5. Step3 — verify everything

reference:
1. "How to Change Interconnect/Public Interface IP or Subnet in Oracle Clusterware", Doc ID: 283684.1
2. "Modifying the VIP or VIP Hostname of a 10g or 11g Oracle Clusterware Node", DOC ID: 276434.1
3. "How to change Public and VIP component address in case of RAC?" http://orcl-experts.info/index.php?name=FAQ&id_cat=9

Blog at WordPress.com.