Search This Blog

Tuesday, March 28, 2023

Oracle dataguard errors : ORA-16853 / ORA-16853:

 













I was getting below error when I ran the 'show configuration' command from the Datagaurd broker.

DGMGRL>  show configuration;

Configuration - renodbdg
  Protection Mode: MaxPerformance
  Members:
  renodbpr - Primary database
    renodbdr - Physical standby database
      Warning: ORA-16809: multiple warnings detected for the member
Fast-Start Failover:  Disabled
Configuration Status:
WARNING   
DGMGRL> 

I ran the 'show database ' command for the standby database and found below errors. 

DGMGRL>  show database renodbdr
Database - renodbdr

  Role:               PHYSICAL STANDBY
  Intended State:     APPLY-ON
  Transport Lag:      1 hour(s) 14 minutes 35 seconds (computed 0 seconds ago)
  Apply Lag:          1 hour(s) 15 minutes (computed 0 seconds ago)
  Average Apply Rate: 2.83 MByte/s
  Real Time Query:    ON
  Instance(s):
    renodbdr1 (apply instance)
    renodbdr2
  Database Warning(s):
    ORA-16853: apply lag has exceeded specified threshold
    ORA-16855: transport lag has exceeded specified threshold

Database Status:
WARNING
DGMGRL> 

Cause : The standby database is RAC database and I see that the SRL logs were only created for one instance.







As we can see from the above, all the Standby redo logs were created for the first instance. 

Our Prim,ary database has 2 redo logs for each node. I will drop the last three standby redo logs from the standby database and will re-create them for the thread 2.

1. Stop the MRP process

SQL> ALTER DATABASE RECOVER  managed standby database cancel;
Database altered.

2. Drop the unwanted standby redo logs.

SQL> alter database drop standby logfile group 8;
Database altered.
SQL> alter database drop standby logfile group 9;
Database altered.
SQL> alter database drop standby logfile group 10;
Database altered.

3) Add the SRLs for the thread 2.

SQL> alter database add standby logfile thread 2 group 8 size 200M;
Database altered.
SQL> alter database add standby logfile thread 2 group 9  size 200M;
Database altered.
SQL> alter database add standby logfile thread 2 group 10  size 200M;
Database altered.

Check the database to see if the SRLs shows correct








4. Start the MRP process

SQL> ALTER DATABASE RECOVER  managed standby database using current logfile disconnect;
Database altered.

5. Check the configuration in the dataguard broker.

DGMGRL for Linux: Release 19.0.0.0.0 - Production on Tue Mar 28 10:51:52 2023Version 19.17.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.
Welcome to DGMGRL, type "help" for information.
Connected to "renodbpr"
Connected as SYSDG.
DGMGRL> connect sys/welcome;
Connected to "renodbpr"
Connected as SYSDBA.
DGMGRL> show configuration;
Configuration - renodbdg
  Protection Mode: MaxPerformance
  Members:
  renodbpr - Primary database
    renodbdr - Physical standby database
Fast-Start Failover:  Disabled
Configuration Status:
SUCCESS   (status updated 26 seconds ago)

DGMGRL> 

that fixed the issue and the configuration looks good in the Dataguard broker.

hope this helps you if you are seeing the same issue :)

Thanks
Sambaiah Sammeta

Oracle Grid and Database RU patch roll-back process

 


In this post, we will see how to roll back the 19.18 RU to 19.17 from the Grid and Database homes.

For this, we will use the below setup with 19c RAC environment for the primary and Standby environments.

Primary Source Environment setup

RAC Database    : RENODBPR ( renodbpr1 & renodbpr2)
GRID Home        : /u01/app/19.3.0.0/grid
RDBMS Home     : /u01/app/oracle/product/19.3.0.0/db_1
Version              : Oracle Database 19c EE - Production Version 19.17.0.0.0
hosts                 : labhost01
                          labhost02

Standby database environment setup
RAC Database    : RENODBDR ( renodbdr1 & renodbdr2)
GRID Home        : /u01/app/19.3.0.0/grid
RDBMS Home    : /u01/app/oracle/product/19.3.0.0/dbhome_1
Version             : Oracle Database 19c EE - Production Version 19.17.0.0.0
hosts                  : labdrhost01
                              labdrhost02

Both the primary database and standby database have 19.18 RU applied.

Primary database patch set level.
[oracle@labhost01 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
34768559;OCW RELEASE UPDATE 19.18.0.0.0 (34768559)
34765931;DATABASE RELEASE UPDATE : 19.18.0.0.230117 (REL-JAN230131) (34765931)
OPatch succeeded.
[oracle@labhost01 ~]$

Standby database patchset level.
[oracle@labdrhost01 trace]$ cd
[oracle@labdrhost01 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
34768559;OCW RELEASE UPDATE 19.18.0.0.0 (34768559)
34765931;DATABASE RELEASE UPDATE : 19.18.0.0.230117 (REL-JAN230131) (34765931)
OPatch succeeded.
 [oracle@labdrhost01 ~]$

Patch roll-back process.

1.  Make sure that the standby database is in sync with the primary database .

2. download the 19.18 RU and stage it on the server.

    Patch 34762026 - GI Release Update 19.18.0.0.230117 (REL-JAN230131)

3. Rollback the patch 

As root user, we can rollback the patch using opatchauto. 

We should rollback the patch in the Standby database servers first and then roll-back it from the Primary database servers.

export ORACLE_HOME=/u01/app/19.3.0.0/grid
export PATH=$ORACLE_HOME/bin:$PATH
$ORACLE_HOME/OPatch/opatchauto rollback /u01/software/334762026

Rollback step on the node 1 of the Standby database server.

[root@labdrhost01 34762026]#  $ORACLE_HOME/OPatch/opatchauto rollback     /u01/software/34762026
   
OPatchauto session is initiated at Mon Mar 27 14:06:00 2023
 
System initialization log file is /u01/app/19.3.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2023-   03-27_02-06-02PM.log.
   
Session log file is /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/opatchauto2023-03-27_02-06-   25PM.log
The id for this session is JU2B
    
Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.3.0.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.3.0.0/dbhome_1
Patch applicability verified successfully on home /u01/app/19.3.0.0/grid

Patch applicability verified successfully on home /u01/app/oracle/product/19.3.0.0/dbhome_1

Executing patch validation checks on home /u01/app/19.3.0.0/grid
Patch validation checks successfully completed on home /u01/app/19.3.0.0/grid

Executing patch validation checks on home /u01/app/oracle/product/19.3.0.0/dbhome_1
Patch validation checks successfully completed on home /u01/app/oracle/product/19.3.0.0/dbhome_1

Verifying SQL patch applicability on home /u01/app/oracle/product/19.3.0.0/dbhome_1
Skipping SQL patch step execution on standby database : renodbdr
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.3.0.0/dbhome_1

Preparing to bring down database service on home /u01/app/oracle/product/19.3.0.0/dbhome_1
Successfully prepared home /u01/app/oracle/product/19.3.0.0/dbhome_1 to bring down database service

Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/19.3.0.0/grid
Prepatch operation log file location: /u01/app/oracle/crsdata/labdrhost01/crsconfig/crs_prepatch_rollback_inplace_labdrhost01_2023-03-27_02-09-31PM.log
CRS service brought down successfully on home /u01/app/19.3.0.0/grid


Performing prepatch operation on home /u01/app/oracle/product/19.3.0.0/dbhome_1
Prepatch operation completed successfully on home /u01/app/oracle/product/19.3.0.0/dbhome_1

Start rolling back binary patch on home /u01/app/oracle/product/19.3.0.0/dbhome_1
Binary patch rolled back successfully on home /u01/app/oracle/product/19.3.0.0/dbhome_1

Performing postpatch operation on home /u01/app/oracle/product/19.3.0.0/dbhome_1
Postpatch operation completed successfully on home /u01/app/oracle/product/19.3.0.0/dbhome_1

Start rolling back binary patch on home /u01/app/19.3.0.0/grid
Binary patch rolled back successfully on home /u01/app/19.3.0.0/grid

Performing postpatch operations on CRS - starting CRS service on home /u01/app/19.3.0.0/grid
Postpatch operation log file location: /u01/app/oracle/crsdata/labdrhost01/crsconfig/crs_postpatch_rollback_inplace_labdrhost01_2023-03-27_02-21-43PM.log
CRS service started successfully on home /u01/app/19.3.0.0/grid

Preparing home /u01/app/oracle/product/19.3.0.0/dbhome_1 after database service restarted
No step execution required.........

Trying to roll back SQL patch on home /u01/app/oracle/product/19.3.0.0/dbhome_1
Skipping SQL patch step execution on standby database : renodbdr
No sqlpatch operations are required on the local node for this home
SQL patch rolled back successfully on home /u01/app/oracle/product/19.3.0.0/dbhome_1

OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:
Host:labdrhost01
RAC Home:/u01/app/oracle/product/19.3.0.0/dbhome_1
Version:19.0.0.0.0
Summary:
==Following patches were SKIPPED:
Patch: /u01/software/34762026/34768569
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /u01/software/34762026/33575402
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /u01/software/34762026/34863894
Reason: This patch is not applicable to this specified target type - "rac_database"

==Following patches were SUCCESSFULLY rolled back:

Patch: /u01/software/34762026/34768559
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2023-03-27_14-14-28PM_1.log
Patch: /u01/software/34762026/34765931
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2023-03-27_14-14-28PM_1.log
Host:labdrhost01
CRS Home:/u01/app/19.3.0.0/grid
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /u01/software/34762026/33575402
Reason: Patch /u01/software/34762026/33575402 is not applied as part of bundle patch 34762026

==Following patches were SUCCESSFULLY rolled back:

Patch: /u01/software/34762026/34768559
Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2023-03-27_14-18-13PM_1.log

Patch: /u01/software/34762026/34768569
Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2023-03-27_14-18-13PM_1.log

Patch: /u01/software/34762026/34863894
Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2023-03-27_14-18-13PM_1.log

Patch: /u01/software/34762026/34765931
Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2023-03-27_14-18-13PM_1.log

OPatchauto session completed at Mon Mar 27 14:24:38 2023
Time taken to complete the session 18 minutes, 38 seconds
[root@labdrhost01 34762026]# 

Rolling back the patch on the 2nd node of the Standby database server.


[root@labdrhost02 ~]# export ORACLE_HOME=/u01/app/19.3.0.0/grid
[root@labdrhost02 ~]# export PATH=$ORACLE_HOME/bin:$PATH
[root@labdrhost02 ~]# $ORACLE_HOME/OPatch/opatchauto rollback /u01/software/34762026

OPatchauto session is initiated at Mon Mar 27 14:24:56 2023

System initialization log file is /u01/app/19.3.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2023-03-27_02-24-58PM.log.

Session log file is /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/opatchauto2023-03-27_02-25-22PM.log
The id for this session is NAGH

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.3.0.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.3.0.0/dbhome_1
Patch applicability verified successfully on home /u01/app/19.3.0.0/grid

Patch applicability verified successfully on home /u01/app/oracle/product/19.3.0.0/dbhome_1

Executing patch validation checks on home /u01/app/19.3.0.0/grid
Patch validation checks successfully completed on home /u01/app/19.3.0.0/grid

Executing patch validation checks on home /u01/app/oracle/product/19.3.0.0/dbhome_1
Patch validation checks successfully completed on home /u01/app/oracle/product/19.3.0.0/dbhome_1

Verifying SQL patch applicability on home /u01/app/oracle/product/19.3.0.0/dbhome_1
Skipping SQL patch step execution on standby database : renodbdr
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.3.0.0/dbhome_1

Preparing to bring down database service on home /u01/app/oracle/product/19.3.0.0/dbhome_1
Successfully prepared home /u01/app/oracle/product/19.3.0.0/dbhome_1 to bring down database service

Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/19.3.0.0/grid
Prepatch operation log file location: /u01/app/oracle/crsdata/labdrhost02/crsconfig/crs_prepatch_rollback_inplace_labdrhost02_2023-03-27_02-28-42PM.log
CRS service brought down successfully on home /u01/app/19.3.0.0/grid

Performing prepatch operation on home /u01/app/oracle/product/19.3.0.0/dbhome_1
Prepatch operation completed successfully on home /u01/app/oracle/product/19.3.0.0/dbhome_1

Start rolling back binary patch on home /u01/app/oracle/product/19.3.0.0/dbhome_1
Binary patch rolled back successfully on home /u01/app/oracle/product/19.3.0.0/dbhome_1

Performing postpatch operation on home /u01/app/oracle/product/19.3.0.0/dbhome_1
Postpatch operation completed successfully on home /u01/app/oracle/product/19.3.0.0/dbhome_1

Start rolling back binary patch on home /u01/app/19.3.0.0/grid
Binary patch rolled back successfully on home /u01/app/19.3.0.0/grid

Performing postpatch operations on CRS - starting CRS service on home /u01/app/19.3.0.0/grid
Postpatch operation log file location: /u01/app/oracle/crsdata/labdrhost02/crsconfig/crs_postpatch_rollback_inplace_labdrhost02_2023-03-27_02-41-07PM.log
CRS service started successfully on home /u01/app/19.3.0.0/grid

Preparing home /u01/app/oracle/product/19.3.0.0/dbhome_1 after database service restarted
No step execution required.........

Trying to roll back SQL patch on home /u01/app/oracle/product/19.3.0.0/dbhome_1
Skipping SQL patch step execution on standby database : renodbdr
No sqlpatch operations are required on the local node for this home
SQL patch rolled back successfully on home /u01/app/oracle/product/19.3.0.0/dbhome_1

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:labdrhost02
RAC Home:/u01/app/oracle/product/19.3.0.0/dbhome_1
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:
Patch: /u01/software/34762026/34768569
Reason: This patch is not applicable to this specified target type - "rac_database"
Patch: /u01/software/34762026/33575402
Reason: This patch is not applicable to this specified target type - "rac_database"
Patch: /u01/software/34762026/34863894
Reason: This patch is not applicable to this specified target type - "rac_database"

==Following patches were SUCCESSFULLY rolled back:

Patch: /u01/software/34762026/34768559
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2023-03-27_14-33-36PM_1.log

Patch: /u01/software/34762026/34765931
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2023-03-27_14-33-36PM_1.log

Host:labdrhost02
CRS Home:/u01/app/19.3.0.0/grid
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /u01/software/34762026/33575402
Reason: Patch /u01/software/34762026/33575402 is not applied as part of bundle patch 34762026

==Following patches were SUCCESSFULLY rolled back:

Patch: /u01/software/34762026/34768559
Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2023-03-27_14-37-25PM_1.log

Patch: /u01/software/34762026/34768569
Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2023-03-27_14-37-25PM_1.log

Patch: /u01/software/34762026/34863894
Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2023-03-27_14-37-25PM_1.log

Patch: /u01/software/34762026/34765931
Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2023-03-27_14-37-25PM_1.log

OPatchauto session completed at Mon Mar 27 14:44:10 2023
Time taken to complete the session 19 minutes, 15 seconds
[root@labdrhost02 ~]#


Rolling back the patch on the First node of the Primary database server.

[root@labhost01 ~]#  export ORACLE_HOME=/u01/app/19.3.0.0/grid
[root@labhost01 ~]#  export PATH=$ORACLE_HOME/bin:$PATH
[root@labhost01 ~]# [root@labhost01 ~]# $ORACLE_HOME/OPatch/opatchauto rollback /u01/software/34762026

OPatchauto session is initiated at Mon Mar 27 14:24:58 2023

System initialization log file is /u01/app/19.3.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2023-03-27_02-25-00PM.log.

Session log file is /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/opatchauto2023-03-27_02-25-22PM.log
The id for this session is FW7Q

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.3.0.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.3.0.0/db_1
Patch applicability verified successfully on home /u01/app/19.3.0.0/grid

Patch applicability verified successfully on home /u01/app/oracle/product/19.3.0.0/db_1

Executing patch validation checks on home /u01/app/19.3.0.0/grid
Patch validation checks successfully completed on home /u01/app/19.3.0.0/grid

Executing patch validation checks on home /u01/app/oracle/product/19.3.0.0/db_1
Patch validation checks successfully completed on home /u01/app/oracle/product/19.3.0.0/db_1

Verifying SQL patch applicability on home /u01/app/oracle/product/19.3.0.0/db_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.3.0.0/db_1

Preparing to bring down database service on home /u01/app/oracle/product/19.3.0.0/db_1
Successfully prepared home /u01/app/oracle/product/19.3.0.0/db_1 to bring down database service

Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/19.3.0.0/grid
Prepatch operation log file location: /u01/app/oracle/crsdata/labhost01/crsconfig/crs_prepatch_rollback_inplace_labhost01_2023-03-27_02-29-52PM.log
CRS service brought down successfully on home /u01/app/19.3.0.0/grid


Performing prepatch operation on home /u01/app/oracle/product/19.3.0.0/db_1
Prepatch operation completed successfully on home /u01/app/oracle/product/19.3.0.0/db_1

Start rolling back binary patch on home /u01/app/oracle/product/19.3.0.0/db_1
Binary patch rolled back successfully on home /u01/app/oracle/product/19.3.0.0/db_1

Performing postpatch operation on home /u01/app/oracle/product/19.3.0.0/db_1
Postpatch operation completed successfully on home /u01/app/oracle/product/19.3.0.0/db_1

Start rolling back binary patch on home /u01/app/19.3.0.0/grid
Binary patch rolled back successfully on home /u01/app/19.3.0.0/grid

Performing postpatch operations on CRS - starting CRS service on home /u01/app/19.3.0.0/grid
Postpatch operation log file location: /u01/app/oracle/crsdata/labhost01/crsconfig/crs_postpatch_rollback_inplace_labhost01_2023-03-27_02-42-19PM.log
CRS service started successfully on home /u01/app/19.3.0.0/grid


Preparing home /u01/app/oracle/product/19.3.0.0/db_1 after database service restarted
No step execution required.........


Trying to roll back SQL patch on home /u01/app/oracle/product/19.3.0.0/db_1
No sqlpatch operations are required on the local node for this home
SQL patch rolled back successfully on home /u01/app/oracle/product/19.3.0.0/db_1

OPatchAuto successful.
--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:labhost01
RAC Home:/u01/app/oracle/product/19.3.0.0/db_1
Version:19.0.0.0.0
Summary:
==Following patches were SKIPPED:

Patch: /u01/software/34762026/34768569
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /u01/software/34762026/33575402
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /u01/software/34762026/34863894
Reason: This patch is not applicable to this specified target type - "rac_database"

==Following patches were SUCCESSFULLY rolled back:

Patch: /u01/software/34762026/34768559
Log: /u01/app/oracle/product/19.3.0.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2023-03-27_14-34-52PM_1.log
Patch: /u01/software/34762026/34765931
Log: /u01/app/oracle/product/19.3.0.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2023-03-27_14-34-52PM_1.log

Host:labhost01
CRS Home:/u01/app/19.3.0.0/grid
Version:19.0.0.0.0
Summary:
==Following patches were SKIPPED:

Patch: /u01/software/34762026/33575402
Reason: Patch /u01/software/34762026/33575402 is not applied as part of bundle patch 34762026

==Following patches were SUCCESSFULLY rolled back:

Patch: /u01/software/34762026/34768559
Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2023-03-27_14-38-34PM_1.log
Patch: /u01/software/34762026/34768569
Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2023-03-27_14-38-34PM_1.log
Patch: /u01/software/34762026/34863894
Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2023-03-27_14-38-34PM_1.log
Patch: /u01/software/34762026/34765931
Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2023-03-27_14-38-34PM_1.log

OPatchauto session completed at Mon Mar 27 14:45:24 2023
Time taken to complete the session 20 minutes, 26 seconds
[root@labhost01 ~]# 


Rolling back the patch on the second node of the Primary database server.

[root@labhost02 ~]#  export ORACLE_HOME=/u01/app/19.3.0.0/grid
[root@labhost02 ~]#  export PATH=$ORACLE_HOME/bin:$PATH
[root@labhost02 ~]# $ORACLE_HOME/OPatch/opatchauto rollback /u01/software/34762026

OPatchauto session is initiated at Mon Mar 27 14:59:19 2023

System initialization log file is /u01/app/19.3.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2023-03-27_02-59-25PM.log.

Session log file is /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/opatchauto2023-03-27_02-59-48PM.log
The id for this session is CF35

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.3.0.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.3.0.0/db_1
Patch applicability verified successfully on home /u01/app/19.3.0.0/grid

Patch applicability verified successfully on home /u01/app/oracle/product/19.3.0.0/db_1

Executing patch validation checks on home /u01/app/19.3.0.0/grid
Patch validation checks successfully completed on home /u01/app/19.3.0.0/grid

Executing patch validation checks on home /u01/app/oracle/product/19.3.0.0/db_1
Patch validation checks successfully completed on home /u01/app/oracle/product/19.3.0.0/db_1

Verifying SQL patch applicability on home /u01/app/oracle/product/19.3.0.0/db_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.3.0.0/db_1


Preparing to bring down database service on home /u01/app/oracle/product/19.3.0.0/db_1
Successfully prepared home /u01/app/oracle/product/19.3.0.0/db_1 to bring down database service

Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/19.3.0.0/grid
Prepatch operation log file location: /u01/app/oracle/crsdata/labhost02/crsconfig/crs_prepatch_rollback_inplace_labhost02_2023-03-27_03-03-11PM.log
CRS service brought down successfully on home /u01/app/19.3.0.0/grid

Performing prepatch operation on home /u01/app/oracle/product/19.3.0.0/db_1
Prepatch operation completed successfully on home /u01/app/oracle/product/19.3.0.0/db_1

Start rolling back binary patch on home /u01/app/oracle/product/19.3.0.0/db_1
Binary patch rolled back successfully on home /u01/app/oracle/product/19.3.0.0/db_1

Performing postpatch operation on home /u01/app/oracle/product/19.3.0.0/db_1
Postpatch operation completed successfully on home /u01/app/oracle/product/19.3.0.0/db_1

Start rolling back binary patch on home /u01/app/19.3.0.0/grid
Binary patch rolled back successfully on home /u01/app/19.3.0.0/grid

Performing postpatch operations on CRS - starting CRS service on home /u01/app/19.3.0.0/grid
Postpatch operation log file location: /u01/app/oracle/crsdata/labhost02/crsconfig/crs_postpatch_rollback_inplace_labhost02_2023-03-27_03-15-17PM.log
CRS service started successfully on home /u01/app/19.3.0.0/grid


Preparing home /u01/app/oracle/product/19.3.0.0/db_1 after database service restarted
No step execution required.........


Trying to roll back SQL patch on home /u01/app/oracle/product/19.3.0.0/db_1
SQL patch rolled back successfully on home /u01/app/oracle/product/19.3.0.0/db_1
OPatchAuto successful.
--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:labhost02
RAC Home:/u01/app/oracle/product/19.3.0.0/db_1
Version:19.0.0.0.0
Summary:
==Following patches were SKIPPED:

Patch: /u01/software/34762026/34768569
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /u01/software/34762026/33575402
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /u01/software/34762026/34863894
Reason: This patch is not applicable to this specified target type - "rac_database"

==Following patches were SUCCESSFULLY rolled back:

Patch: /u01/software/34762026/34768559
Log: /u01/app/oracle/product/19.3.0.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2023-03-27_15-08-03PM_1.log

Patch: /u01/software/34762026/34765931
Log: /u01/app/oracle/product/19.3.0.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2023-03-27_15-08-03PM_1.log

Host:labhost02
CRS Home:/u01/app/19.3.0.0/grid
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /u01/software/34762026/33575402
Reason: Patch /u01/software/34762026/33575402 is not applied as part of bundle patch 34762026

==Following patches were SUCCESSFULLY rolled back:

Patch: /u01/software/34762026/34768559
Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2023-03-27_15-11-44PM_1.log

Patch: /u01/software/34762026/34768569
Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2023-03-27_15-11-44PM_1.log

Patch: /u01/software/34762026/34863894
Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2023-03-27_15-11-44PM_1.log

Patch: /u01/software/34762026/34765931
Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2023-03-27_15-11-44PM_1.log

OPatchauto session completed at Mon Mar 27 15:22:15 2023
Time taken to complete the session 22 minutes, 57 seconds
[root@labhost02 ~]#

4. Run the datapatch for each of the primary database. In our case, we will run the datapatch on node 1 of primary database for the database, renodbpr.

[oracle@labhost01 OPatch]$ ./datapatch -verbose
SQL Patching tool version 19.17.0.0.0 Production on Mon Mar 27 16:29:16 2023
Copyright (c) 2012, 2022, Oracle.  All rights reserved.

Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_10770_2023_03_27_16_29_16/sqlpatch_invocation.log

Connecting to database...OK
Gathering database info...done

Note:  Datapatch will only apply or rollback SQL fixes for PDBs
       that are in an open state, no patches will be applied to closed PDBs.
       Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
       (Doc ID 1585822.1)

Bootstrapping registry and package to current versions...done
Determining current state...done

Current state of interim SQL patches:
  No interim patches found

Current state of release update SQL patches:
  Binary registry:
    19.17.0.0.0 Release_Update 220924224051: Installed
  PDB CDB$ROOT:
    Rolled back to 19.17.0.0.0 Release_Update 220924224051 successfully on 27-MAR-23 03.22.10.246894 PM
  PDB ONEPDB:
    Applied 19.17.0.0.0 Release_Update 220924224051 successfully on 18-JAN-23 09.02.58.264135 PM
  PDB PDB$SEED:
    Rolled back to 19.17.0.0.0 Release_Update 220924224051 successfully on 27-MAR-23 03.22.10.530825 PM

Adding patches to installation queue and performing prereq checks...done
Installation queue:
  For the following PDBs: CDB$ROOT PDB$SEED ONEPDB
    No interim patches need to be rolled back
    No release update patches need to be installed
    No interim patches need to be applied

SQL Patching tool complete on Mon Mar 27 16:29:57 2023
[oracle@labhost01 OPatch]$

5. Check the lsinventory or lapatches for each node

[oracle@labhost02 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
34580338;TOMCAT RELEASE UPDATE 19.0.0.0.0 (34580338)
34444834;OCW RELEASE UPDATE 19.17.0.0.0 (34444834)
34428761;ACFS RELEASE UPDATE 19.17.0.0.0 (34428761)
34419443;Database Release Update : 19.17.0.0.221018 (34419443)
33575402;DBWLM RELEASE UPDATE 19.0.0.0.0 (33575402)
OPatch succeeded.
[oracle@labhost02 ~]$

6.For each database or each PDB (if we have) we can run the utlrp if we see any invalid objects.

7. check the standby database to ensure that they are getting logs applied and it is in SYNC with primary database.

that's it, we completed roll-back of 19.18 PSU on RAC database with Standby database configuration.

Hope this document helps you.

Thanks
Sambaiah Sammeta