12c Data Guard : Resolve Log gap Error: ORA-16724: “cannot resolve gap for one or more standby databases”

This error happened while my Lab standby server was unreachable for few days. Although my setup was built on top of a Windows environment, the resolution process stays the same here for fixing this error (using an RMAN incremental backup) .
For more details about the Data Guard setup used in this example please refer to my previous post >> Part I :  Implementation

• Note : mtldb = Primary        mtlstd = Standby

– Display of the DGMGRL “show configuration” output

DGMGRL> show configuration verbose

Configuration – dg12

  Protection Mode: MaxPerformance

  Members:

  mtldb  – Primary database

    Error: ORA-16724: cannot resolve gap for one or more standby databases

    mtlstd – Physical standby database

  – Check Primary DB SCN Number

SQL> select current_scn from v$database;

CURRENT_SCN

———–
  135966683

  – Check Standby DB SCN Number

@mtlsd> select current_scn from gv$database;

CURRENT_SCN

———–

  114264654 

– Check  Standby checkpoint DB SCN Number

@mtlstd> select min (checkpoint_change#) from v$datafile_header;

MIN(CHECKPOINT_CHANGE#)

———————–

  114264655

STEP 1 STOP THE RECOVERY PROCESS (APPLY OFF)

DGMGRL> edit  DATABASE MTLSTD set STATE=APPLY-OFF;

STEP 2  PERFORM A INCREMENTAL BACKUP FROM PRIMARY DB to an SCN that is a bit behind that of the STANDBY

  RMAN> run

  {allocate channel ch1 type disk;

   backup incremental from scn 114264000 database format       

   ‘E:MSSQL_backupstandby_%d_%t_%c_%p’;

  }

STEP 3 RECOVER STANDBY DB

  A) Move the rman backup piece to the standby server => F:Ora_Backupstandby

  B) Catalog backup piece on the standby server
      — ON STANDBY SERVER 

$ rman target /   

RMAN> catalog start with ‘F:Ora_Backupstandby’;

    List of Cataloged Files

    =======================

    File Name: F:ORA_BACKUPSTANDBYSTANDBY_MTLDB_955116673_1_1

    File Name: F:ORA_BACKUPSTANDBYSTANDBY_MTLDB_955116718_1_1

C) Recover the standby database using the backup piece which has taken from the PRIMARY DB

— RUN FROM STANDBY SERVER 

RMAN> run

{allocate channel ch1 type disk;

  recover database noredo;}

STEP 4 Create standby control from primary DB

@mtldb> alter database create standby controlfile as ‘E:backupstandby.ctl’;

STEP 5 Restore the Control file in the STANDBY DB

— Below steps are in the standby server

A) STOP THE STANDBY DATABASE (if it’s a  RAC database run it on all instances)

$ srvctl stop database -d mtlstd

B) OPEN THE STANDBY INSTANCE in NoMount mode  (if it’s a  RAC database only one instance should be open)

   $ sqlplus / as sysdba

   SQL> startup nomount

C) IDENTIFY THE CONTROL FILE LOCATION

SQL> show parameter control_files

D) Past the copied standby control file from primary DB in STEP4

E) RESTORING CONTROL FILE TO STANDBY DATABASE

RMAN> restore controlfile from ‘F:Ora_Backupstandbystandby.ctl’;

RMAN> alter database mount;

STEP 6 CHANGING CONTROLFILE CONFIGURATION (on standby DB)

$rman target /   

RMAN> CATALOG START WITH ‘+DATA/MTLSTD/DATAFILE/’;

  

/*** STEP 7 (special case)  – If no Datafile has been added during the redo apply lag please skip to STEP9

Check for New Datafile added in primary DB

— IN STANBDY DB

   SQL> SELECT FILE#, NAME FROM V$DATAFILE WHERE CREATION_CHANGE# >      114264655;

   No rows ..

  NOTE: If any files available, we have to copy it from primary to the standby database using RMAN

•    copy the datafile from the primary database (example here file 10 exist only in primary DB).

RMAN>

run

  {allocate channel a1 type disk; backup datafile 10 format ‘E:backupdatafile_%d_%U_%p_%c’;}

• Transfer the rman backup datafile to standby database location then catalog the backup piece in the standby database.

RMAN>

run

{allocate channel a1 type disk; Restore datafile 10;}

STEP 8 Switch the datafile USING RMAN
  This process will change the datafile details of the standby database in the controlfile

RMAN> SWITCH DATABASE TO COPY;

*****/

STEP 9 Clear the standby redo log files

SQL> ALTER DATABASE CLEAR LOGFILE GROUP 4;

SQL> ALTER DATABASE CLEAR LOGFILE GROUP 5;

SQL> ALTER DATABASE CLEAR LOGFILE GROUP 6;

SQL> ALTER DATABASE CLEAR LOGFILE GROUP 7;

  

STEP 10 START THE RECOVEY PROCESS ON STANDBY

$dgmgrl     —– On standby server

DGMGRL> edit database ‘mtlstd’ set state=’APPLY-ON’;

Succeeded.

** FINAL RESULT

DGMGRL> show configuration

Configuration – dg12

  Protection Mode: MaxPerformance

  Members:

  mtldb  – Primary database

    mtlstd – Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:

SUCCESS   (status updated 2 seconds ago)