Oracle EBS 12.2 Online Patching

About this Document

Background

This document will cover ADOP patching concept with cutover in detail consideration.

Intended Audience

This document is primarily meant for database administrators, system administrators, Team Leaders, Technical Managers, etc.

References

https://techgoeasy.com/happens-cutover-phase-adop-r12-2/

 http://appsdbastuff.blogspot.com/2016/07/how-adop-works.html

http://allappsdba.blogspot.com/2017/04/adop-cutover-failed.html

https://allappsdba.blogspot.com/2018/09/adop-cutover-failed.html

ADOP Cutover Fails, Timeout While Starting Services On Secondary Node (Doc ID 2411800.1)

R12.2 ADOP Phase=cutover Fails Timed Out In Adnodemgrctl.sh (Doc ID 1640587.1)

Google

Pythian Blogs

Assumptions

Currently customer is using Oracle EBS r12.2.x.

ADOP Phases –

Online patching allows better prediction of downtime and has significantly reduced downtime compared to r12.1.x. EBS r12.2 stores multiple editions of application code (using Edition based redefinition feature). The complete cycle takes considerably more time than before, however, cutover phase has been kept to a minimum to reduce downtime.

ADOP cycle consists of difference phases and each phase needs to be thoroughly understood before applying patches using adop.

1.                  Prepare –

Creates a virtual copy (patch edition) of Run Edition

FscloneStage and FscloneApply is used to copy files from run edition to patch edition (synchronizes files but not configurations)

Validations and checks are done after copy

2.                  Apply –

Applies patches on the Patch edition of dual filesystem

Need to make sure hotpatch is ‘hotpatch’ applicable, else system may hang when using hotpatch mode.

3.                  Finalize-

Prepares system for cutover phase.

Compilation of objects

4.                  Cutover –

Perform a cutover to Patch Edition and make it run Edition

It’s the most sensitive phase. Users cannot connect to the system during cutover phase.

5.                  Cleanup –

Will remove obsolete definitions

6.                  Abort –

Using this option, we can return to normal runtime operation if a patch cycle has failed.

Abort can be used if system is in below phases

-          Prepare

-          Finalize

-          Cutover

Once cutover completes, we cannot run abort.

CUTOVER phase fails or terminates in the middle. Please describe how would you solve / workaround the problem and complete the switchover.

Cutover phase may fail for different stages like as follows –

1.       Shutting down ICM and apps tier services and acquiring lock on AD_ADOP_SESSIONS

2.       Database level cutover

3.       Switchover of filesystem from run edition to patch edition

4.       Starting up admin server

We need to investigate and find out in which stage cutover has failed, this can be done using below query –

set pagesize 200;

set linesize 160;

col PREPARE_STATUS format a15

col APPLY_STATUS format a15

col CUTOVER_STATUS format a15

col ABORT_STATUS format a15

col STATUS format a15

select NODE_NAME,ADOP_SESSION_ID, PREPARE_STATUS , APPLY_STATUS  ,CUTOVER_STATUS , CLEANUP_STATUS , ABORT_STATUS , STATUS

from AD_ADOP_SESSIONS

order by ADOP_SESSION_ID;

Understanding status codes of cutover –

cutover_status='Y' 'COMPLETED'

cutover_status not in ('N','Y','X') and status='F' 'FAILED'

cutover_status='0' 'CUTOVER STARTED'

cutover_status='1' 'SERVICES SHUTDOWN COMPLETED'

cutover_status='3' 'DB CUTOVER COMPLETED'

cutover_status='D' 'FLIP SNAPSHOTS COMPLETED'

cutover_status='4' 'FS CUTOVER COMPLETED'

cutover_status='5' 'ADMIN STARTUP COMPLETED'

cutover_status='6' 'SERVICES STARTUP COMPLETED'

cutover_status='N' 'NOT STARTED'

cutover_status='X' 'NOT APPLICABLE'

CUTOVER_STATUS column will show either Y for completion status or any of above status codes and accordingly we will understand exactly which stage, cutover has failed.

Each stage will require a different action plan and decision making, let’s start with exploring each stage-

Scenario 1 -

1.       Shutting down ICM and apps tier services acquiring lock on AD_ADOP_SESSIONS

You can manually check and make sure if all application services are down.

ps -ef | grep applmgr

2.       Sometimes a terminated adop session can also create issues to current adop cycle.

Check below output for select query to find problematic session in AD_ADOP_SESSIONS table

select * from ad_adop_sessions where edition_name='LOCK' ;

Unlock session using below command manually to check if error exists again-

begin

AD_ZD_ADOP.UNLOCK_SESSIONS_TABLE('<hostname>',60,2);

end; 

/

UNLOCK_SESSIONS_TABLE procedure takes below arguments

 -----------------

 P_NODE_NAME     

 P_WAIT_INTERVAL 

 P_NUM_TRIES

It is always recommended to have further diagnostics done to find root cause of terminated session, Oracle has provided generic patch p19045166_R12_GENERIC for 12.2 customers to collect these diagnostics.

3.       Monitor long running concurrent programs using below queries –

SELECT DISTINCT c.USER_CONCURRENT_PROGRAM_NAME,

      round(((sysdate-a.actual_start_date)*24*60*60/60),2) AS Process_time,

    a.request_id,a.parent_request_id,a.request_date,a.actual_start_date,a.actual_completion_date,

      (a.actual_completion_date-a.request_date)*24*60*60 AS end_to_end,

      (a.actual_start_date-a.request_date)*24*60*60 AS lag_time,

      d.user_name, a.phase_code,a.status_code,a.argument_text,a.priority

FROM     apps.fnd_concurrent_requests a,

    apps.fnd_concurrent_programs b ,

    apps.FND_CONCURRENT_PROGRAMS_TL c,

    apps.fnd_user d

WHERE   a.concurrent_program_id=b.concurrent_program_id AND

    b.concurrent_program_id=c.concurrent_program_id AND

    a.requested_by=d.user_id AND

    status_code='R' order by Process_time desc;

Scenario 2 -

Database level cutover

It is always recommended to take complete backup of system before applying any patch that should include,

-       Database backup

-       Apps tier backup

However, this may not be suitable for applying smaller patches and therefore, we should use Flashback technology for creating snapshots/before images of database. This simplifies rolling back patch after database editions have been updated.

Flashback uses db_recovery_file_dest location to store flashback logs. They are retained as per retention policy and automatically cleared. However, if we have set a guaranteed restore point, they are retained until we dropped restore point.

A restore point can be created just before starting cutover phase at db level to make sure we can fix failed cutover phase.

Steps to create restore point –

a.       Check flashback is enabled for the database –

SQL> select name,open_mode, FLASHBACK_ON from v$database;

NAME      OPEN_MODE            FLASHBACK_ON

--------- -------------------- ------------------

OAMREPO   READ WRITE           NO

b.       Set db_recovery_file_dest_size first and then db_recovery_file_dest

SQL> alter system set db_recovery_file_dest_size=20g;

System altered.

SQL> alter system set db_recovery_file_dest='/u01/app/oracle/db/12.1.0/fra';

System altered.

SQL>

c.       Enable Flashback as follows –

SQL> alter database flashback on;

Database altered.

SQL> show parameter db_flashback

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------

db_flashback_retention_target        integer     1440

SQL> alter system set db_flashback_retention_target=120;

System altered.

d.       Creating guaranteed restore point

We will follow normal patching cycle until cutover phase as follows

adop phase=prepare

adop phase=apply

adop phase=finalize

Once finalize phase completes, just before starting cutover phase, we will create a guaranteed restore point as follows –

SQL> alter system switch logfile;

System altered.

SQL> create restore point before_change_pwd guarantee flashback database;

Restore point created.

SQL> alter system switch logfile;

System altered.

SQL>

e.       Cutover phase

Execute adop phase=cutover and monitor log for any errors

Suppose cutover phase fails in this case, we have a guaranteed restore point to flashback database to ‘before_patch_19045166’

f.        Rolling back to state before cutover

This can be achieved at database level following steps –

a.       Shutdown database

SQL> shut immediate

Database closed.

Database dismounted.

ORACLE instance shut down.

b.       Start database in mount state

SQL> startup mount;

ORACLE instance started.

Total System Global Area  809500672 bytes

Fixed Size                  2929600 bytes

Variable Size             394267712 bytes

Database Buffers          406847488 bytes

Redo Buffers                5455872 bytes

Database mounted.

c.       Flashback database to restore point before_patch_19045166

SQL> flashback database to restore point before_patch_19045166;

Flashback complete.

d.       Startup database in mount and open with resetlogs option

SQL> alter database open resetlogs;

Database altered.

SQL>

e.       Drop guaranteed restore point

SQL> drop restore point before_patch_19045166;

Restore point dropped.

SQL>

f.        Disable flashback until next patching cutover phase

SQL> alter database flashback off;

Database altered.

SQL> alter system set db_recovery_file_dest='';

System altered.

Scenario 3 -

Switchover of filesystem from run edition to patch edition. This can be observed in cutover logfile under location –

$NE_BASE/EBSApps/log/adop/session_id/cutover_timestamp/

       If adop fails during cutover phase, there are 2 possibilities,

a.    Filesystems are not switched

-       Perform steps in scenario 2 to rollback changes done by patch.

-       No action required as run edition is not updated by the patch.

-       Shutdown all services and start apps tier as normal procedure.

-       Now system is back in finalize phase and you can decide to abort online patching cycle.

b.    Filesystems are switched (Patch edition is now new run edition, and old run edition is now patch edition)

-       Source run file system environment

-       Shutdown apps tier (adstpall.sh)

-       Perform a switch back($FND_TOP/bin/txkADOPCutOverPhaseCtrlScript.pl)

$ perl $AD_TOP/patch/115/bin/txkADOPCutOverPhaseCtrlScript.pl

       -action=ctxupdate

       -contextfile= <full path of new run filesystem>

       -patchcontextfile= <full path of new patch filesystem>

       -outdir= <out directory path)

-       Startup all services from new run file system

Scenario 4-

Starting up admin server

Below references have been observed specifically to cutover failing due to start up issues of admin server.

-       Adop Cutover Failed by "[UNEXPECTED]Error occurred starting Admin Server" on Run File System (Doc ID 2314018.1)

-       ADOP Cutover Fails, Timeout While Starting Services On Secondary Node (Doc ID 2411800.1)

Ideal approach in this is to use complete cutover phase by skipping middle tier restart –

adop phase=cutover mtrestart=no

Once cutover completes, manually start services and further troubleshoot.

Scenario 5-

Wrong configuration in single/multimode setup. Oracle has further provided some notes on cutover failing due to wrong configuration in both single/mutlnode setup.

a.       Wrong options used to mount NFS share where apps tier is deployed

b.       CONTEXT_FILE parameters wrongly set ( like s_appltmp, s_batch_status, s_apctimeout)

c. Supporting Metalink Notes -

- 12.2 ADOP Cutover Fails With forceStartupServices (Doc ID 1573972.1)

- 12.2 E-Business Suite ADOP Cutover, Abort, or Fs_clone Fails With 'ERROR: The value of s_appltmp is not correct in run edition context file' As s_appltmp Has Been Customized To A Shared Location On All Nodes (Doc ID 2220669.1)

- Cutover Phase failed due to wrong configuration of services (Doc ID 2384839.1)

Conclusion –

1.       It is recommended to perform patch analysis before applying any patch to ebs environment

2.       If we are applying patches for major release upgrades, then we must take a complete backup of apps tier and database.

3.       Enable flashback at database level and create a guaranteed restore point before cutover.

4.       Always review logs under below location and try to find solution first before performing switch back

$NE_BASE/EBSApps/log/adop/session_id/cutover_timestamp/

5.       We must review metalink notes and implications of action plan with respect to environment

6.       We can always raise Severity 1 SR for patch failures and get solution.