EBS Cloud Manager Troubleshooting - Creating Backups

EBS Cloud manager is well automated for setups on OCI and there are scenarios where DBA intervention would still be required, let's discuss a classic example of such scenario today (17feb2022)

Task - Create backup of EBS Cloud Manager Environment, ebs r12.2.9, db version 12.1.0.2
When you login to EBS Cloud Manager, simply check top right section - 

Once you click on Create backup, it will ask you for encryption password and apps credentials. Backup is then submitted as a JOB and you can get the details for the running backup under Jobs Tab.

Please note that EBS cloud manager creates an OSS level backup on the Object Storage. In my case, the job failed at - Validate -> EBS cloud backup Application tier validations 

Error Details - 

ERROR : WLS domain size is higher than EBS default threshold: 5120 MB ). Please check and cleanup some of the server log files or any unnecessary file under /u01/install/APPS/fs1/FMW_Home/user_projects/domains/EBS_domain.

Failed with code: 1

[2022/02/17 03:45:46] [APPSTIEREBSVALIDATION] ERROR: Source application tier post-validation failed.

[2022/02/17 03:45:46] [APPSTIEREBSVALIDATION] Updating taskid appsTierEBSValidation status to Failed

Now the error was self explanatory, but this step has to be intervened by a DBA and fixed by clearing older logs under your domain, in my case, I cleared some space for below folders -

Check for EBS_domain size - 

$ du -sh EBS_domain/
5.7G    EBS_domain/

Remove files under - 
/u01/install/APPS/fs1/FMW_Home/user_projects/domains/EBS_domain/servers/oacore_server1/logs
oacore_server1.log00*
/u01/install/APPS/fs1/FMW_Home/user_projects/domains/EBS_domain/servers/AdminServer/logs
EBS_domain.log00*

check current size again -
$ du -sh EBS_domain/
2.1G    EBS_domain/

You may then try the 'Retry' option for Backup to be resumed again.