EBS Cloud Manager -- A DBA Sailing around Linux Administration, OCI Cloud Shell, OS Firewalls...

 

After deploying EBS using cloud manager, we were not able to login to apps and db nodes on OCI as root/opc users.
The EBS Cloud manager guide has mentioned only one way to login to apps and db nodes after deployment on OCI i.e. -
- Login to Cloud Manager as opc
- sudo su - oracle
- ssh apps node ip
- ssh db node ip

We may require connecting to root OS user for some superuser related tasks. In my case we had to check db node port as developers were
not able to connect using sqldeveloper after connecting to the VPN (interesting things coming up for this issue later in this blog).

So we were in this scenario -
1. Port is somewhere blocked.
2. db node ip is pingable
3. We can only login to the db node using os user - oracle
4. We can't check firewall rules without root access.
It all started with setting root password for this db node and we followed below note -

Ref -How to Reset Root Password in Oracle Linux 7 (Doc ID 1954652.1)

1. Launch Cloud Shell on OCI for the specific instance.

Now keep cloud shell open, and reboot the db node (of course after shutting the db and listener) -

2. Reboot the server

3. Press upper key when Grub is loading.. 

5. While booting GRUB 2 Edit Menu Option (by pressing E)

6. Select the line starts with linux16**** (or linuxefi**** for UEFI bios)  and append "rd.break" at the end of the line.
Example:
linux16 **** rd.break

7. Press ctrl+x to boot or start.

8. First we will remount the sysroot file system in read and write mode and then use chroot to got into a chroot jail:
# mount -o remount,rw /sysroot
# chroot /sysroot
9. Type passwd command in the command line and press same password twice for reset root password:
# passwd
10. Make sure that all unlabel files ( including shadow ) get relabeled during booting:
# touch /.autorelabel
11. Type the command to sync:
# sync
12. Type twice exit command to leave & logout.
13. The system will apply some SELinux contexts and reboot.

All the commands in one screen below for your reference -

Now the interesting part. I was initially confused if at all firewall was causing port blocking. Reason was simple, apps node was 
connecting to the database node successfully.

So I checked a couple of things at OCI level -
1. Security List.
2. Route table ( we had 2 VCNs, one where VPN was connected and the second for EBS on Cloud Deployment)
We are getting into OCI networking now :). The only thing that was pending was firewall settings at db node and then came gotcha moment -

# firewall-cmd --get-default-zone
public
# firewall-cmd --get-active-zones
# firewall-cmd --list-all
public
  target: default
  icmp-block-inversion: no
  interfaces:
  sources:
  services: dhcpv6-client ssh
  ports:
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:
        rule family="ipv4" source address="10.3.x.xxx" port port="1521" protocol="tcp" accept


A rich rule is defined by terraform when deploying EBS on Cloud through Cloud Manager and it has db listener port say 1521 open
only for the apps node 
# firewall-cmd --permanent --zone=public --list-rich-rules
rule family="ipv4" source address="10.3.x.xxx" port port="1521" protocol="tcp" accept
So we added another rule to open port and add CIDR range for the VCN that could connect to the db.
# firewall-cmd --permanent --zone=public --add-rich-rule='rule family=ipv4 source address=11.x.x.x/20 port port=1521 protocol=tcp accept'
success
# firewall-cmd --reload
success
# firewall-cmd --permanent --zone=public --list-rich-rules
rule family="ipv4" source address="10.3.x.xxx" port port="1521" protocol="tcp" accept
rule family="ipv4" source address="11.x.x.x/20" port port="1521" protocol="tcp" accept


So we started with OCI Console connection, reset the root password for an OCI db node and added a rich rule to open port for a specific CIDR.