I recently created an instance, but when I tried to ssh to it I got the following error; what!?

     tmelander$  Permission denied (publickey,gssapi-keyex,gssapi-with-mic)

As part of the instance creation I went ahead and selected a public ssh key, and I may have used the incorrect public key or something else.  My bad. Regardless of what happen I was denied from logging into the instance.  If you google the error it could be a number of problems including incorrect permissions on the authorized_keys or even the permissions on your private key on your computer.   In my case the public ssh key was not populated in the authorized_keys file on the instance for the opc account; D’oh!  

My first option of course was to delete the instance and recreate it hoping to get it right, but you may not want to do that for a number of reasons.  So, let’s take the approach and recover access to the instance using the Console Connections option.  I think this is a good exercise to do anyway because this skill can come in handy for that use case where no one can access the instance via ssh.  

Steps to reset the opc account SSH Key

I have consolidated the steps as much as possible.  The official documentation  https://docs.cloud.oracle.com/iaas/Content/Compute/References/serialconsole.htm gives several sections, but it can get a little confusing and frustrating.  If you are like me you just want a number of steps to get the job done without trying to extrapolate a lot of wordy documentation.

Create the Console Connection -

1. Login as an Administrator to the OCI console
2. Navigate to Compute > Instances > then select your instance
3. On the bottom left select Console Connections
4. From a terminal run “cat ~/.ssh/id_rsa.pub” and copy it. If you have another public key you want to use, feel free to use it.
5. Click on Create Console Connection
6. Select the PASTE SSH KEYS option
7. Paste the ssh public key into the field
8. Click on the Create Console Connection button.
9. Once it is Active select the three dots menu on the right and select Connect with SSH, alternately you could use Connect with VNC.
10. Choose LINUX/MAC OS or WINDOWS and copy the command.
11. In a text editor to make it easier and for insurance modify the command and add “-i ~/.ssh/id_rsa” in a couple places; replace id_rsa with your private key if it is different.  For example the changes in red:

ssh -i ~/.ssh/id_rsa -o ProxyCommand='ssh -W %h:%p -i ~/.ssh/id_rsa -p 443 ocid1.instanceconsoleconnection.oc1.iad.abuwcljtb73kpfekbhmaf6j7qa56uorzbqtnej4jgviliiofkgfimgi5d3yq@instance-console.us-ashburn-1.oraclecloud.com' ocid1.instance.oc1.iad.abuwcljtjpcpb3ztejz7r5vaap3cpaznuryrdz6yhv3mejxhvalnd7e3ddoa

Access the instance via the Serial Console –

1. From the previous steps paste the command that was copied and modified into a new terminal window and press enter.
2. Press enter again and you should now see a prompt to login — DO NOT LOGIN.
3. While watching your terminal with the login prompt, in the OCI console click on the Reboot button of your instance > choose confirm OK.  
4. You should see commands running across the terminal and once you see the OS options immediately use your up and down arrows to change the option. If you wait to long it will default to an option and continue to start. If that happens you will have to reboot again.
5. For Linux select the Unbreakable Enterprise Kernel option and press “e” to edit. 
6. This is the tricky part.  Scroll down using your keyboard down arrow to the very bottom.  You should see the cursor in a lighter color; it can be hard to see, but trust me it is there.  
7. Move the cursor until it is between the loglevel=4 and the last line ending in “initrdefi /boot/xxxxx.x86_64.img”.
8. Insert “rw init=/bin/bash”, be sure there is a space between the two other lines (see below).
   
9. Now press CTRL-X and it will reboot to a bash shell as root --- *** BE CAREFUL NOW ***.

Reset the OPC SSH key –

From the same terminal from the previous steps run the following:

1. Execute "/usr/sbin/load_policy -i” - to preserve the SELinux policies for the files you will modify.
2. Execute "/bin/mount -o remount,rw /“ - to remount the root partition with read/write permissions.
3. Execute "cd ~opc/.ssh” - to change to the opc SSH key directory. If you run a ls command you should see one file authorized_keys. If you run “cat authorized_keys” it may be empty or the wrong key; mine was empty.
4. Execute “mv authorized_keys authorized_keys.orig” - as a safety precaution to back up the original file.
5. Execute “vi authorized_keys” - to create a new authorized_keys file.
6. Press “i” to insert some text.
7. Paste in your ssh public key used from the previous step when you created the console connection.
8. Press ESC, enter “:wq”, and press Enter to write and quit the file to save it.
9. Execute “chown opc:opc authorized_keys” to make sure the owner of the file is opc otherwise it will be owned by root.
10. Execute “chmod 600 authorized_keys” to set the proper permissions.
11. Execute "mount -o remount,ro /” to force the OS to flush the filesystem cache. It also guarantees that the filesystem is marked as "cleanly unmounted" when the machine comes back up from a reboot.
12. Execute “/usr/sbin/reboot -f” to reboot the running instance. 
13. Next when you see the same OS options let the server start using the default OS option.
14. After the instance has rebooted open a new terminal and try to ssh in using “ssh opc@myinstance”; change myinstance to your hostname.  Another alternative is to provide the location to your private key like “ssh -i ~/.ssh/my_private_key opc@myinstance”.  
15. If successful you will get a login prompt; yeah!

Important, I recommend closing the terminal window with the serial connection and then deleting the Console Connection immediately using the OCI console. If you don't delete the serial console it will terminate your session automatically after 24 hours and you will have to re-authenticate, but the serial console will still be there. The risk is low since for someone to use this serial console they would need your private key, but as a precautionary measure it is best to delete the serial console as soon as you are finished using it.  

Summary

What we learned was a good skill.  These steps provide you with the knowledge on how to access your instance via ssh even though you have been denied.  This time it may have only been the OPC key issue, but next time it could be something else.  Again this is really just a quick and dirty to gain access to the instance terminal via root to troubleshoot in order to save you going through a lot of sections and verbiage in the official documentation, but I would browse the official docs https://docs.cloud.oracle.com/iaas/Content/Compute/References/serialconsole.htm if you are interested in learning even more.