In Openstack Mitaka, one of our users has been locked out from his Windows instance. The instance was volume-backed (booted from volume).
Due to non-standard system configuration, Openstack mechanisms to reset password didn't work. The only solution we had was to "rescue" the instance - boot it with another system image - and change password.
Unfortunately, the instance was booted from Cinder volume, and standard "nova rescue" mechanism returned the following error:
ERROR (BadRequest): Instance <INSTANCE UUID> cannot be rescued: Cannot rescue a volume-backed instance (HTTP 400)
If you try to detach the volume from instance, you will get error:
ERROR (Forbidden): Can't detach root device volume (HTTP 403)
We can detach the volume from its original instance, attach it to new (rescue) instance and repair what's needed. Afterwards, we will re-attach the volume to original instance.
Unfortunately, this process requires some db-level changes, as nova will not allow detaching of a root volume.
- Connect with console and shutdown the instance. Ensure that you are doing full shutdown - you can hold SHIFT when starting shutdown process (source)
- Make a snapshot of instance volume (in case something goes wrong).
- Create and boot new instance (we use Fedora 26 here) that will be our rescue instance.
- Check broken instance's volume ID.
- Set some variables:
instance=[UUID of instance to rescue] rootvol=[UUID of volume to rescue] rescueinstance=[UUID of rescue instance]
- In MySQL database, mark the volume as non-root, detach it from original instance and attach to rescue instance.
echo "update block_device_mapping \ set boot_index = NULL \ where instance_uuid='$instance' \ and deleted_at is NULL \ and volume_id = '$rootvol';" | mysql nova nova volume-detach $instance $rootvol nova volume-attach $rescueinstance $rootvol
- Log into rescue instance.
- Install some utils to manage ntfs filesystem and Windows passwords:
yum install ntfs-3g ntfsprogs chntpw
- Proceed with your rescue; for example, you can change windows passwords using this instruction: https://www.techrepublic.com/blog/tr-dojo/reset-windows-passwords-with-the-help-of-linux/
- When done, shutdown rescue instance, detach the volume, re-attach it to original instance and set as bootable:
nova volume-detach $rescueinstance $rootvol nova volume-attach $instance $rootvol echo "update block_device_mapping \ set boot_index = 0 \ where instance_uuid='$instance' \ and deleted_at is NULL \ and volume_id = '$rootvol';" | mysql nova
- Boot the instance and ensure your changes were successful.
- Once everything works fine, delete:
- rescue instance,
- snapshot created earlier.