How to repair volume-backed Windows instance

21 November 2017
Lead photo

In Openstack Mitaka, one of our users has been locked out from his Windows instance. The instance was volume-backed (booted from volume).

Due to non-standard system configuration, Openstack mechanisms to reset password didn't work. The only solution we had was to "rescue" the instance - boot it with another system image - and change password.

Unfortunately, the instance was booted from Cinder volume, and standard "nova rescue" mechanism returned the following error:

ERROR (BadRequest): Instance <INSTANCE UUID>  cannot be rescued: 
Cannot rescue a volume-backed instance (HTTP 400)

If you try to detach the volume from instance, you will get error:

ERROR (Forbidden): Can't detach root device volume (HTTP 403)

Solution

We can detach the volume from its original instance, attach it to new (rescue) instance and repair what's needed. Afterwards, we will re-attach the volume to original instance.

Unfortunately, this process requires some db-level changes, as nova will not allow detaching of a root volume.

Procedure

  1. Connect with console and shutdown the instance. Ensure that you are doing full shutdown - you can hold SHIFT when starting shutdown process (source)
  2. Make a snapshot of instance volume (in case something goes wrong).
  3. Create and boot new instance (we use Fedora 26 here) that will be our rescue instance.
  4. Check broken instance's volume ID.
  5. Set some variables:
    instance=[UUID of instance to rescue]
    rootvol=[UUID of volume to rescue]
    rescueinstance=[UUID of rescue instance]

     

  6. In MySQL database, mark the volume as non-root,  detach it from original instance and attach to rescue instance.
    echo "update block_device_mapping \
        set boot_index = NULL \
        where instance_uuid='$instance' \
        and deleted_at is NULL \
        and volume_id = '$rootvol';" | mysql nova
    
    nova volume-detach $instance $rootvol
    nova volume-attach $rescueinstance $rootvol
  7. Log into rescue instance.
  8. Install some utils to manage ntfs filesystem and Windows passwords:
    yum install ntfs-3g ntfsprogs chntpw

     

  9. Proceed with your rescue; for example, you can change windows passwords using this instruction: https://www.techrepublic.com/blog/tr-dojo/reset-windows-passwords-with-the-help-of-linux/
  10. When done, shutdown rescue instance, detach the volume, re-attach it to original instance and set as bootable:
    nova volume-detach $rescueinstance $rootvol
    nova volume-attach $instance $rootvol
    echo "update block_device_mapping \
        set boot_index = 0 \
        where instance_uuid='$instance' \
        and deleted_at is NULL \
        and volume_id = '$rootvol';" | mysql nova
  11. Boot the instance and ensure your changes were successful.
  12. Once everything works fine, delete:
    1. rescue instance,
    2. snapshot created earlier.

 

References