Removing orphaned instances when all else fails…

Working on OpenStack is complex and working on older versions of OpenStack is even more complex. If your instance is spawning but the shared storage hosting the ephemeral disk or block storage oopses/offlines/panics then you can be left with orphaned instances that exist in the database but nowhere else. You try to delete them using nova delete but this doesn’t work because OpenStack can’t locate the files it wants to delete and you get into a real mess.

Some articles indicate that all you need to do is run some variation on:

mysql -D nova -e "delete from instances where instances.uuid = '$uuid'"

but this is bad because it leaves all sorts of information relating to the VM in existence. It appears to have been fixed in later versions of OpenStack – Kilo hasn’t exhibited this problem yet – so what follows is Icehouse-specific, for those people still running this release.

Most of the database info I have stolen from the URL in the comments, I have just added input so you don’t need to drop to a mysql prompt. Feed it your VM UUID and you’re done. If you have reached this page then you’re probably in not such a great place so the usual warnings apply about random bits of bash script on the internet apply. And remember that reset-state is your friend and you should have tried lots of other stuff first.

#! /bin/bash
# This is an Icehouse-specific script
# to remove an instance that is not consuming ANY resources
# ie. It only exists in the database. You need to be VERY
# sure of this fact before using so as not to leave disks
# orphaned or instances running. Use as a last resort after
# deletion and reset-state nova options have failed. Use nova show to
# inspect libvirt xml prior to using.
# Source for db schema:

read -p "Please enter the UUID of the vm you need to clear from the database:" uuid
mysql -D nova -e "select display_name from instances where instances.uuid = '$uuid'"
read -p "Are you sure this is the instance you are looking for? y/n: " response
if [ $response == y ]; then
mysql -D nova -e "delete from instance_faults where instance_faults.instance_uuid = '$uuid'"
mysql -D nova -e "delete from instance_id_mappings where instance_id_mappings.uuid = '$uuid'"
mysql -D nova -e "delete from instance_info_caches where instance_info_caches.instance_uuid = '$uuid'"
mysql -D nova -e "delete from instance_system_metadata where instance_system_metadata.instance_uuid = '$uuid'"
mysql -D nova -e "delete from security_group_instance_association where security_group_instance_association.instance_uuid = '$uuid'"
mysql -D nova -e "delete from block_device_mapping where block_device_mapping.instance_uuid = '$uuid'"
mysql -D nova -e "delete from fixed_ips where fixed_ips.instance_uuid = '$uuid'"
mysql -D nova -e "delete from instance_actions_events where instance_actions_events.action_id in (select id from instance_actions where instance_actions.instance_uuid = '$uuid')"
mysql -D nova -e "delete from instance_actions where instance_actions.instance_uuid = '$uuid'"
mysql -D nova -e "delete from virtual_interfaces where virtual_interfaces.instance_uuid = '$uuid'"
mysql -D nova -e "delete from instances where instances.uuid = '$uuid'"
echo "Ok, done"
echo "Quitting, no changes made"


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s