Advertisement Header

Tuesday, 21 February 2017

Unable to delete openstack instance stuck at Error state

When you attempt to delete instance from either openstack CLI or openstack dashboard and if it get stuck at error state as follows:

GUI:


CLI:

`nova list` will show:
  | 1541a197-9f80-4ee5-a7d6-08e591aa8lokin | sravan0 | ERROR | deleting | NOSTATE  
There may be so many reasons for this error state like attached volume may not able to reclaim, or unable to reclaim associated port by neutron and so on.

Sometimes we can try from CLI after resetting state "nova reset-state <instance>" using "nova delete $instanceid"

When above workaround is not working along with any other options, at last if you want to mark instance as deleted in nova database then we can follow below approach:

Note: Please use this when you hit the wall other wise not recommended:

Step 1: Get instance details using "nova show "$instanceid""

[root@controller ~]# nova show 92b9c565-d908-4dd0-a169-f58b3ce71428
 Property Value
 OS-DCF:diskConfig AUTO
 OS-EXT-AZ:availability_zone nova
 OS-EXT-SRV-ATTR:host kvm-02
 OS-EXT-SRV-ATTR:hypervisor_hostname kvm-02
 OS-EXT-SRV-ATTR:instance_name instance-0000005b
 OS-EXT-STS:power_state 1
 OS-EXT-STS:task_state -
 OS-EXT-STS:vm_state active
 OS-SRV-USG:launched_at 2016-05-16T09:35:14.000000
 OS-SRV-USG:terminated_at -
 accessIPv4
 accessIPv6
 config_drive
 created 2016-05-16T09:35:09Z
 flavor Standard 5 (b6be6358-120a-4fb9-987c-f10a7e82cb09)
 hostId b8b17a365b15d16f31860029f971348e8c8af7f2092652c5e7c36b83
 id 92b9c565-d908-4dd0-a169-f58b3ce71428
 image CentOS-7-x86_64-GenericCloud-1603.qcow2 (7e78fc90-4eed-41a5-8a79-9fef27371278)
 key_name suresh
 metadata {}
 name Jenkins
 net-int network 10.10.10.82, 202.65.153.24
 os-extended-volumes:volumes_attached []
 progress 0
 security_groups default
 status ACTIVE
 tenant_id 87725975eafe46ebb8c54e02acf0b654
 updated 2016-05-16T09:36:04Z
 user_id 670959f49dde4aa582d99afbf6835387

 Step 2: Connect to mysql database

[root@controller ~]# mysql -u root -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 241348
Server version: 5.5.47-MariaDB MariaDB Server

Copyright (c) 2000, 2015, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| cinder             |
| glance             |
| keystone           |
| mysql              |
| neutron            |
| nova               |
| performance_schema |
| test               |
+--------------------+

9 rows in set (0.00 sec)

#Now change the database to use nova:

MariaDB [(none)]> use nova;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [nova]>

Step 3: Get the current status of instance from database

MariaDB [nova]> select uuid, hostname, vm_state, task_state, power_state, deleted from instances where uuid="92b9c565-d908-4dd0-a169-f58b3ce71428";
+--------------------------------------+----------+----------+------------+-------------+---------+
| uuid                                 | hostname | vm_state | task_state | power_state | deleted |
+--------------------------------------+----------+----------+------------+-------------+---------+
| 92b9c565-d908-4dd0-a169-f58b3ce71428 | jenkins  | active   | NULL       |           1 |       0 |
+--------------------------------------+----------+----------+------------+-------------+---------+
1 row in set (0.00 sec)

##Mark instance deleted state to 1 using below query

MariaDB [nova]> update instances set deleted='1', vm_state='deleted', deleted_at='now()' where uuid='77beaec2-636e-4e3e-9867-5df057fc94be' and project_id='87725975eafe46ebb8c54e02acf0b654';

validate the state again by running above status query:

MariaDB [nova]> select uuid, hostname, vm_state, task_state, power_state, deleted from instances where uuid="92b9c565-d908-4dd0-a169-f58b3ce71428";
+--------------------------------------+----------+----------+------------+-------------+---------+
| uuid                                 | hostname | vm_state | task_state | power_state | deleted |
+--------------------------------------+----------+----------+------------+-------------+---------+
| 92b9c565-d908-4dd0-a169-f58b3ce71428 | jenkins  | deleted| NULL       |           0 |       1 |
+--------------------------------------+----------+----------+------------+-------------+---------+
1 row in set (0.00 sec)

Now you can see your power state and deleted db columns set to 0,1 respectively even this instance will be removed from openstack dashboard also.

Hope this post helped...

1 comment:

  1. Nice Post !! Before getting into this we can also give a try by restarting the libvirt and nova compute service as below

    Ubuntu:

    service libvirt-bin restart
    service nova-compute restart

    RedHat:

    systemctl stop openstack-nova-compute.service
    systemctl restart libvirtd.service
    systemctl start openstack-nova-compute.service

    ReplyDelete