Showing posts with label rabbitmq. Show all posts
Showing posts with label rabbitmq. Show all posts

Wednesday, July 6, 2016

Issues starting the nova conductor service OpenStack Mitaka: AMQP server controller1.example.com:5672 closed the connection. Check login credentials: Socket closed




Issues starting the nova conductor service on the controller node. This results as a part of the misconfiguration of the nova conductor in /etc/nova/nova.conf on the controller node of OpenStack.

As usual the best place to see the reasons for problems with the OpenStack Nova services on the controller is at the location /var/log/nova/*. See the logs there.


The below message appears in /var/log/nova/nova-conductor.log

==> nova-conductor.log <==
2016-07-06 06:32:20.891 27161 ERROR oslo.messaging._drivers.impl_rabbit [req-7bff235c-2535-4d18-b9e0-351c6e23557c - - - - -] AMQP server controller1.example.com:5672 closed the connection. Check login credentials: Socket closed

This means that the user mentioned on the Nova Configuration user as the user to be used to access the Rabbitmq either is not present or if present has a password which does not match with the rabbitmq configuration.


My Nova Configuration file looked like this (/etc/nova/nova.conf on the controller server)

[DEFAULT]
rpc_backend = rabbit

[oslo_messaging_rabbit]
rabbit_host = controller1.example.com
rabbit_userid = openstack
rabbit_password = XXXXXXX



The Fix:



1) In case the user for rabbitmq access as mentioned here as "openstack" is not created then create the user on the controller server in the setup as 

rabbitmqctl add_user openstack XXXXXXX

where XXXXXXX is a password which you want to give to the user

2) In case there is a password mismatch between the one given in /etc/nova/nova.conf and the actual rabbitmq password in the rabbitmq password you can set a new password for the user (openstack here) and update the same in the NOVA configuration file /etc/nova/nova.conf

rabbitmqctl change_passwd <user> <password>

Refer to the man pages of rabbitmqctl for more.

root@controller1 nova]# rabbitmqctl --help | grep -i use
Error: could not recognise command
the correct suffix to use after the "@" sign. See rabbitmq-server(1) for
    add_user <username> <password>
    delete_user <username>
    change_password <username> <newpassword>
    clear_password <username>
            authenticate_user <username> <password>
    set_user_tags <username> <tag> ...
    list_users
    set_permissions [-p <vhost>] <user> <conf> <write> <read>
    clear_permissions [-p <vhost>] <username>
    list_user_permissions <username>
channels, protocol, auth_mechanism, user, vhost, timeout, frame_max,
user, vhost, transactional, confirm, consumer_count, messages_unacknowledged,
[root@controller1 nova]#
[root@controller1 nova]#
[root@controller1 nova]# rabbitmqctl change_passwd nova XXXXXXX

OpenStack MItaka Problems starting openstack-nova-consoleauth service "oslo_service.service NotAllowed: Connection.open: (530) NOT_ALLOWED - access to vhost '/' refused for user 'openstack'"

OpenStack MItaka Problems starting openstack-nova-consoleauth service "oslo_service.service NotAllowed: Connection.open: (530) NOT_ALLOWED - access to vhost '/' refused for user 'openstack'"


The Problem: (Which was a result of mis-configuration)


This issue was faced when starting the Openstack Nova Consoleauth service. The service was not starting on the controller node.

For issues related to Nova see the logs /var/log/nova/* on the controller node also. The controller node in the setup is controller1.example.com 


The Main error to look in the file /var/log/nova/nova-consoleauth.log was 

2016-07-06 06:40:25.496 28338 ERROR oslo_service.service NotAllowed: Connection.open: (530) NOT_ALLOWED - access to vhost '/' refused for user 'openstack'




==> nova-consoleauth.log <==
2016-07-06 06:41:14.877 28418 WARNING oslo_reports.guru_meditation_report [-] Guru mediation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2016-07-06 06:41:15.173 28418 INFO oslo_service.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
2016-07-06 06:41:15.288 28418 INFO nova.service [-] Starting consoleauth node (version 13.1.0-1.el7)
2016-07-06 06:41:16.719 28418 ERROR oslo_service.service [req-205c36a5-c723-4688-a78f-fc46089b5a6a - - - - -] Error starting thread.
2016-07-06 06:41:16.719 28418 ERROR oslo_service.service Traceback (most recent call last):
2016-07-06 06:41:16.719 28418 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 680, in run_service
2016-07-06 06:41:16.719 28418 ERROR oslo_service.service     service.start()
2016-07-06 06:41:16.719 28418 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/service.py", line 231, in start
2016-07-06 06:41:16.719 28418 ERROR oslo_service.service     self.rpcserver.start()
2016-07-06 06:41:16.719 28418 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 267, in wrapper
.
.
.

2016-07-06 06:41:16.719 28418 ERROR oslo_service.service     return fun(*args, **kwargs)
2016-07-06 06:41:16.719 28418 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 507, in __call__
2016-07-06 06:41:16.719 28418 ERROR oslo_service.service     self.revive(create_channel())
2016-07-06 06:41:16.719 28418 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 242, in channel
2016-07-06 06:41:16.719 28418 ERROR oslo_service.service     chan = self.transport.create_channel(self.connection)
2016-07-06 06:41:16.719 28418 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 741, in connection
2016-07-06 06:41:16.719 28418 ERROR oslo_service.service     self._connection = self._establish_connection()
2016-07-06 06:41:16.719 28418 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 696, in _establish_connection
2016-07-06 06:41:16.719 28418 ERROR oslo_service.service     conn = self.transport.establish_connection()
2016-07-06 06:41:16.719 28418 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
2016-07-06 06:41:16.719 28418 ERROR oslo_service.service     conn = self.Connection(**opts)
 .
.
.

2016-07-06 06:41:16.719 28418 ERROR oslo_service.service NotAllowed: Connection.open: (530) NOT_ALLOWED - access to vhost '/' refused for user 'openstack'
2016-07-06 06:41:16.719 28418 ERROR oslo_service.service


The Fix:


To get around this ensure that the user mentioned (in my case this rabbitmq_user is opestack) has been allowed the permission to all the rabbitmq things

If not run the command

rabbitmqctl set_permissions <rabbitmq_user> ".*" ".*" ".*"