Friday, September 23, 2016

Neutron OpenvSwitch L2 agent on the nova Node not appearing in the neutron agent-list. MItaka huge time difference between the controller and the nova server



Neutron OpenvSwitch L2 agent on the nova Node not appearing in the neutron agent-list.


root@labnova:~[root@labnova ~]# neutron agent-list
+--------------------------------------+----------------------+------------------------+-------------------+-------+----------------+---------------------------+
| id                                   | agent_type           | host                   | availability_zone | alive | admin_state_up | binary                    |
+--------------------------------------+----------------------+------------------------+-------------------+-------+----------------+---------------------------+
| 31947b08-c7e4-4136-9559-f9d8bc31e1fb | DHCP agent           | labneutron.example.com | nova              | :-)   | True           | neutron-dhcp-agent        |
| 9d6bdf0b-3b11-4f46-9d67-ae49bc2ff968 | Loadbalancerv2 agent | labneutron.example.com |                   | :-)   | True           | neutron-lbaasv2-agent     |
| a94c66c6-d037-4080-ac0d-5cc8e8719729 | L3 agent             | labneutron.example.com | nova              | :-)   | True           | neutron-l3-agent          |
| c0c42fcd-5d6a-4618-8b3a-b71b79a639dc | Metadata agent       | labneutron.example.com |                   | :-)   | True           | neutron-metadata-agent    |
| c96830ca-7807-404c-8168-0627f1950afd | Open vSwitch agent   | labneutron.example.com |                   | :-)   | True           | neutron-openvswitch-agent |
| e49cea55-70cf-49b7-a8fe-91f2f227ebe8 | Open vSwitch agent   | labnova.example.com    |                   | xxx   | True           | neutron-openvswitch-agent |
+--------------------------------------+----------------------+------------------------+-------------------+-------+----------------+---------------------------+

Meanwhile appearing in the /var/log/neutron/server.log on the controller node.
2016-09-23 15:02:13.737 3828 ERROR neutron.db.agents_db [req-9628b672-43b1-4883-89a3-fec7511cb854 - - - - -] Message received from the host: labnova.example.com during the registration of Open vSwitch agent has a timestamp: 2016-09-20T22:55:24.425782. This differs from the current server timestamp: 2016-09-23T19:02:13.736852 by 245209.31107 seconds, which is more than the threshold agent downtime: 75.
2016-09-23 15:02:20.677 3828 WARNING neutron.plugins.ml2.drivers.l2pop.mech_driver [req-9628b672-43b1-4883-89a3-fec7511cb854 - - - - -] Unable to retrieve active L2 agent on host labnova.example.com
2016-09-23 15:02:21.138 3828 WARNING neutron.plugins.ml2.drivers.l2pop.mech_driver [req-9628b672-43b1-4883-89a3-fec7511cb854 - - - - -] Unable to retrieve active L2 agent on host labnova.example.com
2016-09-23 15:02:21.633 3828 WARNING neutron.plugins.ml2.drivers.l2pop.mech_driver [req-9628b672-43b1-4883-89a3-fec7511cb854 - - - - -] Unable to retrieve active L2 agent on host labnova.example.com


On the nova server the service neutron-openvswitch-agent had already been running.

#systemctl –a | grep -i  neutron-openvswitch-agent.service                                                                              loaded    active   running  

There was a date difference of 3 days on the controller and the nova node.
root@labnova:/etc/neutron/plugins/ml2[root@labnova ml2]# date
Tue Sep 20 18:57:26 EDT 2016

At the same time the date was on the controller server as

root@labcontroller:/var/log/neutron[root@labcontroller neutron]# date
Fri Sep 23 15:13:39 EDT 2016


The fix:

The fix was to restart the chronyd on the nova server to sync the time correctly with the NTP Source and restarting the neutron-openvswitch-agent.service running on the nova server labnova.