When a container is stopped, new connections to a published port will
not be routed to that container but any existing connections are
permitted to continue uninterrupted for the duration of the container's
grace period. Unfortunately recent fixes to overlay networks introduced
a regression: existing connections routed over the service mesh to
containers on remote nodes are dropped immediately when the container is
stopped, irrespective of the grace period.
Fix the handling of NetworkDB endpoint table events so that the endpoint
is disabled in the load balancer when a service endpoint transitions to
ServiceDisabled instead of deleting the endpoint and re-adding it. And
fix the other bugged state transitions with the help of a unit test
which exhaustively covers all permutations of endpoint event.
Signed-off-by: Cory Snider <csnider@mirantis.com>
The eventually-consistent nature of NetworkDB means we cannot depend on
events being received in the same order that they were sent. Nor can we
depend on receiving events for all intermediate states. It is possible
for a series of entry UPDATEs, or a DELETE followed by a CREATE with the
same key, to get coalesced into a single UPDATE event on the receiving
node. Watchers of NetworkDB tables therefore need to be prepared to
gracefully handle arbitrary UPDATEs of a key, including those where the
new value may have nothing in common with the previous value.
The libnetwork controller naively handled events for endpoint_table
assuming that an endpoint leave followed by a rejoin of the same
endpoint would always be expressed as a DELETE event followed by a
CREATE. It would handle a coalesced UPDATE as a CREATE, adding a new
service binding without removing the old one. This would
have various side effects, such as having the "transient state" of
having multiple conflicting service bindings where more than one
endpoint is assigned an IP address never settling.
Modify the libnetwork controller to handle an UPDATE by removing the
previous service binding then adding the new one.
Signed-off-by: Cory Snider <csnider@mirantis.com>
