libnet/i/kv/boltdb: fail fast in case of contention

Make sure an error is returned straight away if there's contention on
the underlying db file. This makes sure we don't reintroduce the issue
fixed in d21d088, and it will help detect contention in parallelized
tests if they're badly written. It effectively adds a new error mode to
the daemon, but if anyone faces this error, they should fix their
process manager.

Signed-off-by: Albin Kerouanton <albinker@gmail.com>

libnetwork/internal/kvstore/boltdb/boltdb.go

@@ -3,6 +3,8 @@ package boltdb
 import (
 	"bytes"
 	"encoding/binary"
+	"errors"
+	"fmt"
 	"os"
 	"path/filepath"
 	"sync"
@@ -34,9 +36,20 @@ func New(path, bucket string) (store.Store, error) {
 	}
 
 	db, err := bolt.Open(path, filePerm, &bolt.Options{
-		Timeout: time.Minute,
+		// The bbolt package opens the underlying db file and then issues an
+		// exclusive flock to ensures that it can safely write to the db. If
+		// it fails, it'll re-issue flocks every few ms until Timeout is
+		// reached.
+		// This nanosecond timeout bypasses that retry loop and make sure the
+		// bbolt package returns an ErrTimeout straight away. That way, the
+		// daemon, and unit tests, will fail fast and loudly instead of
+		// silently introducing delays.
+		Timeout: time.Nanosecond,
 	})
 	if err != nil {
+		if errors.Is(err, bolt.ErrTimeout) {
+			return nil, fmt.Errorf("boltdb file %s is already open", path)
+		}
 		return nil, err
 	}