rootless: guide for Bottlerocket OS (sysctl -w user.max_user_namespaces=N)

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

docs/rootless.md

@@ -24,6 +24,12 @@ spec:
 
 See also the [example manifests](#Kubernetes).
 
+### Bottlerocket OS
+
+Needs to run `sysctl -w user.max_user_namespaces=N` (N=positive integer, like 63359) on the host nodes.
+
+See [`../examples/kubernetes/sysctl-userns.privileged.yaml`](../examples/kubernetes/sysctl-userns.privileged.yaml).
+
 <details>
 <summary>Old distributions</summary>
 
@@ -104,6 +110,11 @@ See https://rootlesscontaine.rs/getting-started/common/subuid/
 ### Error `Options:[rbind ro]}]: operation not permitted`
 Make sure to mount an `emptyDir` volume on `/home/user/.local/share/buildkit` .
 
+### Error `fork/exec /proc/self/exe: no space left on device` with `level=warning msg="/proc/sys/user/max_user_namespaces needs to be set to non-zero."`
+Run `sysctl -w user.max_user_namespaces=N` (N=positive integer, like 63359) on the host nodes.
+
+See [`../examples/kubernetes/sysctl-userns.privileged.yaml`](../examples/kubernetes/sysctl-userns.privileged.yaml).
+
 ## Containerized deployment
 
 ### Kubernetes

examples/kubernetes/sysctl-userns.privileged.yaml

@@ -0,0 +1,26 @@
+# Run `sysctl -w user.max_user_namespaces=63359` on all the nodes,
+# for errors like "/proc/sys/user/max_user_namespaces needs to be set to non-zero"
+# on running rootless buildkitd pods.
+#
+# This workaround is known to be needed on Bottlerocket OS.
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  labels:
+    app: sysctl-userns
+  name: sysctl-userns
+spec:
+  selector:
+    matchLabels:
+      app: sysctl-userns
+  template:
+    metadata:
+      labels:
+        app: sysctl-userns
+    spec:
+      containers:
+        - name: sysctl-userns
+          image: busybox
+          command: ["sh", "-euxc", "sysctl -w user.max_user_namespaces=63359 && sleep infinity"]
+          securityContext:
+            privileged: true