test-integration-flaky: Add /flaky-check=Test1,Test2 PR body directive

Allows PR authors to stress-test specific integration tests for
flakiness even when the diff doesn't touch them.

Add a /flaky-check directive on its own line in the PR body:

/flaky-check=TestFoo,TestBar

The GHA integration-flaky job parses the directive from the PR body and
exports the names as FLAKY_EXTRA_TESTS, which the script appends to the
diff-detected set before running the stress loop.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>

.github/PULL_REQUEST_TEMPLATE.md

@@ -4,6 +4,10 @@ https://github.com/moby/moby/blob/master/CONTRIBUTING.md
 
 Make sure commits are signed off (`git commit -s`) with your real name.
 
+To explicitly stress-test specific integration tests for flakiness in CI,
+add a /flaky-check directive anywhere in the PR body (outside of comments):
+  /flaky-check=TestFoo,TestBar
+
 If the PR relates to an existing issue or PR, mention it at the top:
 
 - Fixes: https://github.com/moby/moby/issues/12345678

.github/workflows/.test.yml

@@ -108,6 +108,25 @@ jobs:
         with:
           # Full history needed to diff against target branch and detect new tests (test-integration-flaky).
           fetch-depth: 0
+      -
+        # Extract a /flaky-check=TestA,TestB directive from the PR body and
+        # expose it as FLAKY_EXTRA_TESTS so that hack/make/test-integration-flaky
+        # appends those tests to the ones detected from the diff.
+        name: Extract flaky-check extra tests
+        if: github.event_name == 'pull_request'
+        env:
+          PR_BODY: ${{ github.event.pull_request.body }}
+        run: |
+          flaky_tests=$(
+            printf '%s\n' "$PR_BODY" |
+            tr -d '\r' |
+            grep -oEm1 '^/flaky-check=Test[A-Za-z0-9_]+(,Test[A-Za-z0-9_]+)*$' |
+            sed 's|^/flaky-check=||'
+          )
+          if [ -n "$flaky_tests" ]; then
+            echo "Appending /flaky-check tests: $flaky_tests"
+            echo "FLAKY_EXTRA_TESTS=$flaky_tests" >> "$GITHUB_ENV"
+          fi
       -
         name: Set up runner
         uses: ./.github/actions/setup-runner

Makefile

@@ -57,6 +57,7 @@ DOCKER_ENVS := \
 	-e TEST_SKIP_INTEGRATION \
 	-e TEST_SKIP_INTEGRATION_CLI \
 	-e TESTCOVERAGE \
+	-e FLAKY_EXTRA_TESTS \
 	-e TESTDEBUG \
 	-e TESTDIRS \
 	-e TESTFLAGS \

hack/make/test-integration-flaky

@@ -17,11 +17,21 @@ changed_tests() {
 		|| true
 }
 
+# explicit_tests emits extra test names appended via the /flaky-check=TestA,TestB
+# directive in the PR body.  The workflow extracts that directive and passes it
+# here as the FLAKY_EXTRA_TESTS variable.
+explicit_tests() {
+	if [ -n "${FLAKY_EXTRA_TESTS-}" ]; then
+		echo "$FLAKY_EXTRA_TESTS" | tr ',' '\n'
+	fi
+}
+
 run_integration_flaky() {
 	tests=$(
 		{
 			added_tests
 			changed_tests
+			explicit_tests
 		} | sort -u
 	)