ctest: Add CMAKE coverage tool for CMake listfile coverage

Add a `CMAKE` value for the `CoverageTool` setting, next to `LLVM-COV`.
With it, ctest gives each test its own scratch folder,
`<test-name>_cmake-coverage`, next to the `CTestTestfile.cmake` that
describes the test, and runs the test with CMAKE_LISTFILE_COVERAGE_DIR
set to that folder.  An ambient value of the variable is replaced.
Before the test, ctest removes the folder and the merged file of an
earlier run.

After the test, ctest merges the tracefiles in the folder into one
`<test-name>.cmake-coverage.info` file with a `TN:` record, and removes
the folder.  Coverage problems are warnings and never change the test
result: a folder that cannot be created, an unreadable file, a merged
file that cannot be written, and a test that timed out.  The folder is
kept for inspection when a file was skipped or the write failed.

The merge leaves out the files under ctest's build directory, such as
generated files and `try_compile` projects, unless the build directory
is the source directory.  If no file is left, the test gets no merged
file, as for a test that runs no CMake code.

ctest now also records coverage of the CMake code that it runs itself,
such as `CTestTestfile.cmake` files and dashboard scripts, when
CMAKE_LISTFILE_COVERAGE_DIR is set at startup.  ctest reads the top
`CTestTestfile.cmake` by a relative path, so the collector now makes a
relative listfile path absolute with the current working directory.

Document the tool and add a recipe for CMake's own test suite to
`Tests/RunCMake/README.rst`.

Issue: #28192

Help/command/ctest_test.rst

@@ -214,9 +214,12 @@ The options are:
   .. versionadded:: 4.4
 
   Specify the ``<tool>`` used for collecting coverage during the running
-  of the tests.  If not specified the :variable:`CTEST_TEST_COVERAGE_TOOL`
-  variable will be checked.  See the CTest
-  :ref:`CoverageTool <ctest-CoverageTool>` setting for details.
+  of the tests: ``LLVM-COV`` or ``CMAKE``.  If not specified the
+  :variable:`CTEST_TEST_COVERAGE_TOOL` variable will be checked.  See the
+  CTest :ref:`CoverageTool <ctest-CoverageTool>` setting for details.
+
+  .. versionadded:: 4.6
+    The ``CMAKE`` tool.
 
 ``PRESET <preset>``
   .. versionadded:: 4.4

Help/dev/testing.rst

@@ -109,3 +109,41 @@ the test suite and build test binaries.  One may then run tests normally:
 Note that the configuration with which one drives tests in the second
 build tree is independent of the configuration with which CMake was
 built in the first.
+
+Measuring Coverage of CMake Code
+================================
+
+To see which lines of CMake language code, such as the files in
+``Modules/``, the tests run, use the ``CMAKE`` coverage tool of ``ctest``:
+
+1. Configure the CMake build with ``-DCTEST_TEST_COVERAGE_TOOL=CMAKE``
+   using ``cmake`` version 4.4 or newer.  A plain ``ctest`` run in the
+   build tree is then configured to collect coverage data.
+
+2. Run the tests with the ``ctest`` that the build produced, for example:
+
+   .. code-block:: console
+
+     $ bin/ctest -R '^RunCMake\.Example$'
+
+   For each test, ``ctest`` writes an lcov tracefile named
+   ``<test-name>.cmake-coverage.info`` into the directory of the build
+   tree that holds the test, here ``Tests/RunCMake``.  The tracefile
+   leaves out the files in the build tree, such as the build trees of the
+   test cases.
+
+3. Keep only the records of files in the CMake source tree.  The
+   tracefile can also name files outside both trees, such as the package
+   configuration files that ``find_package()`` reads.  With ``lcov``,
+   where ``<source>`` is the absolute path of the CMake source tree:
+
+   .. code-block:: console
+
+     $ lcov --extract Tests/RunCMake/RunCMake.Example.cmake-coverage.info \
+         '<source>/*' -o example.info
+
+4. Make an HTML report with ``genhtml`` from lcov:
+
+   .. code-block:: console
+
+     $ genhtml -o coverage example.info

Help/envvar/CMAKE_LISTFILE_COVERAGE_DIR.rst

@@ -36,6 +36,10 @@ These modes record coverage:
 * Script mode, :option:`cmake -P`.
 * Install scripts, :option:`cmake --install`.
 * A re-run of configure that the build system starts.
+* :manual:`ctest(1)`, for the CMake language code that it runs itself, such
+  as ``CTestTestfile.cmake`` files and :ref:`Dashboard Client` scripts.
+  To record coverage for each test instead, use the ``CMAKE`` value of the
+  CTest :ref:`CoverageTool <ctest-CoverageTool>` setting.
 
 These modes do not record coverage and do not check the directory:
 
@@ -74,10 +78,12 @@ path that CMake used to read it.  This includes files under
 :variable:`CMAKE_ROOT` and files that CMake generates in the build tree.
 Records can name generated files that CMake deletes later, such as the
 projects that :command:`try_compile` configures, so report tools may need
-to filter them out.  Files that CMake never reads do not appear.  A file
-with no line that can count and no function or macro, such as a file with
-only comments, gets no record, and a process that has no record writes no
-file.
+to filter them out.  The CTest ``CMAKE``
+:ref:`coverage tool <ctest-CoverageTool>` leaves out the files under the
+build tree when it merges the files of a test, except for an in-source
+build.  Files that CMake never reads do not appear.  A file with no line
+that can count and no function or macro, such as a file with only comments,
+gets no record, and a process that has no record writes no file.
 
 Lines:
 

Help/manual/ctest.1.rst

@@ -1697,6 +1697,56 @@ Configuration settings include:
     are removed before running the test.  Any existing ``LLVM_PROFILE_FILE``
     environment variable is ignored.
 
+  ``CMAKE``
+    .. versionadded:: 4.6
+
+    This value records coverage of the CMake language code that each test
+    runs, for example the :option:`cmake -P` scripts and the projects that
+    the test configures.  See the :envvar:`CMAKE_LISTFILE_COVERAGE_DIR`
+    environment variable for what gets recorded.
+
+    Before each test, CTest removes any
+    ``<directory>/<test-name>.cmake-coverage.info`` file and
+    ``<directory>/<test-name>_cmake-coverage`` folder that an earlier run
+    left, and creates that folder again, empty.  ``<directory>``
+    is the absolute path to the directory where the ``CTestTestfile.cmake``
+    that describes the test is located.  CTest runs the test with the
+    :envvar:`CMAKE_LISTFILE_COVERAGE_DIR` environment variable set to the
+    folder, so each :program:`cmake` and :program:`ctest` process of the
+    test records into it.  Any existing value of the variable is ignored.
+
+    After the test, CTest merges the lcov tracefiles in the folder into
+    ``<directory>/<test-name>.cmake-coverage.info`` and removes the folder.
+    The merge adds the hit counts of each line and each function.  The
+    merged file leaves out the files under the ``BuildDirectory`` setting,
+    such as files that CMake generates and the projects that
+    :command:`try_compile` configures, unless that setting is the same as
+    the ``SourceDirectory`` setting.  The merged file has one ``TN:``
+    record that holds the test name, with each character other than a
+    letter, a digit or ``_`` replaced by ``_``.  The file name keeps the
+    real test name.  A test that runs no CMake language code, or only code
+    under the build tree, gets no file.
+
+    Coverage problems do not change the result of a test.  CTest reports
+    them as warnings:
+
+    * If CTest cannot create the folder, it runs the test without the
+      variable, and the test gets no file.
+    * If a file in the folder cannot be read, CTest skips it and merges
+      the others.  It keeps the folder for inspection.
+    * If CTest cannot write the merged file, it keeps the folder.
+    * If a test times out, CTest stops its processes, and their data is
+      lost.  CTest reports that the coverage of the test is incomplete.
+
+    As with ``LLVM-COV``, a test name that contains ``/`` gives nested
+    folders, and with :option:`ctest --repeat`, only the data of the last
+    run of a test is kept.
+
+    The variable set for a test applies only to the test.  CTest records
+    coverage of the CMake language code that it runs itself, such as
+    ``CTestTestfile.cmake`` files and :ref:`Dashboard Client` scripts, only
+    if :envvar:`CMAKE_LISTFILE_COVERAGE_DIR` is set when CTest starts.
+
   If no value is specified, no work is done and nothing is added to the
   environment.
 

Help/release/dev/listfile-coverage.rst

@@ -3,3 +3,7 @@ listfile-coverage
 
 * The :envvar:`CMAKE_LISTFILE_COVERAGE_DIR` environment variable was added
   to record coverage of CMake language code in lcov tracefiles.
+
+* :manual:`ctest(1)` gained a ``CMAKE`` value for its
+  :ref:`CoverageTool <ctest-CoverageTool>` setting to record coverage of
+  CMake language code for each test.

Help/variable/CTEST_TEST_COVERAGE_TOOL.rst

@@ -6,3 +6,7 @@ CTEST_TEST_COVERAGE_TOOL
 Specify the CTest :ref:`CoverageTool <ctest-CoverageTool>` setting
 in a :manual:`ctest(1)` :ref:`Dashboard Client` script,
 or on the :program:`ctest` command line via the :ctest-dashboard-option:`-D` option.
+The value may be ``LLVM-COV`` or ``CMAKE``.
+
+.. versionadded:: 4.6
+  The ``CMAKE`` value.

Source/CTest/cmCTestRunTest.cxx

@@ -7,6 +7,7 @@
 #include <cstddef> // IWYU pragma: keep
 #include <cstdint>
 #include <cstdio>
+#include <functional>
 #include <ratio>
 #include <sstream>
 #include <utility>
@@ -18,18 +19,22 @@
 
 #include <cm3p/json/value.h>
 
+#include "cmsys/Directory.hxx"
 #include "cmsys/FStream.hxx"
 #include "cmsys/Glob.hxx"
 #include "cmsys/RegularExpression.hxx"
+#include "cmsys/String.h"
 
 #include "cmCTest.h"
 #include "cmCTestMemCheckHandler.h"
 #include "cmCTestMultiProcessHandler.h"
 #include "cmDuration.h"
 #include "cmEnvironment.h"
+#include "cmGeneratedFileStream.h"
 #include "cmInstrumentation.h"
 #include "cmInstrumentationQuery.h"
 #include "cmJSONState.h"
+#include "cmLcovTracefile.h"
 #include "cmList.h"
 #include "cmProcess.h"
 #include "cmStringAlgorithms.h"
@@ -940,6 +945,40 @@ bool cmCTestRunTest::ForkProcess()
     cmSystemTools::RemoveFile(profDataPath);
   };
 
+  // Give the test its own folder for CMake listfile coverage.
+  if (coverageTool == "CMAKE"_s) {
+    this->UseCMakeCov = true;
+    // Isolate the test from any ambient CMAKE_LISTFILE_COVERAGE_DIR
+    env.UnPutEnv("CMAKE_LISTFILE_COVERAGE_DIR");
+
+    // Remove the data of an earlier run.  Never remove a regular file or a
+    // symbolic link at the folder path.  A symbolic link blocks the folder,
+    // because ctest would record into the link target and delete its content.
+    std::string const scratch = this->GenerateLLVMPath("_cmake-coverage");
+    bool cleared = true;
+    if (cmSystemTools::FileIsSymlink(scratch)) {
+      cleared = false;
+    } else if (cmSystemTools::FileIsDirectory(scratch)) {
+      cleared = static_cast<bool>(cmSystemTools::RemoveADirectory(scratch));
+    }
+    cmSystemTools::RemoveFile(this->GenerateLLVMPath(".cmake-coverage.info"));
+
+    this->CMakeCovScratchOk =
+      cleared && static_cast<bool>(cmSystemTools::MakeDirectory(scratch));
+    if (this->CMakeCovScratchOk) {
+      cmCTestOptionalLog(
+        this->CTest, HANDLER_VERBOSE_OUTPUT, this->TestHandler->GetQuiet(),
+        this->Index,
+        ": Using environment variable CMAKE_LISTFILE_COVERAGE_DIR="_s, scratch,
+        '\n');
+      env.PutEnv(cmStrCat("CMAKE_LISTFILE_COVERAGE_DIR="_s, scratch));
+    } else {
+      cmCTestLog(this->CTest, WARNING,
+                 "Cannot create CMake listfile coverage folder: ", scratch,
+                 '\n');
+    }
+  }
+
   if (this->UseAllocatedResources) {
     this->SetupResourcesEnvironment(env);
   } else {
@@ -1150,6 +1189,93 @@ void cmCTestRunTest::CollectLLVMCoverage()
   cmSystemTools::RemoveFile(profManifestPath);
 }
 
+void cmCTestRunTest::CollectCMakeCoverage()
+{
+  if (!this->CMakeCovScratchOk) {
+    return;
+  }
+  this->CMakeCovScratchOk = false;
+
+  // Coverage problems are warnings.  An error would make ctest -S fail.
+  std::string const& testName = this->TestProperties->Name;
+  if (this->TestProcess->GetProcessStatus() == cmProcess::State::Expired) {
+    cmCTestLog(this->CTest, WARNING, "CMake listfile coverage of test ",
+               testName, " is incomplete because the test timed out.\n");
+  }
+
+  // List the files with cmsys::Directory, because a glob pattern would
+  // treat "[" and "]" in the test name as special.
+  std::string const scratch = this->GenerateLLVMPath("_cmake-coverage");
+  std::vector<std::string> inputs;
+  {
+    cmsys::Directory dir;
+    if (dir.Load(scratch)) {
+      for (unsigned long i = 0; i < dir.GetNumberOfFiles(); ++i) {
+        std::string const& name = dir.GetFileName(i);
+        if (cmHasLiteralSuffix(name, ".info") && !dir.FileIsDirectory(i)) {
+          inputs.push_back(cmStrCat(scratch, '/', name));
+        }
+      }
+    }
+  }
+  std::sort(inputs.begin(), inputs.end());
+
+  cmLcovTracefile merged;
+  auto const warn = [this](std::string const& warning) {
+    cmCTestLog(this->CTest, WARNING, warning, '\n');
+  };
+  bool keepScratch = !merged.MergeFiles(inputs, warn);
+
+  // Leave out the files in the build tree, such as generated files and
+  // try_compile projects, unless the build tree is the source tree.
+  std::string const sourceDir =
+    this->CTest->GetCTestConfiguration("SourceDirectory");
+  std::string buildDir = this->CTest->GetCTestConfiguration("BuildDirectory");
+  if (buildDir.empty()) {
+    buildDir = this->CTest->GetBinaryDir();
+  }
+  auto const logRemoved = [this](std::string const& path) {
+    cmCTestLog(this->CTest, HANDLER_VERBOSE_OUTPUT, this->GetIndex(),
+               ": Ignoring CMake listfile coverage of a file in the build "
+               "tree: ",
+               path, '\n');
+  };
+  merged.RemoveBuildTreeFiles(sourceDir, buildDir, logRemoved);
+
+  // A test that runs no CMake code, or only code in the build tree or in
+  // files with no lines and no functions, has no data.  That is not a
+  // problem.
+  if (merged.HasData()) {
+    // lcov accepts only letters, digits and "_" in a test name.
+    merged.TestName = testName;
+    for (char& c : merged.TestName) {
+      if (!cmsysString_isalnum(c) && c != '_') {
+        c = '_';
+      }
+    }
+
+    // Close() does not report a failed rename, so check the result too.
+    std::string const output = this->GenerateLLVMPath(".cmake-coverage.info");
+    bool written = false;
+    {
+      cmGeneratedFileStream fout(output, true);
+      merged.Write(fout);
+      written = static_cast<bool>(fout) && fout.Close() &&
+        cmSystemTools::FileExists(output, true);
+    }
+    if (!written) {
+      cmCTestLog(this->CTest, WARNING,
+                 "Cannot write CMake listfile coverage file: ", output, '\n');
+      keepScratch = true;
+    }
+  }
+
+  // Never remove through a symbolic link.
+  if (!keepScratch && !cmSystemTools::FileIsSymlink(scratch)) {
+    cmSystemTools::RemoveADirectory(scratch);
+  }
+}
+
 void cmCTestRunTest::FinalizeTest(bool started)
 {
   // Collect known LLVM coverage files into binary form
@@ -1157,6 +1283,11 @@ void cmCTestRunTest::FinalizeTest(bool started)
     this->CollectLLVMCoverage();
   }
 
+  // Merge the CMake listfile coverage files of the test
+  if (this->UseCMakeCov) {
+    this->CollectCMakeCoverage();
+  }
+
   if (started && this->CTest->GetInstrumentation().HasQuery()) {
     if (this->CTest->GetInstrumentation().HasOption(
           cmInstrumentationQuery::Option::ProcessMetrics)) {

Source/CTest/cmCTestRunTest.h

@@ -136,6 +136,7 @@ private:
   std::string GenerateLLVMPath(std::string fileString);
   std::string GetTestMetricsFile() const;
   void CollectLLVMCoverage();
+  void CollectCMakeCoverage();
   void SetupResourcesEnvironment(cmEnvironment& env);
 
   // Returns "completed/total Test #Index: "
@@ -168,6 +169,9 @@ private:
   int RunCount = 1;  // how many runs it may be given
   bool RunAgain = false;
   bool UseLLVMCov = false;
+  bool UseCMakeCov = false;
+  // Whether this run of the test has a CMake listfile coverage folder.
+  bool CMakeCovScratchOk = false;
   size_t TotalNumberOfTests;
 };
 

Source/cmLcovTracefile.cxx

@@ -5,7 +5,6 @@
 #include <algorithm>
 #include <array>
 #include <cstdio>
-#include <ostream>
 #include <utility>
 #include <vector>
 
@@ -13,6 +12,8 @@
 #include <cmext/algorithm>
 #include <cmext/string_view>
 
+#include "cmsys/FStream.hxx"
+
 #include "cmStringAlgorithms.h"
 #include "cmSystemTools.h"
 
@@ -113,6 +114,55 @@ void cmLcovTracefile::Merge(cmLcovTracefile const& other)
   }
 }
 
+bool cmLcovTracefile::MergeFiles(
+  std::vector<std::string> const& paths,
+  std::function<void(std::string const&)> const& warn)
+{
+  bool allMerged = true;
+  for (std::string const& path : paths) {
+    // Read into a fresh object, so that a bad file adds nothing.
+    cmLcovTracefile tracefile;
+    std::string error;
+    cmsys::ifstream fin(path.c_str());
+    if (!fin || !tracefile.Read(fin, error)) {
+      std::string warning =
+        cmStrCat("Skipping unreadable CMake listfile coverage file: ", path);
+      if (!error.empty()) {
+        warning = cmStrCat(std::move(warning), "\n  ", error);
+      }
+      warn(warning);
+      allMerged = false;
+      continue;
+    }
+    this->Merge(tracefile);
+  }
+  return allMerged;
+}
+
+void cmLcovTracefile::RemoveBuildTreeFiles(
+  std::string const& sourceDir, std::string const& buildDir,
+  std::function<void(std::string const&)> const& removed)
+{
+  if (buildDir.empty()) {
+    return;
+  }
+  std::string const build = cmSystemTools::CollapseFullPath(buildDir);
+  if (!sourceDir.empty() &&
+      cmSystemTools::ComparePath(cmSystemTools::CollapseFullPath(sourceDir),
+                                 build)) {
+    return;
+  }
+  for (auto it = this->Files.begin(); it != this->Files.end();) {
+    std::string const path = cmSystemTools::CollapseFullPath(it->first);
+    if (cmSystemTools::IsSubDirectory(path, build)) {
+      removed(path);
+      it = this->Files.erase(it);
+    } else {
+      ++it;
+    }
+  }
+}
+
 bool cmLcovTracefile::HasData() const
 {
   return std::any_of(

Source/cmLcovTracefile.h

@@ -4,9 +4,11 @@
 
 #include "cmConfigure.h" // IWYU pragma: keep
 
+#include <functional>
 #include <iosfwd>
 #include <map>
 #include <string>
+#include <vector>
 
 /** \brief Hit count of one function in an lcov tracefile.  */
 struct cmLcovFunction
@@ -51,6 +53,20 @@ public:
       name does not change.  */
   void Merge(cmLcovTracefile const& other);
 
+  /** Read the tracefiles at PATHS and merge them into this object.  Skip a
+      file that can not be read, and call WARN with a message for it.
+      Return false if a file was skipped.  */
+  bool MergeFiles(std::vector<std::string> const& paths,
+                  std::function<void(std::string const&)> const& warn);
+
+  /** Remove the files under BUILDDIR, such as generated files and
+      try_compile projects, and call REMOVED with the path of each one.
+      Remove nothing if BUILDDIR is empty or is SOURCEDIR, as in an
+      in-source build.  */
+  void RemoveBuildTreeFiles(
+    std::string const& sourceDir, std::string const& buildDir,
+    std::function<void(std::string const&)> const& removed);
+
   /** True if some file has a line or a function record.  */
   bool HasData() const;
 

Source/cmListFileCoverage.cxx

@@ -132,7 +132,10 @@ cmListFileCoverage::SyntheticCommandScope::~SyntheticCommandScope()
 void cmListFileCoverage::FileParsed(
   std::string const& path, std::vector<cmListFileFunction> const& functions)
 {
-  cmLcovSourceFile& file = this->Files[path];
+  cmLcovSourceFile& file =
+    this->Files[cmSystemTools::FileIsFullPath(path)
+                  ? path
+                  : cmSystemTools::CollapseFullPath(path)];
   for (long line : ExecutableLines(functions)) {
     file.Lines.emplace(line, 0);
   }
@@ -152,7 +155,7 @@ void cmListFileCoverage::CommandRan(cmListFileFunction const& lff,
   if (this->SyntheticCommands.count(&lff)) {
     return;
   }
-  auto file = this->Files.find(path);
+  auto file = this->FindFile(path);
   if (file == this->Files.end()) {
     return;
   }
@@ -165,7 +168,7 @@ void cmListFileCoverage::CommandRan(cmListFileFunction const& lff,
 void cmListFileCoverage::FunctionCalled(std::string const& path,
                                         long definitionLine)
 {
-  auto file = this->Files.find(path);
+  auto file = this->FindFile(path);
   if (file == this->Files.end()) {
     return;
   }
@@ -175,6 +178,15 @@ void cmListFileCoverage::FunctionCalled(std::string const& path,
   }
 }
 
+std::map<std::string, cmLcovSourceFile>::iterator cmListFileCoverage::FindFile(
+  std::string const& path)
+{
+  if (cmSystemTools::FileIsFullPath(path)) {
+    return this->Files.find(path);
+  }
+  return this->Files.find(cmSystemTools::CollapseFullPath(path));
+}
+
 cmLcovTracefile cmListFileCoverage::ToTracefile() const
 {
   cmLcovTracefile tracefile;

Source/cmListFileCoverage.h

@@ -55,7 +55,9 @@ public:
 
   /** Register the executable lines and the function and macro definitions
       of a parsed file.  A second call for the same path adds new lines and
-      functions and keeps existing counts.  */
+      functions and keeps existing counts.  A relative path, as ctest uses
+      for its top CTestTestfile.cmake, is made absolute with the current
+      working directory, here and in CommandRan and FunctionCalled.  */
   void FileParsed(std::string const& path,
                   std::vector<cmListFileFunction> const& functions);
 
@@ -77,6 +79,9 @@ public:
 private:
   explicit cmListFileCoverage(std::string outputPath);
 
+  std::map<std::string, cmLcovSourceFile>::iterator FindFile(
+    std::string const& path);
+
   std::string OutputPath;
   std::map<std::string, cmLcovSourceFile> Files;
   std::set<cmListFileFunction const*> SyntheticCommands;

Source/ctest.cxx

@@ -13,6 +13,7 @@
 #include "cmDocumentationEntry.h"
 #include "cmInstrumentation.h"
 #include "cmInstrumentationQuery.h"
+#include "cmListFileCoverage.h"
 #include "cmStdIoConsole.h"
 #include "cmSystemTools.h"
 
@@ -267,6 +268,15 @@ int main(int argc, char const* const* argv)
   // copy the args to a vector
   auto args = std::vector<std::string>(argv, argv + argc);
 
+  // Record coverage of the CMake code that ctest runs itself.
+  std::string error;
+  if (!cmListFileCoverage::Start(error)) {
+    cmSystemTools::Error(error);
+    return 1;
+  }
+
   // run ctest
-  return cmCTest{}.Run(args);
+  int const ret = cmCTest{}.Run(args);
+  cmListFileCoverage::Finish();
+  return ret;
 }

Tests/CMakeLib/testLcovTracefile.cxx

@@ -6,8 +6,12 @@
 #include <sstream>
 #include <string>
 #include <utility>
+#include <vector>
+
+#include "cmsys/FStream.hxx"
 
 #include "cmLcovTracefile.h"
+#include "cmSystemTools.h"
 
 #include "testCommon.h"
 
@@ -231,6 +235,76 @@ bool testMerge()
   ASSERT_EQUAL(writeText(a), writeText(expected));
   return true;
 }
+
+bool testMergeFiles()
+{
+  std::string const good = "testLcovTracefile-good.info";
+  std::string const bad = "testLcovTracefile-bad.info";
+  {
+    cmsys::ofstream fout(good.c_str());
+    fout << "SF:/x\nDA:1,2\nend_of_record\n";
+  }
+  {
+    cmsys::ofstream fout(bad.c_str());
+    fout << "SF:/y\nDA:1,1\nend_of_record\nSF:/z\nDA:a,1\nend_of_record\n";
+  }
+
+  cmLcovTracefile t;
+  t.Files["/x"].Lines[1] = 1;
+  std::vector<std::string> warnings;
+  auto warn = [&warnings](std::string const& w) { warnings.push_back(w); };
+  ASSERT_TRUE(t.MergeFiles({ good }, warn));
+  ASSERT_TRUE(warnings.empty());
+  ASSERT_TRUE(!t.MergeFiles({ bad, "testLcovTracefile-missing.info" }, warn));
+  cmSystemTools::RemoveFile(good);
+  cmSystemTools::RemoveFile(bad);
+  ASSERT_EQUAL(warnings.size(), 2u);
+  ASSERT_EQUAL(warnings[0],
+               "Skipping unreadable CMake listfile coverage file: "
+               "testLcovTracefile-bad.info\n  line 5: malformed DA record "
+               "\"a,1\"");
+  ASSERT_EQUAL(warnings[1],
+               "Skipping unreadable CMake listfile coverage file: "
+               "testLcovTracefile-missing.info");
+
+  // The bad file adds nothing, not even its good first record.
+  ASSERT_EQUAL(writeText(t),
+               "SF:/x\nFNF:0\nFNH:0\nDA:1,3\nLF:1\nLH:1\nend_of_record\n");
+  return true;
+}
+
+bool testRemoveBuildTreeFiles()
+{
+  auto sample = []() {
+    cmLcovTracefile t;
+    for (char const* path :
+         { "/s/a.cmake", "/b/gen.cmake", "/b/sub/x.cmake", "/bb/y.cmake" }) {
+      t.Files[path].Lines[1] = 1;
+    }
+    return t;
+  };
+
+  std::vector<std::string> removed;
+  auto collect = [&removed](std::string const& path) {
+    removed.push_back(path);
+  };
+
+  cmLcovTracefile t = sample();
+  t.RemoveBuildTreeFiles("/s", "/b", collect);
+  ASSERT_EQUAL(removed.size(), 2u);
+  ASSERT_EQUAL(t.Files.size(), 2u);
+  ASSERT_EQUAL(t.Files.count("/s/a.cmake"), 1u);
+  ASSERT_EQUAL(t.Files.count("/bb/y.cmake"), 1u);
+
+  // An in-source build, or no build tree, keeps every file.
+  removed.clear();
+  t = sample();
+  t.RemoveBuildTreeFiles("/b", "/b/", collect);
+  t.RemoveBuildTreeFiles("/s", "", collect);
+  ASSERT_TRUE(removed.empty());
+  ASSERT_EQUAL(t.Files.size(), 4u);
+  return true;
+}
 }
 
 int testLcovTracefile(int /*unused*/, char* /*unused*/[])
@@ -239,6 +313,7 @@ int testLcovTracefile(int /*unused*/, char* /*unused*/[])
                     testReadIgnoresUnknown, testReadCrlfAndDrive,
                     testReadSameFileTwice, testReadStripsSuffix,
                     testReadErrors, testNameEncoding, testWriteSkipsEmptyFiles,
-                    testHasData, testMerge },
+                    testHasData, testMerge, testMergeFiles,
+                    testRemoveBuildTreeFiles },
                   false);
 }

Tests/RunCMake/CTestCoverage/CoverageBad.cmake

@@ -0,0 +1,6 @@
+# Never write to the file system root.
+if(NOT DEFINED ENV{CMAKE_LISTFILE_COVERAGE_DIR})
+  message(FATAL_ERROR "CMAKE_LISTFILE_COVERAGE_DIR is not set")
+endif()
+# A record with no end_of_record line.
+file(WRITE "$ENV{CMAKE_LISTFILE_COVERAGE_DIR}/bad.info" "SF:/x\nDA:1,1\n")

Tests/RunCMake/CTestCoverage/CoverageBlocked.cmake

@@ -0,0 +1,3 @@
+if(DEFINED ENV{CMAKE_LISTFILE_COVERAGE_DIR})
+  message(FATAL_ERROR "CMAKE_LISTFILE_COVERAGE_DIR is set: $ENV{CMAKE_LISTFILE_COVERAGE_DIR}")
+endif()

Tests/RunCMake/CTestCoverage/CoverageChild.cmake

@@ -0,0 +1 @@
+set(child 1)

Tests/RunCMake/CTestCoverage/CoverageCommentOnly.cmake

@@ -0,0 +1 @@
+# This file has only comments, so it gets no record.

Tests/RunCMake/CTestCoverage/CoverageScript.cmake

@@ -0,0 +1,2 @@
+execute_process(COMMAND ${CMAKE_COMMAND} -P ${CMAKE_CURRENT_LIST_DIR}/CoverageChild.cmake)
+execute_process(COMMAND ${CMAKE_COMMAND} -P ${CMAKE_CURRENT_LIST_DIR}/CoverageChild.cmake)

Tests/RunCMake/CTestCoverage/CoverageToolCMake-bad-check.cmake

@@ -0,0 +1,7 @@
+set(tree "${RunCMake_TEST_BINARY_DIR}")
+# The script's own file is merged.  The scratch folder stays for inspection.
+foreach(path IN ITEMS "bad.cmake-coverage.info" "bad_cmake-coverage/bad.info")
+  if(NOT EXISTS "${tree}/${path}")
+    string(APPEND RunCMake_TEST_FAILED "Missing ${tree}/${path}\n")
+  endif()
+endforeach()

Tests/RunCMake/CTestCoverage/CoverageToolCMake-bad-stderr.txt

@@ -0,0 +1,2 @@
+Skipping unreadable CMake listfile coverage file: [^
+]*bad\.info

Tests/RunCMake/CTestCoverage/CoverageToolCMake-check.cmake

@@ -0,0 +1,68 @@
+set(tree "${RunCMake_TEST_BINARY_DIR}")
+
+# The per-test file merges the files of all three cmake processes.
+set(info "${tree}/test one.cmake-coverage.info")
+if(EXISTS "${info}")
+  file(READ "${info}" actual)
+  string(REPLACE "${RunCMake_SOURCE_DIR}/" "<SRC>/" actual "${actual}")
+  file(READ "${RunCMake_SOURCE_DIR}/CoverageToolCMake-lcov.txt" expected)
+  string(REPLACE "\r\n" "\n" expected "${expected}")
+  if(NOT actual STREQUAL expected)
+    string(APPEND RunCMake_TEST_FAILED
+      "${info} does not match CoverageToolCMake-lcov.txt.\n"
+      "Expected:\n${expected}Actual:\n${actual}")
+  endif()
+else()
+  string(APPEND RunCMake_TEST_FAILED "Missing ${info}\n")
+endif()
+
+# The scratch folders are gone, and the stale data of nocode with them.
+foreach(path IN ITEMS
+    "test one_cmake-coverage"
+    "nocode_cmake-coverage"
+    "nocode.cmake-coverage.info"
+    )
+  if(EXISTS "${tree}/${path}")
+    string(APPEND RunCMake_TEST_FAILED "Unexpected ${tree}/${path}\n")
+  endif()
+endforeach()
+
+# ctest itself records into the ambient directory.  Its tests do not.
+set(cov_dir "${tree}/ambient")
+set(forbid_sf_regex "/CoverageScript\\.cmake$")
+include(${RunCMake_SOURCE_DIR}/../ListFileCoverage/check-lcov.cmake)
+
+# The merge leaves out the records of scripts in the build tree.
+set(info "${tree}/generated.cmake-coverage.info")
+if(EXISTS "${info}")
+  file(READ "${info}" actual)
+  string(REPLACE "${RunCMake_SOURCE_DIR}/" "<SRC>/" actual "${actual}")
+  set(expected "TN:generated
+SF:<SRC>/CoverageChild.cmake
+FNF:0
+FNH:0
+DA:1,1
+LF:1
+LH:1
+end_of_record
+")
+  if(NOT actual STREQUAL expected)
+    string(APPEND RunCMake_TEST_FAILED
+      "${info} does not match.\nExpected:\n${expected}Actual:\n${actual}")
+  endif()
+else()
+  string(APPEND RunCMake_TEST_FAILED "Missing ${info}\n")
+endif()
+# Tests whose records have no lines and no functions write no file.
+foreach(path IN ITEMS
+    "buildonly.cmake-coverage.info"
+    "buildonly_cmake-coverage"
+    "commentonly.cmake-coverage.info"
+    "commentonly_cmake-coverage"
+    "emptyrecord.cmake-coverage.info"
+    "emptyrecord_cmake-coverage"
+    )
+  if(EXISTS "${tree}/${path}")
+    string(APPEND RunCMake_TEST_FAILED "Unexpected ${tree}/${path}\n")
+  endif()
+endforeach()

Tests/RunCMake/CTestCoverage/CoverageToolCMake-insource-check.cmake

@@ -0,0 +1,20 @@
+set(tree "${RunCMake_TEST_BINARY_DIR}")
+set(info "${tree}/insource.cmake-coverage.info")
+if(EXISTS "${info}")
+  file(READ "${info}" actual)
+  set(expected "TN:insource
+SF:${tree}/script.cmake
+FNF:0
+FNH:0
+DA:1,1
+LF:1
+LH:1
+end_of_record
+")
+  if(NOT actual STREQUAL expected)
+    string(APPEND RunCMake_TEST_FAILED
+      "${info} does not match.\nExpected:\n${expected}Actual:\n${actual}")
+  endif()
+else()
+  string(APPEND RunCMake_TEST_FAILED "Missing ${info}\n")
+endif()

Tests/RunCMake/CTestCoverage/CoverageToolCMake-lcov.txt

@@ -0,0 +1,16 @@
+TN:test_one
+SF:<SRC>/CoverageChild.cmake
+FNF:0
+FNH:0
+DA:1,2
+LF:1
+LH:1
+end_of_record
+SF:<SRC>/CoverageScript.cmake
+FNF:0
+FNH:0
+DA:1,1
+DA:2,1
+LF:2
+LH:2
+end_of_record

Tests/RunCMake/CTestCoverage/CoverageToolCMake-noscratch-stderr.txt

@@ -0,0 +1,2 @@
+Cannot create CMake listfile coverage folder: [^
+]*blocked_cmake-coverage

Tests/RunCMake/CTestCoverage/CoverageToolCMake-stdout.txt

@@ -0,0 +1,2 @@
+Using environment variable CMAKE_LISTFILE_COVERAGE_DIR=[^
+]*/CoverageToolCMake/test one_cmake-coverage

Tests/RunCMake/CTestCoverage/CoverageToolCMake-symlink-check.cmake

@@ -0,0 +1,8 @@
+set(tree "${RunCMake_TEST_BINARY_DIR}")
+set(target "${RunCMake_BINARY_DIR}/CoverageToolCMake-symlink-target")
+if(NOT EXISTS "${target}/keep.txt")
+  string(APPEND RunCMake_TEST_FAILED "Missing ${target}/keep.txt\n")
+endif()
+if(NOT IS_SYMLINK "${tree}/linked_cmake-coverage")
+  string(APPEND RunCMake_TEST_FAILED "Missing link ${tree}/linked_cmake-coverage\n")
+endif()

Tests/RunCMake/CTestCoverage/CoverageToolCMake-symlink-stderr.txt

@@ -0,0 +1,2 @@
+Cannot create CMake listfile coverage folder: [^
+]*linked_cmake-coverage

Tests/RunCMake/CTestCoverage/CoverageToolCMake-timeout-result.txt

@@ -0,0 +1 @@
+8

Tests/RunCMake/CTestCoverage/CoverageToolCMake-timeout-stderr.txt

@@ -0,0 +1,2 @@
+CMake listfile coverage of test [^
+]* is incomplete because the test timed out\.

Tests/RunCMake/CTestCoverage/CoverageToolCMake-writefail-check.cmake

@@ -0,0 +1,4 @@
+set(tree "${RunCMake_TEST_BINARY_DIR}")
+if(NOT IS_DIRECTORY "${tree}/wf_cmake-coverage")
+  string(APPEND RunCMake_TEST_FAILED "Missing ${tree}/wf_cmake-coverage\n")
+endif()

Tests/RunCMake/CTestCoverage/CoverageToolCMake-writefail-stderr.txt

@@ -0,0 +1,2 @@
+Cannot write CMake listfile coverage file: [^
+]*wf\.cmake-coverage\.info

Tests/RunCMake/CTestCoverage/RunCMakeTest.cmake

@@ -13,3 +13,133 @@ if(CMake_TEST_CLANG_COVERAGE)
     run_cmake_command(ClangCoverage-coverage ${CMAKE_CTEST_COMMAND} -D ExperimentalCoverage -C Debug -VV)
   endblock()
 endif()
+
+# ctest records its own CMake code when the variable is set at startup.
+block()
+  set(RunCMake_TEST_BINARY_DIR ${RunCMake_BINARY_DIR}/SelfCoverage)
+  set(RunCMake_TEST_NO_CLEAN 1)
+  file(REMOVE_RECURSE "${RunCMake_TEST_BINARY_DIR}")
+  file(MAKE_DIRECTORY "${RunCMake_TEST_BINARY_DIR}")
+  file(WRITE "${RunCMake_TEST_BINARY_DIR}/CTestTestfile.cmake" "
+add_test(t1 \"${CMAKE_COMMAND}\" -E true)
+")
+  set(cov_dir "${RunCMake_TEST_BINARY_DIR}/coverage")
+  run_cmake_command(SelfCoverage ${CMAKE_COMMAND} -E env
+    CMAKE_LISTFILE_COVERAGE_DIR=${cov_dir} ${CMAKE_CTEST_COMMAND})
+endblock()
+
+# The CMAKE coverage tool.  Make a clean test tree that uses it, with the
+# given CTestTestfile.cmake content, and set tree in the caller.
+function(prepare_CoverageToolCMake case testfile)
+  set(tree ${RunCMake_BINARY_DIR}/${case})
+  file(REMOVE_RECURSE "${tree}")
+  file(MAKE_DIRECTORY "${tree}")
+  file(WRITE "${tree}/DartConfiguration.tcl" "
+BuildDirectory: ${tree}
+CTestTestCoverageTool: CMAKE
+")
+  file(WRITE "${tree}/CTestTestfile.cmake" "${testfile}")
+  set(tree "${tree}" PARENT_SCOPE)
+endfunction()
+
+block()
+  prepare_CoverageToolCMake(CoverageToolCMake "
+add_test(\"test one\" \"${CMAKE_COMMAND}\" -P \"${RunCMake_SOURCE_DIR}/CoverageScript.cmake\")
+add_test(nocode \"${CMAKE_COMMAND}\" -E true)
+add_test(generated \"${CMAKE_COMMAND}\" -P \"${RunCMake_BINARY_DIR}/CoverageToolCMake/generated.cmake\")
+add_test(buildonly \"${CMAKE_COMMAND}\" -P \"${RunCMake_BINARY_DIR}/CoverageToolCMake/buildonly.cmake\")
+add_test(commentonly \"${CMAKE_COMMAND}\" -P \"${RunCMake_SOURCE_DIR}/CoverageCommentOnly.cmake\")
+add_test(emptyrecord \"${CMAKE_COMMAND}\" -P \"${RunCMake_BINARY_DIR}/CoverageToolCMake/emptyrecord.cmake\")
+")
+  set(RunCMake_TEST_BINARY_DIR ${tree})
+  set(RunCMake_TEST_NO_CLEAN 1)
+  # Stale data of an earlier run.  The stale file is valid and names a file
+  # outside the build tree, so a missing cleanup shows up as merged data.
+  file(WRITE "${tree}/nocode_cmake-coverage/stale.info"
+    "SF:${RunCMake_SOURCE_DIR}/stale.cmake\nDA:1,1\nend_of_record\n")
+  # Scripts in the build tree.  The merge leaves out their records.
+  file(WRITE "${tree}/generated.cmake"
+    "include(\"${RunCMake_SOURCE_DIR}/CoverageChild.cmake\")\n")
+  file(WRITE "${tree}/buildonly.cmake" "set(x 1)\n")
+  # This script writes a record with no lines for a file in the source tree.
+  # After the merge, only that record is left.
+  file(WRITE "${tree}/emptyrecord.cmake" "
+if(DEFINED ENV{CMAKE_LISTFILE_COVERAGE_DIR})
+  file(WRITE \"\$ENV{CMAKE_LISTFILE_COVERAGE_DIR}/empty.info\"
+    \"SF:${RunCMake_SOURCE_DIR}/CoverageChild.cmake\\nend_of_record\\n\")
+endif()
+")
+  file(WRITE "${tree}/nocode.cmake-coverage.info" "")
+  run_cmake_command(CoverageToolCMake ${CMAKE_COMMAND} -E env
+    CMAKE_LISTFILE_COVERAGE_DIR=${tree}/ambient ${CMAKE_CTEST_COMMAND} -V)
+endblock()
+
+block()
+  prepare_CoverageToolCMake(CoverageToolCMake-insource "
+add_test(insource \"${CMAKE_COMMAND}\" -P \"${RunCMake_BINARY_DIR}/CoverageToolCMake-insource/script.cmake\")
+")
+  set(RunCMake_TEST_BINARY_DIR ${tree})
+  set(RunCMake_TEST_NO_CLEAN 1)
+  # In an in-source build, the merge keeps the files of the build tree.
+  file(APPEND "${tree}/DartConfiguration.tcl" "SourceDirectory: ${tree}\n")
+  file(WRITE "${tree}/script.cmake" "set(x 1)\n")
+  run_cmake_command(CoverageToolCMake-insource ${CMAKE_CTEST_COMMAND})
+endblock()
+
+block()
+  prepare_CoverageToolCMake(CoverageToolCMake-bad "
+add_test(bad \"${CMAKE_COMMAND}\" -P \"${RunCMake_SOURCE_DIR}/CoverageBad.cmake\")
+")
+  set(RunCMake_TEST_BINARY_DIR ${tree})
+  set(RunCMake_TEST_NO_CLEAN 1)
+  run_cmake_command(CoverageToolCMake-bad ${CMAKE_CTEST_COMMAND})
+endblock()
+
+block()
+  prepare_CoverageToolCMake(CoverageToolCMake-writefail "
+add_test(wf \"${CMAKE_COMMAND}\" -P \"${RunCMake_SOURCE_DIR}/CoverageChild.cmake\")
+")
+  set(RunCMake_TEST_BINARY_DIR ${tree})
+  set(RunCMake_TEST_NO_CLEAN 1)
+  # The cleanup does not remove a directory, so the final write fails.
+  file(MAKE_DIRECTORY "${tree}/wf.cmake-coverage.info")
+  run_cmake_command(CoverageToolCMake-writefail ${CMAKE_CTEST_COMMAND})
+endblock()
+
+block()
+  prepare_CoverageToolCMake(CoverageToolCMake-timeout "
+add_test(sleeper \"${CMAKE_COMMAND}\" -E sleep 10)
+set_tests_properties(sleeper PROPERTIES TIMEOUT 1)
+")
+  set(RunCMake_TEST_BINARY_DIR ${tree})
+  set(RunCMake_TEST_NO_CLEAN 1)
+  run_cmake_command(CoverageToolCMake-timeout ${CMAKE_CTEST_COMMAND})
+endblock()
+
+block()
+  prepare_CoverageToolCMake(CoverageToolCMake-noscratch "
+add_test(blocked \"${CMAKE_COMMAND}\" -P \"${RunCMake_SOURCE_DIR}/CoverageBlocked.cmake\")
+")
+  set(RunCMake_TEST_BINARY_DIR ${tree})
+  set(RunCMake_TEST_NO_CLEAN 1)
+  # A regular file blocks the scratch folder.
+  file(WRITE "${tree}/blocked_cmake-coverage" "")
+  run_cmake_command(CoverageToolCMake-noscratch ${CMAKE_CTEST_COMMAND})
+endblock()
+
+if(NOT CMAKE_HOST_WIN32)
+  block()
+    prepare_CoverageToolCMake(CoverageToolCMake-symlink "
+add_test(linked \"${CMAKE_COMMAND}\" -P \"${RunCMake_SOURCE_DIR}/CoverageBlocked.cmake\")
+")
+    set(RunCMake_TEST_BINARY_DIR ${tree})
+    set(RunCMake_TEST_NO_CLEAN 1)
+    # A symbolic link at the scratch folder path points outside the tree.
+    # ctest must not record into the link target or delete its content.
+    set(target "${RunCMake_BINARY_DIR}/CoverageToolCMake-symlink-target")
+    file(REMOVE_RECURSE "${target}")
+    file(WRITE "${target}/keep.txt" "")
+    file(CREATE_LINK "${target}" "${tree}/linked_cmake-coverage" SYMBOLIC)
+    run_cmake_command(CoverageToolCMake-symlink ${CMAKE_CTEST_COMMAND})
+  endblock()
+endif()

Tests/RunCMake/CTestCoverage/SelfCoverage-check.cmake

@@ -0,0 +1,2 @@
+set(expect_sf_regex "/CTestTestfile\\.cmake$")
+include(${RunCMake_SOURCE_DIR}/../ListFileCoverage/check-lcov.cmake)

Tests/RunCMake/ctest_test/RunCMakeTest.cmake

@@ -368,6 +368,32 @@ set(CTEST_TEST_COVERAGE_TOOL "LLVM-COV")
 endfunction()
 run_CTestTestCoverageToolVar()
 
+function(run_CTestTestCoverageToolCMake)
+  set(CASE_CTEST_TEST_ARGS "COVERAGE_TOOL" "CMAKE")
+  run_ctest(TestCTestTestCoverageToolCMake -VV)
+endfunction()
+run_CTestTestCoverageToolCMake()
+
+# A coverage problem is a warning, so the dashboard script does not fail.
+function(run_TestCoverageToolCMakeNoScratch)
+  set(CASE_CTEST_TEST_ARGS "COVERAGE_TOOL" "CMAKE")
+  set(CASE_CTEST_TEST_RAW_ARGS "RETURN_VALUE ret")
+  set(CASE_CMAKELISTS_SUFFIX_CODE "
+add_test(NAME blocked COMMAND \"\${CMAKE_COMMAND}\" -P \"${RunCMake_SOURCE_DIR}/../CTestCoverage/CoverageBlocked.cmake\")
+")
+  # A regular file blocks the scratch folder.
+  set(CASE_TEST_PREFIX_CODE [[
+file(WRITE "${CTEST_SCRIPT_DIRECTORY}-build/blocked_cmake-coverage" "")
+]])
+  set(CASE_TEST_SUFFIX_CODE [[
+if(NOT ret EQUAL 0)
+  message(FATAL_ERROR "ctest_test returned ${ret}")
+endif()
+]])
+  run_ctest(TestCoverageToolCMakeNoScratch)
+endfunction()
+run_TestCoverageToolCMakeNoScratch()
+
 # test for OUTPUT_JUNIT
 run_ctest_test(OutputJUnit OUTPUT_JUNIT junit.xml REPEAT UNTIL_FAIL:2)
 

Tests/RunCMake/ctest_test/TestCTestTestCoverageToolCMake-stdout.txt

@@ -0,0 +1,3 @@
+1: Using environment variable CMAKE_LISTFILE_COVERAGE_DIR=[^
+]*/Tests/RunCMake/ctest_test/[^
+]*_cmake-coverage

Tests/RunCMake/ctest_test/TestCoverageToolCMakeNoScratch-stderr.txt

@@ -0,0 +1,2 @@
+Cannot create CMake listfile coverage folder: [^
+]*blocked_cmake-coverage

Tests/RunCMake/ctest_test/test.cmake.in

@@ -21,3 +21,4 @@ if("@CASE_NAME@" STREQUAL "TestChangingLabels")
 else()
   ctest_test(${ctest_test_args} @CASE_CTEST_TEST_RAW_ARGS@)
 endif()
+@CASE_TEST_SUFFIX_CODE@