Added --query-driver option to enable automatic detection of system include paths from selected compiler (#109)

docs/configuration_file.md

@@ -18,6 +18,7 @@
 * `debug_mode` - add inline debug information in the generated diagrams
 * `add_compile_flags` - add compile flags to all compilation database entries
 * `remove_compile_flags` - remove compile flags from all compilation database entries
+* `query_driver` - name or path to compiler driver, which should be queried for system include paths (e.g. arm-none-eabi-g++)
 
 ### Diagram options
 * `type` - type of diagram, one of [`class`, `sequence`, `package`, `include`]
@@ -61,6 +62,9 @@ add_compile_flags:
 # Remove specified compile flags from all compilation database entries
 remove_compile_flags:
   - '-Wshadow'
+# Compiler driver command to query for system include paths
+query_driver:
+  - arm-none-eabi-g++
 # The directory where *.puml files will be generated
 output_directory: docs/diagrams
 # Set this as default for all diagrams

docs/troubleshooting.md

@@ -15,17 +15,24 @@
 <!-- tocstop -->
 
 ## General issues
+
 ### Diagram generated with PlantUML is cropped
-When generating diagrams with PlantUML without specifying an output file format, the default is PNG. 
+
-Unfortunately PlantUML will not check if the diagram will fit in the default PNG size, and often the diagram
+When generating diagrams with PlantUML without specifying an output file format,
-will be incomplete in the picture. A better option is to specify SVG as output format and then convert 
+the default is PNG.
+Unfortunately PlantUML will not check if the diagram will fit in the default PNG
+size, and often the diagram
+will be incomplete in the picture. A better option is to specify SVG as output
+format and then convert
 to PNG, e.g.:
+
 ```bash
 $ plantuml -tsvg mydiagram.puml
 $ convert +antialias mydiagram.svg mydiagram.png
 ```
 
 ### `clang` produces several warnings during diagram generation
+
 During the generation of the diagram `clang` may report a lot of warnings, which
 do not occur during the compilation with other compiler (e.g. GCC). This can be
 fixed easily by using the `add_compile_flags` config option. For instance,
@@ -63,9 +70,10 @@ remove_compile_flags:
 ```
 
 ### YAML anchors and aliases are not fully supported
+
 `clang-uml` uses [yaml-cpp](https://github.com/jbeder/yaml-cpp) library, which
 currently does not support
-[merging YAML anchor dictionaries](https://github.com/jbeder/yaml-cpp/issues/41), 
+[merging YAML anchor dictionaries](https://github.com/jbeder/yaml-cpp/issues/41),
 e.g. in the following configuration file the `main_sequence_diagram` will work,
 but the `foo_sequence_diagram` will fail with parse error:
 
@@ -98,25 +106,44 @@ yq 'explode(.)' .clang-uml | clang-uml --config -
 
 ## Class diagrams
 ### "fatal error: 'stddef.h' file not found"
-This error means that Clang cannot find some standard headers in the include paths
-specified in the `compile_commands.json`. This typically happens on macOS and sometimes on Linux, when 
-the code was compiled with different Clang version than `clang-uml` itself.
 
-One solution to this issue is to add the following line to your `CMakeLists.txt` file:
+This error means that Clang cannot find some standard headers in the include
+paths  specified in the `compile_commands.json`. This typically happens on macOS
+and sometimes on Linux, when the code was compiled with different Clang version
+than `clang-uml` itself.
+
+One solution to this issue is to add the following line to your `CMakeLists.txt`
+file:
 
 ```cmake
 set(CMAKE_CXX_STANDARD_INCLUDE_DIRECTORIES ${CMAKE_CXX_IMPLICIT_INCLUDE_DIRECTORIES})
 ```
 
-Another option is to make sure that the Clang is installed on the system (even if not used for building your
+Another option is to provide an option (on command line or in configuration
+file) called `query_driver` (inspired by the [clangd](https://clangd.llvm.org/)
+language  server - although much less flexible), which will invoke the
+provider compiler command and query it for its default system paths, which then
+will be added to each compile command in the database. This is especially useful
+on macOS as well as for embedded toolchains, example usage:
+
+```bash
+$ clang-uml --query-driver arm-none-eabi-g++
+```
+
+Another option is to make sure that the Clang is installed on the system (even
+if not used for building your
 project), e.g.:
+
 ```bash
 apt install clang
 ```
 
-If this doesn't help the include paths can be customized using config options:
+If this doesn't help to include paths can be customized using config options:
- * `add_compile_flags` - which adds a list of compile flags such as include paths to each entry of the compilation database
+
- * `remove_compile_flags` - which removes existing compile flags from each entry of the compilation database
+* `add_compile_flags` - which adds a list of compile flags such as include paths
+  to each entry of the compilation database
+* `remove_compile_flags` - which removes existing compile flags from each entry
+  of the compilation database
 
 For instance:
 
@@ -133,28 +160,43 @@ These options can be also passed on the command line, for instance:
 $ clang-uml --add-compile-flag -I/opt/my_toolchain/include \
             --remove-compile-flag -I/usr/include ...
 ```
+
 ## Sequence diagrams
 ### Generated diagram is empty
-In order to generate sequence diagram the `start_from` configuration option must have a valid starting point
+
-for the diagram (e.g. `function`), which must match exactly the function signature in the `clang-uml` model.
+In order to generate sequence diagram the `start_from` configuration option must
+have a valid starting point
+for the diagram (e.g. `function`), which must match exactly the function
+signature in the `clang-uml` model.
 Look for error in the console output such as:
+
 ```bash
 Failed to find participant mynamespace::foo(int) for start_from condition
 ```
-which means that either you have a typo in the function signature in the configuration file, or that the function
+
-was not defined in the translation units you specified in the `glob` patterns for this diagram. Run again the
+which means that either you have a typo in the function signature in the
-`clang-uml` tool with `-vvv` option and look in the console output for any mentions of the function from
+configuration file, or that the function
-which the diagram should start and copy the exact signature into the configuration file.
+was not defined in the translation units you specified in the `glob` patterns
+for this diagram. Run again the
+`clang-uml` tool with `-vvv` option and look in the console output for any
+mentions of the function from
+which the diagram should start and copy the exact signature into the
+configuration file.
 
 ### Generated diagram contains several empty control blocks or calls which should not be there
-Currently the filtering of call expressions and purging empty control blocks (e.g. loops or conditional statements),
+
-within which no interesting calls were included in the diagram is not perfect. In case the regular `namespaces` filter
+Currently the filtering of call expressions and purging empty control blocks (
-is not enough, it is useful to add also a `paths` filter, which will only include participants and call expressions
+e.g. loops or conditional statements),
+within which no interesting calls were included in the diagram is not perfect.
+In case the regular `namespaces` filter
+is not enough, it is useful to add also a `paths` filter, which will only
+include participants and call expressions
 from files in a subdirectory of your project, e.g.:
+
 ```yaml
    include:
-      namespaces:
+     namespaces:
-         - myproject
+       - myproject
-      paths:
+     paths:
-         - src
+       - src
 ```

src/cli/cli_handler.cc

@@ -88,6 +88,11 @@ cli_flow_t cli_handler::parse(int argc, const char **argv)
     app.add_option("--remove-compile-flag", remove_compile_flag,
         "Remove a compilation flag from each entry in the compilation "
         "database");
+#if !defined(_WIN32)
+    app.add_option("--query-driver", query_driver,
+        "Query the specific compiler driver to extract system paths and add to "
+        "compile commands (e.g. arm-none-eabi-g++)");
+#endif
     app.add_option(
         "--add-class-diagram", add_class_diagram, "Add class diagram config");
     app.add_option("--add-sequence-diagram", add_sequence_diagram,
@@ -289,6 +294,10 @@ cli_flow_t cli_handler::handle_post_config_options()
         config.remove_compile_flags.has_value = true;
     }
 
+    if (query_driver) {
+        config.query_driver.set(*query_driver);
+    }
+
     return cli_flow_t::kContinue;
 }
 

src/cli/cli_handler.h

@@ -123,6 +123,9 @@ public:
     bool initialize{false};
     std::optional<std::vector<std::string>> add_compile_flag;
     std::optional<std::vector<std::string>> remove_compile_flag;
+#if !defined(_WIN32)
+    std::optional<std::string> query_driver;
+#endif
     std::optional<std::string> add_class_diagram;
     std::optional<std::string> add_sequence_diagram;
     std::optional<std::string> add_package_diagram;

src/common/compilation_database.cc

@@ -18,6 +18,8 @@
 
 #include "compilation_database.h"
 
+#include "util/query_driver_output_extractor.h"
+
 namespace clanguml::common {
 
 std::unique_ptr<compilation_database>
@@ -77,9 +79,40 @@ compilation_database::getAllCompileCommands() const
     return commands;
 }
 
+std::string compilation_database::guess_language_from_filename(
+    const std::string &filename) const
+{
+    if (util::ends_with(filename, std::string{".c"}))
+        return "c";
+
+    return "c++";
+}
+
 void compilation_database::adjust_compilation_database(
     std::vector<clang::tooling::CompileCommand> &commands) const
 {
+#if !defined(_WIN32)
+    if (config().query_driver && !config().query_driver().empty()) {
+        for (auto &compile_command : commands) {
+            util::query_driver_output_extractor extractor{
+                config().query_driver(),
+                guess_language_from_filename(compile_command.Filename)};
+
+            extractor.execute();
+
+            std::vector<std::string> system_header_args;
+            for (const auto &path : extractor.system_include_paths()) {
+                system_header_args.emplace_back("-isystem");
+                system_header_args.emplace_back(path);
+            }
+
+            compile_command.CommandLine.insert(
+                compile_command.CommandLine.begin() + 1,
+                system_header_args.begin(), system_header_args.end());
+        }
+    }
+#endif
+
     if (config().add_compile_flags && !config().add_compile_flags().empty()) {
         for (auto &compile_command : commands) {
             compile_command.CommandLine.insert(

src/common/compilation_database.h

@@ -61,6 +61,8 @@ public:
 
     const clang::tooling::CompilationDatabase &base() const;
 
+    std::string guess_language_from_filename(const std::string &filename) const;
+
 private:
     void adjust_compilation_database(
         std::vector<clang::tooling::CompileCommand> &commands) const;

src/config/config.h

@@ -228,6 +228,7 @@ struct config : public inheritable_diagram_options {
     option<std::vector<std::string>> add_compile_flags{"add_compile_flags"};
     option<std::vector<std::string>> remove_compile_flags{
         "remove_compile_flags"};
+    option<std::string> query_driver{"query_driver"};
     option<std::string> output_directory{"output_directory"};
 
     option<std::map<std::string, diagram_template>> diagram_templates{

src/main.cc

@@ -20,6 +20,7 @@
 #include "common/compilation_database.h"
 #include "common/generators/generators.h"
 #include "include_diagram/generators/plantuml/include_diagram_generator.h"
+#include "util/query_driver_output_extractor.h"
 #include "util/util.h"
 
 #ifdef ENABLE_BACKWARD_CPP
@@ -81,6 +82,13 @@ int main(int argc, const char *argv[])
             cli.config.compilation_database_dir(), e.what());
         return 1;
     }
+    catch (clanguml::util::query_driver_no_paths &e) {
+        LOG_ERROR("Quering provided compiler driver {} did not provide any "
+                  "paths, please make sure the path is correct and that your "
+                  "compiler is GCC-compatible: {}",
+            cli.config.query_driver(), e.what());
+        return 1;
+    }
 
     return 0;
 }

src/util/query_driver_output_extractor.cc

@@ -0,0 +1,84 @@
+/**
+ * src/util/query_driver_include_extractor.cc
+ *
+ * Copyright (c) 2021-2023 Bartek Kryza <bkryza@gmail.com>
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "query_driver_output_extractor.h"
+
+#include "util.h"
+
+namespace clanguml::util {
+
+query_driver_output_extractor::query_driver_output_extractor(
+    std::string command, std::string language)
+    : command_{std::move(command)}
+    , language_{std::move(language)}
+{
+}
+
+void query_driver_output_extractor::execute()
+{
+    auto cmd =
+        fmt::format("{} -E -v -x {} /dev/null 2>&1", command_, language_);
+
+    LOG_DBG("Executing query driver command: {}", cmd);
+
+    auto driver_output = get_process_output(cmd);
+
+    system_include_paths_.clear();
+    extract_system_include_paths(driver_output);
+
+    if (system_include_paths_.empty()) {
+        throw query_driver_no_paths(fmt::format(
+            "Compiler driver {} did not report any system include paths "
+            "in its output: {}",
+            command_, driver_output));
+    }
+
+    LOG_DBG("Extracted the following paths from compiler driver: {}",
+        fmt::join(system_include_paths_, ","));
+}
+
+void query_driver_output_extractor::extract_system_include_paths(
+    const std::string &output)
+{
+    std::istringstream f(output);
+    std::string line;
+
+    bool in_include_paths_range{false};
+    while (std::getline(f, line)) {
+        line = trim(line);
+        if (line == "#include <...> search starts here:") {
+            in_include_paths_range = true;
+            continue;
+        }
+        if (line == "End of search list.") {
+            break;
+        }
+
+        if (in_include_paths_range) {
+            system_include_paths_.emplace_back(line);
+        }
+    }
+}
+
+const std::vector<std::string> &
+query_driver_output_extractor::system_include_paths() const
+{
+    return system_include_paths_;
+}
+
+} // namespace clanguml::util

src/util/query_driver_output_extractor.h

@@ -0,0 +1,47 @@
+/**
+ * src/util/query_driver_include_extractor.h
+ *
+ * Copyright (c) 2021-2023 Bartek Kryza <bkryza@gmail.com>
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#pragma once
+
+#include <stdexcept>
+#include <string>
+#include <vector>
+
+namespace clanguml::util {
+
+class query_driver_no_paths : public std::runtime_error {
+    using std::runtime_error::runtime_error;
+};
+
+class query_driver_output_extractor {
+public:
+    query_driver_output_extractor(std::string command, std::string language);
+
+    ~query_driver_output_extractor() = default;
+
+    void execute();
+
+    void extract_system_include_paths(const std::string &output);
+
+    const std::vector<std::string> &system_include_paths() const;
+
+private:
+    const std::string command_;
+    const std::string language_;
+    std::vector<std::string> system_include_paths_;
+};
+} // namespace clanguml::util

tests/CMakeLists.txt

@@ -43,7 +43,8 @@ set(TEST_CASES
         test_config
         test_cli_handler
         test_filters
-        test_thread_pool_executor)
+        test_thread_pool_executor
+        test_query_driver_output_extractor)
 
 foreach(TEST_NAME ${TEST_CASES})
     add_executable(${TEST_NAME}

tests/test_query_driver_output_extractor.cc

@@ -0,0 +1,69 @@
+/**
+ * tests/test_query_driver_output_extractor.cc
+ *
+ * Copyright (c) 2021-2023 Bartek Kryza <bkryza@gmail.com>
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#define CATCH_CONFIG_MAIN
+
+#include "util/query_driver_output_extractor.h"
+
+#include "catch.h"
+
+TEST_CASE("Test extract system include paths", "[unit-test]")
+{
+
+    std::string output = R"(###
+Using built-in specs.
+COLLECT_GCC=g++
+OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa
+OFFLOAD_TARGET_DEFAULT=1
+Target: x86_64-linux-gnu
+Configured with: ../src/configure -v
+Thread model: posix
+Supported LTO compression algorithms: zlib zstd
+gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04)
+COLLECT_GCC_OPTIONS='-E' '-v' '-shared-libgcc' '-mtune=generic' '-march=x86-64'
+ /usr/lib/gcc/x86_64-linux-gnu/11/cc1 -E -quiet -v -imultiarch x86_64-linux-gnu /dev/null -mtune=generic -march=x86-64 -fasynchronous-unwind-tables -fstack-protector-strong -Wformat -Wformat-security -fstack-clash-protection -fcf-protection -dumpbase null
+ignoring nonexistent directory "/usr/local/include/x86_64-linux-gnu"
+ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/11/include-fixed"
+ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/11/../../../../x86_64-linux-gnu/include"
+#include "..." search starts here:
+#include <...> search starts here:
+ /usr/lib/gcc/x86_64-linux-gnu/11/include
+ /usr/local/include
+ /usr/include/x86_64-linux-gnu
+ /usr/include
+End of search list.
+# 0 "/dev/null"
+# 0 "<built-in>"
+# 0 "<command-line>"
+# 1 "/usr/include/stdc-predef.h" 1 3 4
+# 0 "<command-line>" 2
+# 1 "/dev/null"
+COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/11/:/usr/lib/gcc/x86_64-linux-gnu/11/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/11/:/usr/lib/gcc/x86_64-linux-gnu/
+LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/11/:/usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/11/../../../../lib/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/11/../../../:/lib/:/usr/lib/
+COLLECT_GCC_OPTIONS='-E' '-v' '-shared-libgcc' '-mtune=generic' '-march=x86-64'
+###)";
+
+    clanguml::util::query_driver_output_extractor extractor("g++", "c++");
+
+    extractor.extract_system_include_paths(output);
+
+    std::vector<std::string> expected = {
+        "/usr/lib/gcc/x86_64-linux-gnu/11/include", "/usr/local/include",
+        "/usr/include/x86_64-linux-gnu", "/usr/include"};
+
+    REQUIRE(extractor.system_include_paths() == expected);
+}