Extend generator script user guide

docs/source/api/generation.rst

@@ -20,9 +20,15 @@ Categories, Parameter Types, and Special Values
 
 .. autoclass:: _GlobalNamespace
 .. autodata:: GLOBAL_NAMESPACE
+
 .. autoclass:: OutputMode
+    :members:
+
 .. autoclass:: CodeStyle
+    :members:
+
 .. autoclass:: ClangFormatOptions
+    :members:
 
 Option Descriptors
 ------------------

docs/source/conf.py

@@ -43,6 +43,10 @@ source_suffix = {
 }
 master_doc = "index"
 nitpicky = True
+myst_enable_extensions = [
+    "colon_fence",
+    "dollarmath"
+]
 
 
 # -- Options for HTML output -------------------------------------------------

docs/source/index.md

@@ -6,7 +6,7 @@
 :caption: User Guide
 
 usage/generator_scripts
-usage/cli_and_build_system
+usage/project_integration
 usage/tips_n_tricks
 ```
 
@@ -225,7 +225,7 @@ To `#include` them, add the prefix `gen/<target name>`:
 ```
 
 For details on how to add *pystencils-sfg* to your CMake project, refer to
-[CLI and Build System Integration](usage/cli_and_build_system.md).
+[the project integration guide](#guide_project_integration).
 
 ## Learn To Use pystencils-sfg
 
@@ -243,7 +243,7 @@ arbitrary C++ glue code.
 ```
 
 ```{card} CLI and Build System Integration
-:link: guide:cli
+:link: guide_project_integration
 :link-type: ref
 
 Learn how to control code generation from the command line

docs/source/usage/examples/guide_generator_scripts/custom_cmdline_args/kernels.py

+from pystencilssfg import SourceFileGenerator
+from argparse import ArgumentParser
+
+parser = ArgumentParser()
+# set up parser ...
+
+with SourceFileGenerator(keep_unknown_argv=True) as sfg:
+    args = parser.parse_args(sfg.context.argv)
+    ...

docs/source/usage/examples/guide_generator_scripts/inline_config/kernels.py

+from pystencilssfg import SourceFileGenerator, SfgConfig
+
+cfg = SfgConfig()
+cfg.output_directory = "gen_src"
+cfg.codestyle.indent_width = 4
+
+with SourceFileGenerator(cfg) as sfg:
+    ...

docs/source/usage/generator_scripts.md

@@ -3,7 +3,7 @@
 
 Writing generator scripts is the primary usage idiom of *pystencils-sfg*.
 A generator script is a Python script, say `kernels.py`, which contains *pystencils-sfg*
-code at the top level that, when executed, emits source code to a pair of files `kernels.h`
+code at the top level that, when executed, emits source code to a pair of files `kernels.hpp`
 and `kernels.cpp`. This guide describes how to write such a generator script, its structure, and how
 it can be used to generate code.
 
@@ -23,14 +23,14 @@ To start, place the following code in a Python script, e.g. `kernels.py`:
 
 The source file is constructed within the context manager's managed region.
 During execution of the script, when the region ends, a header/source file pair
-`kernels.h` and `kernels.cpp` will be written to disk next to your script.
+`kernels.hpp` and `kernels.cpp` will be written to disk next to your script.
 Execute the script as-is and inspect the generated files, which will of course still be empty:
 
 ``````{dropdown} Generated Files
 `````{tab-set}
 
-````{tab-item} kernels.h
-```{literalinclude} examples/guide_generator_scripts/01/kernels.h
+````{tab-item} kernels.hpp
+```{literalinclude} examples/guide_generator_scripts/01/kernels.hpp
 ```
 ````
 
@@ -41,31 +41,110 @@ Execute the script as-is and inspect the generated files, which will of course s
 `````
 ``````
 
-<!-- A few notes on configuration:
-
- - The [SourceFileGenerator](#pystencilssfg.SourceFileGenerator) parses the script's command line arguments
-   for configuration options (refer to [CLI and Build System Integration](cli_and_build_system.md)).
-   If you intend to evaluate command-line parameters inside your
-   generator script, read them from `sfg.context.argv` instead of `sys.argv`.
-   There, all arguments meant for the code generator are already removed.
- - The code generator's configuration is consolidated from a global project configuration which may
-   be provided by the build system; a number of command line arguments; and the
-   [SfgConfiguration](#pystencilssfg.SfgConfiguration) provided in the script.
-   The project configuration may safely be overridden by the latter two; however, conflicts
-   between command-line arguments and the configuration defined in the script will cause
-   an exception to be thrown. -->
-
 ## Using the Composer
 
 The object `sfg` constructed in above snippet is an instance of [SfgComposer](#pystencilssfg.composer.SfgComposer).
 The composer is the central part of the user front-end of *pystencils-sfg*.
 It provides an interface for constructing source files that closely mimics
 C++ syntactic structures within Python.
-Here is an overview of its various functions:
+
+::::{dropdown} Composer API Overview
+```{eval-rst}
+.. currentmodule:: pystencilssfg.composer
+```
+
+Structure and Verbatim Code:
+
+```{eval-rst}
+
+.. autosummary::
+  :nosignatures:
+
+  SfgBasicComposer.prelude
+  SfgBasicComposer.include
+  SfgBasicComposer.namespace
+  SfgBasicComposer.code
+  SfgBasicComposer.define_once
+```
+
+Kernels and Kernel Namespaces:
+
+```{eval-rst}
+
+.. autosummary::
+  :nosignatures:
+
+  SfgBasicComposer.kernels
+  SfgBasicComposer.kernel_namespace
+  SfgBasicComposer.kernel_function
+```
+
+Function definition, parameters, and header inclusion:
+
+```{eval-rst}
+
+.. autosummary::
+  :nosignatures:
+
+  SfgBasicComposer.function
+  SfgBasicComposer.params
+  SfgBasicComposer.require
+```
+
+Variables, expressions, and variable initialization:
+
+```{eval-rst}
+
+.. autosummary::
+  :nosignatures:
+
+  SfgBasicComposer.var
+  SfgBasicComposer.vars
+  SfgBasicComposer.expr
+  SfgBasicComposer.init
+  
+  SfgBasicComposer.map_field
+  SfgBasicComposer.set_param
+```
+
+Parameter mappings:
+
+```{eval-rst}
+
+.. autosummary::
+  :nosignatures:
+
+  SfgBasicComposer.set_param
+  SfgBasicComposer.map_field
+  SfgBasicComposer.map_vector
+```
+
+Control Flow:
+
+```{eval-rst}
+
+.. autosummary::
+  :nosignatures:
+
+  SfgBasicComposer.branch
+  SfgBasicComposer.switch
+```
+
+Kernel Invocation:
+
+```{eval-rst}
+
+.. autosummary::
+  :nosignatures:
+
+  SfgBasicComposer.call
+  SfgBasicComposer.cuda_invoke
+```
+::::
 
 ### Includes and Definitions
 
-With [`SfgComposer.include`](#pystencilssfg.composer.SfgBasicComposer.include), the code generator can be instructed to include header files.
+With {any}`include <SfgBasicComposer.include>`, the code generator can be instructed to include header files.
 As in C++, you can use the `<>` delimiters for system headers, and omit them for project headers.
 
 `````{tab-set}
@@ -75,8 +154,8 @@ As in C++, you can use the `<>` delimiters for system headers, and omit them for
 ```
 ````
 
-````{tab-item} kernels.h
-```{literalinclude} examples/guide_generator_scripts/02/kernels.h
+````{tab-item} kernels.hpp
+```{literalinclude} examples/guide_generator_scripts/02/kernels.hpp
 ```
 ````
 
@@ -112,8 +191,8 @@ the [`sfg.kernel_namespace`](#pystencilssfg.composer.SfgBasicComposer.kernel_nam
 ```
 ````
 
-````{tab-item} kernels.h
-```{literalinclude} examples/guide_generator_scripts/03/kernels.h
+````{tab-item} kernels.hpp
+```{literalinclude} examples/guide_generator_scripts/03/kernels.hpp
 ```
 ````
 
@@ -141,8 +220,8 @@ Use `sfg.function` to create a function, and `sfg.call` to call a kernel:
 ```
 ````
 
-````{tab-item} kernels.h
-```{literalinclude} examples/guide_generator_scripts/04/kernels.h
+````{tab-item} kernels.hpp
+```{literalinclude} examples/guide_generator_scripts/04/kernels.hpp
 ```
 ````
 
@@ -171,8 +250,8 @@ such as, for example, `std::span` or `std::vector`, like this:
 ```
 ````
 
-````{tab-item} kernels.h
-```{literalinclude} examples/guide_generator_scripts/05/kernels.h
+````{tab-item} kernels.hpp
+```{literalinclude} examples/guide_generator_scripts/05/kernels.hpp
 ```
 ````
 
@@ -192,3 +271,77 @@ The pystencils-sfg provides modelling support for a number of C++ standard libra
 (see {any}`pystencilssfg.lang.cpp.std`).
 It also provides the necessary infrastructure for modelling the data structures of any C++ framework
 in a similar manner.
+
+
+## Configuration and Invocation
+
+There are several ways to affect the behavior and output of a generator script.
+For one, the `SourceFileGenerator` itself may be configured from the combination of three
+different configuration sources:
+
+- **Inline Configuration:** The generator script may set up an {any}`SfgConfig` object,
+  which is passed to the `SourceFileGenerator` at its creation; see [Inline Configuration](#inline_config)
+- **Command-Line Options:** The `SourceFileGenerator` parses the command line arguments of
+  the generator script to set some of its configuration options; see [Command-Line Options](#cmdline_options)
+- **Project Configuration:** When embedded into a larger project, using a build system such as CMake, generator scripts
+  may be configured globally within that project by the use of a *configuration module*.
+  Settings specified inside that configuration module are always overridden by the former to configuration sources.
+  For details on configuration modules, refer to the guide on [Project and Build System Integration](#guide_project_integration).
+
+(inline_config)=
+### Inline Configuration
+
+To configure the source file generator within your generator script, import the {any}`SfgConfig` from `pystencilssfg`.
+You may then set up the configuration object before passing it to the `SourceFileGenerator` constructor.
+To illustrate, the following snippet alters the code indentation width and changes the output directory
+of the generator script to `gen_src`:
+
+```{literalinclude} examples/guide_generator_scripts/inline_config/kernels.py
+```
+
+(cmdline_options)=
+### Command-Line Options
+
+The `SourceFileGenerator` consumes a number of command-line parameters that may be passed to the script
+on invocation. These include:
+
+- `--sfg-output-dir <path>`: Set the output directory of the generator script. This corresponds to {any}`SfgConfig.output_directory`.
+- `--sfg-file-extensions <exts>`: Set the file extensions used for the generated files;
+  `exts` must be a comma-separated list not containing any spaces. Corresponds to {any}`SfgConfig.extensions`.
+- `--sfg-output-mode <mode>`: Set the output mode of the generator script. Corresponds to {any}`SfgConfig.output_mode`.
+
+If any configuration option is set to conflicting values on the command line and in the inline configuration,
+the generator script will terminate with an error.
+
+You may examine the full set of possible command line parameters by invoking a generator script
+with the `--help` flag:
+
+```bash
+$ python kernels.py --help
+```
+
+## Adding Custom Command-Line Options
+
+Sometimes, you might want to add your own command-line options to a generator script
+in order to affect its behavior from the shell,
+for instance by using {any}`argparse` to set up an argument parser.
+If you parse your options directly from {any}`sys.argv`,
+as {any}`parse_args <argparse.ArgumentParser.parse_args>` does by default,
+your parser will also receive any options meant for the `SourceFileGenerator`.
+To filter these out of the argument list,
+pass the additional option `keep_unknown_argv=True` to your `SourceFileGenerator`.
+This will instruct it to store any unknown command line arguments into `sfg.context.argv`,
+where you can then retrieve them from and pass on to your custom parser:
+
+```{literalinclude} examples/guide_generator_scripts/custom_cmdline_args/kernels.py
+```
+
+Any SFG-specific arguments will already have been filtered out of this argument list.
+As a consequence of the above, if the generator script is invoked with a typo in some SFG-specific argument,
+which the `SourceFileGenerator` therefore does not recognize,
+that argument will be passed on to your downstream parser instead.
+
+:::{important}
+If you do *not* pass on `sfg.context.argv` to a downstream parser, make sure that `keep_unknown_argv` is set to
+`False` (which is the default), such that typos or illegal arguments will not be ignored.
+:::

docs/source/usage/cli_and_build_system.md → docs/source/usage/project_integration.md

-(guide:cli)=
-# CLI and Build System
+(guide_project_integration)=
+# Project and Build System Integration
 
 ## Command Line Interface
 
@@ -14,7 +14,7 @@ The global CLI may be accessed either through the `sfg-cli` shell command, or us
 
 ### Generator Script CLI
 
-The [SourceFileGenerator][pystencilssfg.SourceFileGenerator] evaluates a generator script's command line arguments,
+The {any}`SourceFileGenerator` evaluates a generator script's command line arguments,
 which can be supplied by the user, but more frequently by the build system.
 
 ## CMake Integration
@@ -72,7 +72,7 @@ path, such that generated header files for a target `<target>` may be included v
 #include "gen/<target>/kernels.h"
 ```
 
-### Project Configuration
+### Configuration Module
 
 The *pystencils-sfg* CMake module reads the scoped variable `PystencilsSfg_CONFIGURATOR_SCRIPT` to find
 the *configuration module* that should be passed to the generator scripts.

pyproject.toml

@@ -23,9 +23,6 @@ requires = [
 build-backend = "setuptools.build_meta"
 
 [project.optional-dependencies]
-interactive = [
-    "ipython>=8.17.2",
-]
 testing = [
     "flake8>=6.1.0",
     "mypy>=1.7.0",
@@ -39,7 +36,8 @@ docs = [
     "sphinx_design",
     "sphinx_autodoc_typehints",
     "sphinx-copybutton",
-    "packaging"
+    "packaging",
+    "clang-format"
 ]
 
 [tool.versioneer]

src/pystencilssfg/composer/basic_composer.py

@@ -319,6 +319,7 @@ class SfgBasicComposer(SfgIComposer):
         return SfgFunctionParams([x.as_variable() for x in args])
 
     def require(self, *includes: str | SfgHeaderInclude) -> SfgRequireIncludes:
+        """Use inside a function body to require the inclusion of headers."""
         return SfgRequireIncludes(
             list(SfgHeaderInclude.parse(incl) for incl in includes)
         )
@@ -443,6 +444,7 @@ class SfgBasicComposer(SfgIComposer):
         return SfgBranchBuilder()
 
     def switch(self, switch_arg: ExprLike) -> SfgSwitchBuilder:
+        """Use inside a function to construct a switch-case statement."""
         return SfgSwitchBuilder(switch_arg)
 
     def map_field(

src/pystencilssfg/generator.py

@@ -10,7 +10,7 @@ from .emission import AbstractEmitter, OutputSpec
 class SourceFileGenerator:
     """Context manager that controls the code generation process in generator scripts.
 
-    **Usage:** The `SourceFileGenerator` must be used as a context manager by calling it within
+    The `SourceFileGenerator` must be used as a context manager by calling it within
     a ``with`` statement in the top-level code of a generator script (see :ref:`guide:generator_scripts`).
     Upon entry to its context, it creates an :class:`SfgComposer` which can be used to populate the generated files.
     When the managed region finishes, the code files are generated and written to disk at the locations
@@ -18,15 +18,16 @@ class SourceFileGenerator:
     Existing copies of the target files are deleted on entry to the managed region,
     and if an exception occurs within the managed region, no files are exported.
 
-    **Configuration:** The `SourceFileGenerator` optionally takes a user-defined configuration
-    object which is merged with configuration obtained from the build system; for details
-    on configuration sources, refer to :class:`SfgConfiguration`.
-
     Args:
-        sfg_config: User configuration for the code generator
+        sfg_config: Inline configuration for the code generator
+        keep_unknown_argv: If `True`, any command line arguments given to the generator script
+            that the `SourceFileGenerator` does not understand are stored in
+            `sfg.context.argv`.
     """
 
-    def __init__(self, sfg_config: SfgConfig | None = None):
+    def __init__(
+        self, sfg_config: SfgConfig | None = None, keep_unknown_argv: bool = False
+    ):
         if sfg_config and not isinstance(sfg_config, SfgConfig):
             raise TypeError("sfg_config is not an SfgConfiguration.")
 
@@ -44,7 +45,13 @@ class SourceFileGenerator:
             allow_abbrev=False,
         )
         CommandLineParameters.add_args_to_parser(parser)
-        sfg_args, script_args = parser.parse_known_args()
+
+        if keep_unknown_argv:
+            sfg_args, script_args = parser.parse_known_args()
+        else:
+            sfg_args = parser.parse_args()
+            script_args = []
+
         cli_params = CommandLineParameters(sfg_args)
 
         config = cli_params.get_config()