Extended documentation: Usage Guides 1

docs/api/frontend.md

 
-# Source File Composition
+# Source File Generator
+
+::: pystencilssfg.generator.SourceFileGenerator
+
+# Code Generator Configuration
+
+::: pystencilssfg.configuration.SfgConfiguration
+
+# Generation Context
+
+::: pystencilssfg.context.SfgContext
+
+# Source File Composition and Components
 
 ::: pystencilssfg.composer.SfgComposer
 
 ::: pystencilssfg.source_components.SfgKernelNamespace
 
 ::: pystencilssfg.source_components.SfgKernelHandle
+
+::: pystencilssfg.source_components.SfgFunction

docs/usage/generator_scripts.md

+
+Generator scripts are the primary way *pystencils-sfg* is meant to be used.
+A generator script is a single Python script, say `kernels.py`, which contains *pystencils-sfg*
+code at the top level such that, when executed, it emits source code to a pair of files `kernels.h`
+and `kernels.cpp`. This guide describes how to write such a generator script, its structure, and how
+it can be used to generate code.
+
+This page gives a general overview over the code generation process, but introduces only the
+convenient high-level interface provided by the [SourceFileGenerator][pystencilssfg.SourceFileGenerator]
+and [SfgComposer][pystencilssfg.SfgComposer] classes.
+For a more in-depth look into building source files, and about using *pystencils-sfg* outside
+of a generator script, please take a look at the [In-Depth Guide](building.md).
+
+## Anatomy
+
+The code generation process in a generator script is controlled by the
+[SourceFileGenerator][pystencilssfg.SourceFileGenerator] context manager.
+It configures the code generator by combining configuration options from the 
+environment (e.g. a CMake build system) with options specified in the script,
+and infers the names of the output files from the script's name.
+It then prepares a code generation [context][pystencilssfg.SfgContext] and a
+[composer][pystencilssfg.SfgComposer].
+The latter is returned when entering the context manager, and provides a convenient
+interface to constructing the source files.
+
+To start, place the following code in a Python script, e.g. `kernels.py`:
+
+```Python
+from pystencilssfg import SourceFileGenerator, SfgConfiguration
+
+sfg_config = SfgConfiguration()
+with SourceFileGenerator(sfg_config) as sfg:
+    pass
+
+```
+
+The source file is constructed within the context manager's managed region.
+During execution of the script, when the region ends, a header/source file pair
+`kernels.h` and `kernels.cpp` will be written to the file system next to your script.
+Execute the script as-is and inspect the generated files, which will of course
+still be empty.
+
+A few notes on configuration:
+
+ - The [SourceFileGenerator][pystencilssfg.SourceFileGenerator] parses the script's command line arguments
+   for configuration options (refer to [CLI and Build System Integration](cli.md)).
+   If you intend to use command-line parameters in your
+   generation script, use [`sfg.context.argv`][pystencilssfg.SfgContext.argv] instead of `sys.argv`.
+   There, all arguments meant for the code generator are already removed.
+ - The code generator's configuration is consolidated from a global project configuration which may
+   be provided by the build system; a number of command line arguments; and the
+   [SfgConfiguration][pystencilssfg.SfgConfiguration] provided in the script.
+   The project configuration may safely be overridden by the latter two; however, conflicts
+   between command-line arguments and the configuration defined in the script will cause
+   an exception to be thrown.
+
+## Using the Composer
+
+The object `sfg` returned by the context manager is an [SfgComposer](pystencilssfg.SfgComposer).
+It is a stateless builder object which provides a convenient interface to construct the source
+file's contents. Here's an overview:
+
+### Includes and Definitions
+
+With [`SfgComposer.include`][pystencilssfg.SfgComposer.include], the code generator can be instructed
+to include header files. 
+
+```Python
+with SourceFileGenerator(sfg_config) as sfg:
+    # ...
+    sfg.include("<vector>")
+    sfg.incldue("custom_header.h")
+```
+
+### Adding Kernels
+
+`pystencils`-generated kernels are managed in
+[kernel namespaces][pystencilssfg.source_components.SfgKernelNamespace].
+The default kernel namespace is called `kernels` and is available via
+[`SfgComposer.kernels`][pystencilssfg.SfgComposer.kernels].
+Adding an existing `pystencils` AST, or creating one from a list of assignments, is possible
+through [`add`][pystencilssfg.source_components.SfgKernelNamespace.add]
+and [`create`][pystencilssfg.source_components.SfgKernelNamespace.create].
+The latter is a wrapper around
+[`pystencils.create_kernel`](
+https://pycodegen.pages.i10git.cs.fau.de/pystencils/sphinx/kernel_compile_and_call.html#pystencils.create_kernel
+).
+Both functions return a [kernel handle][pystencilssfg.source_components.SfgKernelHandle]
+through which the kernel can be accessed, e.g. for calling it in a function.
+
+If required, use [`SfgComposer.kernel_namespace`][pystencilssfg.SfgComposer.kernel_namespace]
+to access other kernel namespaces than the default one.
+
+```Python
+with SourceFileGenerator(sfg_config) as sfg:
+    # ...
+
+    ast = ps.create_kernel(assignments, config)
+    khandle = sfg.kernels.add(ast, "kernel_a")
+    
+    # is equivalent to
+    
+    khandle = sfg.kernels.create(assignments, "kernel_a", config)
+
+    # You may use a different namespace
+    nspace = sfg.kernel_namespace("group_of_kernels")
+    nspace.create(assignments, "kernel_a", config)
+```
+
+### Building Functions
+
+[Functions][pystencilssfg.source_components.SfgFunction] form the link between your `pystencils` kernels
+and your C++ framework. A function in *pystencils-sfg* translates to a simple C++ function, and should
+fulfill just the following tasks:
+
+ - Extract kernel parameters (pointers, sizes, strides, numerical coefficients)
+   from C++ objects (like fields, vectors, other data containers)
+ - Call one or more kernels in sequence or in conditional branches
+
+It is the philosophy of this project that anything more complicated than this should happen in handwritten
+code; these generated functions are merely meant to close the remaining gap.
+
+The composer provides an interface for constructing functions that tries to mimic the look of the generated C++
+code.
+Use [`SfgComposer.function`][pystencilssfg.SfgComposer.function] to create a function,
+and [`SfgComposer.call`][pystencilssfg.SfgComposer.call] to call a kernel by its handle:
+
+```Python
+with SourceFileGenerator(sfg_config) as sfg:
+    # ...
+
+    sfg.function("MyFunction")(
+        sfg.call(khandle)
+    )
+```
+
+Note the special syntax: To mimic the look of a C++ function, the composer uses a sequence of two calls
+to construct the function.
+
+The function body may further be populated with the following things:
+
+#### Parameter Mappings
+
+Extract kernel parameters from C++ objects:
+
+ - [`map_param`][pystencilssfg.SfgComposer.map_param]: Add a single line of code to define one parameter
+   depending on one other.
+ - [`map_field`][pystencilssfg.SfgComposer.map_field] maps a pystencils
+   [`Field`](https://pycodegen.pages.i10git.cs.fau.de/pystencils/sphinx/field.html)
+   to a field data structure providing the necessary pointers, sizes and stride information.
+   The field data structure must be provided as an instance of a subclass of
+   [`SrcField`][pystencilssfg.source_concepts.SrcField].
+   Currently, *pystencils-sfg* provides mappings to 
+   [`std::vector`](https://en.cppreference.com/w/cpp/container/vector)
+   (via [`std_vector_ref`][pystencilssfg.source_components.cpp.std_vector_ref])
+   and
+   [`std::mdspan`](https://en.cppreference.com/w/cpp/container/mdspan)
+   (via [`mdspan_ref`][pystencilssfg.source_components.cpp.mdspan_ref])
+   from the C++ standard library.
+ - [`map_vector`][pystencilssfg.SfgComposer.map_vector] maps a sequence of scalar numerical values
+   (given as `pystencils.TypedSymbol`s) to a vector data type. Currently, only `std::vector` is provided.
+
+#### Conditional Branches
+
+A conditonal branch may be added with [`SfgComposer.branch`][pystencilssfg.SfgComposer.branch]
+using a special syntax:
+
+```Python
+with SourceFileGenerator(sfg_config) as sfg:
+    # ...
+    
+    sfg.function("myFunction")(
+        # ...
+        sfg.branch("condition")(
+            # then-body
+        )(
+            # else-body (may be omitted)
+        )
+    )
+    
+```
\ No newline at end of file

docs/usage/index.md

+These pages provide an overview of how to use the pystencils Source File Generator.
+A basic understanding of [pystencils](https://pycodegen.pages.i10git.cs.fau.de/pystencils/index.html)
+is required.
+
+## Guides
+
+ - [Writing Generator Scripts](generator_scripts.md) explains about the primary interface of *pystencils-sfg*:
+   Generator scripts, which are Python scripts that, when executed, emit *pystencils*-generated code to a header/source
+   file pair with the same name as the script.
+ - [In-Depth: Building Source Files](building.md)
+ - [CLI and Build System Integration](cli.md)
\ No newline at end of file

mkdocs.yml

@@ -36,6 +36,11 @@ markdown_extensions:
 
 nav:
   - Home: index.md
+  - 'Usage Guide':
+    - 'Overview': usage/index.md
+    - 'Writing Generator Scripts': usage/generator_scripts.md
+    - 'In-Depth: Building Source Files': usage/building.md
+    - 'CLI and Build System Integration': usage/cli.md
   - 'API Documentation':
     - 'Overview': api/index.md
     - 'Source File Generator Front-End': api/frontend.md

src/pystencilssfg/composer.py

 from __future__ import annotations
-from typing import TYPE_CHECKING, Optional, Sequence
+from typing import TYPE_CHECKING, Sequence
 from abc import ABC, abstractmethod
 
 from pystencils import Field
@@ -27,7 +27,11 @@ class SfgComposer:
 
     @property
     def kernels(self) -> SfgKernelNamespace:
-        """The default kernel namespace."""
+        """The default kernel namespace. Add kernels like:
+        ```Python
+        sfg.kernels.add(ast, "kernel_name")
+        sfg.kernels.create(assignments, "kernel_name", config)
+        ```"""
         return self._ctx._default_kernel_namespace
 
     def kernel_namespace(self, name: str) -> SfgKernelNamespace:
@@ -63,6 +67,16 @@ class SfgComposer:
         self._ctx.add_function(func)
 
     def function(self, name: str):
+        """Add a function.
+
+        The syntax of this function adder uses a chain of two calls to mimic C++ syntax:
+
+        ```Python
+        sfg.function("FunctionName")(
+            # Function Body
+        )
+        ```
+        """
         if self._ctx.get_function(name) is not None:
             raise ValueError(f"Function {name} already exists.")
 
@@ -74,6 +88,11 @@ class SfgComposer:
         return sequencer
 
     def call(self, kernel_handle: SfgKernelHandle) -> SfgKernelCallNode:
+        """Use inside a function body to generate a kernel call.
+
+        Args:
+            kernel_handle: Handle to a kernel previously added to some kernel namespace.
+        """
         return SfgKernelCallNode(kernel_handle)
 
     def seq(self, *args: SfgCallTreeNode) -> SfgSequence:
@@ -81,18 +100,36 @@ class SfgComposer:
 
     @property
     def branch(self) -> SfgBranchBuilder:
+        """Use inside a function body to create an if/else conditonal branch.
+
+        The syntax is:
+        ```Python
+        sfg.branch("condition")(
+            # then-body
+        )(
+            # else-body (may be omitted)
+        )
+        ```
+        """
         return SfgBranchBuilder()
 
-    def map_field(self, field: Field, src_object: Optional[SrcField] = None) -> SfgDeferredFieldMapping:
-        if src_object is None:
-            raise NotImplementedError("Automatic field extraction is not implemented yet.")
-        else:
-            return SfgDeferredFieldMapping(field, src_object)
+    def map_field(self, field: Field, src_object: SrcField) -> SfgDeferredFieldMapping:
+        """Map a pystencils field to a field data structure, from which pointers, sizes
+        and strides should be extracted.
+
+        Args:
+            field: The pystencils field to be mapped
+            src_object: A `SrcField` object representing a field data structure.
+        """
+        return SfgDeferredFieldMapping(field, src_object)
 
     def map_param(self, lhs: TypedSymbolOrObject, rhs: TypedSymbolOrObject, mapping: str):
+        """Arbitrary parameter mapping: Add a single line of code to define a left-hand
+        side object from a right-hand side."""
         return SfgStatements(mapping, (lhs,), (rhs,))
 
     def map_vector(self, lhs_components: Sequence[TypedSymbolOrObject], rhs: SrcVector):
+        """Extracts scalar numerical values from a vector data type."""
         return make_sequence(*(
             rhs.extract_component(dest, coord) for coord, dest in enumerate(lhs_components)
         ))

src/pystencilssfg/context.py

@@ -3,10 +3,11 @@ from typing import Generator, Sequence
 from .configuration import SfgConfiguration, SfgCodeStyle
 from .tree.visitors import CollectIncludes
 from .source_components import SfgHeaderInclude, SfgKernelNamespace, SfgFunction
+from .exceptions import SfgException
 
 
 class SfgContext:
-    def __init__(self, argv, config: SfgConfiguration):
+    def __init__(self, config: SfgConfiguration, argv: Sequence[str] | None = None):
         self._argv = argv
         self._config = config
         self._default_kernel_namespace = SfgKernelNamespace(self, "kernels")
@@ -20,6 +21,13 @@ class SfgContext:
 
     @property
     def argv(self) -> Sequence[str]:
+        """If this context was created by a `pystencilssfg.SourceFileGenerator`, provides the command
+        line arguments given to the generator script, with configuration arguments for the code generator
+        stripped away.
+
+        Otherwise, throws an exception."""
+        if self._argv is None:
+            raise SfgException("This context provides no command-line arguments.")
         return self._argv
 
     @property

src/pystencilssfg/generator.py

@@ -21,7 +21,7 @@ class SourceFileGenerator:
 
         config = merge_configurations(project_config, cmdline_config, sfg_config)
 
-        self._context = SfgContext(script_args, config)
+        self._context = SfgContext(config, argv=script_args)
 
         from .emitters.cpu.basic_cpu import BasicCpuEmitter
         self._emitter = BasicCpuEmitter(basename, config)

src/pystencilssfg/source_components.py

@@ -56,7 +56,8 @@ class SfgKernelNamespace:
         yield from self._asts.values()
 
     def add(self, ast: KernelFunction, name: str | None = None):
-        """Adds an existing pystencils AST to this namespace."""
+        """Adds an existing pystencils AST to this namespace.
+        If a name is specified, the AST's function name is changed."""
         if name is not None:
             astname = name
         else:
@@ -73,6 +74,14 @@ class SfgKernelNamespace:
         return SfgKernelHandle(self._ctx, astname, self, ast.get_parameters())
 
     def create(self, assignments, name: str | None = None, config: CreateKernelConfig | None = None):
+        """Creates a new pystencils kernel from a list of assignments and a configuration.
+        This is a wrapper around
+        [`pystencils.create_kernel`](
+            https://pycodegen.pages.i10git.cs.fau.de/pystencils/
+            sphinx/kernel_compile_and_call.html#pystencils.create_kernel
+        )
+        with a subsequent call to [`add`][pystencilssfg.source_components.SfgKernelNamespace.add].
+        """
         if config is None:
             config = CreateKernelConfig()
 

src/pystencilssfg/source_concepts/cpp/std_vector.py

@@ -59,7 +59,7 @@ class std_vector(SrcVector, SrcField):
         else:
             return SfgStatements(f"assert( 1 == {stride} );", (), ())
 
-    def extract_component(self, destination: TypedSymbolOrObject, coordinate: int):
+    def extract_component(self, destination: TypedSymbolOrObject, coordinate: int) -> SfgStatements:
         if self._unsafe:
             mapping = f"{destination.dtype} {destination.name} = {self._identifier}[{coordinate}];"
         else:

src/pystencilssfg/source_concepts/source_objects.py

@@ -82,5 +82,5 @@ class SrcField(SrcObject, ABC):
 
 class SrcVector(SrcObject, ABC):
     @abstractmethod
-    def extract_component(self, destination: TypedSymbolOrObject, coordinate: int):
+    def extract_component(self, destination: TypedSymbolOrObject, coordinate: int) -> SfgStatements:
         pass