Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found
Select Git revision
  • fhennig/devel
  • master
  • rangersbach/c-interfacing
  • v0.1a1
  • v0.1a2
  • v0.1a3
  • v0.1a4
7 results

Target

Select target project
No results found
Select Git revision
  • examples
  • frontend-cleanup
  • lbwelding-features
  • master
  • refactor-indexing-params
  • unit_tests
  • v0.1a1
  • v0.1a2
  • v0.1a3
  • v0.1a4
10 results
Show changes

Commits on Source 66

190 files
+ 13358
6734
Compare changes
  • Side-by-side
  • Inline

Files

+1 −0
Original line number Original line Diff line number Diff line
[flake8]
[flake8]
max-line-length=120
max-line-length=120
exclude = src/pystencilssfg/_version.py
+7 −5
Original line number Original line Diff line number Diff line
@@ -7,9 +7,10 @@
**/build
**/build




#   pdm dev environment
#   dev environment
**/.venv
**/.venv
.pdm-python
**/venv
**/.nox


#   build artifacts
#   build artifacts
dist
dist
@@ -17,6 +18,7 @@ dist
*.whl
*.whl
*.egg-info
*.egg-info



# tests and coverage
#   mkdocs
.coverage*
site
htmlcov
 No newline at end of file
coverage.xml                
+63 −26
Original line number Original line Diff line number Diff line
stages:
stages:
  - pretest
  - "Code Quality"
  - test
  - "Tests"
  - "Documentation"
  - deploy
  - deploy


linter:
.nox-base:
  stage: pretest
  image: i10git.cs.fau.de:5005/pycodegen/pycodegen/nox:alpine
  except:
    variables:
      - $ENABLE_NIGHTLY_BUILDS
  image: i10git.cs.fau.de:5005/pycodegen/pycodegen/full
  script:
    - flake8 src/pystencilssfg
  tags:
  tags:
    - docker
    - docker


linter:
  extends: .nox-base
  stage: "Code Quality"
  needs: []
  script:
    - nox --session lint

typechecker:
typechecker:
  stage: pretest
  extends: .nox-base
  except:
  stage: "Code Quality"
    variables:
  needs: []
      - $ENABLE_NIGHTLY_BUILDS
  image: i10git.cs.fau.de:5005/pycodegen/pycodegen/full
  script:
  script:
    - pip install mypy
    - nox --session typecheck
    - mypy src/pystencilssfg 

.testsuite-base:
  stage: "Tests"
  needs: []
  coverage: '/TOTAL.*\s+(\d+%)$/'
  artifacts:
    when: always
    paths:
      - htmlcov
      - coverage.xml
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml

"testsuite-py3.10+cuda":
  extends: .testsuite-base
  image: i10git.cs.fau.de:5005/pycodegen/pycodegen/nox:ubuntu24.04-cuda12.6
  script:
    - nox --session testsuite-3.10
  tags:
  tags:
    - docker
    - docker
    - cuda
    - cudaComputeCapability6.1

"testsuite-py3.13":
  extends: .testsuite-base
  image: i10git.cs.fau.de:5005/pycodegen/pycodegen/nox:alpine
  script:
    - nox --session testsuite-3.13


build-documentation:
build-documentation:
  stage: test
  image: i10git.cs.fau.de:5005/pycodegen/pycodegen/nox:ubuntu24.04
  image: i10git.cs.fau.de:5005/pycodegen/pycodegen/full
  stage: "Documentation"
  needs: []
  script:
  script:
    - pip install mkdocs mkdocs-material mkdocstrings[python]
    - nox -s docs -- --fail-on-warnings
    - mkdocs build
  tags:
    - docker
  artifacts:
  artifacts:
    paths:
    paths:
      - site
      - docs/build/html
    when: always
  tags:
     - docker


pages:
pages:
  image: i10git.cs.fau.de:5005/pycodegen/pycodegen/full
  image: alpine:latest
  stage: deploy
  stage: deploy
  script:
  script:
    - ls -l
    - ls -l
    - mv site public  # folder has to be named "public" for gitlab to publish it
    - mv docs/build/html public  # folder has to be named "public" for gitlab to publish it
  artifacts:
  artifacts:
    paths:
    paths:
      - public
      - public
@@ -51,3 +79,12 @@ pages:
    - docker
    - docker
  only:
  only:
    - master@pycodegen/pystencils-sfg
    - master@pycodegen/pystencils-sfg

cmake-standalone:
  stage: deploy
  needs: []
  script:
    - echo "Publishing Cmake standalone"
  artifacts:
    paths:
      - standalone
+53 −20
Original line number Original line Diff line number Diff line
@@ -13,44 +13,77 @@ As such, any submission of contributions via merge requests is considered as agr


## Developing `pystencils-sfg`
## Developing `pystencils-sfg`


### Prequesites

To develop pystencils-sfg, you will need at least these packages:

 - Python 3.10
 - Git
 - A C++ compiler supporting at least C++20 (gcc >= 10, or clang >= 10)
 - GNU Make
 - CMake
 - Nox

Before continuing, make sure that the above packages are installed on your machine.

### Fork and Clone
### Fork and Clone


To work within the `pystencils-sfg` source tree, first create a *fork* of this repository on GitLab and create
To work within the `pystencils-sfg` source tree, first create a *fork* of this repository
a local clone of your fork.
and clone it to your workstation.


### Set up your dev environment
### Set up your dev environment


`pystencils-sfg` uses [`pdm`](https://pdm-project.org) for managing a virtual development environment.
Create a virtual environment using either `venv` or `virtualenv` and install the pystencils-sfg source tree
Install `pdm` through your system's package manager and run `pdm sync` in your cloned project directory.
into it using an editable install, e.g. by running the following commands in the `pystencils-sfg` project root directory:
It will set up a virtual environment in the subfolder `.venv`, installing all project dependencies into it.

The `pystencils-sfg` package itself is also installed in editable mode.
```bash
You can activate the virtual environment using `eval $(pdm venv activate)`.
python -m virtualenv .venv
source .venv/bin/activate
pip install -e .
```

If you have [nox](https://nox.thea.codes/en/stable/) installed, you can also set up your virtual environment
by running `nox --session dev_env`.


### Code Style and Type Checking
### Code Style and Type Checking


To contribute, please adhere to the Python code style set by [PEP 8](https://peps.python.org/pep-0008/).
To contribute, please adhere to the Python code style set by [PEP 8](https://peps.python.org/pep-0008/).
It is recommended that you use the [black](https://pypi.org/project/black/) formatter to format your source files.
For consistency, format all your source files using the [black](https://pypi.org/project/black/) formatter,
Use flake8 (installed in the `pdm` virtual environment) to check your code style:
and check them regularily using the `flake8` linter through Nox:


```shell
```shell
pdm run flake8 src/pystencilssfg
nox --session lint
# or, if .venv is activated
flake8 src/pystencilssfg
```
```


Further, `pystencils-sfg` takes a rigorous approach to correct static typing.
Further, `pystencils-sfg` is being fully type-checked using [MyPy](https://www.mypy-lang.org/).
All submitted code should contain type annotations ([PEP 484](https://peps.python.org/pep-0484/)) and must be
All submitted code should contain type annotations ([PEP 484](https://peps.python.org/pep-0484/)) and must be
correctly statically typed.
correctly statically typed.
To check types, we use [MyPy](https://www.mypy-lang.org/), which is automatically installed in the dev environment
Regularily check your code for type errors using
and can be invoked as


```shell
```shell
pdm run mypy src/pystencilssfg
nox --session typecheck
# or, if .venv is activated
mypy src/pystencilssfg
```
```


Both `flake8` and `mypy` are also run in the integration pipeline.
Both `flake8` and `mypy` are also run in the integration pipeline.
It is furthermore recommended to run both checkers as a git pre-commit hook.
Such a hook can be installed using the [`install_git_hooks.sh`](install_git_hooks.sh) script located at the project root.


### Test Your Code

We are working toward near-complete test coverage of the module source files.
When you add code, make sure to include test cases for both its desired
and exceptional behavior at the appropriate locations in the [tests](tests) directory.

Unit tests should be placed under a path and filename mirroring the location
of the API they are testing within the *pystencils-sfg* source tree.

In [tests/generator_scripts](tests/generator_scripts), a framework is provided to test entire generator scripts
for successful execution, correctness, and compilability of their output.
Read the documentation within [test_generator_scripts.py](tests/generator_scripts/test_generator_scripts.py)
for more information.

Run the test suite by calling it through Nox:

```shell
nox --session testsuite
```

This will also collect coverage information and produce a coverage report as a HTML site placed in the `htmlcov` folder.
+4 −3
Original line number Original line Diff line number Diff line
# pystencils Source File Generator (pystencils-sfg)
# pystencils Source File Generator (pystencils-sfg)


[![](https://img.shields.io/badge/read-the_docs-brightgreen)](https://pycodegen.pages.i10git.cs.fau.de/pystencils-sfg)
[![documentation](https://img.shields.io/badge/read-the_docs-brightgreen)](https://pycodegen.pages.i10git.cs.fau.de/pystencils-sfg)
[![](https://i10git.cs.fau.de/pycodegen/pystencils-sfg/badges/master/pipeline.svg)](https://i10git.cs.fau.de/pycodegen/pystencils-sfg/commits/master)
[![pipeline](https://i10git.cs.fau.de/pycodegen/pystencils-sfg/badges/master/pipeline.svg)](https://i10git.cs.fau.de/pycodegen-/pystencils-sfg/commits/master)
[![](https://img.shields.io/gitlab/license/pycodegen%2Fpystencils-sfg?gitlab_url=https%3A%2F%2Fi10git.cs.fau.de)](https://i10git.cs.fau.de/pycodegen/pystencils-sfg/-/blob/master/LICENSE)
![coverage](https://i10git.cs.fau.de/pycodegen/pystencils-sfg/badges/master/coverage.svg)
[![licence](https://img.shields.io/gitlab/license/pycodegen%2Fpystencils-sfg?gitlab_url=https%3A%2F%2Fi10git.cs.fau.de)](https://i10git.cs.fau.de/pycodegen/pystencils-sfg/-/blob/master/LICENSE)




A bridge over the semantic gap between code emitted by pystencils and your C/C++/Cuda/HIP framework.
A bridge over the semantic gap between code emitted by pystencils and your C/C++/Cuda/HIP framework.

conftest.py

0 → 100644
+31 −0
Original line number Original line Diff line number Diff line
import pytest
from os import path


DATA_DIR = path.join(path.split(__file__)[0], "tests/data")


@pytest.fixture
def sample_config_module():
    return path.join(DATA_DIR, "project_config.py")


@pytest.fixture
def sfg():
    from pystencilssfg import SfgContext, SfgComposer
    from pystencilssfg.ir import SfgSourceFile, SfgSourceFileType

    return SfgComposer(
        SfgContext(
            header_file=SfgSourceFile("", SfgSourceFileType.HEADER),
            impl_file=SfgSourceFile("", SfgSourceFileType.TRANSLATION_UNIT),
        )
    )


@pytest.fixture(autouse=True)
def prepare_doctest_namespace(doctest_namespace, sfg):
    from pystencilssfg import lang

    doctest_namespace["sfg"] = sfg
    doctest_namespace["lang"] = lang

docs/.gitignore

0 → 100644
+1 −0
Original line number Original line Diff line number Diff line
**/generated

docs/Makefile

0 → 100644
+20 −0
Original line number Original line Diff line number Diff line
# Minimal makefile for Sphinx documentation
#

# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS    ?=
SPHINXBUILD   ?= sphinx-build
SOURCEDIR     = source
BUILDDIR      = build

# Put it first so that "make" without argument is like "make help".
help:
	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

.PHONY: help Makefile

# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/api/composer.md

deleted100644 → 0
+0 −8
Original line number Original line Diff line number Diff line

::: pystencilssfg.composer.SfgComposer

::: pystencilssfg.composer.SfgBasicComposer

::: pystencilssfg.composer.SfgClassComposer

::: pystencilssfg.composer.make_sequence
 No newline at end of file

docs/api/context.md

deleted100644 → 0
+0 −2
Original line number Original line Diff line number Diff line

::: pystencilssfg.context.SfgContext

docs/api/cpp_std.md

deleted100644 → 0
+0 −8
Original line number Original line Diff line number Diff line

::: pystencilssfg.source_concepts.cpp.std_vector_ref

::: pystencilssfg.source_concepts.cpp.mdspan_ref

::: pystencilssfg.source_concepts.cpp.StdVector

::: pystencilssfg.source_concepts.cpp.StdMdspan

docs/api/emission.md

deleted100644 → 0
+0 −14
Original line number Original line Diff line number Diff line

## Output Configuration

::: pystencilssfg.configuration.SfgOutputSpec

## Header-Implementation-Pair Emission

::: pystencilssfg.emission.HeaderImplPairEmitter

## Code Style and `clang-format`

::: pystencilssfg.configuration.SfgCodeStyle

::: pystencilssfg.emission.clang_format.invoke_clang_format
 No newline at end of file

docs/api/generator.md

deleted100644 → 0
+0 −6
Original line number Original line Diff line number Diff line

::: pystencilssfg.generator.SourceFileGenerator

::: pystencilssfg.configuration.SfgConfiguration

::: pystencilssfg.configuration.DEFAULT_CONFIG

docs/api/index.md

deleted100644 → 0
+0 −23
Original line number Original line Diff line number Diff line
# API Documentation

These pages document the public API of *pystencils-sfg*.

### Front End

 - [Source File Generator](generator.md)
 - [Code Generation Context](context.md)
 - [Composer](composer.md)

### Source File Modelling

 - [Source File Components](source_components.md)
 - [Kernel Call Tree](tree.md)

### High-Level Language Concepts

 - [Base Classes](source_objects.md)
 - [C++ Standard Library](cpp_std.md)

### Code Generation

 - [Emission and Printing](emission.md)
 No newline at end of file

docs/api/source_components.md

deleted100644 → 0
+0 −36
Original line number Original line Diff line number Diff line

## Kernels and Kernel Namespaces

::: pystencilssfg.source_components.SfgKernelNamespace

::: pystencilssfg.source_components.SfgKernelHandle

## Includes

::: pystencilssfg.source_components.SfgHeaderInclude

## Functions

::: pystencilssfg.source_components.SfgFunction

## Classes

::: pystencilssfg.source_components.SfgClassKeyword

::: pystencilssfg.source_components.SfgClass

### Visibility

::: pystencilssfg.source_components.SfgVisibility

::: pystencilssfg.source_components.SfgVisibilityBlock

### Members

::: pystencilssfg.source_components.SfgClassMember

::: pystencilssfg.source_components.SfgInClassDefinition

::: pystencilssfg.source_components.SfgConstructor

::: pystencilssfg.source_components.SfgMethod

docs/api/source_objects.md

deleted100644 → 0
+0 −6
Original line number Original line Diff line number Diff line

::: pystencilssfg.source_concepts.SrcObject

::: pystencilssfg.source_concepts.SrcField

::: pystencilssfg.source_concepts.SrcVector

docs/api/tree.md

deleted100644 → 0
+0 −30
Original line number Original line Diff line number Diff line

## Base Classes

::: pystencilssfg.tree.SfgCallTreeNode

::: pystencilssfg.tree.SfgCallTreeLeaf

::: pystencilssfg.tree.SfgEmptyNode

## Utility Nodes

::: pystencilssfg.tree.SfgFunctionParams

::: pystencilssfg.tree.SfgRequireIncludes

## Sequences of Statements

::: pystencilssfg.tree.SfgSequence

::: pystencilssfg.tree.SfgStatements

## Structural and Conditional Constructs

::: pystencilssfg.tree.SfgBlock

## Kernel Calls

::: pystencilssfg.tree.SfgKernelCallNode

docs/css/mkdocstrings.css

deleted100644 → 0
+0 −35
Original line number Original line Diff line number Diff line

h2.doc-heading {
    font-size: x-large
}

h3.doc-heading {
    font-size: large;
}

.doc-class>.doc-heading::before {
    font-size: small;
    content: "class ";
    margin-right: 5pt;
}

.doc-contents {
    border-left: 3pt solid rgb(60, 60, 60);
    padding-left: 10pt;
}

.doc-class .doc-children .doc-attribute>.doc-heading::before {
    font-size: small;
    content: "attribute ";
    margin-right: 5pt;
}

.doc-class .doc-children .doc-function>.doc-heading::before {
    font-size: small;
    content: "function ";
    margin-right: 5pt;
}

.doc-children {
    padding-left: 10pt;
}
 No newline at end of file

docs/index.md

deleted100644 → 0
+0 −94
Original line number Original line Diff line number Diff line
# The pystencils Source File Generator

[![](https://i10git.cs.fau.de/pycodegen/pystencils-sfg/badges/master/pipeline.svg)](https://i10git.cs.fau.de/pycodegen/pystencils-sfg/commits/master)
[![](https://img.shields.io/gitlab/license/pycodegen%2Fpystencils-sfg?gitlab_url=https%3A%2F%2Fi10git.cs.fau.de)](https://i10git.cs.fau.de/pycodegen/pystencils-sfg/-/blob/master/LICENSE)

A bridge over the semantic gap between code emitted by [pystencils](https://pypi.org/project/pystencils/)
and your C/C++/Cuda/HIP framework.

## Installation

### From Git

Install the package into your current Python environment from the git repository using pip
(usage of virtual environments is strongly encouraged!):

```bash
pip install git+https://i10git.cs.fau.de/pycodegen/pystencils-sfg.git
```

### From PyPI

Not yet available.

## Primer

With *pystencils-sfg*, including your *pystencils*-generated kernels with handwritten code becomes straightforward
and intuitive. To illustrate, generating a Jacobi smoother for the two-dimensional Poisson equation
and mapping it onto C++23 `std::mdspan`s takes just a few lines of code:

```python
import sympy as sp

from pystencils import fields, kernel

from pystencilssfg import SourceFileGenerator, SfgComposer
from pystencilssfg.source_concepts.cpp import mdspan_ref

with SourceFileGenerator() as ctx:
    sfg = SfgComposer(ctx)

    u_src, u_dst, f = fields("u_src, u_dst, f(1) : double[2D]", layout="fzyx")
    h = sp.Symbol("h")

    @kernel
    def poisson_jacobi():
        u_dst[0,0] @= (h**2 * f[0, 0] + u_src[1, 0] + u_src[-1, 0] + u_src[0, 1] + u_src[0, -1]) / 4

    poisson_kernel = sfg.kernels.create(poisson_jacobi)

    sfg.function("jacobi_smooth")(
        sfg.map_field(u_src, mdspan_ref(u_src)),
        sfg.map_field(u_dst, mdspan_ref(u_dst)),
        sfg.map_field(f, mdspan_ref(f)),
        sfg.call(poisson_kernel)
    )
```

Take this code, store it into a file `poisson_smoother.py`, and enter the magic words into a terminal:

```shell
python poisson_smoother.py
```

This command will execute the code generator through the `SourceFileGenerator` context manager.
The code generator takes the name of your Python script, replaces `.py` with `.cpp` and `.h`, and writes
`poisson_smoother.cpp` and `poisson_smoother.h` into the current directory, ready to be `#include`d.

The above is what we call a *generator script*; a Python script that, when executed, produces a pair
of source files of the same name, but with different extensions.
Generator scripts are the primary front-end pattern of *pystencils-sfg*; to learn more about them,
read the [Usage Guide](usage/generator_scripts.md).

## CMake Integration

*Pystencils-sfg* comes with a CMake module to register generator scripts for on-the-fly code generation.
With the module loaded, use the function `pystencilssfg_generate_target_sources` inside your `CMakeLists.txt`
to register one or multiple generator scripts; their outputs will automatically be added to the specified target.

```CMake
pystencilssfg_generate_target_sources( <target name> 
    SCRIPTS kernels.py ...
    FILE_EXTENSIONS .h .cpp
)
```

*Pystencils-sfg* makes sure that all generated files are on the project's include path.
To `#include` them, add the prefix `gen/<target name>`:

```C++
#include "gen/<target name>/kernels.h"
```

For details on how to add *pystencils-sfg* to your CMake project, refer to
[CLI and Build System Integration](usage/cli_and_build_system.md).

docs/make.bat

0 → 100644
+35 −0
Original line number Original line Diff line number Diff line
@ECHO OFF

pushd %~dp0

REM Command file for Sphinx documentation

if "%SPHINXBUILD%" == "" (
	set SPHINXBUILD=sphinx-build
)
set SOURCEDIR=source
set BUILDDIR=build

%SPHINXBUILD% >NUL 2>NUL
if errorlevel 9009 (
	echo.
	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
	echo.installed, then set the SPHINXBUILD environment variable to point
	echo.to the full path of the 'sphinx-build' executable. Alternatively you
	echo.may add the Sphinx directory to PATH.
	echo.
	echo.If you don't have Sphinx installed, grab it from
	echo.https://www.sphinx-doc.org/
	exit /b 1
)

if "%1" == "" goto help

%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
goto end

:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%

:end
popd

docs/source/.gitignore

0 → 100644
+5 −0
Original line number Original line Diff line number Diff line
**/_sfg_out
**/.ipynb_checkpoints

mdspan_demo
a.out
Original line number Original line Diff line number Diff line
{% extends "!autosummary/class.rst" %}

   {% block methods %}

   {% if methods %}
   .. rubric:: {{ _('Methods') }}

   .. autosummary::
      :toctree:
   {% for item in methods %}
      ~{{ name }}.{{ item }}
   {%- endfor %}
   {% endif %}
   {% endblock %}

   {% block attributes %}
   {% if attributes %}
   .. rubric:: {{ _('Attributes') }}

   .. autosummary::
      :toctree:
   {% for item in attributes %}
      ~{{ name }}.{{ item }}
   {%- endfor %}
   {% endif %}
   {% endblock %}
+87 −0
Original line number Original line Diff line number Diff line
import pystencilssfg
from pystencilssfg.config import SfgConfig

from os.path import splitext
from pathlib import Path


class DocsPatchedGenerator(pystencilssfg.SourceFileGenerator):
    """Mockup wrapper around SourceFileGenerator for use in documentation
    notebooks to print the generated code directly to the HTML
    instead of writing it to file."""

    scriptname: str = "demo"
    glue: bool = False
    display: bool = True
    output_dir: str | None = None

    @classmethod
    def setup(
        cls,
        scriptname: str,
        glue: bool = False,
        display: bool = True,
        output_dir: str | None = None,
    ):
        cls.scriptname = scriptname
        cls.glue = glue
        cls.display = display
        cls.output_dir = output_dir

    def _scriptname(self) -> str:
        return f"{DocsPatchedGenerator.scriptname}.py"

    def __init__(
        self, sfg_config: SfgConfig | None = None, keep_unknown_argv: bool = False
    ):
        if DocsPatchedGenerator.output_dir:
            sfg_config = sfg_config.copy() if sfg_config is not None else SfgConfig()
            sfg_config.output_directory = DocsPatchedGenerator.output_dir
        super().__init__(sfg_config, keep_unknown_argv=True)

    def __exit__(self, exc_type, exc_value, traceback):
        if exc_type is None:
            self._finish_files()
            emitter = self._get_emitter()

            if DocsPatchedGenerator.output_dir:
                emitter.emit(self._header_file)
                if self._impl_file is not None:
                    emitter.emit(self._impl_file)

            header_code = emitter.dumps(self._header_file)
            header_ext = splitext(self._header_file.name)[1]

            mdcode = ":::::{tab-set}\n"

            mdcode += f"::::{{tab-item}} Generated Header ({header_ext})\n"
            mdcode += ":::{code-block} C++\n\n"
            mdcode += header_code
            mdcode += "\n:::\n::::\n"

            if self._impl_file is not None:
                impl_code = emitter.dumps(self._impl_file)
                impl_ext = splitext(self._impl_file.name)[1]

                mdcode += f"::::{{tab-item}} Generated Implementation ({impl_ext})\n"
                mdcode += ":::{code-block} C++\n\n"
                mdcode += impl_code
                mdcode += "\n:::\n::::\n"

            mdcode += ":::::"
            from IPython.display import Markdown

            mdobj = Markdown(mdcode)

            if self.glue:
                from myst_nb import glue

                glue(f"sfg_out_{self.scriptname}", mdobj, display=False)

            if self.display:
                from IPython.display import display

                display(mdobj)


pystencilssfg.SourceFileGenerator = DocsPatchedGenerator
+73 −0
Original line number Original line Diff line number Diff line
*****************************************
Composer API (``pystencilssfg.composer``)
*****************************************

.. module:: pystencilssfg.composer

.. autoclass:: SfgComposer
    :members:

.. autoclass:: SfgIComposer
    :members:

.. autoclass:: SfgBasicComposer
    :members:

.. autoclass:: SfgClassComposer
    :members:

.. autoclass:: SfgGpuComposer
    :members:

Custom Generators
=================

.. module:: pystencilssfg.composer.custom

.. autoclass:: CustomGenerator
    :members:


Helper Methods and Builders
===========================

.. module:: pystencilssfg.composer.basic_composer

.. autofunction:: make_sequence

.. autoclass:: KernelsAdder
    :members:

.. autoclass:: SfgFunctionSequencer
    :members:
    :inherited-members:

.. autoclass:: SfgNodeBuilder
    :members:

.. autoclass:: SfgBranchBuilder
    :members:

.. autoclass:: SfgSwitchBuilder
    :members:

.. module:: pystencilssfg.composer.class_composer

.. autoclass:: SfgMemberVarSequencer
    :members:
    :inherited-members:

.. autoclass:: SfgMethodSequencer
    :members:
    :inherited-members:

Context and Cursor
==================

.. module:: pystencilssfg.context

.. autoclass:: SfgContext
    :members:

.. autoclass:: SfgCursor
    :members:
+6 −0
Original line number Original line Diff line number Diff line
*********************
Errors and Exceptions
*********************

.. automodule:: pystencilssfg.exceptions
    :members:
+32 −0
Original line number Original line Diff line number Diff line
**************************
Generator Script Interface
**************************

.. autoclass:: pystencilssfg.SourceFileGenerator
    :members:

Configuration
=============

.. module:: pystencilssfg.config

.. autoclass:: SfgConfig
    :members:
    :exclude-members: __init__
    

Categories, Parameter Types, and Special Values
-----------------------------------------------

.. autoclass:: _GlobalNamespace
.. autodata:: GLOBAL_NAMESPACE

.. autoclass:: FileExtensions
    :members:

.. autoclass:: CodeStyle
    :members:

.. autoclass:: ClangFormatOptions
    :members:

docs/source/api/ir.rst

0 → 100644
+11 −0
Original line number Original line Diff line number Diff line
Internal Code Representation (`pystencilssfg.ir`)
=================================================

.. automodule:: pystencilssfg.ir
    :members:

Postprocessing
--------------

.. automodule:: pystencilssfg.ir.postprocessing
    :members:
+49 −0
Original line number Original line Diff line number Diff line
Language Modelling (`pystencilssfg.lang`)
=========================================

.. automodule:: pystencilssfg.lang

Expressions
-----------

.. automodule:: pystencilssfg.lang.expressions
    :members:

Header Files
------------

.. automodule:: pystencilssfg.lang.headers
    :members:

Data Types
----------

.. automodule:: pystencilssfg.lang.types
    :members:

Extraction Protocols
--------------------

.. automodule:: pystencilssfg.lang.extractions
    :members:

C++ Standard Library (``pystencilssfg.lang.cpp``)
-------------------------------------------------

Quick Access
^^^^^^^^^^^^

.. automodule:: pystencilssfg.lang.cpp.std
    :members:

Implementation
^^^^^^^^^^^^^^

.. automodule:: pystencilssfg.lang.cpp
    :members:

GPU Runtime APIs
----------------

.. automodule:: pystencilssfg.lang.gpu
    :members:

docs/source/conf.py

0 → 100644
+94 −0
Original line number Original line Diff line number Diff line
from pystencilssfg import __version__ as sfg_version
from packaging.version import Version

# Configuration file for the Sphinx documentation builder.
#
# For the full list of built-in configuration values, see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html

# -- Project information -----------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information

project = "pystencils-sfg"
copyright = "2024, Frederik Hennig"
author = "Frederik Hennig"

parsed_version = Version(sfg_version)

version = ".".join([parsed_version.public])
release = sfg_version

html_title = f"pystencils-sfg v{version} Documentation"

# -- General configuration ---------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration

extensions = [
    "myst_nb",
    "sphinx.ext.autodoc",
    "sphinx.ext.napoleon",
    "sphinx.ext.autosummary",
    "sphinx.ext.doctest",
    "sphinx.ext.intersphinx",
    "sphinx_autodoc_typehints",
    "sphinx_design",
    "sphinx_copybutton"
]

templates_path = ["_templates"]
exclude_patterns = []
master_doc = "index"


# -- Options for HTML output -------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output

html_theme = "sphinx_book_theme"
html_static_path = ['_static']
html_theme_options = {
   "logo": {
      "image_light": "_static/sfg-logo-light.svg",
      "image_dark": "_static/sfg-logo-dark.svg",
   }
}

#   Intersphinx

intersphinx_mapping = {
    "python": ("https://docs.python.org/3.8", None),
    "numpy": ("https://numpy.org/doc/stable/", None),
    "sympy": ("https://docs.sympy.org/latest/", None),
    "pystencils": ("https://pycodegen.pages.i10git.cs.fau.de/docs/pystencils/2.0dev/", None),
}

#   References

#   Treat `single-quoted` code blocks as references to any
default_role = "any"

#   Autodoc options

autodoc_member_order = "bysource"
autodoc_typehints = "description"
# autodoc_type_aliases = {
#     "VarLike": "pystencilssfg.lang.expressions.VarLike",
#     "ExprLike": "pystencilssfg.lang.expressions.ExprLike"
# }

#   Doctest Setup

doctest_global_setup = '''
from pystencilssfg import SfgContext, SfgComposer
sfg = SfgComposer(SfgContext())
'''


# -- Options for MyST / MyST-NB ----------------------------------------------

nb_execution_mode = "cache"  # do not execute notebooks by default

myst_enable_extensions = [
    "dollarmath",
    "colon_fence",
]
nb_render_markdown_format = "myst"
+330 −0
Original line number Original line Diff line number Diff line
---
jupytext:
  formats: md:myst
  text_representation:
    extension: .md
    format_name: myst
    format_version: 0.13
    jupytext_version: 1.16.4
kernelspec:
  display_name: Python 3 (ipykernel)
  language: python
  name: python3
---

(getting_started_guide)=
# Getting Started

```{code-cell} ipython3
:tags: [remove-cell]

import sys
from pathlib import Path

mockup_path = Path("_util").resolve()
sys.path.append(str(mockup_path))

from sfg_monkeypatch import DocsPatchedGenerator  # monkeypatch SFG for docs
```

This guide will explain the basics of using pystencils-sfg through generator scripts.
Generator scripts are the primary way to run code generation with pystencils-sfg.
A generator script is a Python script that, when executed, produces one or more
C++ source files with the same base name, but different file extensions.

## Writing a Basic Generator Script

To start using pystencils-sfg, create a new empty Python file and populate it with the
following minimal skeleton:

```{code-block} python
from pystencilssfg import SourceFileGenerator

with SourceFileGenerator() as sfg:
    ...
```

The above snippet defines the basic structure of a *generator script*.
When executed, the above will produce two (nearly) empty C++ files
in the current folder, both with the same name as your Python script
but with `.hpp` and `.cpp` file extensions instead.

In the generator script, code generation is orchestrated by the `SourceFileGenerator` context manager.
When entering into the region controlled by the `SourceFileGenerator`,
it supplies us with a *composer object*, customarily called `sfg`.
Through the composer, we can declaratively populate the generated files with code.

## Adding a pystencils Kernel

One of the core applications of pystencils-sfg is to generate and wrap pystencils-kernels
for usage within C++ applications.
To register a kernel, pass its assignments to `sfg.kernels.create`, which returns a *kernel handle* object:

```{code-block} python
src, dst = ps.fields("src, dst: double[1D]")
c = sp.Symbol("c")

@ps.kernel
def scale():
    dst.center @= c * src.center()

#   Register the kernel for code generation
scale_kernel = sfg.kernels.create(scale, "scale_kernel")
```

In order to call the kernel, and expose it to the outside world,
we have to create a wrapper function for it, using `sfg.function`.
In its body, we use `sfg.call` to invoke the kernel:

```{code-block} python
sfg.function("scale")(
    sfg.call(scale_kernel)
)
```

The `function` composer has a special syntax that mimics the generated C++ code.
We call it twice in sequence,
first providing the name of the function, and then populating its body.

Here's our full first generator script:

```{code-cell} ipython3
:tags: [remove-cell]

DocsPatchedGenerator.scriptname = "add_kernel_demo"
DocsPatchedGenerator.glue = True
DocsPatchedGenerator.display = False
```

```{code-cell} ipython3
from pystencilssfg import SourceFileGenerator
import pystencils as ps
import sympy as sp

with SourceFileGenerator() as sfg:
    #   Define a copy kernel
    src, dst = ps.fields("src, dst: double[1D]")
    c = sp.Symbol("c")

    @ps.kernel
    def scale():
        dst.center @= c * src.center()

    #   Register the kernel for code generation
    scale_kernel = sfg.kernels.create(scale, "scale_kernel")

    #   Wrap it in a function
    sfg.function("scale")(
        sfg.call(scale_kernel)
    )
```

When executing the above script, two files will be generated: a C++ header and implementation file containing
the `scale_kernel` and its wrapper function:

:::{glue:md} sfg_out_add_kernel_demo
:format: myst
:::

As you can see, the header file contains a declaration `void scale(...)` of a function
which is defined in the associated implementation file,
and there calls our generated numerical kernel.
As of now, it forwards the entire set of low-level kernel arguments -- array pointers and indexing information --
to the outside.
In numerical applications, this information is most of the time hidden from the user by encapsulating
it in high-level C++ data structures.
Pystencils-sfg offers means of representing such data structures in the code generator, and supports the
automatic extraction of the low-level indexing information from them.

## Mapping Fields to Data Structures

Since C++23 there exists the archetypical [std::mdspan][mdspan], which represents a non-owning n-dimensional view
on a contiguous data array.
Pystencils-sfg offers native support for mapping pystencils fields onto `mdspan` instances in order to
hide their memory layout details.

Import `std` from `pystencilssfg.lang.cpp` and use `std.mdspan.from_field` to create representations
of your pystencils fields as `std::mdspan` objects:

```{code-block} python
from pystencilssfg.lang.cpp import std

...

src_mdspan = std.mdspan.from_field(src)
dst_mdspan = std.mdspan.from_field(dst)
```

Then, inside the wrapper function, instruct the SFG to map the fields onto their corresponding mdspans:

```{code-block} python
sfg.function("scale")(
    sfg.map_field(src, src_mdspan),
    sfg.map_field(dst, dst_mdspan),
    sfg.call(scale_kernel)
)
```

Here's the full script and its output:

```{code-cell} ipython3
:tags: [remove-cell]

DocsPatchedGenerator.setup("generated", False, True, "mdspan_demo")
```

```{code-cell} ipython3
from pystencilssfg import SourceFileGenerator
import pystencils as ps
import sympy as sp

from pystencilssfg.lang.cpp import std

with SourceFileGenerator() as sfg:
    #   Define a copy kernel
    src, dst = ps.fields("src, dst: double[1D]")
    c = sp.Symbol("c")

    @ps.kernel
    def scale():
        dst.center @= c * src.center()

    #   Register the kernel for code generation
    scale_kernel = sfg.kernels.create(scale, "scale_kernel")

    #   Create mdspan objects
    src_mdspan = std.mdspan.from_field(src, layout_policy="layout_left")
    dst_mdspan = std.mdspan.from_field(dst, layout_policy="layout_left")

    #   Wrap it in a function
    sfg.function("scale")(
        sfg.map_field(src, src_mdspan),
        sfg.map_field(dst, dst_mdspan),
        sfg.call(scale_kernel)
    )
```

:::{note}

As of early 2025, `std::mdspan` is still not fully adopted by standard library implementors
(see [cppreference.com][cppreference_compiler_support]);
most importantly, the GNU libstdc++ does not yet ship an implementation of it.
However, a reference implementation is available at https://github.com/kokkos/mdspan.
If you are using the reference implementation, refer to the documentation of {any}`StdMdspan`
for advice on how to configure the header file and namespace where the class is defined.
:::

## Integrating and Compiling the Generated Code

To use the generated functions from handwritten code, include the generated header file
into your C++ application, compile the generated translation unit to an object file,
and link the two together.

Here is an example application frame:

```{code-cell} ipython3
:tags: [remove-input]

from IPython.display import Markdown

output_dir = Path("mdspan_demo")
output_dir.mkdir(exist_ok=True)

code = r"""
#include <mdspan>
#include <memory>
#include <iostream>

// Include the generated header
#include "generated.hpp"

using field_t = std::mdspan< double, std::dextents< uint64_t, 1 >, std::layout_left >;

int main(void){
    constexpr size_t N = 4;
    auto data_src = std::make_unique< double[] >(N);
    auto data_dst = std::make_unique< double[] >(N);

    field_t src { data_src.get(), N };
    field_t dst { data_dst.get(), N };

    for(size_t i = 0; i < N; ++i)
        src[i] = 2.0 * double(i) + 1.0;

    const double c { 4.5 };

    // Call generated function
    scale(c, dst, src);

    // Print output
    for(size_t i = 0; i < N; ++i)
        std::cout << dst[i] << std::endl;

    return 0;
}
"""
(output_dir / "app.cpp").write_text(code)

md = f""":::{{code-block}} C++
:caption: app.cpp

{code}

:::
"""

Markdown(md)
```

The application sets up the data arrays and `mdspan`
views for the `src` and `dst` fields, and initializes `src`.
Then, it invokes the `scale` kernel defined in the above generator script
on these `mdspan` objects and prints the result to stdout.

Save the code into a file `app.cpp`.
We can now compile application frame and generated code together, and link them into an executable,
using the following compiler command:

```{code-block} bash
clang++ -std=c++23 -stdlib=libc++ -I $(python -m pystencils.include -s) generated.cpp app.cpp
```

Let's briefly take a look at the compiler options:
- `std=c++23` ensures the C++23 standard, required for `std::mdspan`;
- `-stdlib=libc++` instructs `clang` to link against the [LLVM C++ standard library](https://libcxx.llvm.org/),
  which implements `std::mdspan`;
- `-I $(python -m pystencils.include -s)` adds the location of the pystencils runtime headers to the compiler's include path;
  the header path is obtained by executing the `python -m pystencils.include -s` subcommand.

:::{note}
The above command requires that at least clang 18 and `libc++` are installed.
On Ubuntu >= 24, you can install these via `apt-get install clang libc++-dev`.
On older systems, install `clang-18 libc++-18-dev` instead.
:::

After succesful compilation, running the executable should yield the following output:

```{code-block} bash
./a.out
```

```{code-cell} ipython3
:tags: [remove-input]

!cd mdspan_demo; clang++ -std=c++23 -stdlib=libc++ -I $(python -m pystencils.include -s) generated.cpp app.cpp
!./mdspan_demo/a.out
```

That's it! We've now gone through all the basic steps of generating code
and integrating it into an application using pystencils-sfg.

## Next Steps

To learn more about using pystencils-sfg's composer API, read [](#composer_guide).
For integrating reflection of your own C++ APIs and field classes in pystencils-sfg,
refer to [](#how_to_cpp_api_modelling).
At [](#guide_project_integration), you can find more information about integrating pystencils-sfg
with your project and build system to run code generation on-the-fly.

[mdspan]: https://en.cppreference.com/w/cpp/container/mdspan
[cppreference_compiler_support]: https://en.cppreference.com/w/cpp/compiler_support

docs/source/index.md

0 → 100644
+58 −0
Original line number Original line Diff line number Diff line
# The pystencils Source File Generator


[![pipeline](https://i10git.cs.fau.de/pycodegen/pystencils-sfg/badges/master/pipeline.svg)](https://i10git.cs.fau.de/pycodegen-/pystencils-sfg/commits/master)
[![coverage](https://i10git.cs.fau.de/pycodegen/pystencils-sfg/badges/master/coverage.svg)](https://i10git.cs.fau.de/pycodegen-/pystencils-sfg/commits/master)
[![licence](https://img.shields.io/gitlab/license/pycodegen%2Fpystencils-sfg?gitlab_url=https%3A%2F%2Fi10git.cs.fau.de)](https://i10git.cs.fau.de/pycodegen/pystencils-sfg/-/blob/master/LICENSE)

*A bridge over the semantic gap between [pystencils](https://pypi.org/project/pystencils/) and C++ HPC frameworks.*

The pystencils Source File Generator is a code generation tool that allows you to
declaratively describe and automatically generate C++ code using its Python API.
It is part of the wider [pycodegen][pycodegen] family of packages for scientific code generation.

The primary purpose of pystencils-sfg is to embed the [pystencils][pystencils] code generator for
high-performance stencil computations into C++ HPC applications and frameworks of all scales.
Its features include:

 - Exporting pystencils kernels to C++ source files for use in larger projects
 - Mapping of symbolic pystencils fields onto a wide variety of n-dimensional array data structures
 - Orchestration of code generation as part of a Makefile or CMake project
 - Declarative description of C++ code structure including functions and classes using the versatile composer API
 - Reflection of C++ APIs in the code generator, including automatic tracking of variables and `#include`s


## Table of Contents

```{toctree}
:maxdepth: 1

installation
getting_started
```

```{toctree}
:maxdepth: 1
:caption: User Guide

usage/how_to_composer
usage/api_modelling
usage/config_and_cli
usage/project_integration
usage/tips_n_tricks
```


```{toctree}
:maxdepth: 1
:caption: API Reference

api/generation
api/composer
api/lang
api/ir
api/errors
```

[pycodegen]: https://pycodegen.pages.i10git.cs.fau.de
[pystencils]: https://pycodegen.pages.i10git.cs.fau.de/docs/pystencils/2.0dev
+45 −0
Original line number Original line Diff line number Diff line
# Installation and Setup

## Prequesites

To use pystencils-sfg, you will need at least Python 3.10.
You will also need the appropriate compilers for building the generated code,
such as
 - a modern C++ compiler (e.g. GCC, clang)
 - `nvcc` for CUDA or `hipcc` for HIP
 - Intel OneAPI or AdaptiveCpp for SYCL

Furthermore, an installation of clang-format for automatic code formatting is strongly recommended. 

## Install the Latest Development Revision

As pystencils-sfg is still unreleased, it can at this time only be obtained directly
from its Git repository.

Create a fresh [virtual environment](https://docs.python.org/3/library/venv.html) or activate
an existing one. Install both the pystencils 2.0 and pystencils-sfg development revisions from Git:

```{code-block} bash
pip install "git+https://i10git.cs.fau.de/pycodegen/pystencils.git@v2.0-dev"
pip install "git+https://i10git.cs.fau.de/pycodegen/pystencils-sfg.git"
```

````{caution}

*pystencils-sfg* is not compatible with the *pystencils 1.3.x* releases available from PyPI;
at the moment, you will still have to manually install the latest version of pystencils 2.0.
````

## Check your Installation

To verify that the SFG was successfully installed, execute the following command:

```{code-block} bash
sfg-cli version
```

You should see an output like `0.1a4+...`.

## Next Steps

Move on to [](#getting_started_guide) for a guide on how to author simple generator scripts.
+420 −0
Original line number Original line Diff line number Diff line
---
file_format: mystnb
kernelspec:
  name: python3
---

(how_to_cpp_api_modelling)=
# How To Reflect C++ APIs

```{code-cell} ipython3
:tags: [remove-cell]

from __future__ import annotations
import sys
from pathlib import Path

mockup_path = Path("../_util").resolve()
sys.path.append(str(mockup_path))

from sfg_monkeypatch import DocsPatchedGenerator  # monkeypatch SFG for docs

from pystencilssfg import SourceFileGenerator
```

Pystencils-SFG is designed to help you generate C++ code that interfaces with pystencils on the one side,
and with your handwritten code on the other side.
This requires that the C++ classes and APIs of your framework or application be represented within the SFG system.
This guide shows how you can use the facilities of the {any}`pystencilssfg.lang` module to model your C++ interfaces
for use with the code generator.

To begin, import the `lang` module:

```{code-cell} ipython3
from pystencilssfg import lang
```

## Defining C++ Types and Type Templates

The first C++ entities that need to be mirrored for the SFGs are the types and type templates a library
or application uses or exposes.

### Non-Templated Types

To define a C++ type, we use {any}`pystencilssfg.lang.cpptype <pystencilssfg.lang.types.cpptype>`:

```{code-cell} ipython3
MyClassTypeFactory = lang.cpptype("my_namespace::MyClass", "MyClass.hpp")
MyClassTypeFactory
```

This defines two properties of the type: its fully qualified name, and the set of headers
that need to be included when working with the type.
Now, whenever this type occurs as the type of a variable given to pystencils-sfg,
the code generator will make sure that `MyClass.hpp` is included into the respective
generated code file.

The object returned by `cpptype` is not the type itself, but a factory for instances of the type.
Even as `MyClass` does not have any template parameters, we can create different instances of it:
`const` and non-`const`, as well as references and non-references.
We do this by calling the factory:

```{code-cell} ipython3
MyClass = MyClassTypeFactory()
str(MyClass)
```

To produce a `const`-qualified version of the type:

```{code-cell} ipython3
MyClassConst = MyClassTypeFactory(const=True)
str(MyClassConst)
```

And finally, to produce a reference instead:

```{code-cell} ipython3
MyClassRef = MyClassTypeFactory(ref=True)
str(MyClassRef)
```

Of course, `const` and `ref` can also be combined to create a reference-to-const.

### Types with Template Parameters

We can add template parameters to our type by the use of
[Python format strings](https://docs.python.org/3/library/string.html#formatstrings):

```{code-cell} ipython3
MyClassTemplate = lang.cpptype("my_namespace::MyClass< {T1}, {T2} >", "MyClass.hpp")
MyClassTemplate
```

Here, the type parameters `T1` and `T2` are specified in braces.
For them, values must be provided when calling the factory to instantiate the type:

```{code-cell} ipython3
MyClassIntDouble = MyClassTemplate(T1="int", T2="double")
str(MyClassIntDouble)
```

The way type parameters are passed to the factory is identical to the behavior of {any}`str.format`,
except that it does not support attribute or element accesses.
In particular, this means that we can also use unnamed, implicit positional parameters:

```{code-cell} ipython3
MyClassTemplate = lang.cpptype("my_namespace::MyClass< {}, {} >", "MyClass.hpp")
MyClassIntDouble = MyClassTemplate("int", "double")
str(MyClassIntDouble)
```

## Creating Variables and Expressions

Type templates and types will not get us far on their own.
To use them in APIs, as function or constructor parameters,
or as class members and local objects,
we need to create *variables* with certain types.

To do so, we need to inject our defined types into the expression framework of pystencils-sfg.
We wrap the type in an interface that allows us to create variables and, later, more complex expressions,
using {any}`lang.CppClass <pystencilssfg.lang.expressions.CppClass>`:

```{code-cell} ipython3
class MyClass(lang.CppClass):
    template = lang.cpptype("my_namespace::MyClass< {T1}, {T2} >", "MyClass.hpp")
```

Instances of `MyClass` can now be created via constructor call, in the same way as above.
This gives us an unbound `MyClass` object, which we can bind to a variable name by calling `var` on it:

```{code-cell} ipython3
my_obj = MyClass(T1="int", T2="void").var("my_obj")
my_obj, str(my_obj.dtype)
```

## Reflecting C++ Class APIs

In the previous section, we showed how to reflect a C++ class in pystencils-sfg in order to create
a variable representing an object of that class.
We can now extend this to reflect the public API of the class, in order to create complex expressions
involving objects of `MyClass` during code generation.

### Public Methods

Assume `MyClass` has the following public interface:

```C++
template< typename T1, typename T2 >
class MyClass {
public:
  T1 & getA();
  std::tuple< T1, T2 > getBoth();

  void replace(T1 a_new, T2 b_new);
}
```

We mirror this in our Python reflection of `CppClass` using methods that create `AugExpr` objects,
which represent C++ expressions annotated with variables they depend on.
A possible implementation might look like this:

```{code-cell} ipython3
---
tags: [remove-cell]
---

class MyClass(lang.CppClass):
    template = lang.cpptype("my_namespace::MyClass< {T1}, {T2} >", "MyClass.hpp")

    def ctor(self, a: lang.AugExpr, b: lang.AugExpr) -> MyClass:
        return self.ctor_bind(a, b)

    def getA(self) -> lang.AugExpr:
        return lang.AugExpr.format("{}.getA()", self)

    def getBoth(self) -> lang.AugExpr:
        return lang.AugExpr.format("{}.getBoth()", self)

    def replace(self, a_new: lang.AugExpr, b_new: lang.AugExpr) -> lang.AugExpr:
        return lang.AugExpr.format("{}.replace({}, {})", self, a_new, b_new)
```

```{code-block} python
class MyClass(lang.CppClass):
    template = lang.cpptype("my_namespace::MyClass< {T1}, {T2} >", "MyClass.hpp")

    def getA(self) -> lang.AugExpr:
        return lang.AugExpr.format("{}.getA()", self)

    def getBoth(self) -> lang.AugExpr:
        return lang.AugExpr.format("{}.getBoth()", self)

    def replace(self, a_new: lang.AugExpr, b_new: lang.AugExpr) -> lang.AugExpr:
        return lang.AugExpr.format("{}.replace({}, {})", self, a_new, b_new)
```

Each method of `MyClass` reflects a method of the same name in its public C++ API.
These methods do not return values, but *expressions*;
here, we use the generic `AugExpr` class to model expressions that we don't know anything
about except how they should be constructed.

We create these expressions using `AugExpr.format`, which takes a format string
and interpolation arguments in the same way as `cpptype`.
Internally, it will analyze the format arguments (e.g. `self`, `a_new` and `b_new` in `replace`),
and combine information from any `AugExpr`s found among them.
These are:
 - **Variables**: If any of the input expression depend on variables, the resulting expression will
   depend on the union of all these variable sets
 - **Headers**: If any of the input expression requires certain header files to be evaluated,
   the resulting expression will require the same header files.

We can see this in action by calling one of the methods on a variable of type `MyClass`:

```{code-cell} ipython3
my_obj = MyClass(T1="int", T2="void").var("my_obj")
expr = my_obj.getBoth()
expr, lang.depends(expr), lang.includes(expr)
```

We can see: the newly created expression `my_obj.getBoth()` depends on the variable `my_obj` and
requires the header `MyClass.hpp` to be included; this header it has inherited from `my_obj`.

### Constructors

Using the `AugExpr` system, we can also model constructors of `MyClass`.
Assume `MyClass` has the constructor `MyClass(T1 a, T2 b)`.
We implement this by adding a `ctor` method to our Python interface:

```{code-block} python
class MyClass(lang.CppClass):
    ...
    
    def ctor(self, a: lang.AugExpr, b: lang.AugExpr) -> MyClass:
        return self.ctor_bind(a, b)
```

Here, we don't use `AugExpr.format`; instead, we use `ctor_bind`, which is exposed by `CppClass`.
This will generate the correct constructor invocation from the type of our `MyClass` object
and also ensure the headers required by `MyClass` are correctly attached to the resulting
expression:

```{code-cell} ipython3
a = lang.AugExpr("int").var("a")
b = lang.AugExpr("double").var("b")
expr = MyClass(T1="int", T2="double").ctor(a, b)
expr, lang.depends(expr), lang.includes(expr)
```

(field_data_structure_reflection)=
## Reflecting Field Data Structures

One key feature of pystencils-sfg is its ability to map symbolic fields
onto arbitrary array data structures
using the composer's {any}`map_field <SfgBasicComposer.map_field>` method.
The APIs of a custom field data structure can naturally be injected into pystencils-sfg
using the modelling framework described above.
However, for them to be recognized by `map_field`,
the reflection class also needs to implement the {any}`SupportsFieldExtraction` protocol.
This requires that the following three methods are implemented:

```{code-block} python
def _extract_ptr(self) -> AugExpr: ...

def _extract_size(self, coordinate: int) -> AugExpr | None: ...

def _extract_stride(self, coordinate: int) -> AugExpr | None: ...
```

The first, `_extract_ptr`, must return an expression that evaluates
to the base pointer of the field's memory buffer.
This pointer has to point at the field entry which pystencils accesses
at all-zero index and offsets (see [](#note-on-ghost-layers)).
The other two, when called with a coordinate $c \ge 0$, shall return
the size and linearization stride of the field in that direction.
If the coordinate is equal or larger than the field's dimensionality,
return `None` instead.

### Sample Field API Reflection

Consider the following class template for a field, which takes its element type
and dimensionality as template parameters
and exposes its data pointer, shape, and strides through public methods:

```{code-block} C++
template< std::floating_point ElemType, size_t DIM >
class MyField {
public:
  size_t get_shape(size_t coord);
  size_t get_stride(size_t coord);
  ElemType * data_ptr();
}
```

It could be reflected by the following class.
Note that in this case we define a custom `__init__` method in order to
intercept the template arguments `elem_type` and `dim`
and store them as instance members.
Our `__init__` then forwards all its arguments up to `CppClass.__init__`.
We then define reflection methods for `shape`, `stride` and `data` -
the implementation of the field extraction protocol then simply calls these methods.

```{code-cell} ipython3
from pystencilssfg.lang import SupportsFieldExtraction
from pystencils.types import UserTypeSpec

class MyField(lang.CppClass, SupportsFieldExtraction):
    template = lang.cpptype(
        "MyField< {ElemType}, {DIM} >",
        "MyField.hpp"
    )

    def __init__(
        self,
        elem_type: UserTypeSpec,
        dim: int,
        **kwargs,
    ) -> None:
        self._elem_type = elem_type
        self._dim = dim
        super().__init__(ElemType=elem_type, DIM=dim, **kwargs)

    #   Reflection of Public Methods
    def get_shape(self, coord: int | lang.AugExpr) -> lang.AugExpr:
        return lang.AugExpr.format("{}.get_shape({})", self, coord)
        
    def get_stride(self, coord: int | lang.AugExpr) -> lang.AugExpr:
        return lang.AugExpr.format("{}.get_stride({})", self, coord)

    def data_ptr(self) -> lang.AugExpr:
        return lang.AugExpr.format("{}.data_ptr()", self)

    #   Field Extraction Protocol that uses the above interface
    def _extract_ptr(self) -> lang.AugExpr:
        return self.data_ptr()

    def _extract_size(self, coordinate: int) -> lang.AugExpr | None:
        if coordinate > self._dim:
            return None
        else:
            return self.get_shape(coordinate)

    def _extract_stride(self, coordinate: int) -> lang.AugExpr | None:
        if coordinate > self._dim:
            return None
        else:
            return self.get_stride(coordinate)
```

Our custom field reflection is now ready to be used.
The following generator script demonstrates what code is generated when an instance of `MyField`
is passed to `sfg.map_field`:


```{code-cell} ipython3
import pystencils as ps
from pystencilssfg.lang.cpp import std

with SourceFileGenerator() as sfg:
    #   Create symbolic fields
    f = ps.fields("f: double[3D]")
    f_myfield = MyField(f.dtype, f.ndim, ref=True).var(f.name)

    #   Create the kernel
    asm = ps.Assignment(f(0), 2 * f(0))
    khandle = sfg.kernels.create(asm)

    #   Create the wrapper function
    sfg.function("invoke")(
        sfg.map_field(f, f_myfield),
        sfg.call(khandle)
    )
```

### Add a Factory Function

In the above example, an instance of `MyField` representing the field `f` is created by the
slightly verbose expression `MyField(f.dtype, f.ndim, ref=True).var(f.name)`.
Having to write this sequence every time, for every field, introduces unnecessary
cognitive load and lots of potential sources of error.
Whenever it is possible to create a field reflection using just information contained in a
pystencils {any}`Field <pystencils.field.Field>` object,
the API reflection should therefore implement a factory method `from_field`:

```{code-cell} ipython3
class MyField(lang.CppClass, SupportsFieldExtraction):
    ...

    @classmethod
    def from_field(cls, field: ps.Field, const: bool = False, ref: bool = False) -> MyField:
        return cls(f.dtype, f.ndim, const=const, ref=ref).var(f.name)

```

The above signature is idiomatic for `from_field`, and you should stick to it as far as possible.
We can now use it inside the generator script:

```{code-block} python
f = ps.fields("f: double[3D]")
f_myfield = MyField.from_field(f)
```

(note-on-ghost-layers)=
### A Note on Ghost Layers

Some care has to be taken when reflecting data structures that model the notion
of ghost layers.
Consider an array with the index space $[0, N_x) \times [0, N_y)$,
its base pointer identifying the entry $(0, 0)$.
When a pystencils kernel is generated with a shell of $k$ ghost layers
(see {any}`CreateKernelConfig.ghost_layers <pystencils.codegen.config.CreateKernelConfig.ghost_layers>`),
it will process only the subspace $[k, N_x - k) \times [k, N_x - k)$.

If your data structure is implemented such that ghost layer nodes have coordinates
$< 0$ and $\ge N_{x, y}$,
you must hence take care that
 - either, `_extract_ptr` returns a pointer identifying the array entry at `(-k, -k)`;
 - or, ensure that kernels operating on your data structure are always generated
   with `ghost_layers = 0`.

In either case, you must make sure that the number of ghost layers in your data structure
matches the expected number of ghost layers of the kernel.
+141 −0
Original line number Original line Diff line number Diff line

(how_to_generator_scripts_config)=
# Generator Script Configuration and Command-Line Interface

There are several ways to affect the behavior and output of a generator script.
For one, the `SourceFileGenerator` itself may be configured from the combination of three
different configuration sources:

- **Inline Configuration:** The generator script may set up an {any}`SfgConfig` object,
  which is passed to the `SourceFileGenerator` at its creation; see [Inline Configuration](#inline_config)
- **Command-Line Options:** The `SourceFileGenerator` parses the command line arguments of
  the generator script to set some of its configuration options; see [Command-Line Options](#cmdline_options)
- **Project Configuration:** When embedded into a larger project, using a build system such as CMake, generator scripts
  may be configured globally within that project by the use of a *configuration module*.
  Settings specified inside that configuration module are always overridden by the two other configuration sources listed above.
  For details on configuration modules, refer to the guide on [Project and Build System Integration](#guide_project_integration).

(inline_config)=
## Inline Configuration

To configure the source file generator within your generator script, import the {any}`SfgConfig` from `pystencilssfg`.
You may then set up the configuration object before passing it to the `SourceFileGenerator` constructor.
To illustrate, the following snippet alters the code indentation width and changes the output directory
of the generator script to `gen_src`:

```{literalinclude} examples/guide_generator_scripts/inline_config/kernels.py
```

For a selection of common configuration options, see [below](#config_options).
The inline configuration will override any values set by the [project configuration](#config_module)
and must not conflict with any [command line arguments](#custom_cli_args).

(config_options)=
## Configuration Options

Here is a selection of common configuration options to be set in the [inline configuration](#inline_config) or
[project configuration](#config_module).

### Output Options

The file extensions of the generated files can be modified through
{any}`cfg.extensions.header <FileExtensions.header>`
and {any}`cfg.extensions.impl <FileExtensions.impl>`;
and the output directory of the code generator can be set through {any}`cfg.output_directory <SfgConfig.output_directory>`.
The [header-only mode](#header_only_mode) can be enabled using {any}`cfg.header_only <SfgConfig.header_only>`.

:::{danger}

When running generator scripts through [CMake](#cmake_integration), the file extensions,
output directory, and header-only mode settings will be managed fully by the pystencils-sfg
CMake module and the (optional) project configuration module.
They should therefore not be set in the inline configuration,
as this will likely lead to errors being raised during code generation.
:::

### Outer Namespace

To specify the outer namespace to which all generated code should be emitted,
set {any}`cfg.outer_namespace <SfgConfig.outer_namespace>`.

### Code Style and Formatting

Pystencils-sfg gives you some options to affect its output code style.
These are controlled by the options in the {any}`cfg.code_style <CodeStyle>` category.

Furthermore, pystencils-sfg uses `clang-format` to beautify generated code.
The behaviour of the clang-format integration is managed by the
the {any}`cfg.clang_format <ClangFormatOptions>` category,
where you can set options to skip or enforce formatting,
or change the formatter binary.
To set the code style used by `clang-format` either create a `.clang-format` file
in any of the parent folders of your generator script,
or modify the {any}`cfg.clang_format.code_style <ClangFormatOptions.code_style>` option.

:::{seealso}
[Clang-Format Style Options](https://clang.llvm.org/docs/ClangFormatStyleOptions.html)
:::

Clang-format will, by default, sort `#include` statements alphabetically and separate
local and system header includes.
To override this, you can set a custom sorting key for `#include` sorting via
{any}`cfg.code_style.includes_sorting_key <CodeStyle.includes_sorting_key>`.

(cmdline_options)=
## Command-Line Options

The `SourceFileGenerator` consumes a number of command-line parameters that may be passed to the script
on invocation. These include:

- `--sfg-output-dir <path>`: Set the output directory of the generator script. This corresponds to {any}`SfgConfig.output_directory`.
- `--sfg-file-extensions <exts>`: Set the file extensions used for the generated files;
  `exts` must be a comma-separated list not containing any spaces. Corresponds to {any}`SfgConfig.extensions`.
- `[--no]--sfg-header-only`: Enable or disable header-only code generation. Corresponds to {any}`SfgConfig.header_only`.

If any configuration option is set to conflicting values on the command line and in the inline configuration,
the generator script will terminate with an error.

You may examine the full set of possible command line parameters by invoking a generator script
with the `--help` flag:

```bash
$ python kernels.py --help
```

(header_only_mode)=
## Header-Only Mode

When the header-only output mode is enabled,
the code generator will emit only a header file and no separate implementation file.
In this case, the composer will automatically place all function, method,
and kernel definitions in the header file.

Header-only code generation can be enabled by setting the `--header-only` command-line flag
or the {any}`SfgConfig.header_only` configuration option.

(custom_cli_args)=
## Adding Custom Command-Line Options

Sometimes, you might want to add your own command-line options to a generator script
in order to affect its behavior from the shell,
for instance by using {any}`argparse` to set up an argument parser.
If you parse your options directly from {any}`sys.argv`,
as {any}`parse_args <argparse.ArgumentParser.parse_args>` does by default,
your parser will also receive any options meant for the `SourceFileGenerator`.
To filter these out of the argument list,
pass the additional option `keep_unknown_argv=True` to your `SourceFileGenerator`.
This will instruct it to store any unknown command line arguments into `sfg.context.argv`,
where you can then retrieve them from and pass on to your custom parser:

```{literalinclude} examples/guide_generator_scripts/custom_cmdline_args/kernels.py
```

Any SFG-specific arguments will already have been filtered out of this argument list.
As a consequence of the above, if the generator script is invoked with a typo in some SFG-specific argument,
which the `SourceFileGenerator` therefore does not recognize,
that argument will be passed on to your downstream parser instead.

:::{important}
If you do *not* pass on `sfg.context.argv` to a downstream parser, make sure that `keep_unknown_argv` is set to
`False` (which is the default), such that typos or illegal arguments will not be ignored.
:::
+22 −0
Original line number Original line Diff line number Diff line
import os
import glob
import subprocess

GROUPS = ["guide_generator_scripts"]
THIS_DIR = os.path.split(__file__)[0]


def main():
    for group in GROUPS:
        group_folder = os.path.join(THIS_DIR, group)

        for group_entry in os.listdir(group_folder):
            scripts_dir = os.path.join(group_folder, group_entry)
            if os.path.isdir(scripts_dir):
                query = os.path.join(scripts_dir, "*.py")
                for script in glob.glob(query):
                    subprocess.run(["python", script], cwd=scripts_dir).check_returncode()


if __name__ == "__main__":
    main()
Original line number Original line Diff line number Diff line
from pystencilssfg import SourceFileGenerator

import pystencils as ps
import sympy as sp

with SourceFileGenerator() as sfg:
    #   Define a copy kernel
    src, dst = ps.fields("src, dst: [1D]")
    c = sp.Symbol("c")

    @ps.kernel
    def scale():
        dst.center @= c * src.center()

    #   Add it to the file
    scale_kernel = sfg.kernels.create(scale, "scale")
Original line number Original line Diff line number Diff line
from pystencilssfg import SourceFileGenerator

import pystencils as ps
import sympy as sp

with SourceFileGenerator() as sfg:
    #   Define a copy kernel
    src, dst = ps.fields("src, dst: [1D]")
    c = sp.Symbol("c")

    @ps.kernel
    def scale():
        dst.center @= c * src.center()

    #   Add it to the file
    scale_kernel = sfg.kernels.create(scale, "scale")

    #   start
    #   ... see above ...
    sfg.function("scale_kernel")(
        sfg.call(scale_kernel)
    )
    #   end
Original line number Original line Diff line number Diff line
from pystencilssfg import SourceFileGenerator

import pystencils as ps
import sympy as sp

with SourceFileGenerator() as sfg:
    #   Define a copy kernel
    src, dst = ps.fields("src, dst: double[1D]")
    c = sp.Symbol("c")

    @ps.kernel
    def scale():
        dst.center @= c * src.center()

    #   Add it to the file
    scale_kernel = sfg.kernels.create(scale, "scale")

    #   start
    import pystencilssfg.lang.cpp.std as std

    sfg.include("<span>")

    sfg.function("scale_kernel")(
        sfg.map_field(src, std.vector.from_field(src)),
        sfg.map_field(dst, std.span.from_field(dst)),
        sfg.call(scale_kernel)
    )
    #   end
+187 −0
Original line number Original line Diff line number Diff line
(guide_project_integration)=
# Project and Build System Integration

(compile_generated_files)=
## Compiling Generated Files

When compiling generated files that include pystencils kernels,
the pystencils runtime headers must be added to the compiler's include search path.
To get the right path, use the `pystencils.include` utility, e.g.:

```{code-block} bash
g++ -I $(python -m pystencils.include -s) [args...] [input-files...]
```

(config_module)=
## Project-Wide Settings using Configuration Modules

When embedding *pystencils-sfg* into a C++ project or build system,
you might want to set a project-wide base configuration for all generator scripts.
In addition, it might be necessary to pass various details about the project
and build setup to the generator scripts.
Both can be achieved by the use of a *configuration module*.

A configuration module is a Python file that defines up to two functions:
- `def configure_sfg(cfg: SfgConfig)` is called to set up the project-wide base configuration.
   It takes an {any}`SfgConfig` object which it may modify to establish the project-wide option set.
- `def project_info() -> Any` is called by *pystencils-sfg* to retrieve an object that encapsulates
  any custom project-specific information.
  This information is passed on to the generator scripts through
  the {any}`sfg.context.project_info <SfgContext.project_info>` attribute.

An example configuration module might look like this:

```Python
from pystencilssfg import SfgConfig

def configure_sfg(cfg: SfgConfig):
    cfg.extensions.header = "h++"
    cfg.extensions.impl = "c++"
    cfg.clang_format.code_style = "llvm"
    ...

def project_info():
    return {
        "project_name": "my-project",
        "float_precision": "float32",
        "use_cuda": False,
        ...
    }
```

Here, `project_info` returns a dictionary, but this is just for illustration;
the function may return any type of arbitrarily complex objects.
For improved API safety, {any}`dataclasses` might be a good tool for setting up
project info objects.

When invoking a generator script, the path to the current configuration module must be passed to it
using the `--sfg-config-module` command-line parameter.
This can be automated by an adequately set up build system, such as GNU Make or CMake.

If you are using pystencils-sfg with CMake through the provided CMake module,
[see below](#cmake_set_config_module) on how to specify a configuration module for your project.

(cmake_integration)=
## CMake Integration

*pystencils-sfg* is shipped with a CMake module for on-the-fly code generation during the CMake build process.

### Add the module

To include the module in your CMake source tree, you must first add the pystencils-sfg *Find-module*
to your CMake module path.
To create the Find-module, navigate to the directory it should be placed in and run the following command:

```shell
sfg-cli cmake make-find-module
```

This will create the `FindPystencilsSfg.cmake` file.
Make sure that its containing directory is added to the CMake module path.

To load pystencils-sfg into CMake, we first need to set the Python interpreter
of the environment SFG is installed in.
There are several ways of doing this:

#### Set Python via a Find-Module Hint

Set the `PystencilsSfg_PYTHON_PATH` hint variable inside your `CMakeLists.txt` to point at the
Python executable which should be used to invoke pystencils-sfg, e.g.:

```CMake
set(PystencilsSfg_PYTHON_PATH ${CMAKE_SOURCE_DIR}/.venv/bin/python)
```

This is the recommended way, especially when other parts of your project also use Python.

#### Set Python via a Cache Variable

On the command line or in a [CMake configure preset](https://cmake.org/cmake/help/latest/manual/cmake-presets.7.html),
set the `PystencilsSfg_PYTHON_INTERPRETER` cache variable to point at the Python executable to be used to invoke pystencils-sfg;
e.g.:

```bash
cmake -S . -B build -DPystencilsSfg_PYTHON_INTERPRETER=`pwd`/.venv/bin/python
```

If both the cache variable and the `PystencilsSfg_PYTHON_PATH` hint are set, the cache variable takes precedence,
so you can use the cache variable to override the hint.

#### Automatically Find a Python Installation

If none of the above is provided, pystencils-sfg will invoke [FindPython](https://cmake.org/cmake/help/latest/module/FindPython.html)
to determine the Python interpreter it should use.
You can affect this process through any of the hints listed in the `FindPython` documentation.

#### Find pystencils-sfg

Finally, call `find_package( PystencilsSfg )` from your `CMakeLists.txt` to load the SFG module.
If SFG as a dependency is not optional, add the `REQUIRED` flag such that the call will fail if
the package cannot be found.

(cmake_add_generator_scripts)=
### Adding Generator Scripts

The primary interaction point in CMake is the function `pystencilssfg_generate_target_sources`,
with the following signature:

```CMake
pystencilssfg_generate_target_sources( <target> 
    SCRIPTS script1.py [script2.py ...]
    [SCRIPT_ARGS arg1 [arg2 ...]]
    [DEPENDS dependency1.py [dependency2.py...]]
    [FILE_EXTENSIONS <header-extension> <impl-extension>]
    [CONFIG_MODULE <path-to-config-module.py>]
    [OUTPUT_DIRECTORY <output-directory>]
    [HEADER_ONLY]
)
```

It registers the generator scripts `script1.py [script2.py ...]` to be executed at compile time using `add_custom_command`
and adds their output files to the specified `<target>`.
Any changes in the generator scripts, or any listed dependency, will trigger regeneration.
The function takes the following options:

 - `SCRIPTS`: A list of generator scripts
 - `SCRIPT_ARGS`: A list of custom command line arguments passed to the generator scripts; see [](#custom_cli_args)
 - `DEPENDS`: A list of dependencies for the generator scripts
 - `FILE_EXTENSION`: The desired extensions for the generated files
 - `CONFIG_MODULE`: Set the configuration module for all scripts registered with this call.
   If set, this overrides the value of `PystencilsSfg_CONFIG_MODULE`
   in the current scope (see [](#cmake_set_config_module))
 - `OUTPUT_DIRECTORY`: Custom output directory for generated files. If `OUTPUT_DIRECTORY` is a relative path,
   it will be interpreted relative to the current build directory.
 - `HEADER_ONLY`: If this option is set, instruct the generator scripts to only generate header files
   (see {any}`SfgConfig.header_only`).

If `OUTPUT_DIRECTORY` is *not* specified, any C++ header files generated by the above call
can be included in any files belonging to `target` via:

```C++
#include "gen/<file1.hpp>"
#include "gen/<file2.hpp>"
/* ... */
```

:::{attention}
If you change the code generator output directory using the `OUTPUT_DIRECTORY` argument,
you are yourself responsible for placing that directory--or any of its parents--on the
include path of your target.
:::


(cmake_set_config_module)=
### Set a Configuration Module

There are two ways of specifying a [configuration module](#config_module) for generator scripts
registered with CMake:
- To set a configuration module for scripts registered with a single call to `pystencilssfg_generate_target_sources`,
  use the `CONFIG_MODULE` function parameter (see [](#cmake_add_generator_scripts)).
- To set a config module for all generator scripts within the current CMake directory and its subdirectories,
  set the scoped variable `PystencilsSfg_CONFIG_MODULE` to point at the respective Python file, e.g.
  `set( PystencilsSfg_CONFIG_MODULE ProjectConfig.py )`.

You might want to populate your configuration module with information about the current
build setup and environment.
For this purpose, take a look at the
[configure_file](https://cmake.org/cmake/help/latest/command/configure_file.html) CMake function.

docs/usage/building.md

deleted100644 → 0
+0 −30
Original line number Original line Diff line number Diff line

## Namespaces

Conceptually, there exist two different kinds of namespaces: *kernel namespaces* for the generated kernels,
and a single *code namespace* for all the generated code.
Both get mapped to standard C++ namespaces, in the end, but they fulfill different purposes in the code generator.

*Kernel namespaces* are used for grouping generated kernels together, e.g. to avoid name collisions.
If, for example, a code generation script combines kernels and functions produced by different components, each
component may create its own kernel namespace to isolate its kernels.

The *code namespace*, in contrast, envelops all the generated code. Its fully qualified name is built from two parts:

 - The *outer namespace* is defined in the [generator configuration][pystencilssfg.SfgConfiguration], typically by
   the global project configuration;
 - The *inner namespace* is defined by the code generation script, e.g. via [`SfgComposer.namespace`][pystencilssfg.SfgComposer.namespace].

These namespaces will finally occur in the generated implementation file as:

```C++
namespace outer_namespace::inner_namespace {

namespace kernels {
    /* kernel definitions */
} // namespace kernels

/* function definitions */

} // namespace outer_namespace::inner_namespace
```
+0 −76
Original line number Original line Diff line number Diff line

## Command Line Interface

*pystencils-sfg* exposes not one, but two command line interfaces:
The *global CLI* offers a few tools meant to be used by build systems,
while the *generator script* command line interface is meant for a build system to communicate
with the code generator during on-the-fly generation.

### Global CLI

The global CLI may be accessed either through the `sfg-cli` shell command, or using `python -m pystencilssfg`.

### Generator Script CLI

The [SourceFileGenerator][pystencilssfg.SourceFileGenerator] evaluates a generator script's command line arguments,
which can be supplied by the user, but more frequently by the build system.

## CMake Integration

*pystencils-sfg* is shipped with a CMake module for on-the-fly code generation during the CMake build process.

### Add the module

To include the module in your CMake source tree, a separate find module is provided.
You can use the global CLI to obtain the find module; simply run

```shell
sfg-cli cmake make-find-module
```

to create the file `FindPystencilsSfg.cmake` in the current directory.
Add it to the CMake module path, and load the *pystencils-sfg* module via *find_package*:

```CMake
find_package( PystencilsSfg )
```

Make sure to set the `Python_ROOT_DIR` cache variable to point to the correct Python interpreter
(i.e. the virtual environment you have installed *pystencils-sfg* into).

### Add generator scripts

The primary interaction point in CMake is the function `pystencilssfg_generate_target_sources`,
with the following signature:

```CMake
pystencilssfg_generate_target_sources( <target> 
    SCRIPTS script1.py [script2.py ...]
    [DEPENDS dependency1.py [dependency2.py...]]
    [FILE_EXTENSIONS <header-extension> <impl-extension>]
    [HEADER_ONLY])
```

It registers the generator scripts `script1.py [script2.py ...]` to be executed at compile time using `add_custom_command`
and adds their output files to the specified `<target>`.
Any changes in the generator scripts, or any listed dependency, will trigger regeneration.
The function takes the following options:

 - `SCRIPTS`: A list of generator scripts
 - `DEPENDS`: A list of dependencies for the generator scripts
 - `FILE_EXTENSION`: The desired extensions for the generated files
 - `HEADER_ONLY`: Toggles header-only code generation

### Include generated files

The `pystencils-sfg` CMake module creates a subfolder `sfg_sources/gen` at the root of the build tree
and writes all generated source files into it. The directory `sfg_sources` is added to the project's include
path, such that generated header files for a target `<target>` may be included via:
```C++
#include "gen/<target>/kernels.h"
```

### Project Configuration

The *pystencils-sfg* CMake module reads the scoped variable `PystencilsSfg_CONFIGURATOR_SCRIPT` to find
the *configuration module* that should be passed to the generator scripts.

docs/usage/generator_scripts.md

deleted100644 → 0
+0 −186
Original line number Original line Diff line number Diff line

Generator scripts are the primary way *pystencils-sfg* is meant to be used.
A generator script is a single Python script, say `kernels.py`, which contains *pystencils-sfg*
code at the top level such that, when executed, it emits source code to a pair of files `kernels.h`
and `kernels.cpp`. This guide describes how to write such a generator script, its structure, and how
it can be used to generate code.

This page gives a general overview over the code generation process, but introduces only the
convenient high-level interface provided by the [SourceFileGenerator][pystencilssfg.SourceFileGenerator]
and [SfgComposer][pystencilssfg.SfgComposer] classes.
For a more in-depth look into building source files, and about using *pystencils-sfg* outside
of a generator script, please take a look at the [In-Depth Guide](building.md).

## Anatomy

The code generation process in a generator script is controlled by the
[SourceFileGenerator][pystencilssfg.SourceFileGenerator] context manager.
It configures the code generator by combining configuration options from the 
environment (e.g. a CMake build system) with options specified in the script,
and infers the names of the output files from the script's name.
It then prepares and returns a code generation [context][pystencilssfg.SfgContext].
This context may then be passed to a [composer][pystencilssfg.SfgComposer],
which provides a convenient interface for constructing the source files.

To start, place the following code in a Python script, e.g. `kernels.py`:

```Python
from pystencilssfg import SourceFileGenerator, SfgConfiguration, SfgComposer

sfg_config = SfgConfiguration()
with SourceFileGenerator(sfg_config) as ctx:
    sfg = SfgComposer(ctx)

```

The source file is constructed within the context manager's managed region.
During execution of the script, when the region ends, a header/source file pair
`kernels.h` and `kernels.cpp` will be written to the file system next to your script.
Execute the script as-is and inspect the generated files, which will of course
still be empty.

A few notes on configuration:

 - The [SourceFileGenerator][pystencilssfg.SourceFileGenerator] parses the script's command line arguments
   for configuration options (refer to [CLI and Build System Integration](cli_and_build_system.md)).
   If you intend to use command-line parameters in your
   generation script, use [`sfg.context.argv`][pystencilssfg.SfgContext.argv] instead of `sys.argv`.
   There, all arguments meant for the code generator are already removed.
 - The code generator's configuration is consolidated from a global project configuration which may
   be provided by the build system; a number of command line arguments; and the
   [SfgConfiguration][pystencilssfg.SfgConfiguration] provided in the script.
   The project configuration may safely be overridden by the latter two; however, conflicts
   between command-line arguments and the configuration defined in the script will cause
   an exception to be thrown.

## Using the Composer

The object `sfg` constructed in above snippet is an instance of [SfgComposer][pystencilssfg.SfgComposer].
The composer is the central part of the user front-end of *pystencils-sfg*.
It provides an interface for constructing source files that attempts to closely mimic
C++ syntactic structures within Python.
Here is an overview of its various functions:

### Includes and Definitions

With [`SfgComposer.include`][pystencilssfg.SfgComposer.include], the code generator can be instructed
to include header files. 

```Python
with SourceFileGenerator(sfg_config) as ctx:
    sfg = SfgComposer(ctx)
    # ...
    sfg.include("<vector>")
    sfg.incldue("custom_header.h")
```

### Adding Kernels

`pystencils`-generated kernels are managed in
[kernel namespaces][pystencilssfg.source_components.SfgKernelNamespace].
The default kernel namespace is called `kernels` and is available via
[`SfgComposer.kernels`][pystencilssfg.SfgComposer.kernels].
Adding an existing `pystencils` AST, or creating one from a list of assignments, is possible
through [`add`][pystencilssfg.source_components.SfgKernelNamespace.add]
and [`create`][pystencilssfg.source_components.SfgKernelNamespace.create].
The latter is a wrapper around
[`pystencils.create_kernel`](
https://pycodegen.pages.i10git.cs.fau.de/pystencils/sphinx/kernel_compile_and_call.html#pystencils.create_kernel
).
Both functions return a [kernel handle][pystencilssfg.source_components.SfgKernelHandle]
through which the kernel can be accessed, e.g. for calling it in a function.

If required, use [`SfgComposer.kernel_namespace`][pystencilssfg.SfgComposer.kernel_namespace]
to access other kernel namespaces than the default one.

```Python
with SourceFileGenerator(sfg_config) as ctx:
    sfg = SfgComposer(ctx)
    # ...

    ast = ps.create_kernel(assignments, config)
    khandle = sfg.kernels.add(ast, "kernel_a")
    
    # is equivalent to
    
    khandle = sfg.kernels.create(assignments, "kernel_a", config)

    # You may use a different namespace
    nspace = sfg.kernel_namespace("group_of_kernels")
    nspace.create(assignments, "kernel_a", config)
```

### Building Functions

[Functions][pystencilssfg.source_components.SfgFunction] form the link between your `pystencils` kernels
and your C++ framework. A function in *pystencils-sfg* translates to a simple C++ function, and should
fulfill just the following tasks:

 - Extract kernel parameters (pointers, sizes, strides, numerical coefficients)
   from C++ objects (like fields, vectors, other data containers)
 - Call one or more kernels in sequence or in conditional branches

It is the philosophy of this project that anything more complicated than this should happen in handwritten
code; these generated functions are merely meant to close the remaining gap.

The composer provides an interface for constructing functions that tries to mimic the look of the generated C++
code.
Use [`SfgComposer.function`][pystencilssfg.SfgComposer.function] to create a function,
and [`SfgComposer.call`][pystencilssfg.SfgComposer.call] to call a kernel by its handle:

```Python
with SourceFileGenerator(sfg_config) as ctx:
    sfg = SfgComposer(ctx)
    # ...

    sfg.function("MyFunction")(
        sfg.call(khandle)
    )
```

Note the special syntax: To mimic the look of a C++ function, the composer uses a sequence of two calls
to construct the function.

The function body may further be populated with the following things:

#### Parameter Mappings

Extract kernel parameters from C++ objects:

 - [`map_param`][pystencilssfg.SfgComposer.map_param]: Add a single line of code to define one parameter
   depending on one other.
 - [`map_field`][pystencilssfg.SfgComposer.map_field] maps a pystencils
   [`Field`](https://pycodegen.pages.i10git.cs.fau.de/pystencils/sphinx/field.html)
   to a field data structure providing the necessary pointers, sizes and stride information.
   The field data structure must be provided as an instance of a subclass of
   [`SrcField`][pystencilssfg.source_concepts.SrcField].
   Currently, *pystencils-sfg* provides mappings to 
   [`std::vector`](https://en.cppreference.com/w/cpp/container/vector)
   (via [`std_vector_ref`][pystencilssfg.source_concepts.cpp.std_vector_ref])
   and
   [`std::mdspan`](https://en.cppreference.com/w/cpp/container/mdspan)
   (via [`mdspan_ref`][pystencilssfg.source_concepts.cpp.mdspan_ref])
   from the C++ standard library.
 - [`map_vector`][pystencilssfg.SfgComposer.map_vector] maps a sequence of scalar numerical values
   (given as `pystencils.TypedSymbol`s) to a vector data type. Currently, only `std::vector` is provided.

#### Conditional Branches

A conditonal branch may be added with [`SfgComposer.branch`][pystencilssfg.SfgComposer.branch]
using a special syntax:

```Python
with SourceFileGenerator(sfg_config) as ctx:
    sfg = SfgComposer(ctx)
    # ...
    
    sfg.function("myFunction")(
        # ...
        sfg.branch("condition")(
            # then-body
        )(
            # else-body (may be omitted)
        )
    )
    
```
 No newline at end of file

docs/usage/index.md

deleted100644 → 0
+0 −14
Original line number Original line Diff line number Diff line
# User Guides

These pages provide an overview of how to use the pystencils Source File Generator.
A basic understanding of [pystencils](https://pycodegen.pages.i10git.cs.fau.de/pystencils/index.html)
is required.

## Guides

 - [Writing Generator Scripts](generator_scripts.md) explains about the primary interface of *pystencils-sfg*:
   Generator scripts, which are Python scripts that, when executed, emit *pystencils*-generated code to a header/source
   file pair with the same name as the script.
 - [In-Depth: Building Source Files](building.md)
 - [CLI and Build System Integration](cli_and_build_system.md)
 - [Tips And Tricks](tips_n_tricks.md): A collection of various tricks that might come in handy when working with *pystencils-sfg*.
 No newline at end of file
Original line number Original line Diff line number Diff line
#!/bin/bash
#!/bin/bash


echo "[Pre-Commit] Checking code style"
echo "[Pre-Commit] Checking code style"
pdm run flake8 src/pystencilssfg
flake8 src/pystencilssfg
status=$?
status=$?


if [ ${status} != 0 ]; then
if [ ${status} != 0 ]; then
@@ -11,7 +11,7 @@ else
fi
fi


echo "[Pre-Commit] Checking types"
echo "[Pre-Commit] Checking types"
pdm run mypy src/pystencilssfg
mypy src/pystencilssfg
status=$?
status=$?
if [ ${status} != 0 ]; then
if [ ${status} != 0 ]; then
    exit 1
    exit 1
Original line number Original line Diff line number Diff line
@@ -11,7 +11,7 @@ execute_process( COMMAND sfg-cli cmake make-find-module


find_package( PystencilsSfg REQUIRED )
find_package( PystencilsSfg REQUIRED )


set( PystencilsSfg_CONFIGURATOR_SCRIPT codegen_config.py )
set( PystencilsSfg_CONFIG_MODULE codegen_config.py )


add_library( genlib )
add_library( genlib )
pystencilssfg_generate_target_sources( genlib SCRIPTS kernels.py FILE_EXTENSIONS .h .cpp )
pystencilssfg_generate_target_sources( genlib SCRIPTS kernels.py FILE_EXTENSIONS .h .cpp )
Original line number Original line Diff line number Diff line
from sys import stderr
from pystencilssfg import SfgConfig
from pystencilssfg import SfgConfiguration


def sfg_config():
    print("sfg_config() called!", file=stderr)


    project_info = {
def configure(cfg: SfgConfig):
        'B': 'A'
    cfg.extensions.header = "h++"
    }
    cfg.extensions.impl = "c++"

    return SfgConfiguration(
        header_extension='hpp',
        impl_extension='cpp',
        outer_namespace='cmake_demo',
        project_info=project_info
    )
Original line number Original line Diff line number Diff line
@@ -5,7 +5,7 @@ import sympy as sp
from pystencils import fields, kernel
from pystencils import fields, kernel


from pystencilssfg import SourceFileGenerator, SfgConfiguration, SfgComposer
from pystencilssfg import SourceFileGenerator, SfgConfiguration, SfgComposer
from pystencilssfg.source_concepts.cpp import mdspan_ref
from pystencilssfg.lang.cpp import mdspan_ref


sfg_config = SfgConfiguration(
sfg_config = SfgConfiguration(
    outer_namespace="make_demo"
    outer_namespace="make_demo"
@@ -22,7 +22,7 @@ Author: Frederik Hennig <frederik.hennig@fau.de>""")
    
    
    sfg.namespace("jacobi")
    sfg.namespace("jacobi")


    u_src, u_dst, f = fields("u_src, u_dst, f(1) : double[2D]", layout="fzyx")
    u_src, u_dst, f = fields("u_src, u_dst, f : double[2D]", layout="fzyx")
    h = sp.Symbol("h")
    h = sp.Symbol("h")


    @kernel
    @kernel
Original line number Original line Diff line number Diff line
@@ -9,13 +9,11 @@ with SourceFileGenerator() as ctx:
    lb_config = LBMConfig(streaming_pattern='esotwist')
    lb_config = LBMConfig(streaming_pattern='esotwist')


    lb_ast_even = create_lb_ast(lbm_config=lb_config, timestep=Timestep.EVEN)
    lb_ast_even = create_lb_ast(lbm_config=lb_config, timestep=Timestep.EVEN)
    lb_ast_even.function_name = "streamCollide_even"


    lb_ast_odd = create_lb_ast(lbm_config=lb_config, timestep=Timestep.ODD)
    lb_ast_odd = create_lb_ast(lbm_config=lb_config, timestep=Timestep.ODD)
    lb_ast_odd.function_name = "streamCollide_odd"


    kernel_even = sfg.kernels.add(lb_ast_even)
    kernel_even = sfg.kernels.add(lb_ast_even, "lb_even")
    kernel_odd = sfg.kernels.add(lb_ast_odd)
    kernel_odd = sfg.kernels.add(lb_ast_odd, "lb_odd")


    sfg.function("myFunction")(
    sfg.function("myFunction")(
        sfg.branch("(timestep & 1) ^ 1")(
        sfg.branch("(timestep & 1) ^ 1")(
Original line number Original line Diff line number Diff line
@@ -2,7 +2,6 @@
from pystencilssfg import SourceFileGenerator, SfgConfiguration, SfgComposer
from pystencilssfg import SourceFileGenerator, SfgConfiguration, SfgComposer
from pystencilssfg.configuration import SfgCodeStyle
from pystencilssfg.configuration import SfgCodeStyle
from pystencilssfg.composer import SfgClassComposer
from pystencilssfg.composer import SfgClassComposer
from pystencilssfg.source_concepts import SrcObject


from pystencils import fields, kernel
from pystencils import fields, kernel


@@ -19,7 +18,6 @@ f, g = fields("f, g(1): double[2D]")


with SourceFileGenerator(sfg_config) as ctx:
with SourceFileGenerator(sfg_config) as ctx:
    sfg = SfgComposer(ctx)
    sfg = SfgComposer(ctx)
    c = SfgClassComposer(ctx)


    @kernel
    @kernel
    def assignments():
    def assignments():
@@ -27,29 +25,29 @@ with SourceFileGenerator(sfg_config) as ctx:


    khandle = sfg.kernels.create(assignments)
    khandle = sfg.kernels.create(assignments)


    c.struct("DataStruct")(
    sfg.struct("DataStruct")(
        SrcObject("coord", "uint32_t"),
        sfg.var("coord", "uint32_t"),
        SrcObject("value", "float")
        sfg.var("value", "float")
    ),
    ),


    c.klass("MyClass", bases=("MyBaseClass",))(
    sfg.klass("MyClass", bases=("MyBaseClass",))(
        # class body sequencer
        # class body sequencer


        c.constructor(SrcObject("a", "int"))
        sfg.constructor(sfg.var("a", "int"))
        .init("a_(a)")
        .init("a_(a)")
        .body(
        .body(
            'cout << "Hi!" << endl;'
            'cout << "Hi!" << endl;'
        ),
        ),


        c.private(
        sfg.private(
            c.var("a_", "int"),
            sfg.var("a_", "int"),


            c.method("getX", returns="int")(
            sfg.method("getX", returns="int")(
                "return 2.0;"
                "return 2.0;"
            )
            )
        ),
        ),


        c.public(
        sfg.public(
            "using xtype = uint8_t;"
            "using xtype = uint8_t;"
        )
        )
    )
    )
Original line number Original line Diff line number Diff line
# type: ignore
# type: ignore
from pystencilssfg import SourceFileGenerator, SfgConfiguration, SfgComposer
from pystencilssfg import SourceFileGenerator, SfgConfiguration, SfgComposer
from pystencilssfg.configuration import SfgCodeStyle
from pystencilssfg.configuration import SfgCodeStyle
from pystencilssfg.types import SrcType
from pystencils.types import PsCustomType
from pystencilssfg.source_concepts import SrcObject
from pystencilssfg.ir.source_components import SfgClass, SfgMemberVariable, SfgConstructor, SfgMethod
from pystencilssfg.source_components import SfgClass, SfgMemberVariable, SfgConstructor, SfgMethod, SfgVisibility


from pystencils import fields, kernel
from pystencils import fields, kernel


@@ -38,7 +37,7 @@ with SourceFileGenerator(sfg_config) as ctx:
        sfg.seq(
        sfg.seq(
            "return -1.0;"
            "return -1.0;"
        ),
        ),
        return_type=SrcType("double"),
        return_type="double",
        inline=True,
        inline=True,
        const=True
        const=True
    ))
    ))
@@ -48,20 +47,20 @@ with SourceFileGenerator(sfg_config) as ctx:
        sfg.seq(
        sfg.seq(
            "return 2.0f;"
            "return 2.0f;"
        ),
        ),
        return_type=SrcType("float"),
        return_type="float",
        inline=False,
        inline=False,
        const=True
        const=True
    ))
    ))


    cls.default.append_member(
    cls.default.append_member(
        SfgMemberVariable(
        SfgMemberVariable(
            "stuff", "std::vector< int >"
            "stuff", PsCustomType("std::vector< int > &")
        )
        )
    )
    )


    cls.default.append_member(
    cls.default.append_member(
        SfgConstructor(
        SfgConstructor(
            [SrcObject("stuff", "std::vector< int > &")],
            [sfg.var("stuff", PsCustomType("std::vector< int > &"))],
            ["stuff_(stuff)"]
            ["stuff_(stuff)"]
        )
        )
    )
    )
+16 −0
Original line number Original line Diff line number Diff line
from pystencils import Target, CreateKernelConfig, no_jit
from lbmpy import create_lb_update_rule, LBMOptimisation
from pystencilssfg import SourceFileGenerator, SfgConfig

sfg_config = SfgConfig()
sfg_config.extensions.impl = "cu"
sfg_config.output_directory = "out/test_cuda"
sfg_config.outer_namespace = "gen_code"

with SourceFileGenerator(sfg_config) as sfg:
    gen_config = CreateKernelConfig(target=Target.CUDA, jit=no_jit)
    opt = LBMOptimisation(field_layout="fzyx")
    update = create_lb_update_rule()
    kernel = sfg.kernels.create(update, "lbm_update", gen_config)

    sfg.function("lb_update")(sfg.call(kernel))
+21 −0
Original line number Original line Diff line number Diff line
from pystencils import Target, CreateKernelConfig, create_kernel, no_jit
from lbmpy import create_lb_update_rule, LBMOptimisation
from pystencilssfg import SourceFileGenerator, SfgConfiguration, SfgOutputMode
from pystencilssfg.lang.cpp import mdspan_ref

sfg_config = SfgConfiguration(
    output_directory="out/test_sycl",
    outer_namespace="gen_code",
    impl_extension="ipp",
    output_mode=SfgOutputMode.INLINE
)

with SourceFileGenerator(sfg_config) as sfg:
    gen_config = CreateKernelConfig(target=Target.SYCL, jit=no_jit)
    opt = LBMOptimisation(field_layout="fzyx")
    update = create_lb_update_rule()
    kernel = sfg.kernels.create(update, "lbm_update", gen_config)

    sfg.function("lb_update")(
        sfg.call(kernel)
    )

mkdocs.yml

deleted100644 → 0
+0 −67

File deleted.

Preview size limit exceeded, changes collapsed.

+3 −0
Original line number Original line Diff line number Diff line
@@ -3,3 +3,6 @@ python_version=3.10


[mypy-pystencils.*]
[mypy-pystencils.*]
ignore_missing_imports=true
ignore_missing_imports=true

[mypy-sympy.*]
ignore_missing_imports=true

noxfile.py

0 → 100644
+102 −0

File added.

Preview size limit exceeded, changes collapsed.

pdm.lock

deleted100644 → 0
+0 −1011

File deleted.

Preview size limit exceeded, changes collapsed.

+42 −14

File changed.

Preview size limit exceeded, changes collapsed.

pytest.ini

0 → 100644
+14 −0
Original line number Original line Diff line number Diff line
[pytest]
testpaths = src/pystencilssfg tests/
python_files = "test_*.py"
#   Need to ignore the generator scripts, otherwise they would be executed
#   during test collection
addopts = 
    --doctest-modules
    --ignore=tests/generator_scripts/source
    --ignore=tests/generator_scripts/deps
    --ignore=tests/generator_scripts/expected
    --ignore=tests/data
    --ignore=tests/integration/cmake_project

doctest_optionflags = NORMALIZE_WHITESPACE IGNORE_EXCEPTION_DETAIL
Original line number Original line Diff line number Diff line

if __name__ == "__main__":
if __name__ == "__main__":
    from .cli import cli_main
    from .cli import cli_main

    cli_main("python -m pystencilssfg")
    cli_main("python -m pystencilssfg")
Original line number Original line Diff line number Diff line

class SfgException(Exception):
class SfgException(Exception):
    pass
    pass

src/pystencilssfg/types.py

deleted100644 → 0
+0 −44

File deleted.

Preview size limit exceeded, changes collapsed.

+77 −0

File added.

Preview size limit exceeded, changes collapsed.

versioneer.py

deleted100644 → 0
+0 −2277

File deleted.

Preview size limit exceeded, changes collapsed.