Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
No results found
Show changes
Commits on Source (2)
Showing
with 462 additions and 115 deletions
......@@ -39,7 +39,7 @@ testsuite:
- pip install "git+https://i10git.cs.fau.de/pycodegen/pystencils.git@v2.0-dev"
- pip install -e .
script:
- pytest -v
- pytest -v --cov=src/pystencilssfg --cov-report=term
- coverage html
- coverage xml
coverage: '/TOTAL.*\s+(\d+%)$/'
......
import pytest
from os import path
@pytest.fixture(autouse=True)
......@@ -9,3 +10,11 @@ def prepare_composer(doctest_namespace):
sfg = SfgComposer(SfgContext())
doctest_namespace["sfg"] = sfg
DATA_DIR = path.join(path.split(__file__)[0], "tests/data")
@pytest.fixture
def sample_config_module():
return path.join(DATA_DIR, "project_config.py")
**/generated
{{ fullname | escape | underline}}
.. currentmodule:: {{ module }}
.. autoclass:: {{ objname }}
:members:
{% extends "!autosummary/class.rst" %}
{% block methods %}
{% if methods %}
.. rubric:: {{ _('Methods') }}
.. autosummary::
:toctree:
{% for item in methods %}
~{{ name }}.{{ item }}
{%- endfor %}
{% endif %}
{% endblock %}
{% block attributes %}
{% if attributes %}
.. rubric:: {{ _('Attributes') }}
.. autosummary::
:toctree:
{% for item in attributes %}
~{{ name }}.{{ item }}
{%- endfor %}
{% endif %}
{% endblock %}
......@@ -5,13 +5,32 @@ Generator Script Interface
.. autoclass:: pystencilssfg.SourceFileGenerator
:members:
.. autoclass:: pystencilssfg.SfgConfiguration
Configuration
=============
.. module:: pystencilssfg.config
.. autoclass:: SfgConfig
:members:
:exclude-members: __init__
Categories, Parameter Types, and Special Values
-----------------------------------------------
.. autoclass:: _GlobalNamespace
.. autodata:: GLOBAL_NAMESPACE
.. autoclass:: pystencilssfg.SfgOutputMode
.. autoclass:: OutputMode
:members:
.. autoclass:: pystencilssfg.SfgCodeStyle
.. autoclass:: CodeStyle
:members:
.. autoattribute:: pystencilssfg.configuration.DEFAULT_CONFIG
.. autoclass:: ClangFormatOptions
:members:
Option Descriptors
------------------
.. autoclass:: Option
#############
API Reference
#############
These pages provide a reference for the public API of *pystencils-sfg*.
.. toctree::
:maxdepth: 1
generation
composer
lang
ir
errors
......@@ -27,6 +27,7 @@ extensions = [
"myst_parser",
"sphinx.ext.autodoc",
"sphinx.ext.napoleon",
"sphinx.ext.autosummary",
"sphinx.ext.doctest",
"sphinx.ext.intersphinx",
"sphinx_autodoc_typehints",
......@@ -42,6 +43,10 @@ source_suffix = {
}
master_doc = "index"
nitpicky = True
myst_enable_extensions = [
"colon_fence",
"dollarmath"
]
# -- Options for HTML output -------------------------------------------------
......@@ -71,6 +76,7 @@ intersphinx_mapping = {
autodoc_member_order = "bysource"
autodoc_typehints = "description"
# autodoc_class_signature = "separated"
# Doctest Setup
......
......@@ -3,9 +3,24 @@
```{toctree}
:maxdepth: 1
:hidden:
:caption: User Guide
usage/index
api/index
usage/generator_scripts
usage/project_integration
usage/tips_n_tricks
```
```{toctree}
:maxdepth: 1
:hidden:
:caption: API Reference
api/generation
api/composer
api/lang
api/ir
api/errors
```
[![pipeline](https://i10git.cs.fau.de/pycodegen/pystencils-sfg/badges/master/pipeline.svg)](https://i10git.cs.fau.de/pycodegen-/pystencils-sfg/commits/master)
......@@ -210,4 +225,34 @@ To `#include` them, add the prefix `gen/<target name>`:
```
For details on how to add *pystencils-sfg* to your CMake project, refer to
[CLI and Build System Integration](usage/cli_and_build_system.md).
[the project integration guide](#guide_project_integration).
## Learn To Use pystencils-sfg
Here is an overview of user guides for pystencils-sfg available on this site.
A basic understanding of [pystencils](https://pycodegen.pages.i10git.cs.fau.de/pystencils/index.html)
is required.
```{card} Writing Generator Scripts
:link: guide:generator_scripts
:link-type: ref
Learn about *generator scripts*, the primary usage idiom of *pystencils-sfg*:
Embedd *pystencils*-generated kernels into C++ source files and augment them with
arbitrary C++ glue code.
```
```{card} CLI and Build System Integration
:link: guide_project_integration
:link-type: ref
Learn how to control code generation from the command line
and how to embedd *pystencils-sfg* into your build system.
```
```{card} Tips and Tricks
:link: guide:tips_n_tricks
:link-type: ref
A collection of various tricks that might come in handy when working with *pystencils-sfg*.
```
*.cpp
*.h
\ No newline at end of file
*.h
*.hpp
\ No newline at end of file
from pystencilssfg import SourceFileGenerator
from argparse import ArgumentParser
parser = ArgumentParser()
# set up parser ...
with SourceFileGenerator(keep_unknown_argv=True) as sfg:
args = parser.parse_args(sfg.context.argv)
...
from pystencilssfg import SourceFileGenerator, SfgConfig
cfg = SfgConfig()
cfg.output_directory = "gen_src"
cfg.codestyle.indent_width = 4
with SourceFileGenerator(cfg) as sfg:
...
......@@ -3,7 +3,7 @@
Writing generator scripts is the primary usage idiom of *pystencils-sfg*.
A generator script is a Python script, say `kernels.py`, which contains *pystencils-sfg*
code at the top level that, when executed, emits source code to a pair of files `kernels.h`
code at the top level that, when executed, emits source code to a pair of files `kernels.hpp`
and `kernels.cpp`. This guide describes how to write such a generator script, its structure, and how
it can be used to generate code.
......@@ -23,14 +23,14 @@ To start, place the following code in a Python script, e.g. `kernels.py`:
The source file is constructed within the context manager's managed region.
During execution of the script, when the region ends, a header/source file pair
`kernels.h` and `kernels.cpp` will be written to disk next to your script.
`kernels.hpp` and `kernels.cpp` will be written to disk next to your script.
Execute the script as-is and inspect the generated files, which will of course still be empty:
``````{dropdown} Generated Files
`````{tab-set}
````{tab-item} kernels.h
```{literalinclude} examples/guide_generator_scripts/01/kernels.h
````{tab-item} kernels.hpp
```{literalinclude} examples/guide_generator_scripts/01/kernels.hpp
```
````
......@@ -41,31 +41,110 @@ Execute the script as-is and inspect the generated files, which will of course s
`````
``````
<!-- A few notes on configuration:
- The [SourceFileGenerator](#pystencilssfg.SourceFileGenerator) parses the script's command line arguments
for configuration options (refer to [CLI and Build System Integration](cli_and_build_system.md)).
If you intend to evaluate command-line parameters inside your
generator script, read them from `sfg.context.argv` instead of `sys.argv`.
There, all arguments meant for the code generator are already removed.
- The code generator's configuration is consolidated from a global project configuration which may
be provided by the build system; a number of command line arguments; and the
[SfgConfiguration](#pystencilssfg.SfgConfiguration) provided in the script.
The project configuration may safely be overridden by the latter two; however, conflicts
between command-line arguments and the configuration defined in the script will cause
an exception to be thrown. -->
## Using the Composer
The object `sfg` constructed in above snippet is an instance of [SfgComposer](#pystencilssfg.composer.SfgComposer).
The composer is the central part of the user front-end of *pystencils-sfg*.
It provides an interface for constructing source files that closely mimics
C++ syntactic structures within Python.
Here is an overview of its various functions:
::::{dropdown} Composer API Overview
```{eval-rst}
.. currentmodule:: pystencilssfg.composer
```
Structure and Verbatim Code:
```{eval-rst}
.. autosummary::
:nosignatures:
SfgBasicComposer.prelude
SfgBasicComposer.include
SfgBasicComposer.namespace
SfgBasicComposer.code
SfgBasicComposer.define_once
```
Kernels and Kernel Namespaces:
```{eval-rst}
.. autosummary::
:nosignatures:
SfgBasicComposer.kernels
SfgBasicComposer.kernel_namespace
SfgBasicComposer.kernel_function
```
Function definition, parameters, and header inclusion:
```{eval-rst}
.. autosummary::
:nosignatures:
SfgBasicComposer.function
SfgBasicComposer.params
SfgBasicComposer.require
```
Variables, expressions, and variable initialization:
```{eval-rst}
.. autosummary::
:nosignatures:
SfgBasicComposer.var
SfgBasicComposer.vars
SfgBasicComposer.expr
SfgBasicComposer.init
SfgBasicComposer.map_field
SfgBasicComposer.set_param
```
Parameter mappings:
```{eval-rst}
.. autosummary::
:nosignatures:
SfgBasicComposer.set_param
SfgBasicComposer.map_field
SfgBasicComposer.map_vector
```
Control Flow:
```{eval-rst}
.. autosummary::
:nosignatures:
SfgBasicComposer.branch
SfgBasicComposer.switch
```
Kernel Invocation:
```{eval-rst}
.. autosummary::
:nosignatures:
SfgBasicComposer.call
SfgBasicComposer.cuda_invoke
```
::::
### Includes and Definitions
With [`SfgComposer.include`](#pystencilssfg.composer.SfgBasicComposer.include), the code generator can be instructed to include header files.
With {any}`include <SfgBasicComposer.include>`, the code generator can be instructed to include header files.
As in C++, you can use the `<>` delimiters for system headers, and omit them for project headers.
`````{tab-set}
......@@ -75,8 +154,8 @@ As in C++, you can use the `<>` delimiters for system headers, and omit them for
```
````
````{tab-item} kernels.h
```{literalinclude} examples/guide_generator_scripts/02/kernels.h
````{tab-item} kernels.hpp
```{literalinclude} examples/guide_generator_scripts/02/kernels.hpp
```
````
......@@ -112,8 +191,8 @@ the [`sfg.kernel_namespace`](#pystencilssfg.composer.SfgBasicComposer.kernel_nam
```
````
````{tab-item} kernels.h
```{literalinclude} examples/guide_generator_scripts/03/kernels.h
````{tab-item} kernels.hpp
```{literalinclude} examples/guide_generator_scripts/03/kernels.hpp
```
````
......@@ -141,8 +220,8 @@ Use `sfg.function` to create a function, and `sfg.call` to call a kernel:
```
````
````{tab-item} kernels.h
```{literalinclude} examples/guide_generator_scripts/04/kernels.h
````{tab-item} kernels.hpp
```{literalinclude} examples/guide_generator_scripts/04/kernels.hpp
```
````
......@@ -171,8 +250,8 @@ such as, for example, `std::span` or `std::vector`, like this:
```
````
````{tab-item} kernels.h
```{literalinclude} examples/guide_generator_scripts/05/kernels.h
````{tab-item} kernels.hpp
```{literalinclude} examples/guide_generator_scripts/05/kernels.hpp
```
````
......@@ -192,3 +271,77 @@ The pystencils-sfg provides modelling support for a number of C++ standard libra
(see {any}`pystencilssfg.lang.cpp.std`).
It also provides the necessary infrastructure for modelling the data structures of any C++ framework
in a similar manner.
## Configuration and Invocation
There are several ways to affect the behavior and output of a generator script.
For one, the `SourceFileGenerator` itself may be configured from the combination of three
different configuration sources:
- **Inline Configuration:** The generator script may set up an {any}`SfgConfig` object,
which is passed to the `SourceFileGenerator` at its creation; see [Inline Configuration](#inline_config)
- **Command-Line Options:** The `SourceFileGenerator` parses the command line arguments of
the generator script to set some of its configuration options; see [Command-Line Options](#cmdline_options)
- **Project Configuration:** When embedded into a larger project, using a build system such as CMake, generator scripts
may be configured globally within that project by the use of a *configuration module*.
Settings specified inside that configuration module are always overridden by the former to configuration sources.
For details on configuration modules, refer to the guide on [Project and Build System Integration](#guide_project_integration).
(inline_config)=
### Inline Configuration
To configure the source file generator within your generator script, import the {any}`SfgConfig` from `pystencilssfg`.
You may then set up the configuration object before passing it to the `SourceFileGenerator` constructor.
To illustrate, the following snippet alters the code indentation width and changes the output directory
of the generator script to `gen_src`:
```{literalinclude} examples/guide_generator_scripts/inline_config/kernels.py
```
(cmdline_options)=
### Command-Line Options
The `SourceFileGenerator` consumes a number of command-line parameters that may be passed to the script
on invocation. These include:
- `--sfg-output-dir <path>`: Set the output directory of the generator script. This corresponds to {any}`SfgConfig.output_directory`.
- `--sfg-file-extensions <exts>`: Set the file extensions used for the generated files;
`exts` must be a comma-separated list not containing any spaces. Corresponds to {any}`SfgConfig.extensions`.
- `--sfg-output-mode <mode>`: Set the output mode of the generator script. Corresponds to {any}`SfgConfig.output_mode`.
If any configuration option is set to conflicting values on the command line and in the inline configuration,
the generator script will terminate with an error.
You may examine the full set of possible command line parameters by invoking a generator script
with the `--help` flag:
```bash
$ python kernels.py --help
```
## Adding Custom Command-Line Options
Sometimes, you might want to add your own command-line options to a generator script
in order to affect its behavior from the shell,
for instance by using {any}`argparse` to set up an argument parser.
If you parse your options directly from {any}`sys.argv`,
as {any}`parse_args <argparse.ArgumentParser.parse_args>` does by default,
your parser will also receive any options meant for the `SourceFileGenerator`.
To filter these out of the argument list,
pass the additional option `keep_unknown_argv=True` to your `SourceFileGenerator`.
This will instruct it to store any unknown command line arguments into `sfg.context.argv`,
where you can then retrieve them from and pass on to your custom parser:
```{literalinclude} examples/guide_generator_scripts/custom_cmdline_args/kernels.py
```
Any SFG-specific arguments will already have been filtered out of this argument list.
As a consequence of the above, if the generator script is invoked with a typo in some SFG-specific argument,
which the `SourceFileGenerator` therefore does not recognize,
that argument will be passed on to your downstream parser instead.
:::{important}
If you do *not* pass on `sfg.context.argv` to a downstream parser, make sure that `keep_unknown_argv` is set to
`False` (which is the default), such that typos or illegal arguments will not be ignored.
:::
# Usage Guides
```{toctree}
:maxdepth: 1
:hidden:
generator_scripts
cli_and_build_system
tips_n_tricks
```
These pages provide an overview of how to use the pystencils Source File Generator.
A basic understanding of [pystencils](https://pycodegen.pages.i10git.cs.fau.de/pystencils/index.html)
is required.
```{card} Writing Generator Scripts
:link: guide:generator_scripts
:link-type: ref
Learn about *generator scripts*, the primary usage idiom of *pystencils-sfg*:
Embedd *pystencils*-generated kernels into C++ source files and augment them with
arbitrary C++ glue code.
```
```{card} CLI and Build System Integration
:link: guide:cli
:link-type: ref
Learn how to control code generation from the command line
and how to embedd *pystencils-sfg* into your build system.
```
```{card} Tips and Tricks
:link: guide:tips_n_tricks
:link-type: ref
A collection of various tricks that might come in handy when working with *pystencils-sfg*.
```
(guide:cli)=
# CLI and Build System
## Command Line Interface
*pystencils-sfg* exposes not one, but two command line interfaces:
The *global CLI* offers a few tools meant to be used by build systems,
while the *generator script* command line interface is meant for a build system to communicate
with the code generator during on-the-fly generation.
### Global CLI
(guide_project_integration)=
# Project and Build System Integration
(config_module)=
## Project-Wide Settings using Configuration Modules
When embedding *pystencils-sfg* into a C++ project or build system,
you might want to set a project-wide base configuration for all generator scripts.
In addition, it might be necessary to pass various details about the project
and build setup to the generator scripts.
Both can be achieved by the use of a *configuration module*.
A configuration module is a Python file that defines up to two functions:
- `def configure_sfg(cfg: SfgConfig)` is called to set up the project-wide base configuration.
It takes an {any}`SfgConfig` object which it may modify to establish the project-wide option set.
- `def project_info() -> Any` is called by *pystencils-sfg* to retrieve an object that encapsulates
any custom project-specific information.
This information is passed on to the generator scripts through
the {any}`sfg.context.project_info <SfgContext.project_info>` attribute.
An example configuration module might look like this:
```Python
from pystencilssfg import SfgConfig
def configure_sfg(cfg: SfgConfig):
cfg.extensions.header = "h++"
cfg.extensions.impl = "c++"
cfg.clang_format.code_style = "llvm"
...
def project_info():
return {
"project_name": "my-project",
"float_precision": "float32",
"use_cuda": False,
...
}
```
The global CLI may be accessed either through the `sfg-cli` shell command, or using `python -m pystencilssfg`.
Here, `project_info` returns a dictionary, but this is just for illustration;
the function may return any type of arbitrarily complex objects.
For improved API safety, {any}`dataclasses` might be a good tool for setting up
project info objects.
### Generator Script CLI
When invoking a generator script, the path to the current configuration module must be passed to it
using the `--sfg-config-module` command-line parameter.
This can be automated by an adequately set up build system, such as GNU Make or CMake.
The [SourceFileGenerator][pystencilssfg.SourceFileGenerator] evaluates a generator script's command line arguments,
which can be supplied by the user, but more frequently by the build system.
If you are using pystencils-sfg with CMake through the provided CMake module,
[see below](#cmake_set_config_module) on how to specify a configuration module for your project.
(cmake_integration)=
## CMake Integration
*pystencils-sfg* is shipped with a CMake module for on-the-fly code generation during the CMake build process.
......@@ -50,7 +84,8 @@ pystencilssfg_generate_target_sources( <target>
SCRIPTS script1.py [script2.py ...]
[DEPENDS dependency1.py [dependency2.py...]]
[FILE_EXTENSIONS <header-extension> <impl-extension>]
[HEADER_ONLY])
[OUTPUT_MODE <standalone|inline|header-only>]
)
```
It registers the generator scripts `script1.py [script2.py ...]` to be executed at compile time using `add_custom_command`
......@@ -61,7 +96,7 @@ The function takes the following options:
- `SCRIPTS`: A list of generator scripts
- `DEPENDS`: A list of dependencies for the generator scripts
- `FILE_EXTENSION`: The desired extensions for the generated files
- `HEADER_ONLY`: Toggles header-only code generation
- `OUTPUT_MODE`: Sets the output mode of the code generator; see {any}`SfgConfig.output_mode`.
### Include generated files
......@@ -72,7 +107,14 @@ path, such that generated header files for a target `<target>` may be included v
#include "gen/<target>/kernels.h"
```
### Project Configuration
(cmake_set_config_module)=
### Set a Configuration Module
To specify a [configuration module](#config_module) for your project,
set the scoped variable `PystencilsSfg_CONFIG_MODULE` to point at the respective Python file.
The pystencils-sfg CMake system will then pass that module to each generator script invocation.
The *pystencils-sfg* CMake module reads the scoped variable `PystencilsSfg_CONFIGURATOR_SCRIPT` to find
the *configuration module* that should be passed to the generator scripts.
You might want to populate your configuration module with information about the current
build setup and environment.
For this purpose, take a look at the
[configure_file](https://cmake.org/cmake/help/latest/command/configure_file.html) CMake function.
......@@ -11,7 +11,7 @@ execute_process( COMMAND sfg-cli cmake make-find-module
find_package( PystencilsSfg REQUIRED )
set( PystencilsSfg_CONFIGURATOR_SCRIPT codegen_config.py )
set( PystencilsSfg_CONFIG_MODULE codegen_config.py )
add_library( genlib )
pystencilssfg_generate_target_sources( genlib SCRIPTS kernels.py FILE_EXTENSIONS .h .cpp )
......
from sys import stderr
from pystencilssfg import SfgConfiguration
from pystencilssfg import SfgConfig
def sfg_config():
print("sfg_config() called!", file=stderr)
project_info = {
'B': 'A'
}
return SfgConfiguration(
header_extension='hpp',
impl_extension='cpp',
outer_namespace='cmake_demo',
project_info=project_info
)
def configure(cfg: SfgConfig):
cfg.extensions.header = "h++"
cfg.extensions.impl = "c++"
......@@ -23,9 +23,6 @@ requires = [
build-backend = "setuptools.build_meta"
[project.optional-dependencies]
interactive = [
"ipython>=8.17.2",
]
testing = [
"flake8>=6.1.0",
"mypy>=1.7.0",
......@@ -39,7 +36,8 @@ docs = [
"sphinx_design",
"sphinx_autodoc_typehints",
"sphinx-copybutton",
"packaging"
"packaging",
"clang-format"
]
[tool.versioneer]
......@@ -51,4 +49,8 @@ tag_prefix = "v"
parentdir_prefix = "pystencilssfg-"
[tool.coverage.run]
include = ["src/pystencilssfg/*"]
omit = [
"setup.py",
"src/pystencilssfg/_version.py",
"integration/*"
]
......@@ -3,6 +3,10 @@ testpaths = src/pystencilssfg tests/
python_files = "test_*.py"
# Need to ignore the generator scripts, otherwise they would be executed
# during test collection
addopts = --doctest-modules --ignore=tests/generator_scripts/scripts --cov=src/pystencilssfg --cov-report=term
addopts =
--doctest-modules
--ignore=tests/generator_scripts/scripts
--ignore=tests/generator_scripts/config
--ignore=tests/data
doctest_optionflags = NORMALIZE_WHITESPACE IGNORE_EXCEPTION_DETAIL
from .configuration import SfgConfiguration, SfgOutputMode, SfgCodeStyle
from .generator import SourceFileGenerator
from .config import SfgConfig
from .generator import SourceFileGenerator, GLOBAL_NAMESPACE, OutputMode
from .composer import SfgComposer
from .context import SfgContext
from .lang import SfgVar, AugExpr
from .exceptions import SfgException
__all__ = [
"SfgConfig",
"GLOBAL_NAMESPACE",
"OutputMode",
"SourceFileGenerator",
"SfgComposer",
"SfgConfiguration",
"SfgOutputMode",
"SfgCodeStyle",
"SfgContext",
"SfgVar",
"AugExpr",
......