Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
No results found
Show changes
Commits on Source (8)
Showing
with 626 additions and 303 deletions
......@@ -276,20 +276,9 @@ typecheck:
# -------------------- Unit Tests ---------------------------------------------------------------------
# Normal test - runs on every commit all but "long run" tests
tests-and-coverage:
.testsuite-base:
stage: "Unit Tests"
needs: []
image: i10git.cs.fau.de:5005/pycodegen/pycodegen/nox:ubuntu24.04-cuda12.6
script:
- mkdir -p ~/.config/matplotlib
- echo "backend:template" > ~/.config/matplotlib/matplotlibrc
- mkdir public
- nox --session "testsuite(cupy12)"
tags:
- docker
- cuda11
- AVX
coverage: /Total coverage:\s\d+.\d+\%/
artifacts:
when: always
......@@ -302,6 +291,28 @@ tests-and-coverage:
path: coverage.xml
junit: report.xml
"testsuite-gpu-py3.10":
extends: .testsuite-base
image: i10git.cs.fau.de:5005/pycodegen/pycodegen/nox:ubuntu24.04-cuda12.6
script:
- mkdir -p ~/.config/matplotlib
- echo "backend:template" > ~/.config/matplotlib/matplotlibrc
- mkdir public
- nox --session "testsuite-3.10(cupy12)"
tags:
- docker
- cuda11
- AVX
"testsuite-cpu-py3.13":
extends: .testsuite-base
image: i10git.cs.fau.de:5005/pycodegen/pycodegen/nox:alpine
script:
- nox --session "testsuite-3.13(cpu)"
tags:
- docker
- AVX
# -------------------- Documentation ---------------------------------------------------------------------
......@@ -322,7 +333,7 @@ build-documentation:
pages:
image: alpine:latest
stage: deploy
needs: ["tests-and-coverage", "build-documentation"]
needs: ["testsuite-gpu-py3.10", "build-documentation"]
script:
- mv docs/build/html public
- mv coverage_report public/coverage_report
......
......@@ -22,7 +22,7 @@ html:
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
clean:
rm -rf source/reference/generated
rm -rf source/api/generated
rm -rf source/api/symbolic/generated
rm -rf source/backend/generated
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
\ No newline at end of file
# Code Generation
## Invocation
```{eval-rst}
.. module:: pystencils.codegen
.. autosummary::
:toctree: generated
:nosignatures:
create_kernel
```
## Configuration
```{eval-rst}
.. module:: pystencils.codegen.config
```
The code generation driver (`create_kernel`, but also `DefaultKernelCreationDriver`) can be configured by
passing it a `CreateKernelConfig` object.
This object can be constructed incrementally:
```Python
cfg = ps.CreateKernelConfig()
cfg.default_dtype = "float32"
cfg.target = ps.Target.X86_AVX
cfg.cpu.openmp.enable = True
cfg.cpu.vectorize.enable = True
cfg.cpu.vectorize.assume_inner_stride_one = True
```
### Options and Option Categories
The following options and option categories are exposed by the configuration object:
#### Target Specification
```{eval-rst}
.. current
.. autosummary::
~CreateKernelConfig.target
```
#### Data Types
```{eval-rst}
.. autosummary::
~CreateKernelConfig.default_dtype
~CreateKernelConfig.index_dtype
```
#### Iteration Space
```{eval-rst}
.. autosummary::
~CreateKernelConfig.ghost_layers
~CreateKernelConfig.iteration_slice
~CreateKernelConfig.index_field
```
#### Kernel Constraint Checks
```{eval-rst}
.. autosummary::
~CreateKernelConfig.allow_double_writes
~CreateKernelConfig.skip_independence_check
```
#### Target-Specific Options
The following categories with target-specific options are exposed:
| | |
|---------------------------|--------------------------|
| {any}`cpu <CpuOptions>` | Options for CPU kernels |
| {any}`gpu <GpuOptions>` | Options for GPU kernels |
| {any}`sycl <SyclOptions>` | Options for SYCL kernels |
#### Kernel Object and Just-In-Time Compilation
```{eval-rst}
.. autosummary::
~CreateKernelConfig.function_name
~CreateKernelConfig.jit
```
### Configuration System Classes
```{eval-rst}
.. autosummary::
:toctree: generated
:nosignatures:
:template: autosummary/recursive_class.rst
CreateKernelConfig
CpuOptions
OpenMpOptions
VectorizationOptions
GpuOptions
SyclOptions
.. autosummary::
:toctree: generated
:nosignatures:
AUTO
.. dropdown:: Implementation Details
.. autosummary::
:toctree: generated
:nosignatures:
:template: autosummary/entire_class.rst
Option
BasicOption
Category
ConfigBase
```
## Target Specification
```{eval-rst}
.. module:: pystencils.codegen.target
.. autosummary::
:toctree: generated
:nosignatures:
:template: autosummary/recursive_class.rst
Target
```
## Code Generation Drivers
```{eval-rst}
.. module:: pystencils.codegen.driver
.. autosummary::
:toctree: generated
:nosignatures:
:template: autosummary/entire_class.rst
DefaultKernelCreationDriver
.. autosummary::
:toctree: generated
:nosignatures:
get_driver
```
## Output Code Objects
```{eval-rst}
.. currentmodule:: pystencils.codegen
.. autosummary::
:toctree: generated
:nosignatures:
:template: autosummary/entire_class.rst
Kernel
GpuKernel
Parameter
GpuThreadsRange
```
pystencils.codegen
==================
.. module:: pystencils.codegen
Invocation
----------
.. autosummary::
:toctree: generated
:nosignatures:
create_kernel
Configuration
-------------
.. autosummary::
:toctree: generated
:nosignatures:
:template: autosummary/entire_class.rst
CreateKernelConfig
CpuOptimConfig
OpenMpConfig
VectorizationConfig
GpuIndexingConfig
.. autosummary::
:toctree: generated
:nosignatures:
AUTO
Target Specification
--------------------
.. autosummary::
:toctree: generated
:nosignatures:
:template: autosummary/recursive_class.rst
Target
Code Generation Drivers
-----------------------
.. autosummary::
:toctree: generated
:nosignatures:
:template: autosummary/entire_class.rst
driver.DefaultKernelCreationDriver
.. autosummary::
:toctree: generated
:nosignatures:
get_driver
Output Code Objects
-------------------
.. autosummary::
:toctree: generated
:nosignatures:
:template: autosummary/entire_class.rst
Kernel
GpuKernel
Parameter
GpuThreadsRange
pystencils.jit
==============
JIT Compilation
===============
.. module:: pystencils.jit
......
# Assignments and AssignmentCollection
```{eval-rst}
.. py:class:: pystencils.Assignment
Monkeypatched variant of `sympy.codegen.ast.Assignment`.
Represents an assignment of an expression to a symbol.
.. autosummary::
:toctree: generated
:nosignatures:
:template: autosummary/recursive_class.rst
pystencils.AssignmentCollection
```
pystencils.field
================
Fields
======
.. module:: pystencils.field
......
# Symbolic Toolbox
:::{toctree}
:maxdepth: 1
field
assignments
sympyextensions
:::
pystencils.sympyextensions
==========================
Extensions to SymPy
===================
.. module:: pystencils.sympyextensions
......@@ -71,7 +71,10 @@ Typed Expressions
.. autoclass:: pystencils.DynamicType
:members:
.. autoclass:: pystencils.sympyextensions.CastFunc
.. autoclass:: pystencils.sympyextensions.typed_sympy.TypeCast
:members:
.. autoclass:: pystencils.sympyextensions.tcast
Integer Operations
......
......@@ -118,10 +118,10 @@ mypy src/pystencils
::::
:::{note}
Type checking is currently restricted to the `codegen`, `jit`, `backend`, and `types` modules,
since most code in the remaining modules is significantly older and is not comprehensively
type-annotated. As more modules are updated with type annotations, this list will expand in the future.
If you think a new module is ready to be type-checked, add an exception clause for it in the `mypy.ini` file.
Type checking is currently restricted only to a few modules, which are listed in the `mypy.ini` file.
Most code in the remaining modules is significantly older and is not comprehensively type-annotated.
As more modules are updated with type annotations, this list will expand in the future.
If you think a new module is ready to be type-checked, add an exception clause to `mypy.ini`.
:::
## Running the Test Suite
......
# Contributor Guide
# Contribution Guide
Welcome to the Contributor's Guide to pystencils!
If you are interested in contributing to the development of pystencils, this is the place to start.
......
......@@ -77,19 +77,19 @@ Topics
.. toctree::
:maxdepth: 1
:caption: Reference Guides
:caption: User Manual
reference/symbolic_language
reference/kernelcreation
reference/gpu_kernels
reference/types
user_manual/symbolic_language
user_manual/kernelcreation
user_manual/gpu_kernels
user_manual/WorkingWithTypes
.. toctree::
:maxdepth: 1
:caption: API
:caption: API Reference
api/field
api/sympyextensions
api/symbolic/index
api/types
api/codegen
api/jit
......
.. _page_v2_migration:
---
jupytext:
formats: md:myst
text_representation:
extension: .md
format_name: myst
kernelspec:
display_name: Python 3 (ipykernel)
language: python
name: python3
mystnb:
execution_mode: cache
---
***************************
Version 2.0 Migration Guide
***************************
(_page_v2_migration)=
# Version 2.0 Migration Guide
With version 2.0, many APIs of *pystencils* will be changed; old interfaces are being deprecated
and new systems are put in place.
This page is a still-incomplete list of these changes, with advice on how to migrate your code
from pystencils 1.x to pystencils 2.0.
Kernel Creation
===============
```{code-cell} ipython3
:tags: [remove-cell]
Configuration
-------------
import pystencils as ps
```
The API of `create_kernel`, and the configuration options of the `CreateKernelConfig`, have changed significantly:
## Kernel Creation
### Configuration
The API of {any}`create_kernel`, and the configuration options of the {any}`CreateKernelConfig`, have changed significantly.
The `CreateKernelConfig` class has been refined to be safe to copy and edit incrementally.
The recommended way of setting up the code generator is now *incremental configuration*:
```{code-cell} ipython3
cfg = ps.CreateKernelConfig()
cfg.default_dtype = "float32"
cfg.cpu.openmp.enable = True
cfg.cpu.openmp.num_threads = 8
cfg.ghost_layers = 2
```
- *Data Types:* `CreateKernelConfig` now takes to parameters to control data types in your kernels:
the ``default_dtype`` is applied to all numerical computations, while the ``index_dtype`` is used
for all index calculations and loop counters.
- *CPU Optimization Options:* Should now be set via the {any}`cpu <CpuOptions>` option category and its subcategories.
.. dropdown:: Deprecated options of `CreateKernelConfig`
:::{dropdown} Deprecated options of `CreateKernelConfig`
- ``data_type``: Use ``default_dtype`` instead
- ``cpu_openmp``: Set OpenMP-Options in the `cpu.openmp <OpenMpOptions>` category instead.
- ``cpu_vectorize_info``: Set vectorization options in the `cpu.vectorize <VectorizationOptions>` category instead
- ``gpu_indexing_params``: Set GPU indexing options in the `gpu <GpuOptions>` category instead
- ``data_type``: Use ``default_dtype`` instead
- ``cpu_openmp``: Set OpenMP-Options via an `OpenMpConfig` in the ``cpu_optim`` (`CpuOptimConfig`) instead.
- ``cpu_vectorize_info``: Set vectorization options via a `VectorizationConfig` in the ``cpu_optim`` option instead
- ``gpu_indexing_params``: Set GPU indexing options via a `GpuIndexingConfig` in the ``gpu_indexing`` option instead
:::
Type Checking
-------------
### Type Checking
The old type checking system of pystencils' code generator has been replaced by a new type inference and validation
mechanism whose rules are much stricter than before.
......@@ -38,24 +66,23 @@ While running `create_kernel`, you may now encounter a `TypificationError` where
If this happens, it is probable that you have been doing some illegal, maybe dangerous, or at least unsafe things with data types
(like inserting integers into a floating-point context without casting them, or mixing types of different precisions or signedness).
If you are sure the error is not your fault, please file an issue at our
`bug tracker <https://i10git.cs.fau.de/pycodegen/pystencils/-/issues>`_.
[bug tracker](https://i10git.cs.fau.de/pycodegen/pystencils/-/issues).
Type System
===========
### Type System
The ``pystencils.typing`` module has been entirely replaced by the new `pystencils.types` module,
The ``pystencils.typing`` module has been entirely replaced by the new {any}`pystencils.types` module,
which is home to a completely new type system.
The primary interaction points with this system are still the `TypedSymbol` class and the `create_type` routine.
The primary interaction points with this system are still the {any}`TypedSymbol` class and the {any}`create_type` routine.
Code using any of these two should not require any changes, except:
- *Importing `TypedSymbol` and `create_type`:* Both `TypedSymbol` and `create_type` should now be imported directly
from the ``pystencils`` namespace.
- *Custom data types:* `TypedSymbol` used to accept arbitrary strings as data types.
This is no longer possible; instead, import `pystencils.types.PsCustomType` and use it to describe
This is no longer possible; instead, import {any}`pystencils.types.PsCustomType` and use it to describe
custom data types unknown to pystencils, as in ``TypedSymbol("xs", PsCustomType("std::vector< int >"))``
All old data type classes (such as ``BasicType``, ``PointerType``, ``StructType``, etc.) have been removed
and replaced by the class hierarchy below `PsType`.
and replaced by the class hierarchy below {any}`PsType`.
Directly using any of these type classes in the frontend is discouraged unless absolutely necessary;
in most cases, `create_type` suffices.
This source diff could not be displayed because it is too large. You can view the blob instead.
---
file_format: mystnb
kernelspec:
display_name: Python 3 (ipykernel)
language: python
name: python3
mystnb:
execution_mode: cache
---
# Working with Data Types
This guide will demonstrate the various options that exist to customize the data types
in generated kernels.
Data types can be modified on different levels of granularity:
Individual fields and symbols,
single subexpressions,
or the entire kernel.
```{code-cell} ipython3
:tags: [remove-cell]
import pystencils as ps
import sympy as sp
```
## Changing the Default Data Types
The pystencils code generator defines two default data types:
- The default *numeric type*, which is applied to all numerical computations that are not
otherwise explicitly typed; the default is `float64`.
- The default *index type*, which is used for all loop and field index calculations; the default is `int64`.
These can be modified by setting the
{any}`default_dtype <CreateKernelConfig.default_dtype>` and
{any}`index_type <CreateKernelConfig.index_dtype>`
options of the code generator configuration:
```{code-cell} ipython3
cfg = ps.CreateKernelConfig()
cfg.default_dtype = "float32"
cfg.index_dtype = "int32"
```
Modifying these will change the way types for [untyped symbols](#untyped-symbols)
and [dynamically typed expressions](#dynamic-typing) are computed.
## Setting the Types of Fields and Symbols
(untyped-symbols)=
### Untyped Symbols
Symbols used inside a kernel are most commonly created using
{any}`sp.symbols <sympy.core.symbol.symbols>` or
{any}`sp.Symbol <sympy.core.symbol.Symbol>`.
These symbols are *untyped*; they will receive a type during code generation
according to these rules:
- Free untyped symbols (i.e. symbols not defined by an assignment inside the kernel) receive the
{any}`default data type <CreateKernelConfig.default_dtype>` specified in the code generator configuration.
- Bound untyped symbols (i.e. symbols that *are* defined in an assignment)
receive the data type that was computed for the right-hand side expression of their defining assignment.
If you are working on kernels with homogenous data types, using untyped symbols will mostly be enough.
### Explicitly Typed Symbols and Fields
If you need more control over the data types in (parts of) your kernel,
you will have to explicitly specify them.
To set an explicit data type for a symbol, use the {any}`TypedSymbol` class of pystencils:
```{code-cell} ipython3
x_typed = ps.TypedSymbol("x", "uint32")
x_typed, str(x_typed.dtype)
```
You can set a `TypedSymbol` to any data type provided by [the type system](#page_type_system),
which will then be enforced by the code generator.
The same holds for fields:
When creating fields through the {any}`fields <pystencils.field.fields>` function,
add the type to the descriptor string; for instance:
```{code-cell} ipython3
f, g = ps.fields("f(1), g(3): float32[3D]")
str(f.dtype), str(g.dtype)
```
When using `Field.create_generic` or `Field.create_fixed_size`, on the other hand,
you can set the data type via the `dtype` keyword argument.
(dynamic-typing)=
### Dynamically Typed Symbols and Fields
Apart from explicitly setting data types,
`TypedSymbol`s and fields can also receive a *dynamic data type* (see {any}`DynamicType`).
There are two options:
- Symbols or fields annotated with {any}`DynamicType.NUMERIC_TYPE` will always receive
the {any}`default numeric type <CreateKernelConfig.default_dtype>` configured for the
code generator.
This is the default setting for fields
created through `fields`, `Field.create_generic` or `Field.create_fixed_size`.
- When annotated with {any}`DynamicType.INDEX_TYPE`, on the other hand, they will receive
the {any}`index data type <CreateKernelConfig.index_dtype>` configured for the kernel.
Using dynamic typing, you can enforce symbols to receive either the standard numeric or
index type without explicitly stating it, such that your kernel definition becomes
independent from the code generator configuration.
## Mixing Types Inside Expressions
Pystencils enforces that all symbols, constants, and fields occuring inside an expression
have the same data type.
The code generator will never introduce implicit casts--if any type conflicts arise, it will terminate with an error.
Still, there are cases where you want to combine subexpressions of different types;
maybe you need to compute geometric information from loop counters or other integers,
or you are doing mixed-precision numerical computations.
In these cases, you might have to introduce explicit type casts when values move from one type context to another.
<!-- 2. Annotate expressions with a specific data type to ensure computations are performed in that type.
TODO: See #97 (https://i10git.cs.fau.de/pycodegen/pystencils/-/issues/97)
-->
(type_casts)=
### Type Casts
Type casts can be introduced into kernels using the {any}`tcast` symbolic function.
It takes an expression and a data type, which is either an explicit type (see [the type system](#page_type_system))
or a dynamic type ({any}`DynamicType`):
```{code-cell} ipython3
x, y = sp.symbols("x, y")
expr1 = ps.tcast(x, "float32")
expr2 = ps.tcast(3 + y, ps.DynamicType.INDEX_TYPE)
str(expr1.dtype), str(expr2.dtype)
```
When a type cast occurs, pystencils will compute the type of its argument independently
and then introduce a runtime cast to the target type.
That target type must comply with the type computed for the outer expression,
which the cast is embedded in.
## Understanding the pystencils Type Inference System
To correctly apply varying data types to pystencils kernels, it is important to understand
how pystencils computes and propagates the data types of symbols and expressions.
Type inference happens on the level of assignments.
For each assignment $x := \mathrm{calc}(y_1, \dots, y_n)$,
the system first attempts to compute a *unique* type for the right-hand side (RHS) $\mathrm{calc}(y_1, \dots, y_n)$.
It searches for any subexpression inside the RHS for which a type is already known --
these might be typed symbols
(whose types are either set explicitly by the user,
or have been determined from their defining assignment),
field accesses,
or explicitly typed expressions.
It then attempts to apply that data type to the entire expression.
If type conflicts occur, the process fails and the code generator raises an error.
Otherwise, the resulting type is assigned to the left-hand side symbol $x$.
:::{admonition} Developer's To Do
It would be great to illustrate this using a GraphViz-plot of an AST,
with nodes colored according to their data types
:::
......@@ -159,15 +159,10 @@ kernel = ps.create_kernel(assignments, cfg).compile()
```
This warns us that the threads range could not be determined automatically.
We can disable this warning by setting `manual_launch_grid` in the GPU indexing options:
We can disable this warning by setting `manual_launch_grid` in the GPU option category:
```{code-cell}
cfg = ps.CreateKernelConfig(
# ... other options ...
gpu_indexing=ps.GpuIndexingConfig(
manual_launch_grid=True
)
)
cfg.gpu.manual_launch_grid = True
```
Now, to execute our kernel, we have to manually specify its launch grid:
......
......@@ -138,7 +138,7 @@ This happens roughly according to the following rules:
We can observe this behavior by setting up a kernel including several fields with different data types:
```{code-cell} ipython3
from pystencils.sympyextensions import CastFunc
from pystencils.sympyextensions import tcast
f = ps.fields("f: float32[2D]")
g = ps.fields("g: float16[2D]")
......@@ -485,13 +485,10 @@ h = sp.Symbol("h")
cfg = ps.CreateKernelConfig(
target=ps.Target.X86_AVX512,
default_dtype="float32",
cpu_optim=ps.CpuOptimConfig(
openmp=True,
vectorize=ps.VectorizationConfig(
assume_inner_stride_one=True
)
)
)
cfg.cpu.openmp.enable = True
cfg.cpu.vectorize.enable = True
cfg.cpu.vectorize.assume_inner_stride_one = True
assignments = [
ps.Assignment(
......
......@@ -42,10 +42,6 @@ Assignments are the fundamental components of pystencils kernels;
they are used both for assigning expressions to symbols
and for writing values to fields.
.. py:class:: pystencils.Assignment
Slightly monkey-patched version of `sympy.codegen.ast.Assignment`.
Assignments are combined and structured inside `assignment collections <pystencils.AssignmentCollection>`.
An assignment collection contains two separate lists of assignments:
......@@ -56,10 +52,9 @@ An assignment collection contains two separate lists of assignments:
into fields.
.. autosummary::
:toctree: generated
:nosignatures:
:template: autosummary/recursive_class.rst
pystencils.Assignment
pystencils.AssignmentCollection
......
......@@ -17,6 +17,12 @@ ignore_errors = False
[mypy-pystencils.jit.*]
ignore_errors = False
[mypy-pystencils.field]
ignore_errors = False
[mypy-pystencils.sympyextensions.typed_sympy]
ignore_errors = False
[mypy-setuptools.*]
ignore_missing_imports=true
......