3.5. Templating#

The control center YAML files are equipped with advanced templating capabilities for dynamic generation and synchronization of configurations at runtime. This enables Continuous Configuration Automation, eliminates data redundancy, and ensures consistency across all project resources. These features are similar to Jinja templating used by other tools (e.g. in conda build recipes) and have some overlaps with YAML anchors and references. However, PyPackIT implements its own templating engine, which offers much more features and flexibility for this use case. For example, PyPackIT templates have access to all other control center configuration values, and can recursively reference and use other templates. They can use JSONPath expressions for complex queries, and allow for the execution of arbitrary Python code to generate values. Moreover, Jinja-templated YAML files are generally not valid YAML before rendering, since template elements may break the YAML data structure. Consequently, these files cannot be viewed, modified, or processed as normal YAML files, complicating their maintenance. In contrast, PyPackIT’s templating syntax is designed to maintain a valid YAML data structure.

3.5.1. Syntax and Behaviour#

Templates can be used in place of any key, value, or sequence element in a YAML file. Similar to Jinja, templates are surrounded by delimiters that denote the beginning and the end of the template. There are four kinds of templates, each with its own start and end delimiters:

Reference templates use ${{ and }}$ delimiters.
Query templates use $[[ and ]]$ delimiters.
Code templates use #{{ and }}# delimiters.
Unpacking templates use *{{ and }}* delimiters.

There must always be at least one space between the template content and each of the delimiters. For example, let ... be a placeholder for the template content. Then, ${{ ...}}$ is not valid and will be treated as a string, whereas ${{ ... }}$ is a valid template. Any other extra whitespace characters (i.e., newlines and tabs) between the content and delimiters are allowed as well.

Example

Consider a template ${{ ... }}$ (where ... is some arbitrary content) that generates the string "generated_value":

Usage as a key in a mapping:
Code 3.5.1.1 Input YAML#
```
${{ ... }}$: some_value
```
Code 3.5.1.2 Output YAML#
```
generated_value: some_value
```
Usage as a value in a mapping:
Code 3.5.1.3 Input YAML#
```
some_key: ${{ ... }}$
```
Code 3.5.1.4 Output YAML#
```
some_key: generated_value
```
Usage as an element in a sequence:
Code 3.5.1.5 Input YAML#
```
- ${{ ... }}$
```
Code 3.5.1.6 Output YAML#
```
- generated_value
```

Templates can also generate any valid YAML data structure, not just strings.

Example

For example, the template may return the sequence [1, 2, 3], in which case the above examples will generate the following outputs, respectively:

Usage as a key in a mapping:
Code 3.5.1.7 Output YAML#
```
[1, 2, 3]: some_value
```
Usage as a value in a mapping:
Code 3.5.1.8 Output YAML#
```
some_key: [1, 2, 3]
```
Usage as an element in a sequence:
Code 3.5.1.9 Output YAML#
```
- [1, 2, 3]
```

Templates are evaluated recursively, so that they can depend on other templates. Before resolving any template, PyPackIT first resolves all templates that are referenced in the current template. You only need to make sure not to create circular references, otherwise you will get an error informing you of the circular path.

3.5.2. Reference Templates#

Reference templates are used to reuse configurations. They use the syntax ${{ <JSONPath> }}$ where <JSONPath> is a JSONPath expression (without the leading $.) pointing to a value in the control center configurations. to extract values from anywhere in the control center configurations.

Example

Reference templates are already heavily used as default values for many configurations in the control center files. For example, the URL of your documentation website’s homepage is defined at $.web.url.home and then reused in several other places, such as the About section of your GitHub repository, which can be set at $.repo.homepage. Looking at the default configuration file .control/vcs.yaml, you can see the following segment, which sets the value of $.repo.homepage to the value of $.web.url.home:

Code 3.5.2.1 .control/vcs.yaml#

repo:
  homepage: ${{ web.url.home }}$

3.5.2.1. Nesting#

Sometimes, the JSONPath you wish to query may itself contain variable parts defined elsewhere in your configurations. Such parts can be replaced with another reference template. PyPackIT will then first resolve these nested templates to build the final JSONPath of the parent template. For each nesting level, you must add an extra { and } to the start and end delimiters, respectively.

Example

Assume you want to get the full name of a team member whose ID is stored under a custom key at $.__data__.target_member_id:

__data__:
  target_member_id: member_1

Your template must then be:

${{ team.${{{ __data__.target_member_id }}}$.name.full }}$

This is first resolved to ${{ team.member_1.name.full }}$ , which will then return the full name of the member.

Further nesting levels are also allowed. For example, assume the following configurations:

Code 3.5.2.2 .control/vcs.yaml#

__data__:
  first_member_id: member_1
  second_member_id: member_2
  target_member_key: first_member_id

Here, first_member_id and second_member_id contain two different member IDs, and target_member_key tells you which one to choose. To get the corresponding member’s full name, you can use the following template:

${{ team.${{{ __data__.${{{{ __data__.target_member_key }}}} }}}$.name.full }}$

This will first resolve to:

${{ team.${{{ __data__.first_member_id }}}$.name.full }}$

then to ${{ team.member_1.name.full }}$ , and finally it returns the full name of the member.

3.5.3. Query Templates#

JSONPath expressions are queries designed to return a sequence of zero or more nodes that match the specified expression, regardless of whether or not that expression specifically points to a single node. In other words, even simple JSONPath expressions like $.path.to.some.node always return a sequence of values. When using reference templates, PyPackIT checks the length of the returned sequence; if it contains only one node, then that node is returned, otherwise the entire sequence is returned. This fails in cases where your query is meant to return a sequence, but it ends up matching only a single node. In such cases, the reference template will incorrectly resolve to the node value, whereas a sequence of node values was expected. Query templates solve this issue by always returning a sequence, regardless of the number of matched nodes. They can be used in place of reference templates where the query must return a sequence, but may end up matching only one node.

Example

Assume you need a list of your project members’ full names for some configuration. You can generate this list using the JSONPath expression $.team.*.name.full. However, if use a reference template ${{ team.*.name.full }}$ and your team has only one member, then you will get a string (i.e. the full name of the only member) instead of a list. Replacing the reference template with the corresponding query template $[[ team.*.name.full ]]$ ensures that the template always resolves to a sequence, even when there is only one member.

You can also use nested reference templates inside query templates the same way they are used inside reference templates.

3.5.4. Code Templates#

Code templates are the most powerful type of template, allowing you to execute Python code to generate a value. They use the syntax #{{ <CODE> }}# where <CODE> corresponds to a Python function body, i.e., any valid Python code ending with a return statement. For example, #{{ return 1 + 1 }}# resolves to the integer 2.

3.5.4.1. JSONPath Queries#

Code templates are a superset of reference and query templates, meaning they can also use JSONPath expressions to query other parts of the control center configurations. For this, a function named get is provided to the local environment in which the code is executed.

JSONPath Resolver Function

get(path: str, default: Any = None, search: bool = False) → Any#

Resolve a JSONPath expression in control center configurations.

Parameters:

path – JSONPath expression to resolve. The JSONPath syntax is the same as described in reference and query templates.
default – Default value to return if no matches are found. By default, None is returned when search is set to False, otherwise, an empty list is returned.
search – Always return a list. Setting this to True will make the resolution work in the same way as query references.

Therefore, any reference template ${{ <JSONPath> }}$ can be expressed as an equivalent code template #{{ return get("<JSONPath>") }}#, and any query template $[[ <JSONPath> ]]$ can be expressed as #{{ return get("<JSONPath>", search=True) }}#.

3.5.4.2. Helper Variables#

Several variables are made available to code templates:

repo_path: pathlib.Path#: Current absolute path to the repository directory on the local machine. This can be used for example to access files in the repository.

ccc: dict#: Current content (i.e., before synchronization) of the metadata.json file. This can be used for example to check whether a configuration has changed.

changelog: controlman.changelog_manager.ChangelogManager#

A changelog manager with three properties:

property contributor: dict#: The contributor.json data structure containing information about the project’s external contributors.

property current_public: dict#: A changelog mapping corresponding to the latest entry in the changelog.json file with a type other than local.

property last_public: dict#: A changelog mapping corresponding to the second-latest entry in the changelog.json file with a type other than local.

hook#: An InlineHooks instance, if defined (see Using a Python File below).

3.5.4.3. Helper Functions#

In addition to the get function, several helper functions are also made available to code templates:

team_members_with_role_ids(role_ids: str | Sequence[str], active_only: bool = True) → list[dict]#

Get team members with specific role IDs.

Parameters:

role_ids – Role ID(s) to filter for, as defined in [$.role]. This can be either a single role ID (as a string) or a sequence of role IDs.
active_only – Only return team members who are active. Default is True.

Returns:

A list of dictionaries (i.e., entity mappings) corresponding to selected team members. Members are sorted according to their priority (highest first) in the given role. If multiple role_ids are provided, the highest priority between all roles is selected for each member. Members with the same priority are sorted alphabetically by their last and first names, in that order.

team_members_with_role_types(role_types: str | Sequence[str], active_only: bool = True) → list[dict]#

Get team members with specific role types.

Parameters:

role_types – Role type(s) to filter for, as defined in [$.role.*.type]. This can be either a single role type (as a string) or a sequence of role types.
active_only – Only return team members who are active. Default is True.

Returns:

A list of dictionaries (i.e., entity mappings) corresponding to selected team members. Members are sorted according to their priority (highest first) in the given role. If multiple role_types are provided, the highest priority between all roles is selected for each member. Members with the same priority are sorted alphabetically by their last and first names, in that order.

team_members_without_role_types(role_types: str | Sequence[str], include_other_roles: bool = True, active_only: bool = True) → list[dict]#

Get team members without specific role types.

Parameters:

role_types – Role type(s) to filter out, as defined in [$.role.*.type]. This can be either a single role type (as a string) or a sequence of role types.
include_other_roles – Whether to include team members that have roles other than the excluded role types
active_only – Only return team members who are active. Default is True.

Returns:

A list of dictionaries (i.e., entity mappings) corresponding to selected team members.

fill_entity(entity: dict) → tuple[dict, dict | None]:#

Fill all missing information for a person, using GitHub API.

Parameters:: entity – The entity mapping representing the person. It must at least contain a GitHub ID.
Returns:: A 2-tuple where the first element is the same entity input dictionary with all available information filled in-place. Note that already defined values will not be replaced. The second tuple element is the raw GitHub user API response.

slugify(string: str, reduce: bool = True) → str:#

Convert a string to a URL-friendly slug. This performs unicode-normalization on the string, converts it to lowercase, and replaces any non-alphanumeric characters with hyphens.

Parameters:

string – The string to slugify.
reduce – If set to True (default), consecutive sequences of hyphens (after replacing non-alphanumeric characters) are reduced to a single hyphen, and any leading and trailing hyphens are stripped.

3.5.4.4. Dependencies#

If your code templates depend on modules not included in the standard library, you can declare them in the requirements.txt file of your control center’s hooks directory. PyPackIT will pip install -r the requirements file during each synchronization event, so that you can import those dependencies in your code templates.

3.5.4.5. Using a Python File#

Maintaining long code templates in YAML files is cumbersome, as they are not processed by IDEs and cannot be easily tested, refactored, or formatted. PyPackIT allows you to write your template codes in a separate Python file, which can then be used in any code template inside YAML files. These reusable code components must be added to a class named Hooks inside a file named cca_inline.py located in the control center’s hooks directory. By default, this class is added to your repository at .control/hooks/cca_inline.py, where it already contains several methods that are used in your default configuration files. You can thus simply add new methods to this class, to be used within your code templates. During synchronization, if this class exists, PyPackIT will automatically instantiate it and make it available to code templates as a variable named hook.

3.5.5. Unpacking Templates#

Sometimes, instead of templating an entire sequence, you may want to insert elements at a certain position, or concatenate the sequence with another. For example, assume you have the following project keywords:

keywords:
  - first main keyword
  - second main keyword

You also have another sequence, containing some extended keywords:

__data__:
  extended_keywords:
    - first extended keyword
    - second extended keyword

If you wish to add your main keywords to the extended list, you cannot simply use a reference or code template like:

__data__:
  extended_keywords:
    - ${{ keywords }}$
    - first extended keyword
    - second extended keyword

as that will incorrectly resolve to:

__data__:
  extended_keywords:
    - - first main keyword
      - second main keyword
    - first extended keyword
    - second extended keyword

Instead, you must wrap the reference template inside an unpacking template:

__data__:
  extended_keywords:
    - *{{ ${{ keywords }}$ }}*
    - first extended keyword
    - second extended keyword

which will correctly resolve to:

__data__:
  extended_keywords:
    - first main keyword
    - second main keyword
    - first extended keyword
    - second extended keyword

Similarly, you can also wrap query and code templates inside unpacking templates. How this works is that PyPackIT first resolves the nested template that is wrapped by the unpacking template, and will then iterate over the resolved value and insert elements one after another at the template’s index. Note that if the nested template returns an empty sequence, nothing will be added. This can also be useful if want to add a single element conditionally.

3.5.6. String Formatting#

Templates can also be used as a part of any string, similar to how Python f-strings work. In this case, the returned values of the templates are always first cast to a string. For unpacking templates, PyPackIT first iterates over the returned value, casts each element to a string, and joins them via “, “ delimiters.

Example

Assume the following configurations:

name: MyProject
team:
  member_1:
    name:
      first: Jane
      last: Doe
  member_2:
    name:
      first: John
      last: Doe

The template defined at $.__data__.project_info:

__data__:
  project_info: >-
    The project ${{ name }}$
    has #{{ return len(get("team")) }}# members;
    their names are: *{{ $[[ team.*.name.full ]]$ }}*

will then resolve to:

__data__:
  project_info: >-
    The project MyProject
    has 2 members;
    their names are: Jane Doe, John Doe

For more complex string compositions, you can instead use a code template that returns the entire string.

3.5.7. Relative Paths#

There may be cases where the absolute JSONPath of a configuration you want to use in a template is unknown or unstable. For example, assume you have a sequence of mappings where some value in each mapping needs to be templated against other values in the same mapping. You can of course use absolute paths and reference each mapping by its sequence index:

__data__:
  dependent_mappings:
    - a: 1
      b: true
      c: >-
        This value depends on
        ${{ __data__.dependent_mappings[0].a }}$ and
        ${{ __data__.dependent_mappings[0].b }}$.
    - a: 2
      b: false
      c: >-
        This value depends on
        ${{ __data__.dependent_mappings[1].a }}$ and
        ${{ __data__.dependent_mappings[1].b }}$.

However, you would then need to update all templates when you insert or remove an element, since indices would change. To simplify such cases, PyPackIT extends the JSONPath syntax to enable using relative paths. These are resolved relative to the path of the value where the template is defined, as follows:

When a JSONPath starts with one or more periods (.), it is considered a relative path.
One period refers to the path of the complex data structure (i.e., mapping or sequence) containing the value with the template. That is, if the template is used in the value of a key in a mapping, . refers to the JSONPath of that mapping. Similarly, if the template is used in the value of an element in a sequence, . refers to the JSONPath of that sequence.
Each additional period refers to the JSONPath of the parent complex data structure of the previous JSONPath, following the same logic.

Therefore, the above example can be rewritten as:

__data__:
  dependent_mappings:
    - a: 1
      b: true
      c: >-
        This value depends on
        ${{ .a }}$ and
        ${{ .b }}$.
    - a: 2
      b: false
      c: >-
        This value depends on
        ${{ .a }}$ and
        ${{ .b }}$.

Now, the templates will always resolve to the values within the same mapping, regardless of the indices. However, there still remains the problem of redundancy, as all elements now have the exact same template. This is where the __temp__ key comes in play. As mentioned earlier, relative paths in templates defined under __temp__ are resolved against the path where that template is referenced, not where it is defined. This means you can further simplify the above example to:

__temp__:
  c: >-
    This value depends on
    ${{ .a }}$ and
    ${{ .b }}$.
__data__:
  dependent_mappings:
    - a: 1
      b: true
      c: ${{ __temp__.c }}$
    - a: 2
      b: false
      c: ${{ __temp__.c }}$

3.5.7.1. Referencing Keys and Indices#

Relative paths can also access the key names of parent mappings or the index values of parent sequences. This is done by adding the __key__ field to end of a relative path. For example, the following template:

__temp__:
  location: >-
    This template is located in
    a mapping named ${{ .__key__ }}$,
    which is at index ${{ ..__key__ }}$
    of a sequence under ${{ ...__key__ }}$
__data__:
  grandparent_sequence:
    - parent_mapping:
        template: ${{ __temp__.location }}$

will resolve to:

__temp__:
  location: >-
    This template is located in
    a mapping named '${{ .__key__ }}$',
    which is at index ${{ ..__key__ }}$
    of a sequence under '${{ ...__key__ }}$'.
__data__:
  grandparent_sequence:
    - parent_mapping:
        template: >-
          This template is located in
          a mapping named 'parent_mapping',
          which is at index 0
          of a sequence under 'grandparent_seqeunce'.