The Newton release of Heat adds support for a yaql intrinsic function, which allows you to evaluate yaql expressions in your Heat templates. Unfortunately, the existing yaql documentation is somewhat limited, and does not offer examples of many of yaql's more advanced features.

I am working on a Fluentd composable service for TripleO. I want to allow each service to specify a logging source configuration fragment, for example:

parameters:
  NovaAPILoggingSource:
    type: json
    description: Fluentd logging configuration for nova-api.
    default:
      tag: openstack.nova.api
      type: tail
      format: |
        /(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d+) (?<pid>\d+) (?<priority>\S+) (?<message>.*)/
      path: /var/log/nova/nova-api.log
      pos_file: /var/run/fluentd/openstack.nova.api.pos

This generally works, but several parts of this fragment are going to be the same across all OpenStack services. I wanted to reduce the above to just the unique attributes, which would look something like:

parameters:
  NovaAPILoggingSource:
    type: json
    description: Fluentd logging configuration for nova-api.
    default:
      tag: openstack.nova.api
      path: /var/log/nova/nova-api.log

This would ultimately give me a list of dictionaries of the form:

[
  {
    "tag": "openstack.nova.api",
    "path": "/var/log/nova/nova-api.log"
  },
  {
    "tag": "openstack.nova.scheduler",
    "path": "/var/log/nova/nova-scheduler.log"
  }
]

I want to iterate over this list, adding default values for attributes that are not explicitly provided.

The yaql language has a select function, somewhat analagous to the SQL select statement, that can be used to construct a new data structure from an existing one. For example, given the above data in a parameter called sources, I could write:

outputs:
  sources:
    yaql:
      data:
        sources: {get_param: sources}
      expression: >
        $.data.sources.select({
          'path' => $.path,
          'tag' => $.tag,
          'type' => $.get('type', 'tail')})

This makes use of the .get method to insert a default value of tail for the type attribute for items that don't specify it explicitly. This would produce a list that looks like:

[
    {
        "path": "/var/log/nova/nova-api.log",
        "tag": "openstack.nova.api",
        "type": "tail"
    },
    {
        "path": "/var/log/nova/nova-scheduler.log",
        "tag": "openstack.nova.scheduler",
        "type": "tail"
    }
]

That works fine, but what if I want to parameterize the default value such that it can be provided as part of the template? I wanted to be able to pass the yaql expression something like this...

outputs:
  sources:
    yaql:
      data:
        sources: {get_param: sources}
        default_type: tail

...and then within the yaql expression, insert the value of default_type into items that don't provide an explicit value for the type attribute.

This is trickier than it might sound at first because within the context of the select method, $ is bound to the local context, which will be an individual item from the list. So while I can ask for $.path, there's no way to refer to items from the top-level context. Or is there?

The operators documentation for yaql mentions the "context pass" operator, ->, but doesn't provide any examples of how it can be used. It turns out that this operator will be the key to our solution. But before we look at that in more detail, we need to introduce the let statement, which can be used to define variables. The let statement isn't mentioned in the documentation at all, but it looks like this:

let(var => value, ...)

By itself, this isn't particularly useful. In fact, if you were to type a bare let statement in a yaql evaluator, you would get an error:

yaql> let(foo => 10, bar => 20)
Execution exception: <yaql.language.contexts.Context object at 0x7fbaf9772e50> is not JSON serializable

This is where the -> operator comes into play. We use that to pass the context created by the let statement into a yaql expression. For example:

yaql> let(foo => 10, bar => 20) -> $foo
10
yaql> let(foo => 10, bar => 20) -> $bar
20

With that in mind, we can return to our earlier task, and rewrite the yaql expression like this:

outputs:
  sources:
    yaql:
      data:
        sources: {get_param: sources}
        default_type: tail
      expression: >
        let(default_type => $.data.default_type) ->
        $.data.sources.select({
          'path' => $.path,
          'tag' => $.tag,
          'type' => $.get('type', $default_type)})

Which will give us exactly what we want. This can of course be extended to support additional default values:

outputs:
  sources:
    yaql:
      data:
        sources: {get_param: sources}
        default_type: tail
        default_format: >
          /some regular expression/
      expression: >
        let(
          default_type => $.data.default_type,
          default_format => $.data.default_format
        ) ->
        $.data.sources.select({
          'path' => $.path,
          'tag' => $.tag,
          'type' => $.get('type', $default_type),
          'format' => $.get('format', $default_format)
        })

Going out on a bit of a tangent, there is another statement not mentioned in the documentation: the def statement lets you defined a yaql function. The general format is:

def(func_name, func_body)

Where func_body is a yaql expresion. For example:

def(upperpath, $.path.toUpper()) ->
$.data.sources.select(upperpath($))

Which would generate:

[
    "/VAR/LOG/NOVA/NOVA-API.LOG", 
    "/VAR/LOG/NOVA/NOVA-SCHEDULER.LOG"
]

This obviously becomes more useful as you use user-defined functions to encapsulate more complex yaql expressions for re-use.

Thanks to sergmelikyan for his help figuring this out.


Heat-kubernetes Demo with Autoscaling

Next week is the Red Hat Summit in Boston, and I'll be taking part in a Project Atomic presentation in which I will discuss various (well, two) options for deploying Atomic into an OpenStack environment, focusing on my heat-kubernetes templates.

As part of that presentation, I've put together a short ...

read more

Heat Hangout

Fri 05 September 2014 by Lars Kellogg-Stedman Tags openstack rdo heat

I ran a Google Hangout this morning on Deploying with Heat. You can find the slides for the presentation on line here, and the Heat templates (as well as slide sources) are available on github.

If you have any questions about the presentation, please feel free to ping me on ...

read more

Visualizing Heat stacks

Tue 02 September 2014 by Lars Kellogg-Stedman Tags openstack heat

I spent some time today learning about Heat autoscaling groups, which are incredibly nifty but a little opaque from the Heat command line, since commands such as heat resource-list don't recurse into nested stacks. It is possible to introspect these resources (you can pass the physical resource id of a ...

read more

Docker plugin bugs

Mon 01 September 2014 by Lars Kellogg-Stedman Tags openstack docker heat

This is a companion to my article on the Docker plugin for Heat.

While writing that article, I encountered a number of bugs in the Docker plugin and elsewhere. I've submitted patches for most of the issues I encountered:

Bugs in the Heat plugin

read more