puccini

Puccini

Deliberately stateless cloud topology management and deployment tools based on TOSCA.

Each tool is a self-contained executable file, allowing them to be easily distributed and easily embedded in toolchains, orchestration, and development environments. They are coded in 100% Go and are very portable, even runnable on WebAssembly.

Check out this live demo of Puccini running in a browser!

Puccini is also available as self-contained shared C libraries that can be called directly from many other languages. See wrappers for Java, Python, and Ruby.

Welcome, first timers! Check out the quickstart guide.

To build Puccini yourself see the build guide.

puccini-tosca

⮕ Documentation

Clout frontend for TOSCA. Parses a TOSCA service template and compiles it to Clout (see below).

Why TOSCA? It's a high-level language made for modeling and validating cloud topologies using reusable and extensible objects. It allows architects to focus on application design and requirements without being bogged down by the ever-changing specificities of the infrastructure.

Puccini can compile several popular TOSCA and TOSCA-like dialects: TOSCA 1.3 (in progress), TOSCA 1.2, TOSCA 1.1, TOSCA 1.0, as well as the more limited grammars of Cloudify DSL 1.3, and OpenStack Heat HOT 2018-08-31.

All these dialects are supported as straightforward YAML files, in the file system or hosted on URLs, as well as packaged in CSAR files. Puccini also includes a simple CSAR creation tool, puccini-csar.

puccini-tosca comes with TOSCA profiles for the Kubernetes and OpenStack cloud infrastructures, as well as BPMN processes. Profiles include node, capability, relationship, policy, and other types. Also included are detailed examples using these profiles to get you started. These profiles would work with any TOSCA-compliant product.

Puccini allows for the inclusion of JavaScript scriptlets in its Clout output (in the metadata section), as an option for self-contained orchestration integration. How do TOSCA, Clout, JavaScript, and cloud infrastructures all fit together in Puccini? Consider this: with a single command line you can take a TOSCA service template, compile it with puccini-tosca, execute JavaScript to generate Kubernetes specifications, then pipe those to kubectl, which will upload the specifications to a running Kubernetes cluster in order to be scheduled. Like so:

 puccini-tosca compile my-app.yaml --exec=kubernetes.generate | kubectl apply -f -

Et voilà, your abstract architectural design became a running deployment.

Note that puccini-js is not a requirement for your toolchain. You can process and consume the Clout output with your own tools.

Standalone Parser

⮕ Documentation

Puccini's TOSCA parser is available as an independent Go library. Its 5 phases do normalization, validation, inheritance, and assignment of TOSCA's many types and templates, resulting in a flat, serializable data structure that can easily be consumed by your program. Validation error messages are precise and useful. It's a very, very fast multi-threaded parser, fast enough that it can be usefully embedded in editors and IDEs for validating TOSCA while typing.

TOSCA is a complex object-oriented language. We put considerable effort into adhering to every aspect of the grammar, especially in regards to value type checking and type inheritance contracts, which are key to delivering the object-oriented promise of extensibility while maintaining reliable base type compatibility. Unfortunately, the TOSCA specification is famously inconsistent and imprecise. For this reason, the Puccini parser also supports quirk modes that enable alternative behaviors based on differing interpretations of the spec.

Compiler

The TOSCA-to-Clout compiler's main role is to take the parsed data structure and dump it into Clout. The next step in the toolchain (which could be puccini-js) would then connect the Clout to your orchestration systems: deploying to your platforms, on-boarding to a service catalog, etc. Thusly Clout functions as an "intermediate representation" (IR) for TOSCA.

By default the compiler also performs topology resolution, which attempts to satisfy requirements with capabilities, thus creating the relationships (Clout edges) between node templates. This feature can be turned off in order to add more processing phases before final resolution. Resolution is handled via the embedded tosca.resolve JavaScript.

Visualization

You can graphically visualize the compiled TOSCA in a dynamic web page. A one-line example:

puccini-tosca compile examples/tosca/requirements-and-capabilities.yaml --exec=assets/tosca/profiles/common/1.0/js/visualize.js > /tmp/puccini.html && xdg-open /tmp/puccini.html

The visualization scriptlet is not embedded by default into the Clout, but can be manually added via puccini-js put (see below) for added portability.

puccini-js

⮕ Documentation

Clout processor for JavaScript. Executes existing JavaScript scriptlets in a Clout file (in the metadata section). For example, it can evaluate TOSCA functions, apply constraints, execute Kubernetes specification generation, translate workflows to BPMN, etc.

The tool can also be used to add/remove scriptlets by manipulating the metadata section in the Clout.

Also supported are implementation-specific JavaScript "plugins" that allow you to extend existing scriptlet functionality without having to modify it. For example, you can add a plugin for Kubernetes to handle custom application needs, such as adding sidecars, routers, loadbalancers, etc. Indeed, Istio support is implemented as a plugin. You can also use puccini-js to add plugins to the Clout file, either storing them permanently or piping through to add and execute them on-the-fly.

For convenience, execution functionality is included in puccini-tosca via the --exec switch. These two commands are equivalent:

puccini-tosca compile my-app.yaml | puccini-js exec my.scriptlet
puccini-tosca compile my-app.yaml --exec=my.scriptlet

TOSCA Functions and Constraints

These are implemented as JavaScript scriptlets so that they can be embedded into the Clout and then be executed by puccini-js, allowing a compiled-from-TOSCA Clout file to be be used independently of its TOSCA source. The Clout lives on its own. The function calls are compiled as "stubs" that can be "coerced" into their evaluated values. This is done via the the tosca.coerce scriptlet:

puccini-js exec tosca.coerce my-clout.yaml

For convenience, this functionality is included in puccini-tosca via the --coerce switch. The following is identical to the above:

puccini-tosca compile --coerce my-clout.yaml

A useful side benefit of Puccini's implementation is it allows you to extend TOSCA by adding your own functions/constraints. Obviously, such custom functions are not part of the TOSCA specification and may be incompatible with other TOSCA implementations.

TOSCA Attributes

WORK IN PROGRESS

TOSCA attributes (as opposed to properties) represent "live" data in a running deployment. And the function get_attribute allows other values to make use of this live data. The implication is that the Clout should be updated to reflect changes to attribute. But also, attribute definitions in TOSCA allow you to define constraints on attributes, so we must also make sure that the new incoming data complies with them before allowing the change to occur.

Our solution has two steps. As an example, let's look at Kubernetes. First, we have JavaScript (kubernetes.update) that extracts these attributes from a Kubernetes cluster (by calling kubectl) and updates the Clout. Second, we run tosca.coerce, which not only calls TOSCA functions but also applies the constraints.

Putting it all together, let's refresh a Clout:

puccini-js exec kubernetes.update my-clout.yaml | puccini-js exec tosca.coerce > coerced-clout.yaml

TOSCA Operations, Notifications, Policy Triggers, and Workflows

WORK IN PROGRESS

In TOSCA, a workflow is grammar for describing a task graph, which is tightly coupled to the topology. It represents the "classical" orchestration paradigm, which procedurally (in serial and/or in parallel) executes individual self-contained tasks. When the workflow reaches a successful ending, there would be a new state for a service instance. Dealing with workflow failures is often painful, resulting in an indeterminate state for the service. It's not always practical to "roll back" the task graph or even reset to the original state. Thus, this paradigm is best avoided if possible.

The building blocks of workflows are "operations" (calling a single operation is the simplest workflow). Notifications and policy triggers are a related feature, as they specify an event or condition that could launch a workflow or an individual operation.

Puccini comes with three different implementations of these features:

1. For OpenStack, Puccini can generate Ansible playbooks that rely on the Ansible OpenStack roles. Custom operation artifacts, if included, are deployed to the virtual machines and executed. Effectively, the combination of TOSCA + Ansible provides an equivalent set of features to Heat/Mistral. However, Ansible is a general-purpose orchestrator that can do a lot more than Heat. The generated playbooks comprise roles that can be imported and used in other playbooks, allowing for custom orchestration integrations. Also note that although Puccini can compile HOT directly, we recommend TOSCA because of its much richer grammar and features.

2. Puccini's BPMN profile lets you generate BPMN2 processes from TOSCA workflows and policy triggers. These allow for tight integration with enterprise process management (called OSS/BSS in the telecommunications industry). The generated processes can also be included as sub-processes within larger business processes.

3. Kubernetes doesn't normally require workflows: its "scheduling" paradigm is a declarative alternative to the classical procedural orchestration paradigm. As it provides a truly cloud-native environment, Kubernetes applications are better off orchestrating themselves, for example by relying on operators to do the heavy lifting. Still, it could make sense to use workflows for certain externally triggered features. Puccini's solution is straightforward: it can generate an Ansible playbook that deploys TOSCA artifacts with kubectl cp and executes them with kubectl exec.

Clout

⮕ Documentation

Introducing the cloud topology ("clou" + "t") representation language, which is simply a representation of a generic graph database in YAML/JSON/XML.

Clout functions as the intermediary format for your deployments. As an analogy, consider a program written in the C language. First, you must compile the C source into machine code for your hardware architecture. Then, you link the compiled object, together with various libraries, into a deployable executable for a specific target platform. Clout is the compiled object in this analogy. If you only care about the final result then you won't see the Clout at all. However, the decoupling allows for a more powerful toolchain. For example, some tools might change your Clout after the initial compilation (to scale out, to optimize, to add platform hooks, debugging features, etc.) and then you just need to "re-link" in order to update your deployment. This can happen without requiring you to update your original source design. It may also possible to "de-compile" some cloud deployments so that you can generate a Clout without any TOSCA "source code".

Clout is essentially a big, unopinionated, implementation-specific dump of vertexes and the edges between them with un-typed, non-validated properties. Rule #1 of Clout is that everything and the kitchen sink should be in one Clout file. Really, anything goes: specifications, configurations, metadata, annotations, source code, documentation, and even text-encoded binaries. (The only exception might be that you might want to store security certificates and keys elsewhere.)

In itself Clout is an unremarkable format. Think of it as a way to gather various deployment specifications for disparate technologies in one place while allowing for the relationships (edges) between entities to be specified and annotated. That's the topology.

Clout is not supposed to be human-readable or human-manageable. The idea is to use tools (Clout frontends and processors) to deal with its complexity. We have some great ones for you here. For example, with Puccini you can use just a little bit of TOSCA to generate a single big Clout file that describes a complex Kubernetes service mesh.

If Clout's file size is an issue, it's good to know that Clout is usually eminently compressible, comprising just text with quite a lot of repetition.

Storage

Orchestrators may choose to store Clout opaquely, as is, in a key-value database or filesystem. This could work well because cloud deployments change infrequently: often all that's needed is to retrieve a Clout, parse and lookup data, and possibly update a TOSCA attribute and store it again. Iterating many Clouts in sequence this way could be done quickly enough even for large environments. Simple solutions are often best.

That said, it could also make sense to store Clout data in a graph database. This would allow for sophisticated queries, using languages such GraphQL and Gremlin, as well as localized transactional updates. This approach could be especially useful for highly composable and dynamic environments in which Clouts combine together to form larger topologies and even relate to data coming from other systems.

Graph databases are quite diverse in features and Clout is very flexible, so one schema will not fit all. Puccini instead comes with examples: see storing in Neo4j and storing in Dgraph.

FAQ

Can Puccini deploy my existing TOSCA to Kubernetes or OpenStack?

If you didn't plan it that way, then: no. Our TOSCA Kubernetes/OpenStack profiles do not make use of TOSCA's Simple Profile or Simple Profile for NFV types (Compute, BlockStorage, VDU, etc.). Still, if you find these so-called "normative" types useful, they are included in Puccini and will be compiled into Clout. You may bring in your own orchestration to deploy them to your cloud environments. But, we encourage you to consider carefully whether this is a good idea. We think it's a dead end.

The notion that a single set of normative types could be used for all the various cloud and container platforms out there is a pipe dream. There may be superficial similarities between them, but the devil is in the details and the amount of detail needed for integrated, scalable, cloud-native deployments keeps growing and diversifying. Thus every platform needs and deserves its own concepts, models, data points, and thus its own profile of interrelated TOSCA types. However, by bringing all these tiny-but-important implementation details into one place we can at least have a lingua franca and common toolchain for all platforms. That's the value proposition of TOSCA and the gist of Puccini.

Why Go?

Go is fast becoming the language of choice for cloud-native solutions. It has the advantage of producing very deployable executables that make it easy to containerize and integrate. Go features garbage collection and easy multi-threading (via lightweight goroutines), but unlike Python, Ruby, and Perl it is a strictly typed language, which encourages good programming practices and reduces the chance for bugs.

Why JavaScript?

JavaScript lets you manipulate the Clout data structures directly using a full-blown, conventional language. It's probably not anyone's favorite language, but it's familiar, mature, standardized (as ECMAScript), and does the job. From a certain angle it's essentially the Scheme language (because it has powerful closures and functions are first class citizens) but with a crusty C syntax.

And because JavaScript is self-contained text, it's trivial to store it in a Clout file, which can then be interpreted and run almost anywhere.

Our chosen ECMAScript engine is goja, which is 100% Go and does not require any external dependencies.

Is there an alternative to JavaScript if I just need to extract data from the Clout?

If the built-in JavaScript support is insufficient or unwanted, you can write your own custom YAML processor in Python, Ruby, etc., to do exactly what you need, e.g.:

puccini-tosca compile my-app.yaml | python myprocessor.py

Also check out yq, a great little tool for extracting YAML and even performing simple manipulations. Example:

puccini-tosca compile examples/tosca/requirements-and-capabilities.yaml | yq r - 'vertexes.(properties.name==light12)'

Can I use simple text templating instead of TOSCA functions and YAML processing?

Nothing is stopping you. You can pipe the input and output to and from the text translator of your choice at any point in the toolchain. Here's an example using gomplate:

puccini-tosca compile my-app.yaml | gomplate | puccini-js exec kubernetes.generate

Your TOSCA my-app.yaml can then include template expressions, such as:

username: "{{strings.ReplaceAll "\"" "\\\"" .Env.USER}}"

Note the proper escaping of quotation marks to avoid invalid YAML. Also, remember that indentation in YAML is significant, so it can be tricky to insert blocks into arbitrary locations. Generally, using text templating to manipulate YAML is not a great idea for these reasons.

Puccini's decision to use an embedded interpreted programming language (JavaScript) is intentional and important. Unlike some Kubernetes tools (Helm), we prefer not treat YAML files as plain text to be manipulated by an anemic text templating language. Perhaps in the future someone will create a templating language specifically for YAML that is designed to work with its special requirements.

If you insist on text templating, a useful convention for your toolchain could be to add a file extension to mark a file for template processing. For example, .yaml.j2 could be recognized as requiring Jinja2 template processing, after which the .j2 extension would be stripped.

Can I compose a single service from several interrelated Clout files?

TOSCA has a feature called "substitution mapping", which is useful for modeling service composition. It is, however, a design feature. The implementation is up to your orchestration toolchain. See our examples here and here.

Puccini intentionally does not support service composition. Each Clout file is its own universe. If you need to create edges between vertexes in one Clout file and vertexes in other Clout files, then it's up to you and your tools to design and implement that integration. The solution could be very elaborate indeed: the two Clouts might represent services with very different lifecycles, that run in different clouds, that are handled by different orchestrators. And the connection might require complex networking to achieve. There's simply no one-size-fits-all way Puccini could do it—namespaces? proxies? catalogs? repositories?—so it insists on not having an opinion.

TOSCA is so complicated! Help!

I know, right? Now imagine writing a parser for it... Not only is it a complex language, but the specification itself (as of version 1.2) has many contradictions, errors, and gaps.

Please join OASIS's TOSCA community to help improve the language!

Meanwhile, we've included examples of TOSCA core grammatical features, with some running commentary. Treat them as your playground. Also, if you have 4 hours to spare, grab some snacks, get comfortable, and watch the author's free online course for TOSCA 1.0: part 1, part 2.

(Author's note: This is my second take at writing a TOSCA parser. The first was AriaTosca, an incubation project under the Apache Software Foundation. I am grateful to Cloudify for funding much of the AriaTosca project. Note, however, that Puccini is a fresh start initiated by myself with no commercial backing. It does not use AriaTosca code and has a radically different architecture as well as very different goals.)

Why is it called "Puccini"?

Giacomo Puccini was the composer of the Tosca opera (based on Victorien Sardou's play, La Tosca), as well as La bohème, Madama Butterfly, and other famous works. The theme here is orchestration, orchestras, composition, and thus operas. Capiche?

How to pronounce "Puccini"?

For a demonstration of its authentic 19th-century Italian pronunciation see this clip.