5.5. Flowgraphs
Hammer has experimental support for flowgraph constructions, similar to tools like mflowgen.
Their intention is to simplify the way flows are constructed and ran in Hammer.
They can be imported via the hammer.flowgraph
module.
5.5.1. Construction
Flowgraphs are consist of a collection of Node
instances linked together via a Graph
instance.
Each Node
“pulls” from a directory to feed in inputs and “pushes” output files to another directory to be used by other nodes.
Node
instances are roughly equivalent to a single call to the hammer-vlsi
CLI tool, so they take in similar attributes:
The action being called
The tool used to perform the action
The pull and push directories
Any required input/output files
Any optional input/output files (the name of the first output file corresponds to the name of the JSON that the action emits)
A driver to run the node with; this enables backwards compatibility with hooks.
Options to specify steps within an action; this enables backwards compatibility with flow control.
start_before_step
start_after_step
stop_before_step
stop_after_step
only_step
A minimal example of a Node
:
from hammer.flowgraph import Node
test = Node(
action="foo",
tool="nop",
pull_dir="foo_dir",
push_dir="/dev/null",
required_inputs=["foo.yml"],
required_outputs=["foo-out.json"],
)
Each Node
has the ability to be “privileged”, meaning that a flow must start with this node.
This only occurs when a flow is being controlled using any of the steps.
5.5.2. Running a Flowgraph
Node
instances are linked together using an adjacency list.
This list can be used to instantiate a Graph
:
from hammer.flowgraph import Graph, Node
root = Node(
"foo", "nop", "foo_dir", "",
["foo.yml"],
["foo-out.json"],
)
child1 = Node(
"bar", "nop", "foo_dir", "bar_dir",
["foo-out.json"],
["bar-out.json"],
)
graph = Graph({root: [child1]})
Using the Hammer CLI tool, separate actions are manually linked via an auxiliary action, such as syn-to-par
.
By using a flowgraph, Graph
instances by default automatically insert auxiliary actions.
This means that actions no longer need to be specified in a flow; the necessary nodes are inserted by the flowgraph tool.
This feature can be disabled by setting auto_auxiliary
to False
in a Graph
.
A Graph
can be run by calling the run
method and passing in a starting node.
When running a flow, each Node
keeps an internal status based on the status of the action being run:
NOT_RUN
: The action has yet to be run.RUNNING
: The action is currently running.COMPLETE
: The action has finished.INCOMPLETE
: The action ran into an error while being run.INVALID
: The action’s outputs have been invalidated (e.g. inputs or attributes have changed).
The interactions between the statuses are described in the following state diagram:
Regardless of whether a flow completes with or without errors, the graph at the time of completion or error is returned, allowing for a graph to be “resumed” once any errors have been fixed.
5.5.3. Visualization
A flowgraph can be visualized in Markdown files via the Mermaid tool.
Calling a Graph
instance’s to_mermaid
method outputs a file named graph-viz.md
.
The file can be viewed in a site like Mermaid’s live editor or using Github’s native support.
The flowgraph below would appear like this:
from hammer.flowgraph import Graph, Node
syn = Node(
"syn", "nop",
os.path.join(td, "syn_dir"), os.path.join(td, "s2p_dir"),
["syn-in.yml"],
["syn-out.json"],
)
s2p = Node(
"syn-to-par", "nop",
os.path.join(td, "s2p_dir"), os.path.join(td, "par_dir"),
["syn-out.json"],
["s2p-out.json"],
)
par = Node(
"par", "nop",
os.path.join(td, "par_dir"), os.path.join(td, "out_dir"),
["s2p-out.json"],
["par-out.json"],
)
g = Graph({
syn: [s2p],
s2p: [par],
par: []
})
Here are the contents of graph-viz.md
after calling g.to_mermaid()
:
```mermaid
stateDiagram-v2
syn --> syn_to_par
syn_to_par --> par
```
Which would render like this:
Note that the separators have been changed to comply with Mermaid syntax.
5.5.4. Caveats
The flowgraph frontend has a number of caveats that prevent it from full parity with the current CLI tool:
If flows are constructed in a Python interactive session, then errors from the underlying tool do not propagate to the flowgraph and thus render the interactive session unusable. This can be worked around by embedding the flow in a Python script and running it from the command line.