Send pipes to multiple processes: fanout (EXPERIMENTAL)

Copies stdin and sends it to multiple pipelines, collating their output

Description

fanout allows the the output of one process to be copied to multiple pipelines (also known as vertices) in a one-to-many approach.

Where normal pipelines would follow the logic of having the output of process A connected to the input of process B. With fanout, you can connect that pipe to many processes (vertices) without worrying about any contentions reading from the pipe nor race conditions.

Each vertex runs in parallel from the moment fanout is invoked. And fanout will wait until all vertices have complete before fanout exits.

Data Types

By default, the datatype of stdout will match the datatype of stdin.

Merging Rules

By default, stdout of each vertex will merged according to the rules of that particular datatype and data structure.

Arrays

With arrays, fanout will preserve the order of the array based on the order of the vertices and NOT by the order in which a vertex completes.

This means that even if vertex B completes before vertex A, vertex B's output will still appear after vertex A in the array.

Maps

Maps in most document formats (eg JSON) do not preserve ordering. So this means vertex B could appear before vertex A, or even overwrite any values defined by the output of vertex A.

The order of precedence for when output from vertices are merged will still follow the order the vertices were defined in the parameters of fanout.

So vertex C and D could overwrite vertex B and A, and vertex B could only overwrite vertex A.

Concatenate

To combine the output of each vertex in fanout as a simple byte stream, use the --concat flag.

The output of each vertex will be written to fanout's stdout in the order in which the vertices were defined in fanout's parameters. So there is no risk of one vertex's output getting jumbled with the output of another vertex.

Usage

<stdin> -> fanout [ flags ] { code block } { code block } { code block } ... -> <stdout>

<stdin> -> fanout [ flags ] --parse {
  { code block }
  { code block }
  { code block }
  ...
} -> <stdout>

See below for a list of optional flags.

Examples

Merge (default behavior)

Arrays

The following example will merge the output of four different vertices into one JSON array:

» dag --datatype json { %[four] } { %[two] } { %[one] } { %[three] }
[
    "four",
    "two",
    "one",
    "three"
]

Maps

The following example will merge the output of four different vertices into one JSON object:

» dag --datatype json { %{4: four} } { %{2: two} } { %{1: one} } { %{3: three} }
{
    "1": "one",
    "2": "two",
    "3": "three",
    "4": "four"
}

Concatenate

Compare the file size of Murex after it has been deflated by different compression algorithms:

» open $MUREX_EXE -> fanout --concat --parse {
  { printf 'File type:\t' }
  { -> file - }

  { printf 'Original size:\t' }
  { -> wc -c }

  { printf 'xz:\t\t' }
  { -> xz -c | wc -c }

  { printf 'bzip2:\t\t' }
  { -> bzip2 -c | wc -c }

  { printf 'gzip:\t\t' }
  { -> gzip -c | wc -c }
}

Flags

• --concat Combines the output of each vertex as a byte stream
• --datatype Defines the datatype for fanout's stdout. Overriding what would otherwise be defined by the stdin for fanout
• --parse Tells fanout to expect one parameter and to parse that for each vertex. The parameter must be surrounded by curly braces: { and }. This flag allows for readability improvements where fanout vertices can occupy dedicated lines without the need for hacks like dangling escapes (ie \\LF)
• -c alias for --append
• -p alias for --parse
• -t alias for --datatype

Detail

fanout is inspired by the theory of directed acyclic graphs (DAG) where:

[DAG]s consist of vertices and edges (also called arcs), with each edge directed from one vertex to another, such that following those directions will never form a closed loop. A directed graph is a DAG if and only if it can be topologically ordered, by arranging the vertices as a linear ordering that is consistent with all edge directions.

(source: Wikipedia)

See Also

• Create Named Pipe: pipe: Manage Murex named pipes
• Read / Write To A Named Pipe: <pipe>: Reads from a Murex named pipe

---

This document was generated from builtins/core/dag/fanout_doc.yaml.