murex Shell Docs

User Guide: Pipeline

Overview of what a “pipeline” is

In the murex docs you’ll often see the term “pipeline”. This refers to any commands sequenced together.

A pipeline can be joined via any pipe token (eg |, ->, =>, ?). But, for the sake of documentation, a pipeline might even be a solitary command.

Examples

Typical murex pipeline:

open: example.json -> [[ /node/0 ]]

Example of a single command pipeline:

top

Pipeline you might see in Bash / Zsh (this is also valid in murex):

cat names.txt | sort | uniq

Pipeline filtering out a specific error from example-cmd

example-cmd ? grep: "File not found"

Detail

A pipeline isn’t a murex specific construct but rather something inherited from Unix. Where murex differs is that it can support sending typed information to compatible functions (unlike standard Unix pipes which are dumb-byte streams).

Wikipedia has a page on Pipeline (Unix):

In Unix-like computer operating systems, a pipeline is a mechanism for inter-process communication using message passing. A pipeline is a set of processes chained together by their standard streams, so that the output text of each process (stdout) is passed directly as input (stdin) to the next one. The second process is started as the first process is still executing, and they are executed concurrently. The concept of pipelines was championed by Douglas McIlroy at Unix’s ancestral home of Bell Labs, during the development of Unix, shaping its toolbox philosophy. It is named by analogy to a physical pipeline. A key feature of these pipelines is their “hiding of internals” (Ritchie & Thompson, 1974). This in turn allows for more clarity and simplicity in the system.

Named Pipes

The drawback with pipes is that it assumes each command runs sequentially one after another and that everything fits neatly into the concept of “output” and “errors”. The moment you need to use background (bg) processes, do anything more specific with data streams (even if just ignore them entirely), or use more than one data stream, then this concept breaks down. This is where named pipes come to the rescue. Named pipes are out of scope for this specific document but you can read more on them in links the links below.

See Also

This site's content is rebuilt automatically from murex's source code after each merge to the master branch. Downloadable murex binaries are also built with the website.

Last built on Thu Oct 13 08:08:18 UTC 2022 against commit ed6b6fced6b6fc609f4be93c5d3348695a762facfecf48f.

Current version is 2.11.2200 which has been verified against 16798 tests cases.