Conceptually, we're dealing with a bunch of distributed processes (
dpN above). These distributed processes may be long-running (such as
dp5) or batch-oriented ones, for example
dp6. There are a number of situations where you want these distributed processes to communicate, very similar to what IPC enables you to do on a single machine. Now,
dnpipes are a simple mechanism to facilitate IPC between distributed processes. What follows is an interface specification as well as a reference implementation for
Interpret the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", "MAY NOT", and "OPTIONAL" in the context of this repo as defined in RFC 2119.
dnpipe is a distributed ordered queue (FIFO) of messages available to a number of participating distributed processes. A distributed process is a useful abstraction provided by systems such as DC/OS (for example a Marathon app or a Metronome job) or Kubernetes (ReplicaSet or a Job) that give a user the illusion that a service or application she is executing on a bunch of commodity machines (the cluster) behaves like one global entity while it really is a collection of locally executed processes. In DC/OS this locally executed process would be a Mesos task and in Kubernetes a pod.
dnpipes implementation MUST support the following operations:
push(TOPIC, MSG)… executed by a publisher, this writes the message
MSG <- pull(TOPIC)… executed by a subscriber, this reads a message from a
reset(TOPIC)… executed by either a publisher or consumer, this removes all messages from a
The following MUST be true at any point in time:
pushis executed by the publisher
MSGMUST be available for
pullto any participant until
resetis triggered and has completed.
pulldoes not remove a message from a
dnpipe, it merely delivers its content to the consumer.
- The way how participants discover a
dnpipeis outside of the scope of this specification.
Note concerning the last point: since there are many ways to implement service discovery in a distributed system we do not expect that an agreement can be found here hence we leave it up to the implementation how to go about it. The next sections shows an example using Kafka and DNS to achieve this.
dnpipe can be useful in a number of situations including but not limited to the following:
- To implement a work queue with, for example: Adron/testing-aws-sqs-worker
- To do event dispatching in microservices, for example: How we build microservices at Karma
- To coordinate Function-as-a-Service executions, for example: Integrate SQS and Lambda: serverless architecture for asynchronous workloads