Documenting Problems That Were Difficult To Find The Answer To

ZeroMQ – What I Want


I have begun experimenting with the Perl CPAN module ZeroMQ-0.20 with a locally compiled zeromq-2.1.11 C library on Ubuntu 11.04 (natty).

This is my first attempt to delve into the world of ZeroMQ.

In my mind I have a goal of a design like the following:

  clients                        workers
   _____       ________           _____
  | REQ |---->| SWITCH |-------->| REP |
  |_____|     |        |       ->|_____|
         \ -->|        |-     /
          X   |        | \   /
         | \->|________|  \ /
   _____/  /           \   X      _____
  | REQ | ||            \ / ---->| REP |
  |_____| ||             X   --->|_____|
         \| \   ________/ \ /
          X  ->| SWITCH |  X
         / \   |        | / \
   _____/   -->|        |-   \    _____
  | REQ |      |        |     -->| REP |

.. where I can have an arbitrary number of clients and workers. Workers can connect and disconnect at any time (as can clients). This requirement for dynamic connections and disconnections appears to cause the ZeroMQ library enormous grief as it has no way of informing the user program that a connection has been lost (and that a message may never be successfully processed).

Switch Implementation

Implementation of the switch is done by using ROUTER sockets. Why? Because only ROUTER sockets record who sent the message – and pass it onto the user program. The REP socket actually records who sent a message but does not make this information available to the user program.

How is sender information passed to the user program for a ROUTER socket? It is done through sub-parts of the request. A message coming from a ROUTER socket appears as follows:

  1. sender from which THIS message was received
  2. sender before that
  3. sender before that
  n. "" (empty message)
  n+1. first data part
  n+2. second data part

For other sockets (except DEALER) the sender information is hidden from the user program. Only the data parts are handed to the user program.

Note that sender information does not contain an IP address or anything like that – it is a unique code that the ZeroMQ internal library uses to track connections.

Perl Code to Fetch Message Parts

I wrote the following code to fetch an entire message and split it up into its parts from a ROUTER socket in Perl:

use Exporter; # ZeroMQ complains if this isn't included

use ZeroMQ;
use ZeroMQ::Constants;

sub get_msg {
    my ( $zsock ) = @_;

    my @addr = (); # stack of sender info
    my $header = 1;
    my @data = (); # data parts of message (at least 1)

    while ( 1 ) {
        my $msg = $zsock->recv();
        my $optval = $zsock->getsockopt(
        my $str = $msg->data();
        if ( length( $str ) == 0 ) {
            $header = 0; # data follows

        if ( $header ) {
            push( @addr, $str );
        } else {
            push( @data, $str );

        last if ( ! $optval ); # no more parts
    return( \@addr, \@data );

ZeroMQ Suggested LRU (Least Recently Used) Method

In the ZeroMQ Guide about Message Broking it is suggested the workers use REQ sockets to initiate a “READY” message to the switch/router/broker and have the broker track which workers are ready to receive messages.


What if a worker is killed (deliberately, though a software bug, or a hardware issue such as loss of network or power failure)? Well – the switch/router/broker will never know. There is no technique in ZeroMQ to inform the user program that a connection has been terminated.

A heartbeat could be used – but this also has problems. For one – a separate heartbeat connection will appear to have a different source header – making it impossible to associate the heartbeat with a currently connected REQ socket.

In short ZeroMQ has serious short comings. Besides being unable to detect connection failure ZeroMQ additionally kills an entire program if a bind() attempt fails – every other known socket library politely informs the user program when this happens.

Another annoyance is that a ROUTER socket is the only way to track routing information for a received message. There may be situations where a REP socket may wish to know where the request came from.


Perhaps there needs to be another library written. I would want the following features:

  • ability to poll for ZMQ_POLLERR (detect connection failure)
  • error code instead of crash for bind error
  • ability to extract sender information from ANY socket type

This is not a fully considered list but I may update this entry from time to time.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: