ZeroMQ is an absolutely fantastic communications library that has allowed
me to very easily create networks/connections between multiple different
applications to communicate in an extremely fast yet directed manner and allow
scaling of the connected entities with almost no second thought.
I have used ZeroMQ within Python and C++, and it has been fantastic leaving
that part of the communication layer to a library rather than writing something
similar myself. However, embedding ZeroMQ into an another event loop has proven
to have some quirks. They are documented in the API, but not so much by other
users/examples found around the web, and caught me by surprise.
This post mainly goes over the issue, the difference between edge and level
triggered and how ZeroMQ works. I am hoping to spend some time documenting how
to embed ZeroMQ correctly inside libev at a later date.
The quirk
ZeroMQ allows you to get an underlying file descriptor (ZMQ_FD) for a ZeroMQ
socket using getsockopt(), this is not the actual ZeroMQ file descriptor
but a stand-in:
int fd = 0;
size_t fd_len = sizeof(fd);
zmq_socket.getsockopt(ZMQ_FD, &fd, &fd_len);
This file descriptor can then be passed on to poll(), select(), kqueue()
or whatever your event notification library of choice may be, the idea being
that if the file descriptor is notified to be available for reading then you
check the sockets ZMQ_EVENTS and find out if you are allowed to read or
write to the ZeroMQ socket, or even possibly neither (false positive). The same
file descriptor is also used internally within the ZeroMQ library for
notification and thus we may receive an event notification that we can't do
anything with.
As stated in the ZeroMQ API guide for getsockopt() under ZMQ_FD:
the ØMQ library shall signal any pending events on the socket in an
edge-triggered fashion by making the file descriptor become ready for
reading.
and further it states:
The ability to read from the returned file descriptor does not necessarily
indicate that messages are available to be read from, or can be written to,
the underlying socket; applications must retrieve the actual event state with
a subsequent retrieval of the ZMQ_EVENTS option.
So far so good, so after we return from our event notification on the file
descriptor we simply ask ZeroMQ whether or not we can read or write (or
possibly both):
int zevents = 0;
size_t zevents_len = sizeof(zevents);
zmq_socket.getsockopt(ZMQ_EVENTS, &zevents, &zevents_len);
if (zevents & ZMQ_POLLIN) {
// We can read from the ZeroMQ socket
}
if (zevents & ZMQ_POLLOUT) {
// We can write to the ZeroMQ socket
}
// If neither of the above is true, then it was a false positive
So now you are happily reading from ZeroMQ and using the data as you wish,
except you start to notice that sometimes it can take a little while for data
to show up, or that if the end-point you are connected to on ZeroMQ sends a lot
of data it is not delivered in a timely fashion. This is where the
documentation for edge triggered comes into play, and that is something that is
different about ZeroMQ compared to for example BSD sockets.
Edge triggered versus level triggered
Edge and level triggered refer to what kind of signal a device will output in
an attempt to gain the attention of another device. Edge and level triggered
have their history in electronics and hardware, where they play a very crucial
role.
For both level and edge triggered assume the data is represented by this simple string:
00000000000111111111111111111111100000000011111111111000000000
The zeroes (0) are where no data is available, and the ones (1) are where data
is available for processing.
Level triggered
Level triggered devices when they require attention, and until they no longer
require attention will signal with a high voltage (going back to early
electronics), and when they no longer require attention signal with a low
voltage.
Data: 00000000001111111111111111111111000000000011111111111000000000
Block: A B C D
+ 5v ____________________ _________
- __________| |_________| |__________
0v
State: 0 1 0 1 0
At this point, once the signal turns to 1 in block B we can start processing
data, and until we are done processing data the signal will stay 1. Once we are
done processing the device will signal that we are done by setting the state to
0 in block C, and the process can repeat itself with block D.
Edge triggered
Edge triggered devices when they require attention will simply turn on the
signal for a short period of time and then turn it back off until all of the
requireding processing is complete at which point they will once again signal
that they need processing.
Data: 00000000001111111111111111111111100000000011111111111000000000
Block: A B C D
+ 5v __ __
- __________| |____________________________| |________________
0v
State: 0 1 0 1 0
In this case when the trigger arrives you have to process all of the data
available. When block B arrives and we get 1 as our signal, we have to process
all of the data, we won't know when we are done other than some other
notification scheme by the device. (Such as a special message that says we are
done). Then when we are done we can go back to listening to waiting for a
trigger.
If however in the time that we are processing the data from the trigger at B
receive more data to process we won't get notified because we weren't
officially done processing data from the device, and thus the device will not
notify us of the new data again until we have processed all of the data.
There are some devices that as soon as you start processing their data, you
re-enable the trigger, so if you receive new data while still processing the
old you will get a new notification.
ZeroMQ uses edge triggered
When multiple message arrive for the ZeroMQ socket, one notification is sent.
You check to see if the ZMQ_EVENTS actually contains ZMQ_POLLIN and then
read a single message from ZeroMQ. Content that you have processed the data
ZeroMQ has for you, you add the file descriptor back to the event notification
library and wait. However, you know you sent two messages, but all you got was
one. The next time you send data to the socket, it returns the second message,
and you have now send three but the third doesn't seem to arrive.
This is where the edge triggered comes in. ZeroMQ only notifies you once, and
after it has notified you it won't notify you again until it receives new
messages, EVEN if you have messages waiting for you. In BSD sockets when you
recv() data from the socket, if it is not all of the data it will simply tell
you, and until all of the data has been read that the socket has to offer it is
going to tell you (level triggered).
This means that when you get a notification from ZeroMQ you have to use a loop
to process all of the messages that are coming your way:
int zevents = 0;
size_t zevents_len = sizeof(zevents);
zmq_socket.getsockopt(ZMQ_EVENTS, &zevents, &zevents_len);
do {
if (zevents & ZMQ_POLLIN) {
// We can read from the ZeroMQ socket
} else {
break;
}
// Check to see if there is more to read ...
zmq_socket.getsockopt(ZMQ_EVENTS, &zevents, &zevents_len);
} while (zevents & ZMQ_POLLIN);
if (zevents & ZMQ_POLLOUT) {
// We can write to the ZeroMQ socket
}
// If neither of the above is true, then it was a false positive
The same is true if you need to send multiple messages, you will need to check
ZMQ_EVENTS for ZMQ_POLLOUT and send messages so long as that holds true, as
soon as it is no longer true you will need to wait until you get triggered
again by using the file descriptor in your event notification library.
Embedding or co-existing with another event loop
Embedding ZeroMQ into a different event loop becomes more difficult when you
realise that it is edge triggered versus level triggered. If you want to allow
fair processing of data between events it becomes harder to accomplish with
ZeroMQ because you are required to process all of the data before going back to
your event loop or requires the setting up of different types of events
depending on the state that the ZeroMQ socket is in to make sure you process
all of the data and don't forget any.
Edge triggered for the ZeroMQ file descriptor was a design decision which
unfortunately made the use of ZeroMQ sockets more difficult and different from
BSD sockets which makes it harder for application programmers to do the right
thing. There are plenty of questions on the internet regarding the proper usage
of ZMQ_FD with various different event notification schemes/implementations.