funcptr - cpp

Build numbers in binaries using waf

2014-06-10T17:32:03+00:00

My build system of choice these days for any C++ project is waf</a>. One of the things I always like havig is the build number included in the final binary, so that with a simple ./binary --version</code> or even ./binary</code> the version is printed that it was built from. This can make it much simpler to debug any potential issues, especially if fixes may have already been made but a bad binary was deployed.</p>

Setup the wscript</h2>
Make sure that your wscript somewhere near the top contains the following:</p>

APPNAME</span> =</span> '</span>myapp</span>'</span></span>
VERSION</span> =</span> '</span>0.0.0</span>'</span></span></code></pre>
Then in your configure(cfg)</code> add the following:</p>
cfg.env.</span>VERSION</span> =</span> VERSION</span></span>
cfg.env.</span>APPNAME</span> =</span> APPNAME</span></span>
</span>
git_version</span> =</span> try_git_version</span>()</span></span>
</span>
if</span> git_version:</span></span>
    cfg.env.</span>VERSION</span> +=</span> '</span>-</span>'</span> +</span> git_version</span></span></code></pre>
The try_git_version()</code> function is fairly simple and looks like this:</p>
def</span> try_git_version</span>()</span>:</span></span>
    import</span> os</span></span>
    import</span> sys</span></span>
</span>
    version</span> =</span> None</span></span>
    try</span>:</span></span>
        version</span> =</span> os.</span>popen</span>(</span>'</span>git describe --always --dirty --long</span>'</span>).</span>read</span>().</span>strip</span>()</span></span>
    except</span> Exception</span> as</span> e:</span></span>
        print</span> e</span></span>
    return</span> version</span></span></code></pre>
It runs git describe --always --dirty --long</code> which will return something
along these lines: 401b85f-dirty</code>. If you have any annoted tags, it will
return the tag name as well.</p>
If git</code> is not installed, or it is not a valid git</code> directory, then it will
simply return None</code>. At that point all we have to go on is the VERSION</code>
variable set at the top of the wscript.</p>
Now that we have our configuration environment set up with the VERSION</code> we
want to get that into a file that we can then include in our C++ source code.</p>
Create a build_version.h.in</code> file</h2>
#</span>ifndef</span> BUILD_VERSION_H_IN_941AD1F24D0A9D</span></span>
#</span>define</span> BUILD_VERSION_H_IN_941AD1F24D0A9D</span></span>
</span>
char</span> VERSION[] </span>=</span> "</span>@VERSION@</span>"</span>;</span></span>
</span>
#</span>endif</span> /* BUILD_VERSION_H_IN_941AD1F24D0A9D */</span></span></code></pre>Add the following to build(ctx)</code></h2>
ctx</span>(</span>features</span>=</span>'</span>subst</span>',</span></span>
        source</span>=</span>'</span>build_version.h.in</span>',</span></span>
        target</span>=</span>'</span>build_version.h</span>',</span></span>
        VERSION</span> =</span> ctx.env</span>[</span>'</span>VERSION</span>'</span>]</span>,</span></span>
        )</span></span></code></pre>
This uses the substitution feature to transform build_version.h.in</code> into
build_version.h</code>, while inserting the version into the file.</p>
Include build_version.h</code> in your source code</h2>
#</span>include</span> "</span>build_version.h</span>"</span></span></code></pre>
And add something along these lines to your main()</code>:</p>
std::cerr </span><<</span> "</span>Version: </span>"</span> <<</span> VERSION </span><<</span> std::endl;</span></span></code></pre>
This will print out the VERSION</code> that has been stored in build_version.h</code>.</p>
Full example</h2>
Check out my mdns-announce</a> project on Github for an example of how this is
implemented.</p>



Embedding ZeroMQ in the libev Event Loop
2013-04-20T18:22:00+00:00
In a previous article on ZeroMQ</a> we went over how ZeroMQ is triggered when
you use the socket that ZeroMQ returns, in that article there was some
discussion of embedding ZeroMQ into another event loop. Let's do that.</p>
libev</a> is an absolutely fantastic library that helps make it easy to write
evented programs. Evented programs work by getting notified that an action has
happened, and acting upon it. Unlike threaded where multiple pieces of work are
being executed at the same time, in an evented system you move every item that
could block to an event loop, that then calls back into user code with a
notification to continue. If one event uses up more than its fair share of CPU
time because it is busy doing a long calculation, every single other event that
is waiting will never get notified.</p>
Now, as previously discussed ZeroMQ is edge triggered, so embedding it into an
event loop that is level triggered doesn't do us much good, because we will miss
certain ZeroMQ notifications.</p>
One way to solve this problem is by looping over ZeroMQ's event system until we
get back a notification that it no longer has anything else for us to process,
that would look something like this1</a></sup>:</p>
int</span> zevents </span>=</span> 0</span>;</span></span>
size_t</span> zevents_len </span>=</span> sizeof</span>(zevents);</span></span>
zmq_socket</span>.</span>getsockopt</span>(ZMQ_EVENTS,</span> &</span>zevents,</span> &</span>zevents_len);</span></span>
</span>
do</span> {</span></span>
    if</span> (zevents </span>&</span> ZMQ_POLLIN) {</span></span>
        // We can read from the ZeroMQ socket</span></span>
    }</span> else</span> {</span></span>
        break</span>;</span></span>
    }</span></span>
</span>
    // Check to see if there is more to read ...</span></span>
    zmq_socket</span>.</span>getsockopt</span>(ZMQ_EVENTS,</span> &</span>zevents,</span> &</span>zevents_len);</span></span>
}</span> while</span> (zevents </span>&</span> ZMQ_POLLIN);</span></span>
</span>
if</span> (zevents </span>&</span> ZMQ_POLLOUT) {</span></span>
    // We can write to the ZeroMQ socket</span></span>
}</span></span>
</span>
// If neither of the above is true, then it was a false positive</span></span></code></pre>
However if we are receiving information from ZeroMQ remote endpoints faster
than we can process them, we end up being stuck in that do</code> ... while</code> loop
forever. If we have other events we want to process, that isn't entirely fair
since they will never ever get called again. Especially in a server application
where it may be servicing thousands of clients this is simply not acceptable.</p>
libev</h2>
libev provides various different event notifications, to be able to get around
edge triggered notifications, and still provide fair round-robin for all events
we are going to have to build on top of multiple different events.</p>
The events used will be:</p>

ev::io</code></a>: This one is pretty self explanatory, this is for getting notified
about input output changes. This is the one we are going to use on the
ZMQ_FD</code>.</li>
ev::prepare</code></a> and ev::check</code></a>: These two are generally used together, they
can be used to change the event loop and or make modifications on the fly to
events that have been registered with the event loop.</li>
ev::idle</code></a>: This is an event that gets fired whenever the event loop has
nothing else to do, so no other events fired, this will fire.</li>
</ul>
Plan of attack</h2>
Since the prepare</code> and check</code> run before and after the loop, we are going to
be using those to do most of the work. We use an io</code> so that we can turn off
the idle</code> when we can actually wait for a result from ZeroMQ's file
descriptor, otherwise we use idle</code> so that we will always get called once
every loop.</p>
In the prepare</code> watcher callback we do the following:</p>

Check to see what events ZeroMQ has for us, and check what events the user
has requested.</li>
If the ZeroMQ has an event for us that we want, and the user has requested
that event, we start the idle watcher.</li>
If ZeroMQ has no events, we start the io watcher.</li>
</ol>
In the check</code> watcher callback we do the following:</p>

Stop both the io</code> and idle</code> watchers, they were only there to make sure
that our check</code> watcher was called.</li>
See what event ZeroMQ has for us, and check that against what the user
wants. Depending on the event, call user defined function write() or user
defined function read().</li>
If this was a spurious wake-up on the part of ZeroMQ we simply ignore it
and let libev go on to other events.</li>
</ol>
We could make all of this work by simply using the prepare</code>, check</code> and
idle</code> watchers, but that would mean libev would be busy-waiting for something
to happen on the ZeroMQ socket. The io</code> watcher is required simply so in times
of nothing happening libev in its library can call into the kernels event
handling mechanism and go to sleep. We can't use just the io</code> watcher due to
the edge-triggered notification, because we'd miss all kinds of ZeroMQ
messages. So all four watchers are required, and play crucial parts in making
this work.</p>
Let's get down to code</h2>
Below you will find example code, it is not complete. Do note that I am using
some C++11isms, error checking code may not be complete/correct and in general
I don't suggest you copy and paste this without reading and understanding what
it does.</p>
The zmq_event</code> class is meant to be used as a base class, inherit from it, and
create the write()</code> and read()</code> functions. These functions will be called
when you are able to read from the ZeroMQ socket, or when you are able to write
to the ZeroMQ socket. You are guaranteed to be able to read one whole ZeroMQ
message, so if it is a multi-part message, do make sure to loop on
ZMQ_SNDMORE</code> as required.</p>
Upon instantiation it will automatically start being notified about events, we
start off with ev::READ</code>. When your sub-class wants to write to ZeroMQ it
should put the messages to be written into a list somewhere, and set
ev::READ | ev::WRITE</code> on watcher_io</code>, by calling
watcher_io.set(socket_fd, ev::READ | ev::WRITE)</code>. write()</code> will then be
called, write a single message to ZeroMQ, and if necessary when finished
writing, unset ev::WRITE</code> using watcher_io.set(socket_fd, ev::READ)</code>. If you
are not finished writing, after writing that singular message you may return
and write()</code> will be called again the next loop iteration. This way if you
have a lot of data to write you don't starve the other events from receiving
their notifications.</p>
zmq_event.h</code></h4>
#</span>include</span> <</span>string</span>></span></span>
</span>
#</span>include</span> <</span>ev++.h</span>></span></span>
#</span>include</span> <</span>zmq.hpp</span>></span></span>
</span>
class</span> zmq_event {</span></span>
    public:</span></span>
        zmq_event</span>(zmq::</span>context_t</span>&</span> context</span>,</span> int</span> type</span>,</span> const</span> std::string</span>&</span> connect</span>);</span></span>
        virtual</span> ~zmq_event</span>();</span></span>
</span>
    protected:</span></span>
        // This gets fired before the event loop, to prepare</span></span>
        void</span> before</span>(ev::prepare</span>&</span> prep</span>,</span> int</span> revents</span>);</span></span>
</span>
        // This is fired after the event loop, but before any other type of events</span></span>
        void</span> after</span>(ev::check</span>&</span> check</span>,</span> int</span> revents</span>);</span></span>
</span>
        // We need to have a no-op function available for those events that we</span></span>
        // want to add to the list, but should never fire an actual event</span></span>
        template</span> <</span>typename</span> T></span></span>
            inline</span> void</span> noop</span>(T</span>&</span> w</span>,</span> int</span> revents</span>) {};</span></span>
</span>
        // Function we are going to call to write to the ZeroMQ socket</span></span>
        virtual</span> void</span> write</span>()</span> =</span> 0</span>;</span></span>
</span>
        // Function we are going to call to read from the ZeroMQ socket</span></span>
        virtual</span> void</span> read</span>()</span> =</span> 0</span>;</span></span>
</span>
        // Some helper function, one to start notifications</span></span>
        void</span> start_notify</span>();</span></span>
</span>
        // And one to stop notifications.</span></span>
        void</span> stop_notify</span>();</span></span>
</span>
        // Our event types</span></span>
        ev::io      watcher_io;</span></span>
        ev::prepare watcher_prepare;</span></span>
        ev::check   watcher_check;</span></span>
        ev::idle    watcher_idle;</span></span>
</span>
        // Our ZeroMQ socket</span></span>
        zmq::</span>socket_t</span> socket;</span></span>
        int</span>           socket_fd </span>= -</span>1</span>;</span></span>
};</span></span></code></pre>zmq_event.cc</code></h4>
#</span>include</span> <</span>stdexcept</span>></span></span>
</span>
#</span>include</span> "</span>zmq_event.h</span>"</span></span>
</span>
zmq_event::</span>zmq_event</span>(zmq::</span>context_t</span>&</span> context,</span> int</span> type,</span> const</span> std::string</span>&</span> connect) :</span> socket</span>(context, type) {</span></span>
    // Get the file descriptor for the socket</span></span>
    size_t</span> fd_len </span>=</span> sizeof</span>(_socket_fd);</span></span>
    socket</span>.</span>getsockopt</span>(ZMQ_FD,</span> &</span>socket_fd,</span> &</span>fd_len);</span></span>
</span>
    // Actually connect to the ZeroMQ endpoint, could replace this with a bind as well ...</span></span>
    socket</span>.</span>bind</span>(</span>connect</span>.</span>c_str</span>());</span></span>
</span>
    // Set up all of our watchers</span></span>
</span>
    // Have our IO watcher check for READ on the ZeroMQ socket</span></span>
    watcher_io</span>.</span>set</span>(socket_fd, ev::READ);</span></span>
</span>
    // This watcher has a no-op callback</span></span>
    watcher_io</span>.</span>set</span><</span>zmq_event,</span> &</span>zmq_event::noop</span>></span>(</span>this</span>);</span></span>
</span>
    // Set up our prepare watcher to call the before() function</span></span>
    watcher_prepare</span>.</span>set</span><</span>zmq_event,</span> &</span>zmq_event::before</span>></span>(</span>this</span>);</span></span>
</span>
    // Set up the check watcher to call the after() function</span></span>
    watcher_check</span>.</span>set</span><</span>zmq_event,</span> &</span>zmq_event::after</span>></span>(</span>this</span>);</span></span>
</span>
    // Set up our idle watcher, once again a no-op</span></span>
    watcher_idle</span>.</span>set</span><</span>zmq_event,</span> &</span>zmq_event::noop</span>></span>(</span>this</span>);</span></span>
</span>
    // Tell libev to start notifying us!</span></span>
    start_notify</span>();</span></span>
}</span></span>
</span>
zmq_event::</span>~</span>zmq_event</span>() {}</span></span>
</span>
zmq_event::</span>before</span>(ev::prepare</span>&</span>,</span> int</span> revents) {</span></span>
    if</span> (EV_ERROR </span>&</span> revents) {</span></span>
        throw</span> std::</span>runtime_error</span>(</span>"</span>libev error</span>"</span>);</span></span>
    }</span></span>
</span>
    // Get any events that may be waiting</span></span>
    uint32_t</span> zevents </span>=</span> 0</span>;</span></span>
    size_t</span> zevents_len </span>=</span> sizeof</span>(zevents);</span></span>
</span>
    // Lucky for us, getting the events available doesn't invalidate the</span></span>
    // events, so that calling this in `before()` and in `after()` will</span></span>
    // give us the same results.</span></span>
    socket</span>.</span>getsockopt</span>(ZMQ_EVENTS,</span> &</span>zevents,</span> &</span>zevents_len);</span></span>
</span>
    // Check what events exists, and check it against what event we want. We</span></span>
    // "abuse" our watcher_io.events for this information.</span></span>
    if</span> ((zevents </span>&</span> ZMQ_POLLOUT)</span> &&</span> (</span>watcher_io</span>.</span>events</span> &</span> ev::WRITE)) {</span></span>
        watcher_idle</span>.</span>start</span>();</span></span>
        return</span>;</span></span>
    }</span></span>
</span>
    if</span> ((zevents </span>&</span> ZMQ_POLLIN)</span> &&</span> (</span>watcher_io</span>.</span>events</span> &</span> ev::READ)) {</span></span>
        watcher_idle</span>.</span>start</span>();</span></span>
        return</span>;</span></span>
    }</span></span>
</span>
    // No events ready to be processed, we'll just go watch some io</span></span>
    watcher_io</span>.</span>start</span>();</span></span>
}</span></span>
</span>
zmq_event::</span>after</span>(ev::check</span>&</span>,</span> int</span> revents) {</span></span>
    if</span> (EV_ERROR </span>&</span> revents) {</span></span>
        throw</span> std::</span>runtime_error</span>(</span>"</span>libev error</span>"</span>);</span></span>
    }</span></span>
</span>
    // Stop both the idle and the io watcher, no point in calling the no-op callback</span></span>
    // One of them will be reactived by before() on the next loop</span></span>
    watcher_idle</span>.</span>stop</span>();</span></span>
    watcher_io</span>.</span>stop</span>();</span></span>
</span>
    // Get the events</span></span>
    uint32_t</span> zevents </span>=</span> 0</span>;</span></span>
    size_t</span> zevents_len </span>=</span> sizeof</span>(zevents);</span></span>
    socket</span>.</span>getsockopt</span>(ZMQ_EVENTS,</span> &</span>zevents,</span> &</span>zevents_len);</span></span>
</span>
    // Check the events and call the users read/write function</span></span>
    if</span> ((zevents </span>&</span> ZMQ_POLLIN)</span> &&</span> (</span>watcher_io</span>.</span>events</span> &</span> ev::READ)) {</span></span>
        this</span>-></span>read</span>();</span></span>
    }</span></span>
</span>
    if</span> ((zevents </span>&</span> ZMQ_POLLOUT)</span> &&</span> (</span>watcher_io</span>.</span>events</span> &</span> ev::WRITE)) {</span></span>
        this</span>-></span>write</span>();</span></span>
    }</span></span>
}</span></span>
</span>
zmq_event::</span>start_notify</span>() {</span></span>
    watcher_check</span>.</span>start</span>();</span></span>
    watcher_prepare</span>.</span>start</span>();</span></span>
}</span></span>
</span>
zmq_event::</span>stop_notify</span>() {</span></span>
    watcher_check</span>.</span>stop</span>();</span></span>
    watcher_prepare</span>.</span>stop</span>();</span></span>
}</span></span></code></pre>Other event loops</h2>
libev is but just one of many event loops that exist out there, hopefully
this shows how it is possible to embed ZeroMQ into an event loop, thereby
making it easier to embed ZeroMQ into any other event loops.</p>



This snippet was from my older article regarding
ZeroMQ edge triggered notifications</a>. I would highly suggest reading that
article for more information and even more background on what is going on. ↩</a></p>
</li>
</ol>
</section>


ZeroMQ - Edge Triggered Notification
2012-09-10T19:01:50+00:00
ZeroMQ</a> is an absolutely fantastic communications library that has allowed
me to very easily create networks/connections between multiple different
applications to communicate in an extremely fast yet directed manner and allow
scaling of the connected entities with almost no second thought.</p>
I have used ZeroMQ within Python and C++, and it has been fantastic leaving
that part of the communication layer to a library rather than writing something
similar myself. However, embedding ZeroMQ into an another event loop has proven
to have some quirks. They are documented in the API, but not so much by other
users/examples found around the web, and caught me by surprise.</p>
This post mainly goes over the issue, the difference between edge and level
triggered and how ZeroMQ works. I am hoping to spend some time documenting how
to embed ZeroMQ correctly inside libev</a> at a later date.</p>
The quirk</h2>
ZeroMQ allows you to get an underlying file descriptor (ZMQ_FD</code>) for a ZeroMQ
socket using getsockopt()</code></a>, this is not the actual ZeroMQ file descriptor
but a stand-in:</p>
int</span> fd </span>=</span> 0</span>;</span></span>
size_t</span> fd_len </span>=</span> sizeof</span>(fd);</span></span>
zmq_socket</span>.</span>getsockopt</span>(ZMQ_FD,</span> &</span>fd,</span> &</span>fd_len);</span></span></code></pre>
This file descriptor can then be passed on to poll()</code>, select()</code>, kqueue()</code>
or whatever your event notification library of choice may be, the idea being
that if the file descriptor is notified to be available for reading then you
check the sockets ZMQ_EVENTS</code></a> and find out if you are allowed to read or
write to the ZeroMQ socket, or even possibly neither (false positive). The same
file descriptor is also used internally within the ZeroMQ library for
notification and thus we may receive an event notification that we can't do
anything with.</p>
As stated in the ZeroMQ API guide for getsockopt()</code> under ZMQ_FD</code></a>:</p>

the ØMQ library shall signal any pending events on the socket in an
edge-triggered</em> fashion by making the file descriptor become ready for
reading.</p>
</blockquote>
and further it states:</p>

The ability to read from the returned file descriptor does not necessarily
indicate that messages are available to be read from, or can be written to,
the underlying socket; applications must retrieve the actual event state with
a subsequent retrieval of the ZMQ_EVENTS</code> option.</p>
</blockquote>
So far so good, so after we return from our event notification on the file
descriptor we simply ask ZeroMQ whether or not we can read or write (or
possibly both):</p>
int</span> zevents </span>=</span> 0</span>;</span></span>
size_t</span> zevents_len </span>=</span> sizeof</span>(zevents);</span></span>
zmq_socket</span>.</span>getsockopt</span>(ZMQ_EVENTS,</span> &</span>zevents,</span> &</span>zevents_len);</span></span>
</span>
if</span> (zevents </span>&</span> ZMQ_POLLIN) {</span></span>
    // We can read from the ZeroMQ socket</span></span>
}</span></span>
</span>
if</span> (zevents </span>&</span> ZMQ_POLLOUT) {</span></span>
    // We can write to the ZeroMQ socket</span></span>
}</span></span>
</span>
// If neither of the above is true, then it was a false positive</span></span></code></pre>
So now you are happily reading from ZeroMQ and using the data as you wish,
except you start to notice that sometimes it can take a little while for data
to show up, or that if the end-point you are connected to on ZeroMQ sends a lot
of data it is not delivered in a timely fashion. This is where the
documentation for edge triggered comes into play, and that is something that is
different about ZeroMQ compared to for example BSD sockets.</p>
Edge triggered versus level triggered</h2>
Edge and level triggered refer to what kind of signal a device will output in
an attempt to gain the attention of another device. Edge and level triggered
have their history in electronics and hardware, where they play a very crucial
role.</p>
For both level and edge triggered assume the data is represented by this simple string:</p>
00000000000111111111111111111111100000000011111111111000000000</span></span></code></pre>
The zeroes (0) are where no data is available, and the ones (1) are where data
is available for processing.</p>
Level triggered</h3>
Level triggered devices when they require attention, and until they no longer
require attention will signal with a high voltage (going back to early
electronics), and when they no longer require attention signal with a low
voltage.</p>
Data:  00000000001111111111111111111111000000000011111111111000000000</span></span>
Block:      A               B              C         D</span></span>
+ 5v              ____________________           _________</span></span>
  -    __________|                    |_________|         |__________</span></span>
  0v</span></span>
State: 0          1                    0          1        0</span></span></code></pre>
At this point, once the signal turns to 1 in block B we can start processing
data, and until we are done processing data the signal will stay 1. Once we are
done processing the device will signal that we are done by setting the state to
0 in block C, and the process can repeat itself with block D.</p>
Edge triggered</h3>
Edge triggered devices when they require attention will simply turn on the
signal for a short period of time and then turn it back off until all of the
requireding processing is complete at which point they will once again signal
that they need processing.</p>
Data:  00000000001111111111111111111111100000000011111111111000000000</span></span>
Block:      A     B              C                 D</span></span>
+ 5v              __                              __</span></span>
  -    __________|  |____________________________|  |________________</span></span>
  0v</span></span>
State: 0          1              0                1        0</span></span></code></pre>
In this case when the trigger arrives you have to process all of the data
available. When block B arrives and we get 1 as our signal, we have to process
all of the data, we won't know when we are done other than some other
notification scheme by the device. (Such as a special message that says we are
done). Then when we are done we can go back to listening to waiting for a
trigger.</p>
If however in the time that we are processing the data from the trigger at B
receive more data to process we won't get notified because we weren't
officially done processing data from the device, and thus the device will not
notify us of the new data again until we have processed all of the data.</p>
There are some devices that as soon as you start processing their data, you
re-enable the trigger, so if you receive new data while still processing the
old you will get a new notification.</p>
ZeroMQ uses edge triggered</h2>
When multiple message arrive for the ZeroMQ socket, one notification is sent.
You check to see if the ZMQ_EVENTS</code> actually contains ZMQ_POLLIN</code> and then
read a single message from ZeroMQ. Content that you have processed the data
ZeroMQ has for you, you add the file descriptor back to the event notification
library and wait. However, you know you sent two messages, but all you got was
one. The next time you send data to the socket, it returns the second message,
and you have now send three but the third doesn't seem to arrive.</p>
This is where the edge triggered comes in. ZeroMQ only notifies you once, and
after it has notified you it won't notify you again until it receives new
messages, EVEN</em> if you have messages waiting for you. In BSD sockets when you
recv()</code> data from the socket, if it is not all of the data it will simply tell
you, and until all of the data has been read that the socket has to offer it is
going to tell you (level triggered).</p>
This means that when you get a notification from ZeroMQ you have to use a loop
to process all of the messages that are coming your way:</p>
int</span> zevents </span>=</span> 0</span>;</span></span>
size_t</span> zevents_len </span>=</span> sizeof</span>(zevents);</span></span>
zmq_socket</span>.</span>getsockopt</span>(ZMQ_EVENTS,</span> &</span>zevents,</span> &</span>zevents_len);</span></span>
</span>
do</span> {</span></span>
    if</span> (zevents </span>&</span> ZMQ_POLLIN) {</span></span>
        // We can read from the ZeroMQ socket</span></span>
    }</span> else</span> {</span></span>
        break</span>;</span></span>
    }</span></span>
</span>
    // Check to see if there is more to read ...</span></span>
    zmq_socket</span>.</span>getsockopt</span>(ZMQ_EVENTS,</span> &</span>zevents,</span> &</span>zevents_len);</span></span>
}</span> while</span> (zevents </span>&</span> ZMQ_POLLIN);</span></span>
</span>
if</span> (zevents </span>&</span> ZMQ_POLLOUT) {</span></span>
    // We can write to the ZeroMQ socket</span></span>
}</span></span>
</span>
// If neither of the above is true, then it was a false positive</span></span></code></pre>
The same is true if you need to send multiple messages, you will need to check
ZMQ_EVENTS</code> for ZMQ_POLLOUT</code> and send messages so long as that holds true, as
soon as it is no longer true you will need to wait until you get triggered
again by using the file descriptor in your event notification library.</p>
Embedding or co-existing with another event loop</h2>
Embedding ZeroMQ into a different event loop becomes more difficult when you
realise that it is edge triggered versus level triggered. If you want to allow
fair processing of data between events it becomes harder to accomplish with
ZeroMQ because you are required to process all of the data before going back to
your event loop or requires the setting up of different types of events
depending on the state that the ZeroMQ socket is in to make sure you process
all of the data and don't forget any.</p>
Edge triggered for the ZeroMQ file descriptor was a design decision which
unfortunately made the use of ZeroMQ sockets more difficult and different from
BSD sockets which makes it harder for application programmers to do the right
thing. There are plenty of questions on the internet regarding the proper usage
of ZMQ_FD</code> with various different event notification schemes/implementations.</p>


OpenSSL as a Filter (or non-blocking OpenSSL)
2012-04-08T16:50:14+00:00
OpenSSL</a> as a library at first glance is complicated, and then you realise
that a lot of the documentation</a> seems to be incomplete or missing.
Generally the individual man pages for the various functions are not bad, and
they will give you relevant information to help you along, but it can be hard
to find.</p>
The idea is to write a filter in such a way that OpenSSL can easily be disabled
or removed. We don't want to rely on OpenSSL specific functionality in our
code, for instance in the future we may want to change to using Botan</a> or
another SSL/TLS library without having to change a lot of core functionality.</p>
Possible methods</h2>
There are various ways of doing non-blocking OpenSSL, the main one is to simply
set the underlying socket to non-blocking and pass it into OpenSSL, at that
point the functions SSL_write()</code> and SSL_read()</code> will enter various error
states that can be read by SSL_get_error()</code></a>, more specifically the
function will return either SSL_ERROR_WANT_READ</code> or SSL_ERROR_WANT_WRITE</code>.</p>
The other method, the one used below, was described by Marc Lehmann in a post
to the libev mailling list</a> and uses memory BIO</a> objects. Specifically
he linked to some Perl sample code; the OpenSSL setup</a>, and the function
called for new incoming data/data to be written: dotls function</a>.</p>

The code is very new and might still be buggy, but it outlines the
principles: use a memory stream, which will avoid all issues with blocking in
openssl</p>
</blockquote>
It shows the idea pretty clearly but unless you know Perl it can be really
confusing as to what is going on.</p>
The SSL Filter implementation</h2>
The SSL_CTX</code></a> that is passed in contains all of the initialisation that is
done for OpenSSL in general, which is not shown here. The SSLFilter</code> is
created for each time you want to have a socket start doing SSL.</p>
// openssl_filter.h</span></span>
</span>
class</span> SSLFilter {</span></span>
    public:</span></span>
        SSLFilter</span>(SSL_CTX</span>*</span> ctxt</span>,</span></span>
                  std::string</span>*</span> nread</span>,</span></span>
                  std::string</span>*</span> nwrite</span>,</span></span>
                  std::string</span>*</span> aread</span>,</span></span>
                  std::string</span>*</span> awrite</span>);</span></span>
        virtual</span> ~SSLFilter</span>();</span></span>
</span>
        void</span> update</span>();</span></span>
</span>
    private:</span></span>
        bool</span> continue_ssl_</span>(</span>int</span> function_return</span>);</span></span>
</span>
        SSL </span>*</span> ssl;</span></span>
        BIO </span>*</span> rbio;</span></span>
        BIO </span>*</span> wbio;</span></span>
</span>
        std::string</span>*</span> nread;</span></span>
        std::string</span>*</span> nwrite;</span></span>
        std::string</span>*</span> aread;</span></span>
        std::string</span>*</span> awrite;</span></span>
};</span></span></code></pre>
The class contains mainly various different bits of state, when you create the
filter you pass in a pointer to four different strings, they are used as
follows:</p>

nread</code>: This contains data to be processed by the filter, from the network</li>
nwrite</code>: This contains data that has been processed by the filter, and has
to be sent to the network</li>
aread</code>: This contains data that has been processed by the filter, and is
ready to be processed by the application</li>
awrite</code>: This contains data that the application wants to send out to the
network and is ready for processing by the filter.</li>
</ul>
The other private variables contain various different pieces of state and are
created by the filters constructors.</p>

ssl</code>: Contains the state for the current SSL connection.</li>
rbio</code>: Contains the data that the SSL functions will read from, data put in
here is copied from nread</code>.</li>
wbio</code>: Contains the data that the SSL functions will write to, data from
this BIO is copied to nwrite</code>.</li>
</ul>
The implementation is the really interesting part.</p>
  1</span>// openssl_filter.cc</span></span>
  2</span></span>
  3</span>#</span>include</span> <</span>stdexcept</span>></span></span>
  4</span></span>
  5</span>#</span>include</span> "</span>openssl_filter.h</span>"</span></span>
  6</span></span>
  7</span>SSLFilter::</span>SSLFilter</span>(SSL_CTX</span>*</span> ctxt, </span></span>
  8</span>                     std::string</span>*</span> nread, </span></span>
  9</span>                     std::string</span>*</span> nwrite,</span></span>
 10</span>                     std::string</span>*</span> aread,</span></span>
 11</span>                     std::string</span>*</span> awrite)</span></span>
 12</span>                      :</span></span>
 13</span>                     nread</span>(nread), </span></span>
 14</span>                     nwrite</span>(nwrite), </span></span>
 15</span>                     aread</span>(aread), </span></span>
 16</span>                     awrite</span>(awrite) {</span></span>
 17</span></span>
 18</span>    rbio </span>=</span> BIO_new</span>(</span>BIO_s_mem</span>());</span></span>
 19</span>    wbio </span>=</span> BIO_new</span>(</span>BIO_s_mem</span>());</span></span>
 20</span>    </span></span>
 21</span>    ssl </span>=</span> SSL_new</span>(ctxt);</span></span>
 22</span>    </span></span>
 23</span>    SSL_set_accept_state</span>(ssl);</span></span>
 24</span>    SSL_set_bio</span>(ssl, rbio, wbio);</span></span>
 25</span>}</span></span>
 26</span></span>
 27</span>SSLFilter::</span>~</span>SSLFilter</span>() {</span></span>
 28</span>    SSL_free</span>(_ssl);</span></span>
 29</span>}</span></span>
 30</span></span>
 31</span>void</span> SSLFilter::</span>update</span>(Filter::FilterDirection) {</span></span>
 32</span>    // If we have data from the network to process, put it the memory BIO for OpenSSL</span></span>
 33</span>    if</span> (</span>!</span>nread</span>-></span>empty</span>()) {</span></span>
 34</span>        int</span> written </span>=</span> BIO_write</span>(rbio,</span> nread</span>-></span>c_str</span>(),</span> nread</span>-></span>length</span>());</span></span>
 35</span>        if</span> (written </span>></span> 0</span>)</span> nread</span>-></span>erase</span>(</span>0</span>, written);</span></span>
 36</span>    }</span></span>
 37</span></span>
 38</span>    // If the application wants to write data out to the network, process it with SSL_write</span></span>
 39</span>    if</span> (</span>!</span>awrite</span>-></span>empty</span>()) {</span></span>
 40</span>        int</span> written </span>=</span> SSL_write</span>(ssl,</span> awrite</span>-></span>c_str</span>(),</span> awrite</span>-></span>length</span>());</span></span>
 41</span></span>
 42</span>        if</span> (</span>!</span>continue_ssl_</span>()) {</span></span>
 43</span>            throw</span> std::</span>runtime_error</span>(</span>"</span>An SSL error occured.</span>"</span>);</span></span>
 44</span>        }</span></span>
 45</span></span>
 46</span>        if</span> (written </span>></span> 0</span>)</span> awrite</span>-></span>erase</span>(</span>0</span>, written);</span></span>
 47</span>    }</span></span>
 48</span></span>
 49</span>    // Read data for the application from the encrypted connection and place it in the string for the app to read</span></span>
 50</span>    while</span> (</span>1</span>) {</span></span>
 51</span>        char *</span>readto </span>=</span> new</span> char</span>[</span>1024</span>];</span></span>
 52</span>        int</span> read </span>=</span> SSL_read</span>(ssl, readto,</span> 1024</span>);</span></span>
 53</span>       </span></span>
 54</span>        if</span> (</span>!</span>continue_ssl_</span>()) {</span></span>
 55</span>            delete</span> readto;</span></span>
 56</span>            throw</span> std::</span>runtime_error</span>(</span>"</span>An SSL error occured.</span>"</span>);</span></span>
 57</span>        }</span></span>
 58</span></span>
 59</span>        if</span> (read </span>></span> 0</span>) {</span></span>
 60</span>            size_t</span> cur_size </span>=</span> aread</span>-></span>length</span>();</span></span>
 61</span>            aread</span>-></span>resize</span>(cur_size </span>+</span> read);</span></span>
 62</span>            std::</span>copy</span>(readto, readto </span>+</span> read,</span> aread</span>-></span>begin</span>()</span> +</span> cur_size);</span></span>
 63</span>        }</span></span>
 64</span></span>
 65</span>        delete</span> readto;</span></span>
 66</span></span>
 67</span>        if</span> (</span>static_cast</span><size_t></span>(read)</span> !=</span> 1024</span> ||</span> written </span>==</span> 0</span>)</span> break</span>;</span></span>
 68</span>    }</span></span>
 69</span></span>
 70</span>    // Read any data to be written to the network from the memory BIO and copy it to nwrite</span></span>
 71</span>    while</span> (</span>1</span>) {</span></span>
 72</span>        char *</span>readto </span>=</span> new</span> char</span>[</span>1024</span>];</span></span>
 73</span>        int</span> read </span>=</span> BIO_read</span>(wbio, readto,</span> 1024</span>);</span></span>
 74</span>        </span></span>
 75</span>        if</span> (read </span>></span> 0</span>) {</span></span>
 76</span>            size_t</span> cur_size </span>=</span> nwrite</span>-></span>length</span>();</span></span>
 77</span>            nwrite</span>-></span>resize</span>(cur_size </span>+</span> read);</span></span>
 78</span>            std::</span>copy</span>(readto, readto </span>+</span> read,</span> nwrite</span>-></span>begin</span>()</span> +</span> cur_size);</span></span>
 79</span>        }</span></span>
 80</span></span>
 81</span>        delete</span> readto;</span></span>
 82</span></span>
 83</span>        if</span> (</span>static_cast</span><size_t></span>(read)</span> !=</span> 1024</span> ||</span> read </span>==</span> 0</span>)</span> break</span>;</span></span>
 84</span>    }</span></span>
 85</span>}</span></span>
 86</span></span>
 87</span>bool</span> SSLFilter::</span>continue_ssl_</span>(</span>int</span> function_return</span>) {</span></span>
 88</span>    int</span> err </span>=</span> SSL_get_error</span>(ssl, function_return);</span></span>
 89</span></span>
 90</span>    if</span> (err </span>==</span> SSL_ERROR_NONE </span>||</span> err </span>==</span> SSL_ERROR_WANT_READ) {</span></span>
 91</span>        return</span> true</span>;</span></span>
 92</span>    }</span></span>
 93</span></span>
 94</span>    if</span> (err </span>==</span> SSL_ERROR_SYSCALL) {</span></span>
 95</span>        ERR_print_errors_fp</span>(stderr);</span></span>
 96</span>        perror</span>(</span>"</span>syscall error: </span>"</span>);</span></span>
 97</span>        return</span> false</span>;</span></span>
 98</span>    }</span></span>
 99</span></span>
100</span>    if</span> (err </span>==</span> SSL_ERROR_SSL) {</span></span>
101</span>        ERR_print_errors_fp</span>(stderr);</span></span>
102</span>        return</span> false</span>;</span></span>
103</span>    }</span></span>
104</span>    return</span> true</span>;</span></span>
105</span>}</span></span></code></pre>Explanation of the SSL Filter class</h3>
Line 7 - 25</strong>, we create the two memory BIO's and then call SSL_new()</code> using
the passed in SSL_CTX</code>, set the accept set for SSL and then we hook up the two
BIO objects to the ssl</code> object.</p>
Line 27 - 29</strong>, All we do in the destructor is SSL_free()</code> the SSL state,
which automatically takes care of freeing the two memory BIOs that were given
to it.</p>
Line 33 - 36</strong>, if nread</code> is not empty we copy the data into rbio</code>. rbio</code>
is used by the SSL functions to read from (as if it were reading from a
socket). We read into the memory BIO as much as possible, technically this
should be everything, but it is possible it won't read everything due to memory
constraints, next time SSLFilter::update()</code> is called it will get emptied as
much as possible again.</p>
Line 39 - 47</strong>, we see if there is anything that the app wants to write out
to the remote client, if there is we call SSL_write()</code>. After the call to
SSL_write()</code> we call continue_ssl_()</code> to see if we can safely continue using
SSL.</p>
Line 50 - 68</strong>, this is where try to read as much data from SSL_read()</code> as
we can and place it in aread</code>. This is data that has been decrypted by OpenSSL
and can be used by the application. After the call to SSL_read()</code> we call
continue_ssl_()</code> to see if we can safely continue using SSL.</p>
Line 71 - 84</strong>, if OpenSSL has to write something to the network, either
directly due to something the app did (by filling awrite</code>) or because of
something the remote client sent, it will place it in its memory buffer
(wbio</code>), what we do here is drain that memory buffer and fill nwrite</code>.</p>
Line 87 - 105</strong>, the continue_ssl_()</code> function gets the SSL errors using
SSL_get_error</code></a> and checks that either it is SSL_ERROR_NONE</code> or
SSL_ERROR_WANT_READ</code> both of which are acceptable errors, any other errors and
we check to see if it is an error due to a system call or due to an SSL error,
we print out the errors to standard error and return false. At that point the
calling code is free to go about its business as it pleases (most likely
throwing an error).</p>
Notes regarding this sample code ...</h4>
The only thing missing in this example code is checking to see if the SSL
connection has gone into an SSL_shutdown()</code> mode. This should be added so that
you cleanly shut down an SSL connection, but it is not absolutely required,
especially since it seems to be shaky as to whether or not it will work,
especially if you are using SSLv3 or SSLv2.</p>
How to use the SSL Filter</h3>
Here are the steps to using this filter:</p>

At program startup set up the SSL_CTX</code> structure as required, so that it
be used in the SSLFilter</code></li>
Set up socket/event loop and accept() new socket.</li>
Create four new std::string</code>'s and create a new SSLFilter, pass in the
SSL_CTX</code> you created, and the four buffers.</li>
Add the new socket to the event loop to wait for reading.</li>
</ol>
Now upon receiving a read ready status from the event loop, the following should be done:</p>

Read data into nread</code> buffer.</li>
call SSLFilter::update()</code></li>
See if aread</code> is not empty, process the data as you please</li>
Write data to awrite</code> as needed, if data is written to awrite</code>, call
SSLFilter::update(WRITE)</code> to process the data</li>
See if nwrite</code> is not empty, if so add socket to event loop for write
readiness.</li>
</ol>
Once you receive a write ready status from the event loop, you should do the following:</p>

Write data in nwrite</code> to the socket.</li>
If nwrite</code> is empty, remove the socket from the event loop for writing (so
that you don't have the event loop notifying you of the ability to write,
even-though you have nothing to write).</li>
</ol>
Now it becomes simple to add OpenSSL to any open socket, in a way that is easy
to refactor later and or disable depending on parameters passed in as arguments
to the program. If you set nread</code> to the same string as aread</code> and nwrite</code>
to awrite</code> and don't create an SSLFilter you have bypassed the filter...</p>

funcptr - cpp

Build numbers in binaries using waf

Full example</h2> Check out my mdns-announce</a> project on Github for an example of how this is implemented.</p>

Embedding ZeroMQ in the libev Event Loop

ZeroMQ - Edge Triggered Notification

OpenSSL as a Filter (or non-blocking OpenSSL)

`Full example</h2> Check out my mdns-announce</a> project on Github for an example of how this is implemented.</p>`