funcptr - development

Python Packaging and Distribution

2018-05-21T14:00:00+00:00

There's been many a discussion online related to a variety of tools related to Python packaging and distribution. There is pip</a>, pipenv</a> and poetry</a> that have been the tools under discussion.</p>

As an open source maintainer as part of the Pylons Project</a>, while I would love to be writing code I end up spending a lot of time dealing with user questions around packaging/distributing their source code using the software I've helped build, and as we move forward myself and other maintainers were wondering if we were actually helping users move forward in the best way possible using best of breed tools.1</a></sup></p>

As the Python community has moved from easy_install</a> to pip, we too have kept the documentation up to date. We went from python setup.py develop</code> to pip install -e .</code> to create editable installs of local projects, and try to let people know the pitfalls of using both easy_install and pip in the same project (mostly with an answer that falls in line with: remove your virtual environment and start over, just use pip).</p>

As part of Pyramid</a> we have developed and maintain various different cookiecutter</a> templates, and our goal is to attempt to provide templates that are both useful, but also follow best practices that are being adopted within the community at large so that newcomers can use their existing skills/knowledge and those that are starting with us walk away with a knowledge and experience that applies not just to development of Pyramid applications, but also applies to the broader community as a whole.</p>

pip</h2> Pip is a great tool that has simplified installation of packages, it supports using binary distributions named wheels</a> and has a way to easily install software from the Python Packaging Index</a>. It has a rather naive dependency resolution process, but for the most part it works and works well. It replaced easy_install as the tool to use for installing packages.</p> While you can use a requirements.txt</code> file with pip to install a "blessed" list of software there is no good way to "lock" the dependencies of dependencies without manually adding it to the list of requirements. This ends up making it very difficult to manage, and it is very difficult to know that what has been tested is what the user is actually going to get because packages may be updated at anytime, and re-creating the same exact environment is difficult and fraught with errors.</p> This is where Pipfile</a> is supposed to help. This is a project to add a new, more descriptive requirements file, as well as allowing for a lockfile that would lock not just your primary packages you have listed, but also all dependencies of dependencies all the way down the tree. This helps with reproducibility and allows for the same installation on two different systems to have the exact same software/dependencies installed.</p> pipenv</h2> While pip is a great tool, and with the Pipfile changes it would allow for locking of dependencies, there is one more puzzle piece missing. When installing packages while you can install them into the global namespace, the recommended way is to install all packages for a particular tool/project into a virtual environment.</p> Normally you'd invoke virtualenv</a>, to create this environment and then you'd make sure to install all packages within it, thereby isolating it from the rest of the system.</p> pipenv automates this for you, as well as using Pipfile</code> it also supports locking using Pipfile.lock</code> and provides a bunch of tooling around adding/removing dependencies from a local project.</p> pipenv allows you to easily create an environment and manage dependencies, but it makes no effort to solve the problem of distributing and building a package that may be installed by third parties.</p> poetry</h2> Poetry is a similar project to pipenv, with a major difference being that it was built to help with distributing/developing applications and building a distributable package that may then be installed using pip.</p> Instead of using a Pipfile</code> it uses a recently standardised pyproject.toml</code> file instead. Like pipenv it also supports locking, and it provides tooling around adding/removing dependencies as well as managing what versions are required.</p> Ultimately those dependencies are going to end up as metadata in a distributable package.</p> Poetry makes it easier to manage a software development project, whether that is for an application using various libraries for internal use, or for libraries that are going to be distributed to other developers.</p> The divide</h2> This is where the divide really starts, while you can use pipenv with a standard setuptools</a> project, any dependencies you add to the Pipfile using pipenv's tooling will not be listed as a dependencies for your project when you distribute it, this either means you need to duplicate the list in both setup.py</code> as well as the Pipfile</code>, or you have to add your current project as an editable install within your Pipfile</code> which means your Pipfile</code> is now not as easily distributable.</p> There are work-arounds that people have used, such as having setup.py</code> read a requirements.txt</code>, so that you could have all your requirements listed in a text file, and not in setup.py</code>, but asking to do the same with a Pipfile</code> in pipenv was met with a "Do not do this."</a>.</p> poetry explicitly allows you to add dependencies in one place, and those dependency listings are then automatically inserted into the package metadata that is created when you build your distributable package.</p> The two use cases</h2> There are two competing use cases, one is the deployment of software packages and being able to run them, but not as a developer, the other is a developer of software packages that needs to define dependencies for the project to run.</p> pipenv solves the deployment case. If I was a user I could very simply grab a known good Pipfile.lock</code> and use pipenv to install a known good set of software, this is great when I am deploying a project. It is the use case that many in the Python Packaging Authority</a> also seem to be optimizing for.</p> The other use case is for developers that are building new software, either by using a list of existing packages and deploying privately, or people developing software for other developers to be published on the Python Packaging Index.</p> This latter group of people is under represented due to it likely being much smaller, and existing tools like setuptools</code> and setup.py</code> already providing a "good enough" experience. Innovation in this area is something that readily needs to be improved upon to make it easier to create new libraries/packages that follow best practices. The amount of copy and pastes people have done for adding a setup.py</code> to their projects or to make something work is long. It's all a little bit of black magic, and there is a great many things that have been carried over because of cargo cult programming.</p> Explicit mentions by the Python Packaging Authority</h2> Reading the packaging guide on managing dependencies</a>, pipenv is the recommended tool:</p> This tutorial walks you through the use of Pipenv to manage dependencies for an application. It will show you how to install and use the necessary tools and make strong recommendations on best practices.</p> </blockquote> this language, along with what packaging.python.org</code> implies as a URL makes it difficult as a project maintainer to recommend alternate tools, becuase even if those tools are superior for the use case we are recommending them for it is always going to lead to questions from users, such as:</p> Why are you not using pipenv, the official tool recommended by Python.org?</p> </blockquote> We get similar questions about easy_install</code> vs pip</code> all of the time, as well as why people should switch, and we can point to various bits of documentation that explains why pip is a better choice.</p> If we were to recommend an alternate the appeal to authority that python.org</code> implies is going to make it much more difficult, and the question will become "why is the Pylons Project not using recommended tooling?"</p> poetry is listed as a footnote on that page, alongside pip-tools</a> and hatch</a>, and is mentioned only for doing library development, with no mention of other requirements that may make it a much better tool for developing locally.</p> Deployment is not development</h2> If I am using pipenv with a non-installable project (no setup.py) I end up having to figure out how to get the code, and the Pipfile</code>/Pipfile.lock</code> to my environment I am deploying into. pipenv's install provides a way to make sure to only install if the Pipfile.lock</code> is up to date or otherwise will fail to continue. If you are using a local project though, and it uses setup.py</code> the only way that the Pipfile.lock</code> will contain any sub-dependencies of your setup.py</code> project is if you install it as editable</code>. Otherwise sub-dependencies are not locked.2</a></sup></p> If I am using poetry I get an pip installable project, but it doesn't contain any hard pins or lock files. I'd have to distribute pyproject.lock</code> as well as my wheel. This gets me a little closer, but still no lock file that includes my newly produced wheel, and has all of its dependencies locked.</p> The Python Packaging Authority based on Twitter conversations with its members and the documentation on packaging.python.org</code> suggest using pipenv for development. pipenv is particular ill-suited for development if the goal is to create a package to be deployed to production. With two locations to define dependencies it leaves people scratching their heads as to which is canonical, and if a dependency is added to Pipfile</code> but not setup.py</code> it may leave a developer thinking their package is ready for distribution when in reality it is missing a dependency that is required to run/use said distribution.</p> At this point using both projects seems like a win-win. Use poetry to build/develop a package, then use pipenv in the integration phase to create a Pipfile.lock</code> that is used to deploy in production. This way you get the best of both worlds. A great tool that can help you register entry points and another that can help you with deploying a known good set of dependencies.</p> Interestingly, even the pipenv docs seem to agree that it is a deployment tool:</p> Specify your target Python version in your Pipfile’s [requires] section. Ideally, you should only have one target Python version, as this is a deployment tool.</p> -- Pipenv - General Recommendations & Version Control</a></p> </blockquote> Use pipenv if you have a script that requires a couple of dependencies and doesn't need all of the extra overhead of packaging metadata/packaging. Use poetry if you want to build a distributable project that can easily be deployed by others, and use both if you develop a project and need a known good environment to deploy.</p> In summary</h2> There will likely never be a time that one single tool is considered good enough, and competition between tools is a way to keep advancing forward. Packaging in the Python community for a long time has been difficult. Wheels has made things a little better. pip has made management of installing new packages easier and improved upon easy_install. Here's to the next evolution.</p> Now, can we talk about standardising on pyproject.toml</code> since that is already where "project" metadata needs to go, might as well re-use the name instead of having two different names/files. Oh, and PEP 517</a> can't come soon enough so that alternate tools like flit</a> can be used instead of setuptools/setup.py</code>.</p> We created an issue named Support poetry, flit, pipenv, or ...?</a> that attempts to go over the pros and cons of the various tools and how we currently support our users in our documentation on building projects using pyramid, including how to create a project that is distributable. Pyramid heavily uses pkg_resources</code></a> and entry points. The way to register the entry points is to have an installable package. The framework is flexible enough that there is no requirement for entry points, but at that point you are in territory where the default tooling provided by the project will not work, and some of the convenience tools/functionality that Pyramid provides it's users/developers is not available. ↩</a></p> </li> See documentation for Editable Dependencies (e.g. -e .</code>)</a> which as of this writing states: Sub-dependencies are not</strong> added to the Pipfile.lock if you leave the -e option out.</blockquote> ↩</a></p> </li> </ol> </section>



Build numbers in binaries using waf
2014-06-10T17:32:03+00:00
My build system of choice these days for any C++ project is waf</a>. One of
the things I always like havig is the build number included in the final
binary, so that with a simple ./binary --version</code> or even ./binary</code> the
version is printed that it was built from. This can make it much simpler to
debug any potential issues, especially if fixes may have already been made but
a bad binary was deployed.</p>
Setup the wscript</h2>
Make sure that your wscript somewhere near the top contains the following:</p>
APPNAME</span> =</span> '</span>myapp</span>'</span></span>
VERSION</span> =</span> '</span>0.0.0</span>'</span></span></code></pre>
Then in your configure(cfg)</code> add the following:</p>
cfg.env.</span>VERSION</span> =</span> VERSION</span></span>
cfg.env.</span>APPNAME</span> =</span> APPNAME</span></span>
</span>
git_version</span> =</span> try_git_version</span>()</span></span>
</span>
if</span> git_version:</span></span>
    cfg.env.</span>VERSION</span> +=</span> '</span>-</span>'</span> +</span> git_version</span></span></code></pre>
The try_git_version()</code> function is fairly simple and looks like this:</p>
def</span> try_git_version</span>()</span>:</span></span>
    import</span> os</span></span>
    import</span> sys</span></span>
</span>
    version</span> =</span> None</span></span>
    try</span>:</span></span>
        version</span> =</span> os.</span>popen</span>(</span>'</span>git describe --always --dirty --long</span>'</span>).</span>read</span>().</span>strip</span>()</span></span>
    except</span> Exception</span> as</span> e:</span></span>
        print</span> e</span></span>
    return</span> version</span></span></code></pre>
It runs git describe --always --dirty --long</code> which will return something
along these lines: 401b85f-dirty</code>. If you have any annoted tags, it will
return the tag name as well.</p>
If git</code> is not installed, or it is not a valid git</code> directory, then it will
simply return None</code>. At that point all we have to go on is the VERSION</code>
variable set at the top of the wscript.</p>
Now that we have our configuration environment set up with the VERSION</code> we
want to get that into a file that we can then include in our C++ source code.</p>
Create a build_version.h.in</code> file</h2>
#</span>ifndef</span> BUILD_VERSION_H_IN_941AD1F24D0A9D</span></span>
#</span>define</span> BUILD_VERSION_H_IN_941AD1F24D0A9D</span></span>
</span>
char</span> VERSION[] </span>=</span> "</span>@VERSION@</span>"</span>;</span></span>
</span>
#</span>endif</span> /* BUILD_VERSION_H_IN_941AD1F24D0A9D */</span></span></code></pre>Add the following to build(ctx)</code></h2>
ctx</span>(</span>features</span>=</span>'</span>subst</span>',</span></span>
        source</span>=</span>'</span>build_version.h.in</span>',</span></span>
        target</span>=</span>'</span>build_version.h</span>',</span></span>
        VERSION</span> =</span> ctx.env</span>[</span>'</span>VERSION</span>'</span>]</span>,</span></span>
        )</span></span></code></pre>
This uses the substitution feature to transform build_version.h.in</code> into
build_version.h</code>, while inserting the version into the file.</p>
Include build_version.h</code> in your source code</h2>
#</span>include</span> "</span>build_version.h</span>"</span></span></code></pre>
And add something along these lines to your main()</code>:</p>
std::cerr </span><<</span> "</span>Version: </span>"</span> <<</span> VERSION </span><<</span> std::endl;</span></span></code></pre>
This will print out the VERSION</code> that has been stored in build_version.h</code>.</p>
Full example</h2>
Check out my mdns-announce</a> project on Github for an example of how this is
implemented.</p>


User sessions, what data should be stored where?
2013-08-25T13:05:00+00:00
This article has been updated. For the old version please check
Archive.org</a></em></p>
A couple of days ago on reddit.com</a>'s /r/netsec</a> a poster by the name
of Dan Weber</a> posted what he believed to be an attack on PHP sessions:
Hacking PHP sessions by running out of memory</a> (reddit link)</a>. The way
the "attack" works is as follows:</p>

Create a new session</li>
Assign some new data to the session, in this case a username which is used
to signify that the user is logged in.</li>
Check to verify that the user should be logged in</li>
If the user should not be logged in, destroy the session.</li>
</ol>
The "attack" would be to run the PHP</a> script out of memory on number 3, since
once something is set on the session it is immediately stored, so even if the
user is not supposed to be logged in, they are now logged in since their
session says they are.</p>
I wouldn't necessarily call this a PHP hack, this is really just bad practice
in terms of programming, the logic should be reversed.</p>

Create a new session</li>
Verify the user should be logged in</li>
If so, set the session data</li>
If the user should not be logged in, don't set the session data</li>
</ol>
That would solve the problem at hand, and now there is no way for the user to
trick the PHP script into believing she is logged in when that is not the case.</p>
However as the discussion went on on Reddit, it became even more clear that
there are no good resources on what you should store in the user session, and
what you shouldn't store in the user session. Some of these things may seem
like common knowledge, but sadly this is something every single new person to
programming has to learn on their own.</p>
Some assumptions</h2>
Let's get this out of the way, this is in no way limited to PHP, but it is the
one I will be using as an example. This can all apply to Ruby (Ruby on
Rails</a>), Python (Pyramid</a>) or many other frameworks.</p>
The basic problem is that generally writing to the session is not an atomic
transaction based on the page accessed, so the assumption made in this article
is that when you write to the session it is instantly committed, and there is
no way to roll it back upon failure. If there was, our first example listed
wouldn't be able to occur since upon running out of memory at step 3, the
session would have been rolled back and cleaned up.</p>
What should you store in the users session?</h2>
You should only really store anything in the session that if it were made
public it would do very little harm.</p>
Really the list of items to be stored in a session are as follows:</p>

The users unique id (The ID that allows you to retrieve the users
information from storage)</li>
Temporary state (i.e. Flash messages)</li>
CSRF token</li>
</ol>
More importantly, don't store permission bits, or group memberships, or
anything that is used to allow/deny access to particular resources. You want to
store just enough information that upon a user accessing your site you are able
to retrieve the users information from storage, and based upon that information
from storage you then make decisions such as permissions/group memberships.</p>
Why is this important?</h3>
One of the things that Dan Weber brought up in the Reddit post was storing the
users permission level and group membership in the session. If your code is
then relying on the session to always contain the right permission level, then
there is no way to expire someones access to the data.</p>

A user logs in</li>
A user gets various permissions, and they are set in the session</li>
The user has been fired from his job, and an administrator removes the
users permissions</li>
Since the user is still logged in, the users permission bits are still
set, and he continues to have access to parts of the site/information he
shouldn't have access to.</li>
</ol>
If instead on every page visit we simply pull out the users unique id and
verify the permissions upon access, as soon as the permissions are revoked by
the administrator the user no longer has access to the various resources.</p>
Temporary state</h3>
There has to be an easy way to remember something from page visit to page visit
that isn't considered detrimental if the information gets lost. One of those
things is flash messages. Flash messages are generally used to provide the user
indication that something has changed, they are shown once and then never
again.</p>
Storing these as session data makes sense. If the flash message gets set,
great, if it doesn't get set, it doesn't matter. Flash messages are simply a
notification tool, if the user misses them it isn't important.</p>
What you shouldn't store in the session</h2>
Definitely don't store any kind of permission bits, groups a user is a part of
or anything that would allow the user access that they normally would not be
able to access.</p>
On each page access check what permissions the user has. While it may mean a
little more heavy lifting server side it provides extra security, and the means
to enforce changes in permissions instantly.</p>
Good secure programming practices</h2>
Keep secure programming practices in mind at all times, always consider how the
information you are storing/processing is accessed/viewed/administered. More
importantly think about the access controls that are in place, and how one
could expire access to a particular resource without requiring a co-operative
client.</p>
The ordering of how variables are set, and when they are set are very
important.  $_SESSION['isadmin'] = True</code> at the top of a PHP script, and then
removing it by checking to see if the user is actually an administrator later
on in the script is a bad idea.</p>
Always order your code so that if a failure does occur there is no chance that
a critical section of your code is executed by accident, or that information is
stored in a half-verified state. This is especially important for access
control.</p>


Embedding ZeroMQ in the libev Event Loop
2013-04-20T18:22:00+00:00
In a previous article on ZeroMQ</a> we went over how ZeroMQ is triggered when
you use the socket that ZeroMQ returns, in that article there was some
discussion of embedding ZeroMQ into another event loop. Let's do that.</p>
libev</a> is an absolutely fantastic library that helps make it easy to write
evented programs. Evented programs work by getting notified that an action has
happened, and acting upon it. Unlike threaded where multiple pieces of work are
being executed at the same time, in an evented system you move every item that
could block to an event loop, that then calls back into user code with a
notification to continue. If one event uses up more than its fair share of CPU
time because it is busy doing a long calculation, every single other event that
is waiting will never get notified.</p>
Now, as previously discussed ZeroMQ is edge triggered, so embedding it into an
event loop that is level triggered doesn't do us much good, because we will miss
certain ZeroMQ notifications.</p>
One way to solve this problem is by looping over ZeroMQ's event system until we
get back a notification that it no longer has anything else for us to process,
that would look something like this1</a></sup>:</p>
int</span> zevents </span>=</span> 0</span>;</span></span>
size_t</span> zevents_len </span>=</span> sizeof</span>(zevents);</span></span>
zmq_socket</span>.</span>getsockopt</span>(ZMQ_EVENTS,</span> &</span>zevents,</span> &</span>zevents_len);</span></span>
</span>
do</span> {</span></span>
    if</span> (zevents </span>&</span> ZMQ_POLLIN) {</span></span>
        // We can read from the ZeroMQ socket</span></span>
    }</span> else</span> {</span></span>
        break</span>;</span></span>
    }</span></span>
</span>
    // Check to see if there is more to read ...</span></span>
    zmq_socket</span>.</span>getsockopt</span>(ZMQ_EVENTS,</span> &</span>zevents,</span> &</span>zevents_len);</span></span>
}</span> while</span> (zevents </span>&</span> ZMQ_POLLIN);</span></span>
</span>
if</span> (zevents </span>&</span> ZMQ_POLLOUT) {</span></span>
    // We can write to the ZeroMQ socket</span></span>
}</span></span>
</span>
// If neither of the above is true, then it was a false positive</span></span></code></pre>
However if we are receiving information from ZeroMQ remote endpoints faster
than we can process them, we end up being stuck in that do</code> ... while</code> loop
forever. If we have other events we want to process, that isn't entirely fair
since they will never ever get called again. Especially in a server application
where it may be servicing thousands of clients this is simply not acceptable.</p>
libev</h2>
libev provides various different event notifications, to be able to get around
edge triggered notifications, and still provide fair round-robin for all events
we are going to have to build on top of multiple different events.</p>
The events used will be:</p>

ev::io</code></a>: This one is pretty self explanatory, this is for getting notified
about input output changes. This is the one we are going to use on the
ZMQ_FD</code>.</li>
ev::prepare</code></a> and ev::check</code></a>: These two are generally used together, they
can be used to change the event loop and or make modifications on the fly to
events that have been registered with the event loop.</li>
ev::idle</code></a>: This is an event that gets fired whenever the event loop has
nothing else to do, so no other events fired, this will fire.</li>
</ul>
Plan of attack</h2>
Since the prepare</code> and check</code> run before and after the loop, we are going to
be using those to do most of the work. We use an io</code> so that we can turn off
the idle</code> when we can actually wait for a result from ZeroMQ's file
descriptor, otherwise we use idle</code> so that we will always get called once
every loop.</p>
In the prepare</code> watcher callback we do the following:</p>

Check to see what events ZeroMQ has for us, and check what events the user
has requested.</li>
If the ZeroMQ has an event for us that we want, and the user has requested
that event, we start the idle watcher.</li>
If ZeroMQ has no events, we start the io watcher.</li>
</ol>
In the check</code> watcher callback we do the following:</p>

Stop both the io</code> and idle</code> watchers, they were only there to make sure
that our check</code> watcher was called.</li>
See what event ZeroMQ has for us, and check that against what the user
wants. Depending on the event, call user defined function write() or user
defined function read().</li>
If this was a spurious wake-up on the part of ZeroMQ we simply ignore it
and let libev go on to other events.</li>
</ol>
We could make all of this work by simply using the prepare</code>, check</code> and
idle</code> watchers, but that would mean libev would be busy-waiting for something
to happen on the ZeroMQ socket. The io</code> watcher is required simply so in times
of nothing happening libev in its library can call into the kernels event
handling mechanism and go to sleep. We can't use just the io</code> watcher due to
the edge-triggered notification, because we'd miss all kinds of ZeroMQ
messages. So all four watchers are required, and play crucial parts in making
this work.</p>
Let's get down to code</h2>
Below you will find example code, it is not complete. Do note that I am using
some C++11isms, error checking code may not be complete/correct and in general
I don't suggest you copy and paste this without reading and understanding what
it does.</p>
The zmq_event</code> class is meant to be used as a base class, inherit from it, and
create the write()</code> and read()</code> functions. These functions will be called
when you are able to read from the ZeroMQ socket, or when you are able to write
to the ZeroMQ socket. You are guaranteed to be able to read one whole ZeroMQ
message, so if it is a multi-part message, do make sure to loop on
ZMQ_SNDMORE</code> as required.</p>
Upon instantiation it will automatically start being notified about events, we
start off with ev::READ</code>. When your sub-class wants to write to ZeroMQ it
should put the messages to be written into a list somewhere, and set
ev::READ | ev::WRITE</code> on watcher_io</code>, by calling
watcher_io.set(socket_fd, ev::READ | ev::WRITE)</code>. write()</code> will then be
called, write a single message to ZeroMQ, and if necessary when finished
writing, unset ev::WRITE</code> using watcher_io.set(socket_fd, ev::READ)</code>. If you
are not finished writing, after writing that singular message you may return
and write()</code> will be called again the next loop iteration. This way if you
have a lot of data to write you don't starve the other events from receiving
their notifications.</p>
zmq_event.h</code></h4>
#</span>include</span> <</span>string</span>></span></span>
</span>
#</span>include</span> <</span>ev++.h</span>></span></span>
#</span>include</span> <</span>zmq.hpp</span>></span></span>
</span>
class</span> zmq_event {</span></span>
    public:</span></span>
        zmq_event</span>(zmq::</span>context_t</span>&</span> context</span>,</span> int</span> type</span>,</span> const</span> std::string</span>&</span> connect</span>);</span></span>
        virtual</span> ~zmq_event</span>();</span></span>
</span>
    protected:</span></span>
        // This gets fired before the event loop, to prepare</span></span>
        void</span> before</span>(ev::prepare</span>&</span> prep</span>,</span> int</span> revents</span>);</span></span>
</span>
        // This is fired after the event loop, but before any other type of events</span></span>
        void</span> after</span>(ev::check</span>&</span> check</span>,</span> int</span> revents</span>);</span></span>
</span>
        // We need to have a no-op function available for those events that we</span></span>
        // want to add to the list, but should never fire an actual event</span></span>
        template</span> <</span>typename</span> T></span></span>
            inline</span> void</span> noop</span>(T</span>&</span> w</span>,</span> int</span> revents</span>) {};</span></span>
</span>
        // Function we are going to call to write to the ZeroMQ socket</span></span>
        virtual</span> void</span> write</span>()</span> =</span> 0</span>;</span></span>
</span>
        // Function we are going to call to read from the ZeroMQ socket</span></span>
        virtual</span> void</span> read</span>()</span> =</span> 0</span>;</span></span>
</span>
        // Some helper function, one to start notifications</span></span>
        void</span> start_notify</span>();</span></span>
</span>
        // And one to stop notifications.</span></span>
        void</span> stop_notify</span>();</span></span>
</span>
        // Our event types</span></span>
        ev::io      watcher_io;</span></span>
        ev::prepare watcher_prepare;</span></span>
        ev::check   watcher_check;</span></span>
        ev::idle    watcher_idle;</span></span>
</span>
        // Our ZeroMQ socket</span></span>
        zmq::</span>socket_t</span> socket;</span></span>
        int</span>           socket_fd </span>= -</span>1</span>;</span></span>
};</span></span></code></pre>zmq_event.cc</code></h4>
#</span>include</span> <</span>stdexcept</span>></span></span>
</span>
#</span>include</span> "</span>zmq_event.h</span>"</span></span>
</span>
zmq_event::</span>zmq_event</span>(zmq::</span>context_t</span>&</span> context,</span> int</span> type,</span> const</span> std::string</span>&</span> connect) :</span> socket</span>(context, type) {</span></span>
    // Get the file descriptor for the socket</span></span>
    size_t</span> fd_len </span>=</span> sizeof</span>(_socket_fd);</span></span>
    socket</span>.</span>getsockopt</span>(ZMQ_FD,</span> &</span>socket_fd,</span> &</span>fd_len);</span></span>
</span>
    // Actually connect to the ZeroMQ endpoint, could replace this with a bind as well ...</span></span>
    socket</span>.</span>bind</span>(</span>connect</span>.</span>c_str</span>());</span></span>
</span>
    // Set up all of our watchers</span></span>
</span>
    // Have our IO watcher check for READ on the ZeroMQ socket</span></span>
    watcher_io</span>.</span>set</span>(socket_fd, ev::READ);</span></span>
</span>
    // This watcher has a no-op callback</span></span>
    watcher_io</span>.</span>set</span><</span>zmq_event,</span> &</span>zmq_event::noop</span>></span>(</span>this</span>);</span></span>
</span>
    // Set up our prepare watcher to call the before() function</span></span>
    watcher_prepare</span>.</span>set</span><</span>zmq_event,</span> &</span>zmq_event::before</span>></span>(</span>this</span>);</span></span>
</span>
    // Set up the check watcher to call the after() function</span></span>
    watcher_check</span>.</span>set</span><</span>zmq_event,</span> &</span>zmq_event::after</span>></span>(</span>this</span>);</span></span>
</span>
    // Set up our idle watcher, once again a no-op</span></span>
    watcher_idle</span>.</span>set</span><</span>zmq_event,</span> &</span>zmq_event::noop</span>></span>(</span>this</span>);</span></span>
</span>
    // Tell libev to start notifying us!</span></span>
    start_notify</span>();</span></span>
}</span></span>
</span>
zmq_event::</span>~</span>zmq_event</span>() {}</span></span>
</span>
zmq_event::</span>before</span>(ev::prepare</span>&</span>,</span> int</span> revents) {</span></span>
    if</span> (EV_ERROR </span>&</span> revents) {</span></span>
        throw</span> std::</span>runtime_error</span>(</span>"</span>libev error</span>"</span>);</span></span>
    }</span></span>
</span>
    // Get any events that may be waiting</span></span>
    uint32_t</span> zevents </span>=</span> 0</span>;</span></span>
    size_t</span> zevents_len </span>=</span> sizeof</span>(zevents);</span></span>
</span>
    // Lucky for us, getting the events available doesn't invalidate the</span></span>
    // events, so that calling this in `before()` and in `after()` will</span></span>
    // give us the same results.</span></span>
    socket</span>.</span>getsockopt</span>(ZMQ_EVENTS,</span> &</span>zevents,</span> &</span>zevents_len);</span></span>
</span>
    // Check what events exists, and check it against what event we want. We</span></span>
    // "abuse" our watcher_io.events for this information.</span></span>
    if</span> ((zevents </span>&</span> ZMQ_POLLOUT)</span> &&</span> (</span>watcher_io</span>.</span>events</span> &</span> ev::WRITE)) {</span></span>
        watcher_idle</span>.</span>start</span>();</span></span>
        return</span>;</span></span>
    }</span></span>
</span>
    if</span> ((zevents </span>&</span> ZMQ_POLLIN)</span> &&</span> (</span>watcher_io</span>.</span>events</span> &</span> ev::READ)) {</span></span>
        watcher_idle</span>.</span>start</span>();</span></span>
        return</span>;</span></span>
    }</span></span>
</span>
    // No events ready to be processed, we'll just go watch some io</span></span>
    watcher_io</span>.</span>start</span>();</span></span>
}</span></span>
</span>
zmq_event::</span>after</span>(ev::check</span>&</span>,</span> int</span> revents) {</span></span>
    if</span> (EV_ERROR </span>&</span> revents) {</span></span>
        throw</span> std::</span>runtime_error</span>(</span>"</span>libev error</span>"</span>);</span></span>
    }</span></span>
</span>
    // Stop both the idle and the io watcher, no point in calling the no-op callback</span></span>
    // One of them will be reactived by before() on the next loop</span></span>
    watcher_idle</span>.</span>stop</span>();</span></span>
    watcher_io</span>.</span>stop</span>();</span></span>
</span>
    // Get the events</span></span>
    uint32_t</span> zevents </span>=</span> 0</span>;</span></span>
    size_t</span> zevents_len </span>=</span> sizeof</span>(zevents);</span></span>
    socket</span>.</span>getsockopt</span>(ZMQ_EVENTS,</span> &</span>zevents,</span> &</span>zevents_len);</span></span>
</span>
    // Check the events and call the users read/write function</span></span>
    if</span> ((zevents </span>&</span> ZMQ_POLLIN)</span> &&</span> (</span>watcher_io</span>.</span>events</span> &</span> ev::READ)) {</span></span>
        this</span>-></span>read</span>();</span></span>
    }</span></span>
</span>
    if</span> ((zevents </span>&</span> ZMQ_POLLOUT)</span> &&</span> (</span>watcher_io</span>.</span>events</span> &</span> ev::WRITE)) {</span></span>
        this</span>-></span>write</span>();</span></span>
    }</span></span>
}</span></span>
</span>
zmq_event::</span>start_notify</span>() {</span></span>
    watcher_check</span>.</span>start</span>();</span></span>
    watcher_prepare</span>.</span>start</span>();</span></span>
}</span></span>
</span>
zmq_event::</span>stop_notify</span>() {</span></span>
    watcher_check</span>.</span>stop</span>();</span></span>
    watcher_prepare</span>.</span>stop</span>();</span></span>
}</span></span></code></pre>Other event loops</h2>
libev is but just one of many event loops that exist out there, hopefully
this shows how it is possible to embed ZeroMQ into an event loop, thereby
making it easier to embed ZeroMQ into any other event loops.</p>



This snippet was from my older article regarding
ZeroMQ edge triggered notifications</a>. I would highly suggest reading that
article for more information and even more background on what is going on. ↩</a></p>
</li>
</ol>
</section>


ZeroMQ - Edge Triggered Notification
2012-09-10T19:01:50+00:00
ZeroMQ</a> is an absolutely fantastic communications library that has allowed
me to very easily create networks/connections between multiple different
applications to communicate in an extremely fast yet directed manner and allow
scaling of the connected entities with almost no second thought.</p>
I have used ZeroMQ within Python and C++, and it has been fantastic leaving
that part of the communication layer to a library rather than writing something
similar myself. However, embedding ZeroMQ into an another event loop has proven
to have some quirks. They are documented in the API, but not so much by other
users/examples found around the web, and caught me by surprise.</p>
This post mainly goes over the issue, the difference between edge and level
triggered and how ZeroMQ works. I am hoping to spend some time documenting how
to embed ZeroMQ correctly inside libev</a> at a later date.</p>
The quirk</h2>
ZeroMQ allows you to get an underlying file descriptor (ZMQ_FD</code>) for a ZeroMQ
socket using getsockopt()</code></a>, this is not the actual ZeroMQ file descriptor
but a stand-in:</p>
int</span> fd </span>=</span> 0</span>;</span></span>
size_t</span> fd_len </span>=</span> sizeof</span>(fd);</span></span>
zmq_socket</span>.</span>getsockopt</span>(ZMQ_FD,</span> &</span>fd,</span> &</span>fd_len);</span></span></code></pre>
This file descriptor can then be passed on to poll()</code>, select()</code>, kqueue()</code>
or whatever your event notification library of choice may be, the idea being
that if the file descriptor is notified to be available for reading then you
check the sockets ZMQ_EVENTS</code></a> and find out if you are allowed to read or
write to the ZeroMQ socket, or even possibly neither (false positive). The same
file descriptor is also used internally within the ZeroMQ library for
notification and thus we may receive an event notification that we can't do
anything with.</p>
As stated in the ZeroMQ API guide for getsockopt()</code> under ZMQ_FD</code></a>:</p>

the ØMQ library shall signal any pending events on the socket in an
edge-triggered</em> fashion by making the file descriptor become ready for
reading.</p>
</blockquote>
and further it states:</p>

The ability to read from the returned file descriptor does not necessarily
indicate that messages are available to be read from, or can be written to,
the underlying socket; applications must retrieve the actual event state with
a subsequent retrieval of the ZMQ_EVENTS</code> option.</p>
</blockquote>
So far so good, so after we return from our event notification on the file
descriptor we simply ask ZeroMQ whether or not we can read or write (or
possibly both):</p>
int</span> zevents </span>=</span> 0</span>;</span></span>
size_t</span> zevents_len </span>=</span> sizeof</span>(zevents);</span></span>
zmq_socket</span>.</span>getsockopt</span>(ZMQ_EVENTS,</span> &</span>zevents,</span> &</span>zevents_len);</span></span>
</span>
if</span> (zevents </span>&</span> ZMQ_POLLIN) {</span></span>
    // We can read from the ZeroMQ socket</span></span>
}</span></span>
</span>
if</span> (zevents </span>&</span> ZMQ_POLLOUT) {</span></span>
    // We can write to the ZeroMQ socket</span></span>
}</span></span>
</span>
// If neither of the above is true, then it was a false positive</span></span></code></pre>
So now you are happily reading from ZeroMQ and using the data as you wish,
except you start to notice that sometimes it can take a little while for data
to show up, or that if the end-point you are connected to on ZeroMQ sends a lot
of data it is not delivered in a timely fashion. This is where the
documentation for edge triggered comes into play, and that is something that is
different about ZeroMQ compared to for example BSD sockets.</p>
Edge triggered versus level triggered</h2>
Edge and level triggered refer to what kind of signal a device will output in
an attempt to gain the attention of another device. Edge and level triggered
have their history in electronics and hardware, where they play a very crucial
role.</p>
For both level and edge triggered assume the data is represented by this simple string:</p>
00000000000111111111111111111111100000000011111111111000000000</span></span></code></pre>
The zeroes (0) are where no data is available, and the ones (1) are where data
is available for processing.</p>
Level triggered</h3>
Level triggered devices when they require attention, and until they no longer
require attention will signal with a high voltage (going back to early
electronics), and when they no longer require attention signal with a low
voltage.</p>
Data:  00000000001111111111111111111111000000000011111111111000000000</span></span>
Block:      A               B              C         D</span></span>
+ 5v              ____________________           _________</span></span>
  -    __________|                    |_________|         |__________</span></span>
  0v</span></span>
State: 0          1                    0          1        0</span></span></code></pre>
At this point, once the signal turns to 1 in block B we can start processing
data, and until we are done processing data the signal will stay 1. Once we are
done processing the device will signal that we are done by setting the state to
0 in block C, and the process can repeat itself with block D.</p>
Edge triggered</h3>
Edge triggered devices when they require attention will simply turn on the
signal for a short period of time and then turn it back off until all of the
requireding processing is complete at which point they will once again signal
that they need processing.</p>
Data:  00000000001111111111111111111111100000000011111111111000000000</span></span>
Block:      A     B              C                 D</span></span>
+ 5v              __                              __</span></span>
  -    __________|  |____________________________|  |________________</span></span>
  0v</span></span>
State: 0          1              0                1        0</span></span></code></pre>
In this case when the trigger arrives you have to process all of the data
available. When block B arrives and we get 1 as our signal, we have to process
all of the data, we won't know when we are done other than some other
notification scheme by the device. (Such as a special message that says we are
done). Then when we are done we can go back to listening to waiting for a
trigger.</p>
If however in the time that we are processing the data from the trigger at B
receive more data to process we won't get notified because we weren't
officially done processing data from the device, and thus the device will not
notify us of the new data again until we have processed all of the data.</p>
There are some devices that as soon as you start processing their data, you
re-enable the trigger, so if you receive new data while still processing the
old you will get a new notification.</p>
ZeroMQ uses edge triggered</h2>
When multiple message arrive for the ZeroMQ socket, one notification is sent.
You check to see if the ZMQ_EVENTS</code> actually contains ZMQ_POLLIN</code> and then
read a single message from ZeroMQ. Content that you have processed the data
ZeroMQ has for you, you add the file descriptor back to the event notification
library and wait. However, you know you sent two messages, but all you got was
one. The next time you send data to the socket, it returns the second message,
and you have now send three but the third doesn't seem to arrive.</p>
This is where the edge triggered comes in. ZeroMQ only notifies you once, and
after it has notified you it won't notify you again until it receives new
messages, EVEN</em> if you have messages waiting for you. In BSD sockets when you
recv()</code> data from the socket, if it is not all of the data it will simply tell
you, and until all of the data has been read that the socket has to offer it is
going to tell you (level triggered).</p>
This means that when you get a notification from ZeroMQ you have to use a loop
to process all of the messages that are coming your way:</p>
int</span> zevents </span>=</span> 0</span>;</span></span>
size_t</span> zevents_len </span>=</span> sizeof</span>(zevents);</span></span>
zmq_socket</span>.</span>getsockopt</span>(ZMQ_EVENTS,</span> &</span>zevents,</span> &</span>zevents_len);</span></span>
</span>
do</span> {</span></span>
    if</span> (zevents </span>&</span> ZMQ_POLLIN) {</span></span>
        // We can read from the ZeroMQ socket</span></span>
    }</span> else</span> {</span></span>
        break</span>;</span></span>
    }</span></span>
</span>
    // Check to see if there is more to read ...</span></span>
    zmq_socket</span>.</span>getsockopt</span>(ZMQ_EVENTS,</span> &</span>zevents,</span> &</span>zevents_len);</span></span>
}</span> while</span> (zevents </span>&</span> ZMQ_POLLIN);</span></span>
</span>
if</span> (zevents </span>&</span> ZMQ_POLLOUT) {</span></span>
    // We can write to the ZeroMQ socket</span></span>
}</span></span>
</span>
// If neither of the above is true, then it was a false positive</span></span></code></pre>
The same is true if you need to send multiple messages, you will need to check
ZMQ_EVENTS</code> for ZMQ_POLLOUT</code> and send messages so long as that holds true, as
soon as it is no longer true you will need to wait until you get triggered
again by using the file descriptor in your event notification library.</p>
Embedding or co-existing with another event loop</h2>
Embedding ZeroMQ into a different event loop becomes more difficult when you
realise that it is edge triggered versus level triggered. If you want to allow
fair processing of data between events it becomes harder to accomplish with
ZeroMQ because you are required to process all of the data before going back to
your event loop or requires the setting up of different types of events
depending on the state that the ZeroMQ socket is in to make sure you process
all of the data and don't forget any.</p>
Edge triggered for the ZeroMQ file descriptor was a design decision which
unfortunately made the use of ZeroMQ sockets more difficult and different from
BSD sockets which makes it harder for application programmers to do the right
thing. There are plenty of questions on the internet regarding the proper usage
of ZMQ_FD</code> with various different event notification schemes/implementations.</p>


IPv6 -- getaddrinfo() and bind() ordering with V6ONLY
2012-08-07T22:39:56+00:00
Recently I ran into an issue that took me a while to sort out, and it is
regarding inconsistent behaviour on various OS's with regards to IPv6 sockets
(AF_INET6</code></a>1</a></sup>) and calling bind(2)</code></a> after getting the
results back from getaddrinfo(3)</code></a>.</p>
A call to getaddrinfo()</code> with the hints set to AF_UNSPEC</code></a> in ai_family</code>
and AI_PASSIVE</code></a> in ai_flags</code> will return to us 1 or more results that we
can bind()</code> to. Sample code for that looks like this:</p>
struct</span> addrinfo hints,</span> *</span>addrlist;</span></span>
</span>
memset</span>(</span>&</span>hints</span>,</span> 0</span>,</span> sizeof</span>(hints));</span></span>
</span>
// Ask for TCP</span></span>
hints.ai_socktype </span>=</span> SOCK_STREAM;</span></span>
</span>
// Any family works for us ...</span></span>
hints.ai_family </span>=</span> AF_UNSPEC;</span></span>
</span>
// Set some hints</span></span>
hints.ai_flags </span>=</span> </span></span>
    AI_PASSIVE    </span>|</span> // We want to use this with bind</span></span>
    AI_ADDRCONFIG;</span>  // Only return IPv4 or IPv6 if they are configured</span></span>
</span>
int</span> rv;</span></span>
</span>
if</span> ((rv </span>=</span> getaddrinfo</span>(</span>0</span>,</span> "</span>7020</span>"</span>,</span> &</span>hints</span>,</span> &</span>addrlist</span>))</span> !=</span> 0</span>) {</span></span>
    fprintf</span>(stderr,</span> "</span>getaddrinfo: </span>%s</span>"</span>,</span> gai_strerror</span>(rv))</span>;</span></span>
    return</span> 1</span>;</span></span>
}</span></span>
</span>
// Use the list in *addrlist</span></span>
for</span> (addr </span>=</span> addrlist; addr </span>!=</span> 0</span>; addr </span>=</span> addr</span>-</span>></span>ai_next) {</span></span>
    // use *addr as appropriate</span></span>
}</span></span>
</span>
// Clean up the memory from getaddrinfo()</span></span>
freeaddrinfo</span>(addrlist);</span></span></code></pre>
On Linux there are two entries returned when the host it is run on has both
IPv4 and IPv6 enabled. An AF_INET</code> which was followed by an AF_INET6</code>. Now,
it is not said that you are required to use all of the results that are
returned, but if you want to listen on all address families it is off course
suggested.</p>
Following the steps below for each of the returned results should result in
having 1 or more different sockets that are bound to a single port.</p>

Create the socket()</li>
Set any socket options you want (SO_REUSEADDR</code> for example)</li>
Then bind() the socket</li>
After that call listen() (followed off course by accept() on the socket)</li>
</ol>
Only for some unknown reason (and errno</code> is no help) bind()</code> fails when you
get to the AF_INET6</code>, which was returned second. Searching online as to why
the bind would fail doesn't give you any good results and the thing that is
even worse is that if you run the same code on another platform such as
FreeBSD</a>, OpenIndiana</a> or Mac OS X</a> no such failure exists. However
I started suspecting something was up when I started looking at the output from
netstat -lan | grep 7020</code> on Mac OS X. Where 7020 is the port I passed into
getaddrinfo()</code>.</p>
tcp46      0      0  *.7020                 *.*  LISTEN     </span></span>
tcp4       0      0  *.7020                 *.*  LISTEN </span></span></code></pre>
Wait a minute ... one of the sockets is on both IPv4 and on IPv6. Some more
time spent searching the internet I came across RFC 3493 section 5.3</a>,
which is titled "IPV6_V6ONLY</code> option for AF_INET6</code> Sockets".</p>

As stated in section <3.7 Compatibility with IPv4 Nodes>, AF_INET6</code> sockets
may be used for both IPv4 and IPv6 communications.  Some applications may
want to restrict their use of an AF_INET6</code> socket to IPv6 communications
only.</p>
</blockquote>
This was going down the right route, so I changed my code so that in the steps
listed above in number 2 I added the following code if the socket type is
AF_INET6</code>:</p>
if</span> (</span>setsockopt</span>(sockfd, IPPROTO_IPV6, IPV6_V6ONLY,</span> &</span>yes</span>,</span> sizeof</span>(</span>int</span>))</span> ==</span> -</span>1</span>) {</span></span>
    close</span>(sockfd)</span>;</span></span>
    fprintf</span>(stderr,</span> "</span>setsockopt: </span>%s</span> IPV6_V6ONLY</span>\n</span>"</span>,</span> strerror</span>(errno))</span>;</span></span>
    continue</span>;</span></span>
}</span></span></code></pre>
The RFC 3493 section 5.3 also states that this option should be turned off by
default, which means that all IPv6 sockets can also communicate over IPv4. Thus
technically setting the option manually in code the best way to fix the issue.
FreeBSD has had this feature turned on (as in IPv6 sockets can only communicate
with IPv6 and NOT IPv4) since 5.x.</p>
The biggest issue is that the remaining operating systems (OS X and
OpenIndiana) don't have the same behaviour as Linux which makes troubleshooting
this issue more difficult than it should be. The issue is that the RFC doesn't
specify what exactly the operating should do when it encounters a request to
bind to the same port on IPv4 and IPv6. The only place where I have found this
documented is in "IPv6 Network Programming"</a> under "Tips in IPv6
Programming" chapter 4, section 4, appropriately titled "bind(2) Ordering and
Conflicts".</p>

If you get a bind()</code> error when attempting to bind to an AF_INET6</code> socket
please make sure that you set the socket option IPV6_V6ONLY</code> on the AF_INET6</code>
socket. The default as required by RFC 3493 is to have that option be off. The
default is wrong, and the RFC should have been more specific regarding what the
right behaviour is when attempting to bind on an AF_INET6</code> socket when already
bound on an AF_INET</code> while IPV6_V6ONLY</code> is set to false.</p>
The full code that I used for testing, along with a little bit more information
is available as a gist on github</a>.</p>



The old BSD style socket()</code></a> called for defines
starting with PF_</code> such as PF_INET</code> and PF_INET6</code> with the PF</code> standing for
protocol family. POSIX starts them with AF_</code>, and calls them an address
family. On almost every operating system PF_INET</code> is the same as AF_INET</code>. If
the define doesn't exist you can always create it. ↩</a></p>
</li>
</ol>
</section>


OpenSSL as a Filter (or non-blocking OpenSSL)
2012-04-08T16:50:14+00:00
OpenSSL</a> as a library at first glance is complicated, and then you realise
that a lot of the documentation</a> seems to be incomplete or missing.
Generally the individual man pages for the various functions are not bad, and
they will give you relevant information to help you along, but it can be hard
to find.</p>
The idea is to write a filter in such a way that OpenSSL can easily be disabled
or removed. We don't want to rely on OpenSSL specific functionality in our
code, for instance in the future we may want to change to using Botan</a> or
another SSL/TLS library without having to change a lot of core functionality.</p>
Possible methods</h2>
There are various ways of doing non-blocking OpenSSL, the main one is to simply
set the underlying socket to non-blocking and pass it into OpenSSL, at that
point the functions SSL_write()</code> and SSL_read()</code> will enter various error
states that can be read by SSL_get_error()</code></a>, more specifically the
function will return either SSL_ERROR_WANT_READ</code> or SSL_ERROR_WANT_WRITE</code>.</p>
The other method, the one used below, was described by Marc Lehmann in a post
to the libev mailling list</a> and uses memory BIO</a> objects. Specifically
he linked to some Perl sample code; the OpenSSL setup</a>, and the function
called for new incoming data/data to be written: dotls function</a>.</p>

The code is very new and might still be buggy, but it outlines the
principles: use a memory stream, which will avoid all issues with blocking in
openssl</p>
</blockquote>
It shows the idea pretty clearly but unless you know Perl it can be really
confusing as to what is going on.</p>
The SSL Filter implementation</h2>
The SSL_CTX</code></a> that is passed in contains all of the initialisation that is
done for OpenSSL in general, which is not shown here. The SSLFilter</code> is
created for each time you want to have a socket start doing SSL.</p>
// openssl_filter.h</span></span>
</span>
class</span> SSLFilter {</span></span>
    public:</span></span>
        SSLFilter</span>(SSL_CTX</span>*</span> ctxt</span>,</span></span>
                  std::string</span>*</span> nread</span>,</span></span>
                  std::string</span>*</span> nwrite</span>,</span></span>
                  std::string</span>*</span> aread</span>,</span></span>
                  std::string</span>*</span> awrite</span>);</span></span>
        virtual</span> ~SSLFilter</span>();</span></span>
</span>
        void</span> update</span>();</span></span>
</span>
    private:</span></span>
        bool</span> continue_ssl_</span>(</span>int</span> function_return</span>);</span></span>
</span>
        SSL </span>*</span> ssl;</span></span>
        BIO </span>*</span> rbio;</span></span>
        BIO </span>*</span> wbio;</span></span>
</span>
        std::string</span>*</span> nread;</span></span>
        std::string</span>*</span> nwrite;</span></span>
        std::string</span>*</span> aread;</span></span>
        std::string</span>*</span> awrite;</span></span>
};</span></span></code></pre>
The class contains mainly various different bits of state, when you create the
filter you pass in a pointer to four different strings, they are used as
follows:</p>

nread</code>: This contains data to be processed by the filter, from the network</li>
nwrite</code>: This contains data that has been processed by the filter, and has
to be sent to the network</li>
aread</code>: This contains data that has been processed by the filter, and is
ready to be processed by the application</li>
awrite</code>: This contains data that the application wants to send out to the
network and is ready for processing by the filter.</li>
</ul>
The other private variables contain various different pieces of state and are
created by the filters constructors.</p>

ssl</code>: Contains the state for the current SSL connection.</li>
rbio</code>: Contains the data that the SSL functions will read from, data put in
here is copied from nread</code>.</li>
wbio</code>: Contains the data that the SSL functions will write to, data from
this BIO is copied to nwrite</code>.</li>
</ul>
The implementation is the really interesting part.</p>
  1</span>// openssl_filter.cc</span></span>
  2</span></span>
  3</span>#</span>include</span> <</span>stdexcept</span>></span></span>
  4</span></span>
  5</span>#</span>include</span> "</span>openssl_filter.h</span>"</span></span>
  6</span></span>
  7</span>SSLFilter::</span>SSLFilter</span>(SSL_CTX</span>*</span> ctxt, </span></span>
  8</span>                     std::string</span>*</span> nread, </span></span>
  9</span>                     std::string</span>*</span> nwrite,</span></span>
 10</span>                     std::string</span>*</span> aread,</span></span>
 11</span>                     std::string</span>*</span> awrite)</span></span>
 12</span>                      :</span></span>
 13</span>                     nread</span>(nread), </span></span>
 14</span>                     nwrite</span>(nwrite), </span></span>
 15</span>                     aread</span>(aread), </span></span>
 16</span>                     awrite</span>(awrite) {</span></span>
 17</span></span>
 18</span>    rbio </span>=</span> BIO_new</span>(</span>BIO_s_mem</span>());</span></span>
 19</span>    wbio </span>=</span> BIO_new</span>(</span>BIO_s_mem</span>());</span></span>
 20</span>    </span></span>
 21</span>    ssl </span>=</span> SSL_new</span>(ctxt);</span></span>
 22</span>    </span></span>
 23</span>    SSL_set_accept_state</span>(ssl);</span></span>
 24</span>    SSL_set_bio</span>(ssl, rbio, wbio);</span></span>
 25</span>}</span></span>
 26</span></span>
 27</span>SSLFilter::</span>~</span>SSLFilter</span>() {</span></span>
 28</span>    SSL_free</span>(_ssl);</span></span>
 29</span>}</span></span>
 30</span></span>
 31</span>void</span> SSLFilter::</span>update</span>(Filter::FilterDirection) {</span></span>
 32</span>    // If we have data from the network to process, put it the memory BIO for OpenSSL</span></span>
 33</span>    if</span> (</span>!</span>nread</span>-></span>empty</span>()) {</span></span>
 34</span>        int</span> written </span>=</span> BIO_write</span>(rbio,</span> nread</span>-></span>c_str</span>(),</span> nread</span>-></span>length</span>());</span></span>
 35</span>        if</span> (written </span>></span> 0</span>)</span> nread</span>-></span>erase</span>(</span>0</span>, written);</span></span>
 36</span>    }</span></span>
 37</span></span>
 38</span>    // If the application wants to write data out to the network, process it with SSL_write</span></span>
 39</span>    if</span> (</span>!</span>awrite</span>-></span>empty</span>()) {</span></span>
 40</span>        int</span> written </span>=</span> SSL_write</span>(ssl,</span> awrite</span>-></span>c_str</span>(),</span> awrite</span>-></span>length</span>());</span></span>
 41</span></span>
 42</span>        if</span> (</span>!</span>continue_ssl_</span>()) {</span></span>
 43</span>            throw</span> std::</span>runtime_error</span>(</span>"</span>An SSL error occured.</span>"</span>);</span></span>
 44</span>        }</span></span>
 45</span></span>
 46</span>        if</span> (written </span>></span> 0</span>)</span> awrite</span>-></span>erase</span>(</span>0</span>, written);</span></span>
 47</span>    }</span></span>
 48</span></span>
 49</span>    // Read data for the application from the encrypted connection and place it in the string for the app to read</span></span>
 50</span>    while</span> (</span>1</span>) {</span></span>
 51</span>        char *</span>readto </span>=</span> new</span> char</span>[</span>1024</span>];</span></span>
 52</span>        int</span> read </span>=</span> SSL_read</span>(ssl, readto,</span> 1024</span>);</span></span>
 53</span>       </span></span>
 54</span>        if</span> (</span>!</span>continue_ssl_</span>()) {</span></span>
 55</span>            delete</span> readto;</span></span>
 56</span>            throw</span> std::</span>runtime_error</span>(</span>"</span>An SSL error occured.</span>"</span>);</span></span>
 57</span>        }</span></span>
 58</span></span>
 59</span>        if</span> (read </span>></span> 0</span>) {</span></span>
 60</span>            size_t</span> cur_size </span>=</span> aread</span>-></span>length</span>();</span></span>
 61</span>            aread</span>-></span>resize</span>(cur_size </span>+</span> read);</span></span>
 62</span>            std::</span>copy</span>(readto, readto </span>+</span> read,</span> aread</span>-></span>begin</span>()</span> +</span> cur_size);</span></span>
 63</span>        }</span></span>
 64</span></span>
 65</span>        delete</span> readto;</span></span>
 66</span></span>
 67</span>        if</span> (</span>static_cast</span><size_t></span>(read)</span> !=</span> 1024</span> ||</span> written </span>==</span> 0</span>)</span> break</span>;</span></span>
 68</span>    }</span></span>
 69</span></span>
 70</span>    // Read any data to be written to the network from the memory BIO and copy it to nwrite</span></span>
 71</span>    while</span> (</span>1</span>) {</span></span>
 72</span>        char *</span>readto </span>=</span> new</span> char</span>[</span>1024</span>];</span></span>
 73</span>        int</span> read </span>=</span> BIO_read</span>(wbio, readto,</span> 1024</span>);</span></span>
 74</span>        </span></span>
 75</span>        if</span> (read </span>></span> 0</span>) {</span></span>
 76</span>            size_t</span> cur_size </span>=</span> nwrite</span>-></span>length</span>();</span></span>
 77</span>            nwrite</span>-></span>resize</span>(cur_size </span>+</span> read);</span></span>
 78</span>            std::</span>copy</span>(readto, readto </span>+</span> read,</span> nwrite</span>-></span>begin</span>()</span> +</span> cur_size);</span></span>
 79</span>        }</span></span>
 80</span></span>
 81</span>        delete</span> readto;</span></span>
 82</span></span>
 83</span>        if</span> (</span>static_cast</span><size_t></span>(read)</span> !=</span> 1024</span> ||</span> read </span>==</span> 0</span>)</span> break</span>;</span></span>
 84</span>    }</span></span>
 85</span>}</span></span>
 86</span></span>
 87</span>bool</span> SSLFilter::</span>continue_ssl_</span>(</span>int</span> function_return</span>) {</span></span>
 88</span>    int</span> err </span>=</span> SSL_get_error</span>(ssl, function_return);</span></span>
 89</span></span>
 90</span>    if</span> (err </span>==</span> SSL_ERROR_NONE </span>||</span> err </span>==</span> SSL_ERROR_WANT_READ) {</span></span>
 91</span>        return</span> true</span>;</span></span>
 92</span>    }</span></span>
 93</span></span>
 94</span>    if</span> (err </span>==</span> SSL_ERROR_SYSCALL) {</span></span>
 95</span>        ERR_print_errors_fp</span>(stderr);</span></span>
 96</span>        perror</span>(</span>"</span>syscall error: </span>"</span>);</span></span>
 97</span>        return</span> false</span>;</span></span>
 98</span>    }</span></span>
 99</span></span>
100</span>    if</span> (err </span>==</span> SSL_ERROR_SSL) {</span></span>
101</span>        ERR_print_errors_fp</span>(stderr);</span></span>
102</span>        return</span> false</span>;</span></span>
103</span>    }</span></span>
104</span>    return</span> true</span>;</span></span>
105</span>}</span></span></code></pre>Explanation of the SSL Filter class</h3>
Line 7 - 25</strong>, we create the two memory BIO's and then call SSL_new()</code> using
the passed in SSL_CTX</code>, set the accept set for SSL and then we hook up the two
BIO objects to the ssl</code> object.</p>
Line 27 - 29</strong>, All we do in the destructor is SSL_free()</code> the SSL state,
which automatically takes care of freeing the two memory BIOs that were given
to it.</p>
Line 33 - 36</strong>, if nread</code> is not empty we copy the data into rbio</code>. rbio</code>
is used by the SSL functions to read from (as if it were reading from a
socket). We read into the memory BIO as much as possible, technically this
should be everything, but it is possible it won't read everything due to memory
constraints, next time SSLFilter::update()</code> is called it will get emptied as
much as possible again.</p>
Line 39 - 47</strong>, we see if there is anything that the app wants to write out
to the remote client, if there is we call SSL_write()</code>. After the call to
SSL_write()</code> we call continue_ssl_()</code> to see if we can safely continue using
SSL.</p>
Line 50 - 68</strong>, this is where try to read as much data from SSL_read()</code> as
we can and place it in aread</code>. This is data that has been decrypted by OpenSSL
and can be used by the application. After the call to SSL_read()</code> we call
continue_ssl_()</code> to see if we can safely continue using SSL.</p>
Line 71 - 84</strong>, if OpenSSL has to write something to the network, either
directly due to something the app did (by filling awrite</code>) or because of
something the remote client sent, it will place it in its memory buffer
(wbio</code>), what we do here is drain that memory buffer and fill nwrite</code>.</p>
Line 87 - 105</strong>, the continue_ssl_()</code> function gets the SSL errors using
SSL_get_error</code></a> and checks that either it is SSL_ERROR_NONE</code> or
SSL_ERROR_WANT_READ</code> both of which are acceptable errors, any other errors and
we check to see if it is an error due to a system call or due to an SSL error,
we print out the errors to standard error and return false. At that point the
calling code is free to go about its business as it pleases (most likely
throwing an error).</p>
Notes regarding this sample code ...</h4>
The only thing missing in this example code is checking to see if the SSL
connection has gone into an SSL_shutdown()</code> mode. This should be added so that
you cleanly shut down an SSL connection, but it is not absolutely required,
especially since it seems to be shaky as to whether or not it will work,
especially if you are using SSLv3 or SSLv2.</p>
How to use the SSL Filter</h3>
Here are the steps to using this filter:</p>

At program startup set up the SSL_CTX</code> structure as required, so that it
be used in the SSLFilter</code></li>
Set up socket/event loop and accept() new socket.</li>
Create four new std::string</code>'s and create a new SSLFilter, pass in the
SSL_CTX</code> you created, and the four buffers.</li>
Add the new socket to the event loop to wait for reading.</li>
</ol>
Now upon receiving a read ready status from the event loop, the following should be done:</p>

Read data into nread</code> buffer.</li>
call SSLFilter::update()</code></li>
See if aread</code> is not empty, process the data as you please</li>
Write data to awrite</code> as needed, if data is written to awrite</code>, call
SSLFilter::update(WRITE)</code> to process the data</li>
See if nwrite</code> is not empty, if so add socket to event loop for write
readiness.</li>
</ol>
Once you receive a write ready status from the event loop, you should do the following:</p>

Write data in nwrite</code> to the socket.</li>
If nwrite</code> is empty, remove the socket from the event loop for writing (so
that you don't have the event loop notifying you of the ability to write,
even-though you have nothing to write).</li>
</ol>
Now it becomes simple to add OpenSSL to any open socket, in a way that is easy
to refactor later and or disable depending on parameters passed in as arguments
to the program. If you set nread</code> to the same string as aread</code> and nwrite</code>
to awrite</code> and don't create an SSLFilter you have bypassed the filter...</p>


Magic Properties on Google AppEngine
2010-03-24T02:15:33+00:00
Update to my previous Google AppEngine auto-updating properties post</a>.</p>
In my last post I was talking about:</p>

[...] however there is still an issue, for now the MagicProperty is only
updated the first time the property it affects is set</p>
</blockquote>
I fixed that issue, it has a local cache that is added to the model, so if the
value does change, the cache would no longer be valid and the value is
recreated, otherwise the already computed value is returned. There was however
another issue I had to deal with, and that was the fact that the cache was not
being written to the datastore, this meant that upon getting the data back from
the datastore we recomputed it anyway. Now the programmer has to do some extra
work, they have to add a cache unindexed property to the model, and pass it
into the MagicProperty</code> to be used as the caching variable. Within the
save()</code> they have to prime the cache by setting a temporary variable equal to
the to be computed variable just in case the cache variables have not been
primed, this has not yet been documented other than in my SVN history.</p>
We still don't allow setting of the magic property, however upon returning from
the datastore it calls make_value_from_datastore</code>, which returns a
_MagicDatastore</code>, as long as we set with an instance of _MagicDatastore</code> it will
allow it to go through, so we can get the value from the datastore without
having to recompute the value.</p>
Read the code for an example on how to use this.</p>
Code is added below, feel free to use it as you wish under the license that is
attached.</p>
  1</span>###</span></span>
  2</span> # Copyright (c) 2010 Delta Regeer;</span></span>
  3</span> #</span></span>
  4</span> # Permission to use, copy, modify, and distribute this software for any</span></span>
  5</span> # purpose with or without fee is hereby granted, provided that the above</span></span>
  6</span> # copyright notice and this permission notice appear in all copies.</span></span>
  7</span> #</span></span>
  8</span> # THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES</span></span>
  9</span> # WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF</span></span>
 10</span> # MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR</span></span>
 11</span> # ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES</span></span>
 12</span> # WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN</span></span>
 13</span> # ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF</span></span>
 14</span> # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.</span></span>
 15</span> #</span></span>
 16</span>###</span></span>
 17</span></span>
 18</span>import</span> logging</span></span>
 19</span>import</span> hashlib</span></span>
 20</span>from</span> google.appengine.ext</span> import</span> db</span></span>
 21</span></span>
 22</span></span>
 23</span>def</span> MagicProperty</span>(</span>prop</span>,</span> magic_func=</span>None</span>,</span> cache_prop=</span>None</span>,</span> pass_instance=</span>False</span>,</span> *args</span>,</span> **kw</span>)</span>:</span></span>
 24</span>    if</span> magic_func:</span></span>
 25</span>        # No pants required.</span></span>
 26</span>        return</span> _MagicProperty</span>(</span>prop</span>,</span> magic_func</span>,</span> cache_prop</span>,</span> pass_instance</span>,</span> *</span>args</span>,</span> **</span>kw</span>)</span></span>
 27</span>    else</span>:</span></span>
 28</span>        # We are putting some pants on the function.</span></span>
 29</span>        def</span> pants</span>(</span>magic_func</span>)</span>:</span></span>
 30</span>            return</span> _MagicProperty</span>(</span>prop</span>,</span> magic_func</span>,</span> cache_prop</span>,</span> pass_instance</span>,</span> *</span>args</span>,</span> **</span>kw</span>)</span></span>
 31</span>        return</span> pants</span></span>
 32</span></span>
 33</span>class</span> _MagicDatastore</span>():</span></span>
 34</span>    """</span>This is an internal class for _MagicProperty.</span>"""</span></span>
 35</span>    </span></span>
 36</span>    def</span> __init__</span>(</span>self</span>,</span> val</span>)</span>:</span></span>
 37</span>        self</span>.value</span> =</span> val</span></span>
 38</span>    </span></span>
 39</span>    def</span> retval</span>(</span>self</span>)</span>:</span></span>
 40</span>        return</span> self</span>.value</span></span>
 41</span></span>
 42</span>class</span> _MagicProperty</span>(</span>db</span>.</span>Property</span>):</span></span>
 43</span>    """</span>MagicProperty which will modify output based on a function that it is given.</span></span>
 44</span>    </span></span>
 45</span>    This has several ways in which it may be called:</span></span>
 46</span>    </span></span>
 47</span>    In this example we create the model, no caching is done, so any data that is returned from the datastore</span></span>
 48</span>    will be erased the first time that the MagicProperty is accessed and recomputed.</span></span>
 49</span>    </span></span>
 50</span>        class MagicTest(db.Model):</span></span>
 51</span>            title = db.StringProperty(required=True)</span></span>
 52</span>            chars = utils.MagicProperty(title, len, required=True)</span></span>
 53</span>    </span></span>
 54</span>        mytest = MagicTest(title="It was for the good of the school!")</span></span>
 55</span>        </span></span>
 56</span>        >>> </span>print mytest.title</span></span>
 57</span>        It was for the good of the school!</span></span>
 58</span>        >>> </span>print mytest.chars</span></span>
 59</span>        34</span></span>
 60</span>        </span></span>
 61</span>        mytest = MagicTest.all().get()</span></span>
 62</span>        </span></span>
 63</span>        >>> </span>print mytest.title</span></span>
 64</span>        Hello</span></span>
 65</span>        >>> </span>print mytest.chars		</span></span>
 66</span>        5	# Do note, this is recalculated the first time it is called, as long as title does not change it won't be recomputed.</span></span>
 67</span>    </span></span>
 68</span>    In this example we create the model, and we also create a caching property so that even when we get values back </span></span>
 69</span>    from the datastore we use the cached computed value rather than running the function again. Do note that this requires overriding put()</span></span>
 70</span>    to prime the cache as there is currently no way to specify that certain properties should be "saved" before others.</span></span>
 71</span>    </span></span>
 72</span>        class MagicTesting(db.Model):</span></span>
 73</span>            title = db.StringProperty(required=True)</span></span>
 74</span>            cache = db.UnindexedProperty()</span></span>
 75</span>            chars = utils.MagicProperty(title, len, cache_prop=cache, required=True)</span></span>
 76</span>        </span></span>
 77</span>            def put(self, *args, **kw):</span></span>
 78</span>                prime_cache = self.chars</span></span>
 79</span>                super(MagicTesting).put(*args, **kw)</span></span>
 80</span>        </span></span>
 81</span>        mytesting = MagicTesting(title="Get the pocket knife out of my boot.")</span></span>
 82</span>    </span></span>
 83</span>        >>> </span>print mytesting.title</span></span>
 84</span>        Get the pocket knife out of my boot.</span></span>
 85</span>        >>> </span>print mytesting.chars</span></span>
 86</span>        36</span></span>
 87</span>        </span></span>
 88</span>        mytesting = MagicTest.all().get()</span></span>
 89</span>        </span></span>
 90</span>        >>> </span>print mytesting.title</span></span>
 91</span>        How are you?</span></span>
 92</span>        >>> </span>print mytesting.chars	</span></span>
 93</span>        12	# This is not recomputed so long as the title has not changed, however it uses more datastore space to store a hash.</span></span>
 94</span>        </span></span>
 95</span>        </span></span>
 96</span>    Inspired by: </span></span>
 97</span>    http://appengine-cookbook.appspot.com/recipe/custom-model-properties-are-cute</span></span>
 98</span>    http://code.google.com/appengine/articles/extending_models.html</span></span>
 99</span>    http://googleappengine.blogspot.com/2009/07/writing-custom-property-classes.html</span></span>
100</span>    </span></span>
101</span>    """</span></span>
102</span>    def</span> __init__</span>(</span>self</span>,</span> prop</span>,</span> magic_func</span>,</span> cache_prop</span>,</span> pass_instance</span>,</span> *args</span>,</span> **kw</span>)</span>:</span></span>
103</span>        """</span></span>
104</span>        Extra parameters you can give this initializer.</span></span>
105</span>        </span></span>
106</span>            prop		= Property to be acted upon</span></span>
107</span>            magic_func	= The function to be called when the property is accessed</span></span>
108</span>            cache_prop	= The property that can hold our cache, I suggest it is an db.UnindexedProperty() since it just stores sha1 hashes</span></span>
109</span>        """</span></span>
110</span>        super</span>(</span>_MagicProperty</span>,</span> self</span>).</span>__init__</span>(</span>*</span>args</span>,</span> **</span>kw</span>)</span></span>
111</span>        self</span>.magic_func</span> =</span> magic_func</span></span>
112</span>        self</span>.magic_prop</span> =</span> prop</span></span>
113</span>        self</span>.magic_cache</span> =</span> cache_prop</span></span>
114</span>        self</span>.magic_pass</span> =</span> pass_instance</span></span>
115</span>        </span></span>
116</span>    def</span> get_cache_val</span>(</span>self</span>,</span> model_instance</span>,</span> class_instance</span>)</span>:</span></span>
117</span>        if</span> self</span>.magic_cache</span> is not</span> None</span>:</span></span>
118</span>            return</span> self</span>.magic_cache.</span>__get__</span>(</span>model_instance</span>,</span> class_instance</span>)</span></span>
119</span>        return</span> getattr</span>(</span>model_instance</span>,</span> self</span>.</span>attr_name</span>()</span> +</span> "</span>orig</span>",</span> None</span>)</span></span>
120</span>            </span></span>
121</span>    def</span> set_cache_val</span>(</span>self</span>,</span> model_instance</span>,</span> val</span>)</span>:</span></span>
122</span>        val</span> =</span> hashlib.</span>sha1</span>(</span>val</span>).</span>hexdigest</span>()</span></span>
123</span>        </span></span>
124</span>        if</span> self</span>.magic_cache</span> is not</span> None</span>:</span></span>
125</span>            self</span>.magic_cache.</span>__set__</span>(</span>model_instance</span>,</span> val</span>)</span></span>
126</span>        setattr</span>(</span>model_instance</span>,</span> self</span>.</span>attr_name</span>()</span> +</span> "</span>orig</span>",</span> val</span>)</span></span>
127</span>    </span></span>
128</span>    def</span> attr_name</span>(</span>self</span>)</span>:</span></span>
129</span>        # In google.appengine.ex.db there is an explicit warning not to use this method, so we test for it first.</span></span>
130</span>        if</span> self</span>._attr_name:</span></span>
131</span>            return</span> self</span>.</span>_attr_name</span>()</span></span>
132</span>        else</span>:</span></span>
133</span>            return</span> "</span>_</span>"</span> +</span> self</span>.name</span></span>
134</span>    </span></span>
135</span>    def</span> __get__</span>(</span>self</span>,</span> model_instance</span>,</span> class_instance</span>)</span>:</span></span>
136</span>        if</span> model_instance</span> is</span> None</span>:</span></span>
137</span>            return</span> self</span></span>
138</span>        </span></span>
139</span>        cur</span> =</span> self</span>.magic_prop.</span>__get__</span>(</span>model_instance</span>,</span> class_instance</span>)</span></span>
140</span>        cur</span> =</span> cur.</span>encode</span>(</span>'</span>utf-8</span>'</span>)		</span></span>
141</span>        last</span> =</span> self</span>.</span>get_cache_val</span>(</span>model_instance</span>,</span> class_instance</span>)</span></span>
142</span>        if</span> last</span> ==</span> hashlib.</span>sha1</span>(</span>cur</span>).</span>hexdigest</span>():</span></span>
143</span>            logging.</span>info</span>(</span>"</span>Cache hit: </span>%s</span>"</span> %</span> (cur)</span>)</span></span>
144</span>            return</span> getattr</span>(</span>model_instance</span>,</span> self</span>.</span>attr_name</span>()</span>,</span> None</span>)</span></span>
145</span>        </span></span>
146</span>        logging.</span>info</span>(</span>"</span>Cache miss: </span>%s</span>"</span> %</span> (cur)</span>)</span></span>
147</span>        </span></span>
148</span>        magic_done</span> = u</span>""</span></span>
149</span>        if</span> self</span>.magic_pass:</span></span>
150</span>            magic_done</span> =</span> self</span>.</span>magic_func</span>(</span>model_instance</span>,</span> cur</span>)</span></span>
151</span>        else</span>:</span></span>
152</span>            magic_done</span> =</span> self</span>.</span>magic_func</span>(</span>cur</span>)</span></span>
153</span>        </span></span>
154</span>        # Set the attribute in the model</span></span>
155</span>        setattr</span>(</span>model_instance</span>,</span> self</span>.</span>attr_name</span>()</span>,</span> magic_done</span>)</span></span>
156</span>        self</span>.</span>set_cache_val</span>(</span>model_instance</span>,</span> cur</span>)</span></span>
157</span>        </span></span>
158</span>        return</span> magic_done</span></span>
159</span>        </span></span>
160</span>    </span></span>
161</span>    def</span> __set__</span>(</span>self</span>,</span> model_instance</span>,</span> value</span>)</span>:</span></span>
162</span>        if</span> isinstance</span>(</span>value</span>,</span> _MagicDatastore</span>):</span></span>
163</span>            setattr</span>(</span>model_instance</span>,</span> self</span>.</span>attr_name</span>()</span>,</span> value.</span>retval</span>())</span></span>
164</span>        else</span>:</span></span>
165</span>            raise</span> db.</span>DerivedPropertyError</span>(</span>"</span>MagicProperty is magic. Magic may not be modified.</span>"</span>)</span></span>
166</span>    </span></span>
167</span>    def</span> make_value_from_datastore</span>(</span>self</span>,</span> value</span>)</span>:</span></span>
168</span>        return</span> _MagicDatastore</span>(</span>value</span>)</span></span></code></pre>


Auto-updating Properties in Google AppEngine Models
2010-02-21T04:03:42+00:00
EDIT: I have an updated version of this code available in a newer post, please
go to Magic Properties on Google AppEngine</a> for the updated information.
For the generic information that has not changed, and the idea behind the code
please continue below.</p>

In Google AppEngine a model may contain many different properties, from this
model you can generate a form using djangoforms. This is all good and well
until you have a required property on a model that needs to have part of it be
calculated based upon something the user types in. For example, a blog entry
generally has a "slug" which is generated from the title, now in Google
AppEngine's models, if a property is required it has to be set at object
creation time.</p>
Djangoforms has a form.save(committed=False)</code> that allows one to get back an
unsaved Model</code>, however since slug is not in the form (since it is auto
generated) it cannot be set at model creation time. Even adding a hidden field
to the form with the name of the property is not going to satisfy the model
since a value of None</code> is considered to be empty, which means it does not meet
the required=True</code> status.</p>
After trying a multitude of different options I ran across an article about
extending models</a> that specifically talked about adding custom properties
that were saved into the DB, and could be searched on, this got me digging and
I found a blog post on the Google AppEngine blogspot</a> about the exact same
thing, using yet again a different example. The one that got me on the right
track was the following post by Rodrigo Moraes on his custom model
properties</a>.</p>
I came up with the following fairly quickly after also taking a look at
googleappengine.ext.db</code> within the Google AppEngine framework, however there is
still an issue, for now the MagicProperty</code> is only updated the first time the
property it affects is set. I am not yet sure how to work around that, I will
have to do some more digging, maybe there is a way to get notified if another
property gets modified or updated.</p>
import</span> logging</span></span>
from</span> google.appengine.ext</span> import</span> db</span></span>
</span>
</span>
def</span> MagicProperty</span>(</span>prop</span>,</span> magic_func=</span>None</span>,</span> *args</span>,</span> **kw</span>)</span>:</span></span>
    if</span> magic_func:</span></span>
        # No pants required.</span></span>
        return</span> _MagicProperty</span>(</span>prop</span>,</span> magic_func</span>,</span> *</span>args</span>,</span> **</span>kw</span>)</span></span>
    else</span>:</span></span>
        # We are putting some pants on the function.</span></span>
        def</span> pants</span>(</span>magic_func</span>)</span>:</span></span>
            return</span> _MagicProperty</span>(</span>prop</span>,</span> magic_func</span>,</span> *</span>args</span>,</span> **</span>kw</span>)</span></span>
        return</span> pants</span></span>
</span>
</span>
class</span> _MagicProperty</span>(</span>db</span>.</span>Property</span>):</span></span>
    """</span>MagicProperty which will modify output based on a function that it is given.</span></span>
    </span></span>
    Inspired by: </span></span>
    http://appengine-cookbook.appspot.com/recipe/custom-model-properties-are-cute</span></span>
    http://code.google.com/appengine/articles/extending_models.html</span></span>
    http://googleappengine.blogspot.com/2009/07/writing-custom-property-classes.html</span></span>
    </span></span>
    """</span></span>
    def</span> __init__</span>(</span>self</span>,</span> prop</span>,</span> magic_func</span>,</span> *args</span>,</span> **kw</span>)</span>:</span></span>
        super</span>(</span>_MagicProperty</span>,</span> self</span>).</span>__init__</span>(</span>*</span>args</span>,</span> **</span>kw</span>)</span></span>
        self</span>.magic_func</span> =</span> magic_func</span></span>
        self</span>.magic_prop</span> =</span> prop</span></span>
    </span></span>
    def</span> attr_name</span>(</span>self</span>)</span>:</span></span>
        # In google.appengine.ex.db there is an explicit warning not to use this method, so we test for it first.</span></span>
        if</span> self</span>._attr_name:</span></span>
            return</span> self</span>.</span>_attr_name</span>()</span></span>
        else</span>:</span></span>
            return</span> "</span>_</span>"</span> +</span> self</span>.name</span></span>
    </span></span>
    def</span> __get__</span>(</span>self</span>,</span> model_instance</span>,</span> class_instance</span>)</span>:</span></span>
        if</span> model_instance</span> is</span> None</span>:</span></span>
            return</span> self</span></span>
        </span></span>
        #</span> TODO</span>: If the property that this one is functioning on is changed, we need a way to know that so that we recalculate</span></span>
        </span></span>
        # Original __get__ in google.appengine.ext.db has getattr to retrieve the value from the model instance</span></span>
        magic_done</span> =</span> getattr</span>(</span>model_instance</span>,</span> self</span>.</span>attr_name</span>()</span>,</span> None</span>)</span></span>
        if</span> magic_done</span> is</span> None</span>:</span></span>
            magic_done</span> =</span> self</span>.</span>magic_func</span>(</span>self</span>.magic_prop.</span>__get__</span>(</span>model_instance</span>,</span> class_instance</span>))</span></span>
            </span></span>
            # Set the attribute in the model</span></span>
            setattr</span>(</span>model_instance</span>,</span> self</span>.</span>_attr_name</span>()</span>,</span> magic_done</span>)</span></span>
        return</span> magic_done</span></span>
        </span></span>
    </span></span>
    def</span> __set__</span>(</span>self</span>,</span> *args</span>)</span>:</span></span>
        raise</span> db.</span>DerivedPropertyError</span>(</span>"</span>MagicProperty is magic. Magic may not be modified.</span>"</span>)</span></span></code></pre>
The approach that the DateTimeProperty</a> takes is one that I can't use
because my class is dependant upon an external property, not whether or not a
value was set when the object was instantiated.</p>
Google AppEngine is posing some rather interesting challenges. Although, for
slugs it may not be entirely bad that it can only be set once, as that means
title changes on the article won't cause the slug to change possibly breaking
old links. This definitely requires more research and more thinking.</p>


Property class on AppEngine does not support a tuple for choices
2010-02-15T04:56:59+00:00
Google AppEngine</a> does not allow one to set the choices field of a model
property</a></a> to a tuple; this workaround will have have a form give the
user the choices with the user friendly items but have the backend store the
non-user friendly version.</p>
# Set up our non-user friendly and user friendly choices</span></span>
REDIRECT_CHOICES</span> =</span> (</span></span>
    (</span>301</span>,</span> "</span>301 - Permanent Redirect</span>"</span>),</span></span>
    (</span>302</span>,</span> "</span>302 - Temporary Redirect</span>"</span>),</span></span>
    (</span>403</span>,</span> "</span>403 - Access Denied</span>"</span>),</span></span>
    (</span>404</span>,</span> "</span>404 - File Not Found</span>"</span>),</span></span>
    (</span>500</span>,</span> "</span>500 - Internal Server Error</span>"</span>)</span></span>
)</span></span>
</span>
# Sample model that has the required Property with the choices set to the</span></span>
# non-user friendly values</span></span>
class</span> Shorturl</span>(</span>db</span>.</span>Model</span>):</span></span>
    uripath</span> =</span> db.</span>StringProperty</span>(</span>verbose_name</span>=</span>"</span>Path</span>",</span> required</span>=</span>True</span>)</span></span>
    httpcode</span> =</span> db.</span>IntegerProperty</span>(</span></span>
        verbose_name</span>=</span>"</span>HTTP code</span>",</span></span>
        required</span>=</span>True</span>,</span></span>
        default</span>=</span>301</span>,</span></span>
        choices</span>=</span>[</span>x</span>[</span>0</span>]</span> for</span> x</span> in</span> REDIRECT_CHOICES</span>]</span></span>
    )</span></span>
    location</span> =</span> db.</span>StringProperty</span>(</span>verbose_name</span>=</span>"</span>Location</span>"</span>)</span></span>
</span>
# This is our djangoform that has the choices set as required so that the user</span></span>
# gets the user friendly version and not just simply the numbers.</span></span>
class</span> ShorturlForm</span>(</span>djangoforms</span>.</span>ModelForm</span>):</span></span>
    httpcode</span> =</span> forms.</span>IntegerField</span>(</span></span>
        widget</span>=</span>forms.</span>Select</span>(</span>choices</span>=</span>REDIRECT_CHOICES</span>)</span>,</span></span>
        required</span>=</span>True</span>,</span></span>
        label</span>=</span>'</span>HTTP code</span>'</span></span>
    )</span></span>
</span>
    class</span> Meta</span>:</span></span>
        model</span> =</span> Shorturl</span></span></code></pre>
Given the above tuple it will set the choices in the model for Google AppEngine
and pass it to the field on the Django form allowing user friendly output. This
will allow the user to pick the right redirect choice without having to know
that various different redirect codes or have to look them up.</p>


addnumbers.asm updated
2008-02-03T09:59:04+00:00
If you have no clue what I am talking about, check out my previous post</a>.
I have fixed the flaws, and now it will parse the commands it is handed on
the command line, and stick them into an integer. It will now print it to
stdout.</p>
This has been an awesome learning experience for me. Especially with regards
to how to do recursion, how to debug a pure assembly program with gdb and
whatnot.</p>
Here comes the code, as I said before, see my previous thread if you don't
know what I am talking about, as well as instructions on how to compile the
program.</p>
; File: addnumbers2.asm</span></span>
; Delta Regeer</span></span>
; </span></span>
; 2008-01-27</span></span>
; </span></span>
; Function:</span></span>
;	Add numbers together that are provided as arguments to the program in argv[1] and argv[2].</span></span>
;</span></span>
; Known limitations:</span></span>
;	This will hopefully be fixed in the next revision. Floating point numbers will not work.</span></span>
;	Any input that is larger than an integer will cause overflows, and thus will not work.</span></span>
 </span></span>
section .data</span></span>
</span>
	; Define some strings that are going to be used throughout the program</span></span>
</span>
	; This string is to let the user know they failed to provide the proper amount of arguments.</span></span>
args	</span>db</span>	"Program </span>addnumbers</span>:</span> ", </span>0xa</span>, </span>0x9</span>, "addnumbers <number </span>1</span>> <number </span>2</span>>", </span>0xa</span>, </span>0x9</span>, "Arguments </span>1</span> and</span> 2</span> are required.", </span>0xa</span>, </span>0x9</span>, "Anything that will cause addition to overflow an </span>int</span> (</span>2</span>,</span>147</span>,</span>483</span>,</span>647</span>), will fail! :P", </span>0xa</span></span>
largs	</span>equ</span>	$ - args</span></span>
</span>
	; This string contains part of the output that we are going to send to the terminal. The last two</span></span>
	; bytes will be filled automatically by the program, before it is output to stdout.</span></span>
msg	</span>db</span>	'</span>Answer</span>:</span> ',	</span>0</span></span>
lmsg	</span>equ</span>	$ - msg</span></span>
</span>
num1	</span>dd</span>	0</span></span>
num2	</span>dd</span>	0</span></span>
</span>
section .bss</span></span>
	; This is where I am going to store the output of my conversion from an integer to a char</span></span>
answer	</span>resb</span>	64</span></span>
</span>
section .text</span></span>
global start				</span>; Linker defined entry point. Mac OS X this is start. </span></span>
global _start				</span>; FreeBSD and others _start.</span></span>
</span>
</span>
_start</span>:</span></span>
start</span>:</span></span>
	push</span>	ebp</span>			; </span></span>
	mov</span>	ebp</span>, </span>esp</span> 		; Set up the stack frame</span></span>
</span>
	mov</span>	ecx</span>, [</span>ebp</span> + </span>4</span>]		</span>; Get argc, we check if it set to at least 3</span></span>
	mov</span>	edx</span>, </span>ebp</span>		; Put the base pointer into edx, so we can use that in </span></span>
					; our dereferences coming up</span></span>
	add</span>	edx</span>, </span>8</span>			; Add 8. We want to skip ebp and argc</span></span>
</span>
	cmp</span>	ecx</span>, </span>3</span>			; Check if we have at least 3 arguments to the program. </span></span>
					; At least two arguments are required, and the 3rd one is </span></span>
					; the name of the program</span></span>
	jl</span>	exit			</span>; If the value in ecx is less than 3, jump to exit</span></span>
</span>
	mov</span>	esi</span>, </span>1</span>			; Set the index to 1</span></span>
	</span></span>
	mov</span>	eax</span>, [</span>edx</span> + </span>esi</span> * </span>4</span>]	</span>; Move the pointer to the character array into eax</span></span>
	push</span>	eax</span>			; Push eax onto the stack</span></span>
	push</span>	num1			</span>; Push the pointer to num1 onto the stack</span></span>
	call</span>	ctoi			</span>; Call my char to int function</span></span>
	add</span>	esp</span>, </span>byte</span> 8</span>		; Put the stack pointer back to where it was.</span></span>
</span>
	inc</span>	esi</span>			; Increase the index</span></span>
	</span></span>
	mov</span>	eax</span>, [</span>edx</span> + </span>esi</span> * </span>4</span>]	</span>; Move the pointer to the character array into eax</span></span>
	push</span>	eax</span>			; Push eax onto the stack</span></span>
	push</span>	num2			</span>; Push the pointer to num2 onto the stack</span></span>
	call</span>	ctoi			</span>; Call my char to int function</span></span>
	add</span>	esp</span>, </span>byte</span> 8</span>		; Put the stack pointer back to where it was.</span></span>
</span>
	mov</span>	eax</span>, [num1]		</span>; Move value stored in num1 into eax</span></span>
	add</span>	eax</span>, [num2]		</span>; Add num2 to eax, this will now be stored in eax</span></span>
</span>
	push</span>	eax</span>			; Push the new calculated number onto the stack</span></span>
	call</span>	itoa			</span>; Convert the integer to a character array</span></span>
</span>
	push</span>	dword</span> lmsg		</span>; Push the length of the string</span></span>
	push</span>	msg			</span>; Push the location of the string in memory</span></span>
	push</span>	dword</span> 0x1</span>		; Push the file descriptor to write to</span></span>
	mov</span>	eax</span>,</span>4</span>			; Move the syscall number into eax</span></span>
	push</span>	eax</span>			; Push the syscall onto the stack</span></span>
	int</span>	0x80</span>			; Interrupt 80, go to kernel</span></span>
	add</span>	esp</span>, </span>byte</span> 16</span>		; Clean up the stack</span></span>
 </span></span>
	push</span>	answer			</span>; Push answer onto the stack</span></span>
	call</span>	len			</span>; Get it's length</span></span>
</span>
	push</span>	edi</span>			; Push the length onto the stack</span></span>
	push</span>	answer			</span>; Push the pointer to the character string onto the stack</span></span>
	push</span>	dword</span> 0x1</span>		; Push the file descriptor to write to</span></span>
	mov</span>	eax</span>,</span>4</span>			; Push the syscall number into eax</span></span>
	push</span>	eax</span>			; Push the syscall onto the stack</span></span>
	int</span>	0x80</span>			; Interrupt 80, go to kernel</span></span>
	add</span>	esp</span>, </span>byte</span> 16</span>		; Clean up the stack</span></span>
</span>
	jmp</span>	done			</span>; Program is done. Jump to done</span></span>
</span>
exit</span>:</span></span>
	; This label is jumped to when we want to exit the program and let the user know how</span></span>
	; to run the program. Like for instance what paramaters to send the program.</span></span>
				; Call sys_write</span></span>
	push</span>	dword</span> largs		</span>; Push the length of the string</span></span>
	push</span>	dword</span> args		</span>; Push the location of the string in memory</span></span>
	push</span>	dword</span> 0x1</span>		; Push the file descriptor to write to</span></span>
	mov</span>	eax</span>,</span>4</span>			; Move the syscall number into eax</span></span>
	push</span>	eax</span>			; Push the syscall onto the stack</span></span>
	int</span>	0x80</span>			; Interrupt 80, go to kernel</span></span>
	add</span>	esp</span>, </span>byte</span> 16</span>		; Clean up the stack</span></span>
</span>
done</span>:</span></span>
	; This is the label we jump to when we want to exit the program, we set the exit code</span></span>
	; to 0.</span></span>
				; Call sys_exit</span></span>
	push</span>	dword</span> 0x0</span>		; Push the value to return to the operating system</span></span>
	mov</span>	eax</span>,</span>1</span>			; Move the syscall number into eax</span></span>
	push</span>	eax</span>			; Push the syscall onto the stack</span></span>
	int</span>	0x80</span>			; Interrupt 80, go to kernel</span></span>
</span>
	; We never return to this function, so no need to clean the stack.</span></span>
</span>
ctoi</span>:</span></span>
	; char to i. We actually convert entire character array's to integers.</span></span>
	;</span></span>
	; We get two paramaters on the stack. The first one we grab is the pointer to the place to store</span></span>
	; the number. The second is the pointer to the character array.</span></span>
</span>
	push</span>	ebp</span>			; Push the old base pointer onto the stack</span></span>
	mov</span>	ebp</span>, </span>esp</span>		; Create a new base pointer</span></span>
	push</span>	esi</span>			; Store all the original registers</span></span>
	push</span>	eax</span></span>
	push</span>	ebx</span></span>
	push</span>	ecx</span>	</span></span>
	push</span>	edx</span>			; Push edx, so that we can overwrite it</span></span>
	sub</span>	esp</span>, </span>4</span>			; We get another storage space on the stack</span></span>
	mov</span>	[</span>esp</span>], </span>dword</span> 10</span>		; This is the number we are going to multiply by</span></span>
</span>
	mov</span>	eax</span>, [</span>ebp</span> + </span>12</span>]		</span>; Move the pointer to the character array into eax</span></span>
	push</span>	eax</span>			; Push the pointer to the character array onto the stack</span></span>
	call</span>	len			</span>; Call the string length versoin</span></span>
	add</span>	esp</span>, </span>byte</span> 4</span>		; Reclaim the space we lost when we pushed eax onto the stack</span></span>
	</span></span>
	mov</span>	ebx</span>, [</span>ebp</span> + </span>8</span>]		</span>; This is where we are going to store the numbers</span></span>
	mov</span>	esi</span>, [</span>ebp</span> + </span>12</span>]		</span>; This is the pointer to the character array</span></span>
	movzx</span>	ecx</span>, </span>di</span>			; move with extended zero edi.</span></span>
	mov</span>	edi</span>, </span>0</span>			; Clean up edi</span></span>
</span>
	ctoi_loop</span>:</span></span>
	mov</span>	eax</span>, [</span>ebx</span>]		</span>; Move the value stored in ebx into eax</span></span>
	mul</span>	dword</span> [</span>esp</span>]		</span>; Move it over a 10s place.</span></span>
	mov</span>	[</span>ebx</span>], </span>eax</span>		; Move the new number back into ebx</span></span>
</span>
	movzx</span>	eax</span>, </span>byte</span> [</span>esi</span> + </span>edi</span>]	</span>; Move the character into eax</span></span>
	movsx</span>	eax</span>, </span>al</span>			; We just want the lower part of the character</span></span>
</span>
	sub</span>	eax</span>, </span>0x30</span>		; Subtract 0x30, ASCII 0 so that it is an actual number</span></span>
	add</span>	[</span>ebx</span>], </span>eax</span>		; Add the new number to the old number that has been multiplied by 10</span></span>
	inc</span>	edi</span>			; Increase the counter</span></span>
	loop</span> ctoi_loop			</span>; Loop into cx is 0</span></span>
</span>
	add</span>	esp</span>, </span>byte</span> 4</span></span>
	pop</span>	edx</span>			; Restore all the registers</span></span>
	pop</span>	ecx</span></span>
	pop</span>	ebx</span></span>
	pop</span>	eax</span></span>
	pop</span>	esi</span></span>
	mov</span>	esp</span>, </span>ebp</span>		; Make esp the original base pointer again</span></span>
	pop</span>	ebp</span>			; Pop the original base pointer into the register</span></span>
	ret</span>				; Return caller</span></span>
</span>
itoa</span>:</span></span>
	; Recursive function. This is going to convert the integer to the character.</span></span>
	push</span>	ebp</span>			; Setup a new stack frame</span></span>
	mov</span>	ebp</span>, </span>esp</span></span>
	push</span>	eax</span>			; Save the registers</span></span>
	push</span>	ebx</span></span>
	push</span>	ecx</span></span>
	push</span>	edx</span></span>
</span>
	mov</span>	eax</span>, [</span>ebp</span> + </span>8</span>]		</span>; eax is going to contain the integer</span></span>
	mov</span>	ebx</span>, </span>dword</span> 10</span>		; This is our "stop" value as well as our value to divide with</span></span>
	mov</span>	ecx</span>, answer		</span>; Put a pointer to answer into ecx</span></span>
	push</span>	ebx</span>			; Push ebx on the field for our "stop" value</span></span>
</span>
	itoa_loop</span>:</span></span>
	cmp</span>	eax</span>, </span>ebx</span>		; Compare eax, and ebx</span></span>
	jl</span>	itoa_unroll		</span>; Jump if eax is less than ebx (which is 10)</span></span>
	xor</span>	edx</span>, </span>edx</span>		; Clear edx</span></span>
	div</span>	ebx</span>			; Divide by ebx (10)</span></span>
	push</span>	edx</span>			; Push the remainder onto the stack</span></span>
	jmp</span>	itoa_loop		</span>; Jump back to the top of the loop</span></span>
	itoa_unroll</span>:</span>			</span></span>
	add</span>	al</span>, </span>0x30</span>		; Add 0x30 to the bottom part of eax to make it an ASCII char</span></span>
	mov</span>	[</span>ecx</span>], </span>byte</span> al</span>		; Move the ASCII char into the memory references by ecx</span></span>
	inc</span>	ecx</span>			; Increment ecx</span></span>
	pop</span>	eax</span>			; Pop the next variable from the stack</span></span>
	cmp</span>	eax</span>, </span>ebx</span>		; Compare if eax is ebx</span></span>
	jne</span>	itoa_unroll		</span>; If they are not equal, we jump back to the unroll loop</span></span>
				; else we are done, and we execute the next few commands</span></span>
	mov</span>	[</span>ecx</span>], </span>byte</span> 0xa</span>		; Add a newline character to the end of the character array</span></span>
	inc</span>	ecx</span>			; Increment ecx</span></span>
	mov</span>	[</span>ecx</span>], </span>byte</span> 0</span>		; Add a null byte to ecx, so that when we pass it to our</span></span>
				; len function it will properly give us a length</span></span>
</span>
	pop</span>	edx</span>			; Restore registers</span></span>
	pop</span>	ecx</span></span>
	pop</span>	ebx</span></span>
	pop</span>	eax</span></span>
	mov</span>	esp</span>, </span>ebp</span>		</span></span>
	pop</span>	ebp</span></span>
	ret</span></span>
</span>
len</span>:</span></span>
	; Returns the length of a string. The string has to be null terminated. Otherwise this function</span></span>
	; will fail miserably. </span></span>
	; Upon return. edi will contain the length of the string.</span></span>
</span>
	push</span>	ebp</span>			; Save the previous stack pointer. We restore it on return</span></span>
	mov</span>	ebp</span>, </span>esp</span>		; We setup a new stack frame</span></span>
	push</span>	eax</span>			; Save registers we are going to use. edi returns the length of the string</span></span>
	push</span> 	ecx</span></span>
	</span></span>
	mov</span>	ecx</span>,  [</span>ebp</span> + </span>8</span>]		</span>; Move the pointer to eax; we want an offset of one, to jump over the return address</span></span>
</span>
	mov</span>	edi</span>, </span>0</span>			; Set the counter to 0. We are going to increment this each loop</span></span>
</span>
	len_loop</span>:</span>			; Just a quick label to jump to</span></span>
	movzx</span>	eax</span>, </span>byte</span> [</span>ecx</span> + </span>edi</span>]	</span>; Move the character to eax.</span></span>
	movsx</span>	eax</span>, </span>al</span>			; Move al to eax. al is part of eax.</span></span>
	inc</span>	di</span>			; Increase di.</span></span>
	cmp</span>	eax</span>, </span>0</span>			; Compare eax to 0.</span></span>
	jnz</span> 	len_loop		</span>; If it is not zero, we jump back to len_loop and repeat.</span></span>
</span>
	dec</span>	di</span>			; Remove one from the count</span></span>
</span>
	pop</span>	ecx</span>			; Restore registers</span></span>
	pop</span>	eax</span></span>
	mov</span>	esp</span>, </span>ebp</span>		; Set esp back to what ebp used to be.</span></span>
	pop</span>	ebp</span>			; Restore the stack frame</span></span>
	ret</span>				; Return to caller</span></span></code></pre>


Introduction to Intel Assembly
2008-01-27T03:42:44+00:00
This semester I am taking a class on Intel assembly, because I want more of an
insight into how the computer works, and it will allow me to better reverse
engineer new viruses and spyware. The class is also required if one is a
Software Engineering major, so that means I have to take it.</p>
We had our first assignment. Sum two numbers and then output them to the
screen. We were supposed to write inline assembly using Visual Studio C++,
but if we are to do an assembly class, then we should learn how to write
assembly, not have some parts assembly and other parts the compiler. Sure
it makes it easy as you will get immediate access to the standard C
library, but if you want that, you can just link against it.</p>
The following code examples were written on Mac OS X, and will work on
FreeBSD. Linux uses a different calling convention for its syscalls, and as
such this code will not run on Linux, unless it is modified. Do note, you
need an Intel Mac for this to work. This is Intel assembly.</p>
Compile the code with (Mac OS X):</p>
nasm</span> -f</span> macho addnumbers.asm</span></span>
ld</span> -o</span> addnumbers addnumbers.o</span></span></code></pre>
or (FreeBSD):</p>
nasm</span> -f</span> elf addnumbers.asm</span></span>
ld</span> -o</span> addnumbers addnumbers.o</span></span></code></pre>
Then you can run it with:</p>
./addnumbers</span> 1 5</span></span></code></pre>
As you can see in the comments of the source code, there are still some
limitations, but the rest of the source code should be made readable by the
comments that are provided.</p>
Porting to Linux: Please double check that all the syscall numbers are the
same. There are some differences between Linux and FreeBSD/Mac OS X in that
regard.</p>
; File: addnumbers.asm</span></span>
; Delta Regeer</span></span>
; </span></span>
; 2008-01-27</span></span>
; </span></span>
; Function:</span></span>
;	Add numbers together that are provided as arguments to the program in argv[1] and argv[2].</span></span>
;</span></span>
; Known limitations:</span></span>
;	As of right now, the numbers that are provided may not add up to anything more than 9.</span></span>
;	This will hopefully be fixed in the next revision. Floating point numbers will not work.</span></span>
;</span></span>
; Todo:</span></span>
;	Write conversion routine, to convert a string of numbers into a real integer on which</span></span>
;	math may be performed.</span></span>
 </span></span>
</span>
</span>
section .data</span></span>
</span>
	; Define some strings that are going to be used throughout the program</span></span>
</span>
	; This string is to let the user know they failed to provide the proper amount of arguments.</span></span>
args	</span>db</span>	"You failed to provide the proper amount of arguments", </span>0xa</span></span>
largs	</span>equ</span>	$ - args</span></span>
</span>
	; This string contains part of the output that we are going to send to the terminal. The last two</span></span>
	; bytes will be filled automatically by the program, before it is output to stdout.</span></span>
msg	</span>db</span>	'</span>Answer</span>:</span> ',	</span>0</span>,	</span>0</span></span>
lmsg	</span>equ</span>	$ - msg</span></span>
</span>
section .text</span></span>
global start				</span>; Linker defined entry point. Mac OS X this is start. FreeBSD and others _start.</span></span>
global _start</span></span>
</span>
_start</span>:</span></span>
start</span>:</span></span>
</span>
; Start the program here.</span></span>
</span>
	add</span>	esp</span>, </span>byte</span> 8</span>		; We don't care about argc or argv[0]</span></span>
	</span></span>
	pop</span> 	ecx</span>			; Get the first argument or argv[1]</span></span>
	jecxz</span>	exit			</span>; If there was no argument. Exit. Let the user know why</span></span>
</span>
	; Change the number from a character to an actual dword</span></span>
	mov</span>	eax</span>, </span>dword</span> [</span>ecx</span>]	</span>; Move the character into eax so we can manipulate it</span></span>
	sub</span>	eax</span>, </span>0x30</span>		; Remove 0x30 from the character. To make it an actual number, not an ASCII number.</span></span>
</span>
	pop</span>	ecx</span>			; get the second argument or argv[2]</span></span>
	jecxz</span>	exit			</span>; If there was no second argument. Exit. Let the user know why</span></span>
</span>
	mov</span>	ebx</span>, </span>dword</span> [</span>ecx</span>]	</span>; Move the character into ebx so we can manipulate it</span></span>
	sub</span>	ebx</span>, </span>0x30</span>		; Remove 0x30 from the character. To make it an actual number, not an ASCII number.</span></span>
</span>
	add</span>	eax</span>, </span>ebx</span>		; Add the two numbers together</span></span>
	add</span>	eax</span>, </span>0x30</span>		; Make it an ASCII number again</span></span>
</span>
	mov</span>	[msg+lmsg-</span>2</span>], </span>eax</span>	; Replace the null character in the msg with the answer</span></span>
	mov</span>	[msg+lmsg-</span>1</span>], </span>dword</span> 0xa</span>	; Add an newline character so that when it spits it out it is neatly formatted</span></span>
				</span></span>
				; Call sys_write</span></span>
	push</span>	dword</span> lmsg		</span>; Push the length of the string</span></span>
	push</span>	msg			</span>; Push the location of the string in memory</span></span>
	push</span>	dword</span> 0x1</span>		; Push the file descriptor to write to</span></span>
	mov</span>	eax</span>,</span>4</span>			; Move the syscall number into eax</span></span>
	push</span>	eax</span>			; Push the syscall onto the stack</span></span>
	int</span>	0x80</span>			; Interrupt 80, go to kernel</span></span>
	add</span>	esp</span>, </span>byte</span> 16</span>		; Advance the stack pointer</span></span>
 </span></span>
	jmp</span>	done			</span>; Program is done. Jump to done</span></span>
</span>
exit</span>:</span></span>
				; Call sys_write</span></span>
	push</span>	dword</span> largs		</span>; Push the length of the string</span></span>
	push</span>	dword</span> args		</span>; Push the location of the string in memory</span></span>
	push</span>	dword</span> 0x1</span>		; Push the file descriptor to write to</span></span>
	mov</span>	eax</span>,</span>4</span>			; Move the syscall number into eax</span></span>
	push</span>	eax</span>			; Push the syscall onto the stack</span></span>
	int</span>	0x80</span>			; Interrupt 80, go to kernel</span></span>
	add</span>	esp</span>, </span>byte</span> 16</span>		; Advance esp past the part we were just at</span></span>
</span>
done</span>:</span></span>
				; sys_exit</span></span>
	push</span>	dword</span> 0x1</span>		; Push the value to return to the operating system</span></span>
	mov</span>	eax</span>,</span>1</span>			; Move the syscall number into eax</span></span>
	push</span>	eax</span>			; Push the syscall onto the stack</span></span>
	int</span>	0x80</span>			; Interrupt 80, go to kernel</span></span>
</span>
	; We never return to this function, so no need to clean the stack.</span></span></code></pre>


X11 mouse button pressed
2007-03-16T04:51:34+00:00
For a Rube Goldberg device I needed a way to know if a mouse button was
pressed on the root window of an X11 screen. Here is the code for it:</p>
#</span>include</span> <</span>stdio.h</span>></span></span>
#</span>include</span> <</span>stdlib.h</span>></span></span>
#</span>include</span> <</span>string.h</span>></span></span>
</span>
#</span>include</span> <</span>unistd.h</span>></span></span>
</span>
#</span>include</span> <</span>X11/Xlib.h</span>></span></span>
#</span>include</span> <</span>X11/Xutil.h</span>></span></span>
</span>
int</span> main</span>() {</span></span>
</span>
        Display </span>*</span>dsp </span>=</span> XOpenDisplay</span>( NULL )</span>;</span></span>
        if</span>(</span> !</span>dsp ){</span> return</span> 1</span>; }</span></span>
</span>
        int</span> screenNumber </span>=</span> DefaultScreen</span>(dsp)</span>;</span></span>
</span>
        long</span> eventMask </span>=</span> StructureNotifyMask;</span></span>
        XSelectInput</span>( dsp,</span> DefaultRootWindow</span>(dsp), eventMask )</span>;</span></span>
</span>
        XEvent evt;</span></span>
</span>
        eventMask </span>=</span> ButtonPressMask</span>|</span>ButtonReleaseMask;</span></span>
        XSelectInput</span>(dsp,</span> DefaultRootWindow</span>(dsp), eventMask)</span>;</span> // override prev</span></span>
</span>
        do</span>{</span></span>
                XNextEvent</span>( dsp,</span> &</span>evt )</span>;</span>   // calls XFlush()</span></span>
                if</span> ( (</span>evt</span>.</span>type</span> ==</span> ButtonRelease)</span> ||</span> (</span>evt</span>.</span>type</span> ==</span> ButtonPress) )</span></span>
                exit</span>(</span>2</span>)</span>;</span></span>
        }</span> while</span>(</span> 1</span> );</span></span>
</span>
return</span> 0</span>;</span></span>
}</span></span></code></pre>
Compile:</p>
gcc</span> -lX11 -L/usr/local/include</span> x11.c</span> -o</span> x11-mousebutton</span></span></code></pre>
Upon a mouse button being pressed it will exit 2, so using that return value
in some script you can execute a command. For example, to open a CD rom
drive so a domino falls over.</p>

funcptr - development

Python Packaging and Distribution

Build numbers in binaries using waf

Full example</h2> Check out my mdns-announce</a> project on Github for an example of how this is implemented.</p>

User sessions, what data should be stored where?

Embedding ZeroMQ in the libev Event Loop

ZeroMQ - Edge Triggered Notification

IPv6 -- getaddrinfo() and bind() ordering with V6ONLY

OpenSSL as a Filter (or non-blocking OpenSSL)

Magic Properties on Google AppEngine

Auto-updating Properties in Google AppEngine Models

Property class on AppEngine does not support a tuple for choices

addnumbers.asm updated

Introduction to Intel Assembly

X11 mouse button pressed

`Full example</h2> Check out my mdns-announce</a> project on Github for an example of how this is implemented.</p>`