funcptr - python

Python Packaging and Distribution

2018-05-21T14:00:00+00:00

There's been many a discussion online related to a variety of tools related to Python packaging and distribution. There is pip</a>, pipenv</a> and poetry</a> that have been the tools under discussion.</p>

As an open source maintainer as part of the Pylons Project</a>, while I would love to be writing code I end up spending a lot of time dealing with user questions around packaging/distributing their source code using the software I've helped build, and as we move forward myself and other maintainers were wondering if we were actually helping users move forward in the best way possible using best of breed tools.1</a></sup></p>

As the Python community has moved from easy_install</a> to pip, we too have kept the documentation up to date. We went from python setup.py develop</code> to pip install -e .</code> to create editable installs of local projects, and try to let people know the pitfalls of using both easy_install and pip in the same project (mostly with an answer that falls in line with: remove your virtual environment and start over, just use pip).</p>

As part of Pyramid</a> we have developed and maintain various different cookiecutter</a> templates, and our goal is to attempt to provide templates that are both useful, but also follow best practices that are being adopted within the community at large so that newcomers can use their existing skills/knowledge and those that are starting with us walk away with a knowledge and experience that applies not just to development of Pyramid applications, but also applies to the broader community as a whole.</p>

pip</h2> Pip is a great tool that has simplified installation of packages, it supports using binary distributions named wheels</a> and has a way to easily install software from the Python Packaging Index</a>. It has a rather naive dependency resolution process, but for the most part it works and works well. It replaced easy_install as the tool to use for installing packages.</p> While you can use a requirements.txt</code> file with pip to install a "blessed" list of software there is no good way to "lock" the dependencies of dependencies without manually adding it to the list of requirements. This ends up making it very difficult to manage, and it is very difficult to know that what has been tested is what the user is actually going to get because packages may be updated at anytime, and re-creating the same exact environment is difficult and fraught with errors.</p> This is where Pipfile</a> is supposed to help. This is a project to add a new, more descriptive requirements file, as well as allowing for a lockfile that would lock not just your primary packages you have listed, but also all dependencies of dependencies all the way down the tree. This helps with reproducibility and allows for the same installation on two different systems to have the exact same software/dependencies installed.</p> pipenv</h2> While pip is a great tool, and with the Pipfile changes it would allow for locking of dependencies, there is one more puzzle piece missing. When installing packages while you can install them into the global namespace, the recommended way is to install all packages for a particular tool/project into a virtual environment.</p> Normally you'd invoke virtualenv</a>, to create this environment and then you'd make sure to install all packages within it, thereby isolating it from the rest of the system.</p> pipenv automates this for you, as well as using Pipfile</code> it also supports locking using Pipfile.lock</code> and provides a bunch of tooling around adding/removing dependencies from a local project.</p> pipenv allows you to easily create an environment and manage dependencies, but it makes no effort to solve the problem of distributing and building a package that may be installed by third parties.</p> poetry</h2> Poetry is a similar project to pipenv, with a major difference being that it was built to help with distributing/developing applications and building a distributable package that may then be installed using pip.</p> Instead of using a Pipfile</code> it uses a recently standardised pyproject.toml</code> file instead. Like pipenv it also supports locking, and it provides tooling around adding/removing dependencies as well as managing what versions are required.</p> Ultimately those dependencies are going to end up as metadata in a distributable package.</p> Poetry makes it easier to manage a software development project, whether that is for an application using various libraries for internal use, or for libraries that are going to be distributed to other developers.</p> The divide</h2> This is where the divide really starts, while you can use pipenv with a standard setuptools</a> project, any dependencies you add to the Pipfile using pipenv's tooling will not be listed as a dependencies for your project when you distribute it, this either means you need to duplicate the list in both setup.py</code> as well as the Pipfile</code>, or you have to add your current project as an editable install within your Pipfile</code> which means your Pipfile</code> is now not as easily distributable.</p> There are work-arounds that people have used, such as having setup.py</code> read a requirements.txt</code>, so that you could have all your requirements listed in a text file, and not in setup.py</code>, but asking to do the same with a Pipfile</code> in pipenv was met with a "Do not do this."</a>.</p> poetry explicitly allows you to add dependencies in one place, and those dependency listings are then automatically inserted into the package metadata that is created when you build your distributable package.</p> The two use cases</h2> There are two competing use cases, one is the deployment of software packages and being able to run them, but not as a developer, the other is a developer of software packages that needs to define dependencies for the project to run.</p> pipenv solves the deployment case. If I was a user I could very simply grab a known good Pipfile.lock</code> and use pipenv to install a known good set of software, this is great when I am deploying a project. It is the use case that many in the Python Packaging Authority</a> also seem to be optimizing for.</p> The other use case is for developers that are building new software, either by using a list of existing packages and deploying privately, or people developing software for other developers to be published on the Python Packaging Index.</p> This latter group of people is under represented due to it likely being much smaller, and existing tools like setuptools</code> and setup.py</code> already providing a "good enough" experience. Innovation in this area is something that readily needs to be improved upon to make it easier to create new libraries/packages that follow best practices. The amount of copy and pastes people have done for adding a setup.py</code> to their projects or to make something work is long. It's all a little bit of black magic, and there is a great many things that have been carried over because of cargo cult programming.</p> Explicit mentions by the Python Packaging Authority</h2> Reading the packaging guide on managing dependencies</a>, pipenv is the recommended tool:</p> This tutorial walks you through the use of Pipenv to manage dependencies for an application. It will show you how to install and use the necessary tools and make strong recommendations on best practices.</p> </blockquote> this language, along with what packaging.python.org</code> implies as a URL makes it difficult as a project maintainer to recommend alternate tools, becuase even if those tools are superior for the use case we are recommending them for it is always going to lead to questions from users, such as:</p> Why are you not using pipenv, the official tool recommended by Python.org?</p> </blockquote> We get similar questions about easy_install</code> vs pip</code> all of the time, as well as why people should switch, and we can point to various bits of documentation that explains why pip is a better choice.</p> If we were to recommend an alternate the appeal to authority that python.org</code> implies is going to make it much more difficult, and the question will become "why is the Pylons Project not using recommended tooling?"</p> poetry is listed as a footnote on that page, alongside pip-tools</a> and hatch</a>, and is mentioned only for doing library development, with no mention of other requirements that may make it a much better tool for developing locally.</p> Deployment is not development</h2> If I am using pipenv with a non-installable project (no setup.py) I end up having to figure out how to get the code, and the Pipfile</code>/Pipfile.lock</code> to my environment I am deploying into. pipenv's install provides a way to make sure to only install if the Pipfile.lock</code> is up to date or otherwise will fail to continue. If you are using a local project though, and it uses setup.py</code> the only way that the Pipfile.lock</code> will contain any sub-dependencies of your setup.py</code> project is if you install it as editable</code>. Otherwise sub-dependencies are not locked.2</a></sup></p> If I am using poetry I get an pip installable project, but it doesn't contain any hard pins or lock files. I'd have to distribute pyproject.lock</code> as well as my wheel. This gets me a little closer, but still no lock file that includes my newly produced wheel, and has all of its dependencies locked.</p> The Python Packaging Authority based on Twitter conversations with its members and the documentation on packaging.python.org</code> suggest using pipenv for development. pipenv is particular ill-suited for development if the goal is to create a package to be deployed to production. With two locations to define dependencies it leaves people scratching their heads as to which is canonical, and if a dependency is added to Pipfile</code> but not setup.py</code> it may leave a developer thinking their package is ready for distribution when in reality it is missing a dependency that is required to run/use said distribution.</p> At this point using both projects seems like a win-win. Use poetry to build/develop a package, then use pipenv in the integration phase to create a Pipfile.lock</code> that is used to deploy in production. This way you get the best of both worlds. A great tool that can help you register entry points and another that can help you with deploying a known good set of dependencies.</p> Interestingly, even the pipenv docs seem to agree that it is a deployment tool:</p> Specify your target Python version in your Pipfile’s [requires] section. Ideally, you should only have one target Python version, as this is a deployment tool.</p> -- Pipenv - General Recommendations & Version Control</a></p> </blockquote> Use pipenv if you have a script that requires a couple of dependencies and doesn't need all of the extra overhead of packaging metadata/packaging. Use poetry if you want to build a distributable project that can easily be deployed by others, and use both if you develop a project and need a known good environment to deploy.</p> In summary</h2> There will likely never be a time that one single tool is considered good enough, and competition between tools is a way to keep advancing forward. Packaging in the Python community for a long time has been difficult. Wheels has made things a little better. pip has made management of installing new packages easier and improved upon easy_install. Here's to the next evolution.</p> Now, can we talk about standardising on pyproject.toml</code> since that is already where "project" metadata needs to go, might as well re-use the name instead of having two different names/files. Oh, and PEP 517</a> can't come soon enough so that alternate tools like flit</a> can be used instead of setuptools/setup.py</code>.</p> We created an issue named Support poetry, flit, pipenv, or ...?</a> that attempts to go over the pros and cons of the various tools and how we currently support our users in our documentation on building projects using pyramid, including how to create a project that is distributable. Pyramid heavily uses pkg_resources</code></a> and entry points. The way to register the entry points is to have an installable package. The framework is flexible enough that there is no requirement for entry points, but at that point you are in territory where the default tooling provided by the project will not work, and some of the convenience tools/functionality that Pyramid provides it's users/developers is not available. ↩</a></p> </li> See documentation for Editable Dependencies (e.g. -e .</code>)</a> which as of this writing states: Sub-dependencies are not</strong> added to the Pipfile.lock if you leave the -e option out.</blockquote> ↩</a></p> </li> </ol> </section>



Build numbers in binaries using waf
2014-06-10T17:32:03+00:00
My build system of choice these days for any C++ project is waf</a>. One of
the things I always like havig is the build number included in the final
binary, so that with a simple ./binary --version</code> or even ./binary</code> the
version is printed that it was built from. This can make it much simpler to
debug any potential issues, especially if fixes may have already been made but
a bad binary was deployed.</p>
Setup the wscript</h2>
Make sure that your wscript somewhere near the top contains the following:</p>
APPNAME</span> =</span> '</span>myapp</span>'</span></span>
VERSION</span> =</span> '</span>0.0.0</span>'</span></span></code></pre>
Then in your configure(cfg)</code> add the following:</p>
cfg.env.</span>VERSION</span> =</span> VERSION</span></span>
cfg.env.</span>APPNAME</span> =</span> APPNAME</span></span>
</span>
git_version</span> =</span> try_git_version</span>()</span></span>
</span>
if</span> git_version:</span></span>
    cfg.env.</span>VERSION</span> +=</span> '</span>-</span>'</span> +</span> git_version</span></span></code></pre>
The try_git_version()</code> function is fairly simple and looks like this:</p>
def</span> try_git_version</span>()</span>:</span></span>
    import</span> os</span></span>
    import</span> sys</span></span>
</span>
    version</span> =</span> None</span></span>
    try</span>:</span></span>
        version</span> =</span> os.</span>popen</span>(</span>'</span>git describe --always --dirty --long</span>'</span>).</span>read</span>().</span>strip</span>()</span></span>
    except</span> Exception</span> as</span> e:</span></span>
        print</span> e</span></span>
    return</span> version</span></span></code></pre>
It runs git describe --always --dirty --long</code> which will return something
along these lines: 401b85f-dirty</code>. If you have any annoted tags, it will
return the tag name as well.</p>
If git</code> is not installed, or it is not a valid git</code> directory, then it will
simply return None</code>. At that point all we have to go on is the VERSION</code>
variable set at the top of the wscript.</p>
Now that we have our configuration environment set up with the VERSION</code> we
want to get that into a file that we can then include in our C++ source code.</p>
Create a build_version.h.in</code> file</h2>
#</span>ifndef</span> BUILD_VERSION_H_IN_941AD1F24D0A9D</span></span>
#</span>define</span> BUILD_VERSION_H_IN_941AD1F24D0A9D</span></span>
</span>
char</span> VERSION[] </span>=</span> "</span>@VERSION@</span>"</span>;</span></span>
</span>
#</span>endif</span> /* BUILD_VERSION_H_IN_941AD1F24D0A9D */</span></span></code></pre>Add the following to build(ctx)</code></h2>
ctx</span>(</span>features</span>=</span>'</span>subst</span>',</span></span>
        source</span>=</span>'</span>build_version.h.in</span>',</span></span>
        target</span>=</span>'</span>build_version.h</span>',</span></span>
        VERSION</span> =</span> ctx.env</span>[</span>'</span>VERSION</span>'</span>]</span>,</span></span>
        )</span></span></code></pre>
This uses the substitution feature to transform build_version.h.in</code> into
build_version.h</code>, while inserting the version into the file.</p>
Include build_version.h</code> in your source code</h2>
#</span>include</span> "</span>build_version.h</span>"</span></span></code></pre>
And add something along these lines to your main()</code>:</p>
std::cerr </span><<</span> "</span>Version: </span>"</span> <<</span> VERSION </span><<</span> std::endl;</span></span></code></pre>
This will print out the VERSION</code> that has been stored in build_version.h</code>.</p>
Full example</h2>
Check out my mdns-announce</a> project on Github for an example of how this is
implemented.</p>


Generating stylesheets for pygments
2011-11-27T00:00:00+00:00
Update: this blog is no longer using the same technology for deployment as
before, the below information is available for historical reasons.</p>

For this blog I am using Python markdown</a> with the Pygments</a> plugin
that is available. Pygments does all of the code highlighting and outputs the
required span</code> tags and inputs them within the page.</p>
If you were to look at the <div><pre></pre></div></code> block that is generated
you'd see something like this:</p>
<div class="codehilite"><pre><span class="k">def</span> <span class="nf">function</span><span class="p">(</span><span class="n">var</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span></span></span>
<span class="k">if</span> <span class="n">var</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span><span class="p">:</span></span></span>
    <span class="k">print</span> <span class="n">var</span></span></span>
</pre></div></span></span></code></pre>
which is generated from the following snippet of code:</p>
def</span> function</span>(</span>var=</span>None</span>)</span>:</span></span>
    if</span> var</span> is not</span> None</span>:</span></span>
        print</span> var</span></span></code></pre>
As you see the top css class</code> is codehilite</code>, and then underneath that there
are a bunch of other classes such as k</code>, n</code>, o</code>, and so forth. Those are all
classes that can be styled. Now one could write such a stylesheet by hand or we
can simply use one of the stylesheets that Pygments ships with.</p>
The documentation for Pygments tells you how to get built-in styles</a>, which
is as simple as:</p>
>>></span> from</span> pygments.styles</span> import</span> get_all_styles</span></span>
>>></span> styles</span> =</span> list</span>(</span>get_all_styles</span>())</span></span>
>>></span> styles</span></span>
['</span>monokai</span>'</span>,</span> '</span>manni</span>'</span>,</span> '</span>perldoc</span>'</span>,</span> '</span>borland</span>'</span>,</span> '</span>colorful</span>'</span>,</span> '</span>default</span>'</span>,</span> ...</span>]</span> </span></span></code></pre>
These are all the styles that are available to you the user, now how do we get
the style into an actual .css</code> file that we can link to in our sites HTML
file? We use the pygmentize</code> command line tool:</p>
pygmentize</span> -f</span> html</span> -S</span> trac</span> -a</span> .codehilite</span> ></span> pygments.css</span></span></code></pre>
Feel free to replace trac</code> with any of the styles you have installed on your
system. -f</code> tells it what formatter to use, since we want it to output the css
we ask it to use the HTML formatter, -S</code> tells pygmentize</code> what style to
use, and -a</code> prepends the .codehilite</code> class before every rule (this way we
don't pollute the whole CSS namespace with all the aforementioned class's such
as k</code>, o</code>, ow</code> and others).</p>


Magic Properties on Google AppEngine
2010-03-24T02:15:33+00:00
Update to my previous Google AppEngine auto-updating properties post</a>.</p>
In my last post I was talking about:</p>

[...] however there is still an issue, for now the MagicProperty is only
updated the first time the property it affects is set</p>
</blockquote>
I fixed that issue, it has a local cache that is added to the model, so if the
value does change, the cache would no longer be valid and the value is
recreated, otherwise the already computed value is returned. There was however
another issue I had to deal with, and that was the fact that the cache was not
being written to the datastore, this meant that upon getting the data back from
the datastore we recomputed it anyway. Now the programmer has to do some extra
work, they have to add a cache unindexed property to the model, and pass it
into the MagicProperty</code> to be used as the caching variable. Within the
save()</code> they have to prime the cache by setting a temporary variable equal to
the to be computed variable just in case the cache variables have not been
primed, this has not yet been documented other than in my SVN history.</p>
We still don't allow setting of the magic property, however upon returning from
the datastore it calls make_value_from_datastore</code>, which returns a
_MagicDatastore</code>, as long as we set with an instance of _MagicDatastore</code> it will
allow it to go through, so we can get the value from the datastore without
having to recompute the value.</p>
Read the code for an example on how to use this.</p>
Code is added below, feel free to use it as you wish under the license that is
attached.</p>
  1</span>###</span></span>
  2</span> # Copyright (c) 2010 Delta Regeer;</span></span>
  3</span> #</span></span>
  4</span> # Permission to use, copy, modify, and distribute this software for any</span></span>
  5</span> # purpose with or without fee is hereby granted, provided that the above</span></span>
  6</span> # copyright notice and this permission notice appear in all copies.</span></span>
  7</span> #</span></span>
  8</span> # THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES</span></span>
  9</span> # WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF</span></span>
 10</span> # MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR</span></span>
 11</span> # ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES</span></span>
 12</span> # WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN</span></span>
 13</span> # ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF</span></span>
 14</span> # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.</span></span>
 15</span> #</span></span>
 16</span>###</span></span>
 17</span></span>
 18</span>import</span> logging</span></span>
 19</span>import</span> hashlib</span></span>
 20</span>from</span> google.appengine.ext</span> import</span> db</span></span>
 21</span></span>
 22</span></span>
 23</span>def</span> MagicProperty</span>(</span>prop</span>,</span> magic_func=</span>None</span>,</span> cache_prop=</span>None</span>,</span> pass_instance=</span>False</span>,</span> *args</span>,</span> **kw</span>)</span>:</span></span>
 24</span>    if</span> magic_func:</span></span>
 25</span>        # No pants required.</span></span>
 26</span>        return</span> _MagicProperty</span>(</span>prop</span>,</span> magic_func</span>,</span> cache_prop</span>,</span> pass_instance</span>,</span> *</span>args</span>,</span> **</span>kw</span>)</span></span>
 27</span>    else</span>:</span></span>
 28</span>        # We are putting some pants on the function.</span></span>
 29</span>        def</span> pants</span>(</span>magic_func</span>)</span>:</span></span>
 30</span>            return</span> _MagicProperty</span>(</span>prop</span>,</span> magic_func</span>,</span> cache_prop</span>,</span> pass_instance</span>,</span> *</span>args</span>,</span> **</span>kw</span>)</span></span>
 31</span>        return</span> pants</span></span>
 32</span></span>
 33</span>class</span> _MagicDatastore</span>():</span></span>
 34</span>    """</span>This is an internal class for _MagicProperty.</span>"""</span></span>
 35</span>    </span></span>
 36</span>    def</span> __init__</span>(</span>self</span>,</span> val</span>)</span>:</span></span>
 37</span>        self</span>.value</span> =</span> val</span></span>
 38</span>    </span></span>
 39</span>    def</span> retval</span>(</span>self</span>)</span>:</span></span>
 40</span>        return</span> self</span>.value</span></span>
 41</span></span>
 42</span>class</span> _MagicProperty</span>(</span>db</span>.</span>Property</span>):</span></span>
 43</span>    """</span>MagicProperty which will modify output based on a function that it is given.</span></span>
 44</span>    </span></span>
 45</span>    This has several ways in which it may be called:</span></span>
 46</span>    </span></span>
 47</span>    In this example we create the model, no caching is done, so any data that is returned from the datastore</span></span>
 48</span>    will be erased the first time that the MagicProperty is accessed and recomputed.</span></span>
 49</span>    </span></span>
 50</span>        class MagicTest(db.Model):</span></span>
 51</span>            title = db.StringProperty(required=True)</span></span>
 52</span>            chars = utils.MagicProperty(title, len, required=True)</span></span>
 53</span>    </span></span>
 54</span>        mytest = MagicTest(title="It was for the good of the school!")</span></span>
 55</span>        </span></span>
 56</span>        >>> </span>print mytest.title</span></span>
 57</span>        It was for the good of the school!</span></span>
 58</span>        >>> </span>print mytest.chars</span></span>
 59</span>        34</span></span>
 60</span>        </span></span>
 61</span>        mytest = MagicTest.all().get()</span></span>
 62</span>        </span></span>
 63</span>        >>> </span>print mytest.title</span></span>
 64</span>        Hello</span></span>
 65</span>        >>> </span>print mytest.chars		</span></span>
 66</span>        5	# Do note, this is recalculated the first time it is called, as long as title does not change it won't be recomputed.</span></span>
 67</span>    </span></span>
 68</span>    In this example we create the model, and we also create a caching property so that even when we get values back </span></span>
 69</span>    from the datastore we use the cached computed value rather than running the function again. Do note that this requires overriding put()</span></span>
 70</span>    to prime the cache as there is currently no way to specify that certain properties should be "saved" before others.</span></span>
 71</span>    </span></span>
 72</span>        class MagicTesting(db.Model):</span></span>
 73</span>            title = db.StringProperty(required=True)</span></span>
 74</span>            cache = db.UnindexedProperty()</span></span>
 75</span>            chars = utils.MagicProperty(title, len, cache_prop=cache, required=True)</span></span>
 76</span>        </span></span>
 77</span>            def put(self, *args, **kw):</span></span>
 78</span>                prime_cache = self.chars</span></span>
 79</span>                super(MagicTesting).put(*args, **kw)</span></span>
 80</span>        </span></span>
 81</span>        mytesting = MagicTesting(title="Get the pocket knife out of my boot.")</span></span>
 82</span>    </span></span>
 83</span>        >>> </span>print mytesting.title</span></span>
 84</span>        Get the pocket knife out of my boot.</span></span>
 85</span>        >>> </span>print mytesting.chars</span></span>
 86</span>        36</span></span>
 87</span>        </span></span>
 88</span>        mytesting = MagicTest.all().get()</span></span>
 89</span>        </span></span>
 90</span>        >>> </span>print mytesting.title</span></span>
 91</span>        How are you?</span></span>
 92</span>        >>> </span>print mytesting.chars	</span></span>
 93</span>        12	# This is not recomputed so long as the title has not changed, however it uses more datastore space to store a hash.</span></span>
 94</span>        </span></span>
 95</span>        </span></span>
 96</span>    Inspired by: </span></span>
 97</span>    http://appengine-cookbook.appspot.com/recipe/custom-model-properties-are-cute</span></span>
 98</span>    http://code.google.com/appengine/articles/extending_models.html</span></span>
 99</span>    http://googleappengine.blogspot.com/2009/07/writing-custom-property-classes.html</span></span>
100</span>    </span></span>
101</span>    """</span></span>
102</span>    def</span> __init__</span>(</span>self</span>,</span> prop</span>,</span> magic_func</span>,</span> cache_prop</span>,</span> pass_instance</span>,</span> *args</span>,</span> **kw</span>)</span>:</span></span>
103</span>        """</span></span>
104</span>        Extra parameters you can give this initializer.</span></span>
105</span>        </span></span>
106</span>            prop		= Property to be acted upon</span></span>
107</span>            magic_func	= The function to be called when the property is accessed</span></span>
108</span>            cache_prop	= The property that can hold our cache, I suggest it is an db.UnindexedProperty() since it just stores sha1 hashes</span></span>
109</span>        """</span></span>
110</span>        super</span>(</span>_MagicProperty</span>,</span> self</span>).</span>__init__</span>(</span>*</span>args</span>,</span> **</span>kw</span>)</span></span>
111</span>        self</span>.magic_func</span> =</span> magic_func</span></span>
112</span>        self</span>.magic_prop</span> =</span> prop</span></span>
113</span>        self</span>.magic_cache</span> =</span> cache_prop</span></span>
114</span>        self</span>.magic_pass</span> =</span> pass_instance</span></span>
115</span>        </span></span>
116</span>    def</span> get_cache_val</span>(</span>self</span>,</span> model_instance</span>,</span> class_instance</span>)</span>:</span></span>
117</span>        if</span> self</span>.magic_cache</span> is not</span> None</span>:</span></span>
118</span>            return</span> self</span>.magic_cache.</span>__get__</span>(</span>model_instance</span>,</span> class_instance</span>)</span></span>
119</span>        return</span> getattr</span>(</span>model_instance</span>,</span> self</span>.</span>attr_name</span>()</span> +</span> "</span>orig</span>",</span> None</span>)</span></span>
120</span>            </span></span>
121</span>    def</span> set_cache_val</span>(</span>self</span>,</span> model_instance</span>,</span> val</span>)</span>:</span></span>
122</span>        val</span> =</span> hashlib.</span>sha1</span>(</span>val</span>).</span>hexdigest</span>()</span></span>
123</span>        </span></span>
124</span>        if</span> self</span>.magic_cache</span> is not</span> None</span>:</span></span>
125</span>            self</span>.magic_cache.</span>__set__</span>(</span>model_instance</span>,</span> val</span>)</span></span>
126</span>        setattr</span>(</span>model_instance</span>,</span> self</span>.</span>attr_name</span>()</span> +</span> "</span>orig</span>",</span> val</span>)</span></span>
127</span>    </span></span>
128</span>    def</span> attr_name</span>(</span>self</span>)</span>:</span></span>
129</span>        # In google.appengine.ex.db there is an explicit warning not to use this method, so we test for it first.</span></span>
130</span>        if</span> self</span>._attr_name:</span></span>
131</span>            return</span> self</span>.</span>_attr_name</span>()</span></span>
132</span>        else</span>:</span></span>
133</span>            return</span> "</span>_</span>"</span> +</span> self</span>.name</span></span>
134</span>    </span></span>
135</span>    def</span> __get__</span>(</span>self</span>,</span> model_instance</span>,</span> class_instance</span>)</span>:</span></span>
136</span>        if</span> model_instance</span> is</span> None</span>:</span></span>
137</span>            return</span> self</span></span>
138</span>        </span></span>
139</span>        cur</span> =</span> self</span>.magic_prop.</span>__get__</span>(</span>model_instance</span>,</span> class_instance</span>)</span></span>
140</span>        cur</span> =</span> cur.</span>encode</span>(</span>'</span>utf-8</span>'</span>)		</span></span>
141</span>        last</span> =</span> self</span>.</span>get_cache_val</span>(</span>model_instance</span>,</span> class_instance</span>)</span></span>
142</span>        if</span> last</span> ==</span> hashlib.</span>sha1</span>(</span>cur</span>).</span>hexdigest</span>():</span></span>
143</span>            logging.</span>info</span>(</span>"</span>Cache hit: </span>%s</span>"</span> %</span> (cur)</span>)</span></span>
144</span>            return</span> getattr</span>(</span>model_instance</span>,</span> self</span>.</span>attr_name</span>()</span>,</span> None</span>)</span></span>
145</span>        </span></span>
146</span>        logging.</span>info</span>(</span>"</span>Cache miss: </span>%s</span>"</span> %</span> (cur)</span>)</span></span>
147</span>        </span></span>
148</span>        magic_done</span> = u</span>""</span></span>
149</span>        if</span> self</span>.magic_pass:</span></span>
150</span>            magic_done</span> =</span> self</span>.</span>magic_func</span>(</span>model_instance</span>,</span> cur</span>)</span></span>
151</span>        else</span>:</span></span>
152</span>            magic_done</span> =</span> self</span>.</span>magic_func</span>(</span>cur</span>)</span></span>
153</span>        </span></span>
154</span>        # Set the attribute in the model</span></span>
155</span>        setattr</span>(</span>model_instance</span>,</span> self</span>.</span>attr_name</span>()</span>,</span> magic_done</span>)</span></span>
156</span>        self</span>.</span>set_cache_val</span>(</span>model_instance</span>,</span> cur</span>)</span></span>
157</span>        </span></span>
158</span>        return</span> magic_done</span></span>
159</span>        </span></span>
160</span>    </span></span>
161</span>    def</span> __set__</span>(</span>self</span>,</span> model_instance</span>,</span> value</span>)</span>:</span></span>
162</span>        if</span> isinstance</span>(</span>value</span>,</span> _MagicDatastore</span>):</span></span>
163</span>            setattr</span>(</span>model_instance</span>,</span> self</span>.</span>attr_name</span>()</span>,</span> value.</span>retval</span>())</span></span>
164</span>        else</span>:</span></span>
165</span>            raise</span> db.</span>DerivedPropertyError</span>(</span>"</span>MagicProperty is magic. Magic may not be modified.</span>"</span>)</span></span>
166</span>    </span></span>
167</span>    def</span> make_value_from_datastore</span>(</span>self</span>,</span> value</span>)</span>:</span></span>
168</span>        return</span> _MagicDatastore</span>(</span>value</span>)</span></span></code></pre>


Auto-updating Properties in Google AppEngine Models
2010-02-21T04:03:42+00:00
EDIT: I have an updated version of this code available in a newer post, please
go to Magic Properties on Google AppEngine</a> for the updated information.
For the generic information that has not changed, and the idea behind the code
please continue below.</p>

In Google AppEngine a model may contain many different properties, from this
model you can generate a form using djangoforms. This is all good and well
until you have a required property on a model that needs to have part of it be
calculated based upon something the user types in. For example, a blog entry
generally has a "slug" which is generated from the title, now in Google
AppEngine's models, if a property is required it has to be set at object
creation time.</p>
Djangoforms has a form.save(committed=False)</code> that allows one to get back an
unsaved Model</code>, however since slug is not in the form (since it is auto
generated) it cannot be set at model creation time. Even adding a hidden field
to the form with the name of the property is not going to satisfy the model
since a value of None</code> is considered to be empty, which means it does not meet
the required=True</code> status.</p>
After trying a multitude of different options I ran across an article about
extending models</a> that specifically talked about adding custom properties
that were saved into the DB, and could be searched on, this got me digging and
I found a blog post on the Google AppEngine blogspot</a> about the exact same
thing, using yet again a different example. The one that got me on the right
track was the following post by Rodrigo Moraes on his custom model
properties</a>.</p>
I came up with the following fairly quickly after also taking a look at
googleappengine.ext.db</code> within the Google AppEngine framework, however there is
still an issue, for now the MagicProperty</code> is only updated the first time the
property it affects is set. I am not yet sure how to work around that, I will
have to do some more digging, maybe there is a way to get notified if another
property gets modified or updated.</p>
import</span> logging</span></span>
from</span> google.appengine.ext</span> import</span> db</span></span>
</span>
</span>
def</span> MagicProperty</span>(</span>prop</span>,</span> magic_func=</span>None</span>,</span> *args</span>,</span> **kw</span>)</span>:</span></span>
    if</span> magic_func:</span></span>
        # No pants required.</span></span>
        return</span> _MagicProperty</span>(</span>prop</span>,</span> magic_func</span>,</span> *</span>args</span>,</span> **</span>kw</span>)</span></span>
    else</span>:</span></span>
        # We are putting some pants on the function.</span></span>
        def</span> pants</span>(</span>magic_func</span>)</span>:</span></span>
            return</span> _MagicProperty</span>(</span>prop</span>,</span> magic_func</span>,</span> *</span>args</span>,</span> **</span>kw</span>)</span></span>
        return</span> pants</span></span>
</span>
</span>
class</span> _MagicProperty</span>(</span>db</span>.</span>Property</span>):</span></span>
    """</span>MagicProperty which will modify output based on a function that it is given.</span></span>
    </span></span>
    Inspired by: </span></span>
    http://appengine-cookbook.appspot.com/recipe/custom-model-properties-are-cute</span></span>
    http://code.google.com/appengine/articles/extending_models.html</span></span>
    http://googleappengine.blogspot.com/2009/07/writing-custom-property-classes.html</span></span>
    </span></span>
    """</span></span>
    def</span> __init__</span>(</span>self</span>,</span> prop</span>,</span> magic_func</span>,</span> *args</span>,</span> **kw</span>)</span>:</span></span>
        super</span>(</span>_MagicProperty</span>,</span> self</span>).</span>__init__</span>(</span>*</span>args</span>,</span> **</span>kw</span>)</span></span>
        self</span>.magic_func</span> =</span> magic_func</span></span>
        self</span>.magic_prop</span> =</span> prop</span></span>
    </span></span>
    def</span> attr_name</span>(</span>self</span>)</span>:</span></span>
        # In google.appengine.ex.db there is an explicit warning not to use this method, so we test for it first.</span></span>
        if</span> self</span>._attr_name:</span></span>
            return</span> self</span>.</span>_attr_name</span>()</span></span>
        else</span>:</span></span>
            return</span> "</span>_</span>"</span> +</span> self</span>.name</span></span>
    </span></span>
    def</span> __get__</span>(</span>self</span>,</span> model_instance</span>,</span> class_instance</span>)</span>:</span></span>
        if</span> model_instance</span> is</span> None</span>:</span></span>
            return</span> self</span></span>
        </span></span>
        #</span> TODO</span>: If the property that this one is functioning on is changed, we need a way to know that so that we recalculate</span></span>
        </span></span>
        # Original __get__ in google.appengine.ext.db has getattr to retrieve the value from the model instance</span></span>
        magic_done</span> =</span> getattr</span>(</span>model_instance</span>,</span> self</span>.</span>attr_name</span>()</span>,</span> None</span>)</span></span>
        if</span> magic_done</span> is</span> None</span>:</span></span>
            magic_done</span> =</span> self</span>.</span>magic_func</span>(</span>self</span>.magic_prop.</span>__get__</span>(</span>model_instance</span>,</span> class_instance</span>))</span></span>
            </span></span>
            # Set the attribute in the model</span></span>
            setattr</span>(</span>model_instance</span>,</span> self</span>.</span>_attr_name</span>()</span>,</span> magic_done</span>)</span></span>
        return</span> magic_done</span></span>
        </span></span>
    </span></span>
    def</span> __set__</span>(</span>self</span>,</span> *args</span>)</span>:</span></span>
        raise</span> db.</span>DerivedPropertyError</span>(</span>"</span>MagicProperty is magic. Magic may not be modified.</span>"</span>)</span></span></code></pre>
The approach that the DateTimeProperty</a> takes is one that I can't use
because my class is dependant upon an external property, not whether or not a
value was set when the object was instantiated.</p>
Google AppEngine is posing some rather interesting challenges. Although, for
slugs it may not be entirely bad that it can only be set once, as that means
title changes on the article won't cause the slug to change possibly breaking
old links. This definitely requires more research and more thinking.</p>


Property class on AppEngine does not support a tuple for choices
2010-02-15T04:56:59+00:00
Google AppEngine</a> does not allow one to set the choices field of a model
property</a></a> to a tuple; this workaround will have have a form give the
user the choices with the user friendly items but have the backend store the
non-user friendly version.</p>
# Set up our non-user friendly and user friendly choices</span></span>
REDIRECT_CHOICES</span> =</span> (</span></span>
    (</span>301</span>,</span> "</span>301 - Permanent Redirect</span>"</span>),</span></span>
    (</span>302</span>,</span> "</span>302 - Temporary Redirect</span>"</span>),</span></span>
    (</span>403</span>,</span> "</span>403 - Access Denied</span>"</span>),</span></span>
    (</span>404</span>,</span> "</span>404 - File Not Found</span>"</span>),</span></span>
    (</span>500</span>,</span> "</span>500 - Internal Server Error</span>"</span>)</span></span>
)</span></span>
</span>
# Sample model that has the required Property with the choices set to the</span></span>
# non-user friendly values</span></span>
class</span> Shorturl</span>(</span>db</span>.</span>Model</span>):</span></span>
    uripath</span> =</span> db.</span>StringProperty</span>(</span>verbose_name</span>=</span>"</span>Path</span>",</span> required</span>=</span>True</span>)</span></span>
    httpcode</span> =</span> db.</span>IntegerProperty</span>(</span></span>
        verbose_name</span>=</span>"</span>HTTP code</span>",</span></span>
        required</span>=</span>True</span>,</span></span>
        default</span>=</span>301</span>,</span></span>
        choices</span>=</span>[</span>x</span>[</span>0</span>]</span> for</span> x</span> in</span> REDIRECT_CHOICES</span>]</span></span>
    )</span></span>
    location</span> =</span> db.</span>StringProperty</span>(</span>verbose_name</span>=</span>"</span>Location</span>"</span>)</span></span>
</span>
# This is our djangoform that has the choices set as required so that the user</span></span>
# gets the user friendly version and not just simply the numbers.</span></span>
class</span> ShorturlForm</span>(</span>djangoforms</span>.</span>ModelForm</span>):</span></span>
    httpcode</span> =</span> forms.</span>IntegerField</span>(</span></span>
        widget</span>=</span>forms.</span>Select</span>(</span>choices</span>=</span>REDIRECT_CHOICES</span>)</span>,</span></span>
        required</span>=</span>True</span>,</span></span>
        label</span>=</span>'</span>HTTP code</span>'</span></span>
    )</span></span>
</span>
    class</span> Meta</span>:</span></span>
        model</span> =</span> Shorturl</span></span></code></pre>
Given the above tuple it will set the choices in the model for Google AppEngine
and pass it to the field on the Django form allowing user friendly output. This
will allow the user to pick the right redirect choice without having to know
that various different redirect codes or have to look them up.</p>

funcptr - python

Python Packaging and Distribution

Build numbers in binaries using waf

Full example</h2> Check out my mdns-announce</a> project on Github for an example of how this is implemented.</p>

Generating stylesheets for pygments

Magic Properties on Google AppEngine

Auto-updating Properties in Google AppEngine Models

Property class on AppEngine does not support a tuple for choices

`Full example</h2> Check out my mdns-announce</a> project on Github for an example of how this is implemented.</p>`