I was working on some code and realized that ManyToManyFields always show up last in a ModelForm, regardless if you specify the order in the Meta fields property. Apparently this is due to be fixed in Django 1.1 which is right around the corner.. unfortunately I’m on 1.0 and feel anxious about moving 10+ sites to 1.1 beta right at this moment. I also felt it was a bit inappropriate to diff my Django source.

Please be aware that this is a hack. It is not an optimal solution. Stuff like form.as_* will NOT work with the solution below. All this does is add a property called ordered_fields to your ModelForm which implements the correct behavior.

# models.py
class Bar(models.Model):
	pass

class Foo(models.Model):
        # ordinarily, Django 1.0 would list the regular_field first and all M2Ms last
	many_to_many = models.ManyToManyField('Bar', related_name='foos')
	regular_field = models.CharField(max_length=10)

# forms.py
class FooForm(forms.ModelForm):
	def ordered_fields(self):
		list = []
		for field_name in self._meta.fields:
			list.append(self[field_name])
		return list

	class Meta:
		model = Foo
		fields = ('many_to_many',
			     'regular_field',)

#Template:

<form action="{{ app_path }}" method="post" id="foo-form">
	<fieldset class="module aligned ()">
	{% for field in form.ordered_fields %}
		<div class="form-row thin wide ()">

			{{ field.label_tag }}
			{{ field }}
			{{ field.errors }}
			{% if field.help_text %}
			<p class="help">{{ field.help_text }}</p>
			{% endif %}
		</div>
	{% endfor %}
	</fieldset>
	<div class="submit-row aligned ()">
		<input class="default" type="submit" value="Send" />
	</div>
</form>

The other day I was working on a Django ModelForm that had an ImageField in it and to my great surprise the ImageField would not validate in the ModelForm, even when I had selected an image. The solution was simple enough: add request.FILES to the instantiation parameters of the ModelForm.. DUH!

    #ProfileForm is a ModelForm
    form = ProfileForm(request.POST, request.FILES, instance=profile)

I have been working to get the new version of DEV360.com out the door and found myself in need of displaying RSS items from my Wordpress-powered blog on the DEV360 site which is developed in Django.

There were two libraries that I found for parsing RSS; RSS.py by Mark Nottingham which unfortunately didn’t support RSS 2.0, and Mark Pilgrim’s feedparser.py. I settled on the latter since it contains more features, has easier syntax and has better unit-test coverage.

For what I’m doing, a template tag works out best for rendering the RSS items; let’s fast-forward to the code:

foo.html:

{% load rss_feed %}
{% rss_feed %}

[application]/templatetags.py:

import feedparser
from datetime import datetime
from django.template import Library, Node
register = Library()

@register.simple_tag
def rss_feed(itemCount=5):
	channel = feedparser.parse('http://blog.dev360.com/feed/')

	html = []
	html.append('<div id="rss-feed">')
	html.append('	<h1>Latest blog posts</h1>')
	html.append('	<ul>')
	for entry in channel.entries[:itemCount]:
		url = entry.id
		title = entry.title
		date = datetime.strptime(entry.date, "%a, %d %b %Y %H:%M:%S +0000");
		html.append('<li><p><strong><a href="%s">%s</a></strong>' % (url, title));
		html.append('%s</p></li>' % (date.strftime("%B %d, %Y")))

	html.append('	</ul>')
	html.append('</div>')
	return "\n".join(html)

I thought I’d share a technique that I have grown very fond of over the last year - all made possible by jQuery’s syntactic bliss.

On many occassions in the past, I have found myself writing roll-over code that looks something like this:

<ul class="navigation">
  <li onmouseover="this.className='selected';" onmouseout="this.className='';" class="selected"><a href="#">Link 1</a></li>
  <li onmouseover="this.className='selected';" onmouseout="this.className='';"><a href="#">Link 2</a></li>
  <li onmouseover="this.className='selected';" onmouseout="this.className='';"><a href="#">Link 3</a></li>
</ul>

As you can see, the onmouseover/onmouseout makes the code less readable and takes up a lot of unnecessary space. One way to get rid of it is obviously to take advantage of the :hover pseudo class on the a element but if you want the hovered item to be the only item marked with the selected class then obviously that solution doesn’t cut it.

jQuery’s event wiring, along with its selector syntax allows you to remove the onmouse**** tags and earn some well deserved HTML karma.

JS

$lastActiveItem = null;
jQuery(document).ready(function(){
	$('.navigation li').mouseover(function(){
		$lastActiveItem = $('.navigation li.selected');
		$('.navigation li').removeClass('selected');
		$(this).addClass('selected');
	});

	$('.navigation li').mouseout(function(){
		$('.navigation li').removeClass('selected')
		$lastActiveItem.addClass('selected')
	});
});

I wrote a small command-line utility with Twill and Python that I use to scrape prices from our booking engine at work. The Python script worked fine when I ran it on my Mac/Python2.5 but Twill (or mechanize, to be exact) was giving deprecation warnings when I tried to run it on Windows/Python2.6.1.

C:\Users\Administrator\twill\other_packages\_mechanize_dist\_auth.py:14: DeprecationWarning: the md5 module is deprecated; use hashlib instead import re, base64, urlparse, posixpath, md5, sha, sys, copy
C:\Users\Administrator\twill\other_packages\_mechanize_dist\_auth.py:14: DeprecationWarning: the sha module is deprecated; use the hashlib module instead import re, base64, urlparse, posixpath, md5, sha, sys, copy

The solution is trivial; to supress deprecation warnings in python, just add an ingore filter before the line where you import Twill (or any other offending library). The alert reader may notice that the second part of the filterwarnings argument (’.*’) is a regular expression; this would allow you to tailor exactly which deprecation warnings you wish to propagate to the command-line.

import warnings
warnings.filterwarnings('ignore', '.*')

from twill.commands import *
import twill

I just read about a new search AI dubbed Wolfram Alpha (WolframAlpha.com) in an article that was published on Sunday May 3 in The Independent. It talks about this new search engine AI invented by Dr. Stephen Wolfram which supposedly “understands and responds to ordinary language in the same way a person does”. Skimming through the article, I can’t remember getting this excited about a site since learning about the MetaWeb/Freebase API, but after scratching the surface a little bit, the excitement faded away very quickly.

Let me start by saying that I can’t say that I’m very knowledgeable or up to date when it comes to the academic research on the subject - apart from following a few AI and Semantic Web blogs. Now, the thing that immediately raised my eyebrows was a little time line of the history of the Internet which listed a handful of milestones - with the most recent item reading “2009 Dr Stephen Wolfram launches Wolfram Alpha.”, listed right after the traditional stuff that you would expect to see (ARPANET, first PC etc). For a site that is not even public (as of Monday May 4, 2009), it seems awfully rash to have it on a list like that.

Reading the fine print confirmed more and more of my skepticism. For instance, Dr. Wolfram admits that information in the engine is first “curated” or assessed by experts AND it also requires approximately 1,000 people for maintaining the site’s databases and keeping current with new information. All this begs to question how exactly is this an AI or anything revolutionary? And to top that off, the end of the article mentions that Google already is working on something similar which is already live. Wow! - Revolutionary - let’s credit Dr. Wolfram and say a loud Amen.

A person’s past is perhaps the best way to look for clues and Dr. Wolfram is no exception. This is where Dr. Dubya appears to have a solid track record as a self-indulgent hype machine. Consider for instance the overwhelmingly negative feedback and concerns about originality for his supposedly epic ‘10 years in the making’ book ironically titled “A New Kind of Science”. A New Kind of Search-Engine? - No, I doubt so.

A few months ago when I wanted to log into my EC2 Fedora 8 instance, Putty as well as the Terminal in OS X started greeting me with the message ‘Server refused to allocate pty’. Apparently the new version of the EC2 Bundling files had changed the FSTAB file which caused this error to occur.

The only way I was able to connect to the server was with SCP and the Windows client WinSCP actually somehow managed to give me SSH access. For some reason I post-poned this fix for the longest time and just used WinSCP’s SSH access, but 3 days ago I decided to do something about it. First, I backed up my /etc/fstab file and used WinSCP to edit it to say:

/dev/sda1               /                       ext3    defaults 1 1
/dev/sda2               /mnt                    ext3    defaults 0 0
/dev/sda3               swap                    swap    defaults 0 0
none                    /dev/pts                devpts  gid=5,mode=620 0 0
none                    /dev/shm                tmpfs   defaults 0 0
none                    /proc                   proc    defaults 0 0
none                    /sys                    sysfs   defaults 0 0

After that, just reboot your instance and you will be able to log in as usual again.

By default, Mac OS X runs Apache in 64-bit mode. Unfortunately, there are a handful of apache mods that are only available as 32-bit downloads and they can be a bit challenging to modify and recompile in 64-bit unless you are confident with the compiler.

Mod_python is one of those mods - and it also happens to be something that I rely on for my development environment on the Mac. The following is the command to change from 64-bit to 32-bit:

sudo mv /usr/sbin/httpd /usr/sbin/httpd.bkp
sudo lipo -thin i386 /usr/sbin/httpd.bkp -output /usr/sbin/httpd.i386
sudo ln -s /usr/sbin/httpd.i386 /usr/sbin/httpd

Unfortunately, I have noticed that Apache goes back to 64-bit mode each time I run a system update. It probably restores the original httpd file. It might go away if you use the actual file and not the sym-link, but I just run the command again.

Use this command if you want to revert back to 64-bit:

sudo mv /usr/sbin/httpd.bkp /usr/sbin/httpd

A few weeks ago I had an interesting exchange with a friend of mine concerning code generation. This post echoes some of the thoughts and perhaps reservations that I have towards the practice.

So what exactly do I mean by the term code generation? Code generation is the practice whereby a tool or a script generates source code from templates as opposed to typing the code by hand. One example would be of a script that examines the columns in a table and creates a stored procedure. A more extreme case would be an MDA/MCD tool like Visual Paradigm.

Learning the hard way

A few months ago I was tasked with developing a large application for one of my clients that made use of extensive model taxonomies that also required a management interface.

With a good ORM and some thought behind the persistence mechanism, we were able to leverage a lot of reusable code that gave us a unified, DRY approach to persistence, validation, search queries, caching, lazy row-fetching and transactional operations. Only one daunting task remained and that was the UI. Again, we did our due diligence and came up with some useful generic base classes for Save/Update/List views but even so, the amount of fields on each Save/Update view was staggering and it was incredibly time-consuming to add them all. At some point when I had done about 10% of the work, I started realizing how genuinely tedious the task was. Integers would always be rendered as asp:TextBox accompanied by an asp:RegularExpressionValidator, one-to-many relationships would use a select-box or auto-complete widget and so forth and so on.

Knowing that my models’ members were lush with juicy meta data defined via attributes, I decided to use introspection to generate the code for the views. It took me about 2 weeks to write an IronPython script that could make sense of all the various types, validation attributes and implicit rules in my DLL. Once I had that, the rest was pretty easy. A single command would spit out thousands of lines of HTML and C#, but it still required modifications here and there since there were spots of non-standard behavior.

We now we have some 60 views that contain a lot of tedious code. As soon as something changes in our model, we still have to update the templates to reflect new validation rules or object definitions. It’s not as simple as re-generating the views because that would overwrite changes that have been made. I’m not ashamed of the quality of the code - with the abstractions that are already there its far exceeding the level of quality that you would expect from our average .NET developer. What bothers me is the sheer size and complexity which threatens to turn this code into an unmaintainable mess and a recipe for failure in the long run.

What went wrong? For starters, the problem wasn’t how to create hundreds of lines of HTML in each view – the appropriate question was why. To use a common anology – I was curing the symptom, not the disease. The real problem I was trying to solve was the severe GUI/Model impedance mismatch in my web-framework, ASP.NET. Had I been brave enough then I could just have used those two weeks wrangling with ViewState to come up with a well thought out, Reflection-based Form abstraction like the one found in Django.

Conclusion

In many instances code generation is an appropriate solution to a repetitive task, but first you have to ask yourself why you want to use it. If it’s solving a time-consuming, reoccurring task of low complexity then go ahead and use it, but implore yourself to look beyond the immediate problem at hand. You will be surprised to find out that frequently, the problems you are trying to solve are far more systemic and can be solved more elegantly by adopting a useful abstraction instead.

Code-generation can be a divisive and destructive approach to programming when applied systematically in a development environment or development methodology. Programming should not be viewed as a necessary evil as this fosters the wrong mind-set. It tends to put developers in two camps – one that knows how to solve problems and one that just knows a tool - most people who interview and hire programmers can attest to this.

Last time I had to use S3 uploads in Django was in 0.96 and I am happy to report that it is simpler than ever to upload files to S3 in Django. The recent storage-engine refactoring in Django 1.0 makes it a breeze to implement - no need for custom field types or anything like that.

David Larlet has put together a very good collection of Django storage-engines that looks very well-written compared to some naive implementations that I happened to stumble across elsewhere on the web. In addition, it includes a library for MogileFS - looks very exciting indeed and I hope I’ll have an opportunity (or reason rather) to play around with it one day!

To use, simply download Amazons S3 api, along with Larlet’s storage-engines (I just placed them in site-packages), and then your model would look something like the following:

import S3Storage

class UserProfile(models.Model):
    storage = S3Storage()
    profile_image = models.ImageField(upload_to='some-dir', storage=storage)