500e3b17d24c

Merge.
[view raw] [browse files]
author Steve Losh <steve@stevelosh.com>
date Wed, 29 Jun 2011 22:31:20 -0400
parents 16794453cea5 (diff) ed31fd9aa251 (current diff)
children c830c178d33a 2060e7f9bdfc
branches/tags (none)
files

Changes

--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/content/blog/2011/06/django-advice.html	Wed Jun 29 22:31:20 2011 -0400
@@ -0,0 +1,1055 @@
+    {% extends "_post.html" %}
+
+    {% hyde
+        title: "Django Advice"
+        snip: "Some useful things I've learned."
+        created: 2011-06-30 08:30:00
+        flattr: true
+    %}
+
+    {% block article %}
+
+For the past year and a half or so I've been working full-time at [Dumbwaiter
+Design][] doing [Django][] development. I've picked up a bunch of useful tricks along
+the way that help me work, and I figured I'd share them.
+
+I'm sure there are better ways to do some of the things that I mention.  If you know
+of any feel free to hit me up on [Twitter][] and let me know.
+
+Also: this entry was written over several months, so if there are inconsistencies let
+me know and I'll try to fix them.
+
+[Dumbwaiter Design]: http://dwaiter.com/
+[Django]: {{links.django}}
+[Twitter]: http://twitter.com/stevelosh
+
+[TOC]
+
+Vagrant
+-------
+
+I used to develop Django sites by running them on my OS X laptop locally and
+deploying to a Linode VPS.  I had a whole section of this post written up about
+tricks and tips for working with that setup.
+
+Then I found [Vagrant][].
+
+I just deleted the entire section of this post I wrote.
+
+Vagrant gives you a better way of working.  You need to use it.
+
+[Vagrant]: http://vagrantup.com/
+
+### Why Vagrant?
+
+If you haven't used it before, Vagrant is basically a tool for managing
+[VirtualBox][] VMs.  It makes it easy to start, pause, and resume VMs.  Instead of
+installing Django in a virtualenv and developing against that, you run a VM which
+runs your site and develop against that.
+
+This may not sound like much, but it's kind of a big deal.  The critical difference
+is that you can now develop against the same setup that you'll be using in
+production.
+
+This cuts out a huge amount of pain that stems from OS differences.  Here are a few
+examples off the top of my head:
+
+* URLField and MacPorts Python 2.5 on OS X.  There's a [bug][] where using
+  verify\_exists will crash your site every time you save a model, unless you set
+  a particular environment variable with no debug information.  Yeah, I spent
+  a couple of hours tracking that one down at work.  Awesome.
+* Installing PIL on OS X is no picnic.  [homebrew][] makes things better, if you use it,
+  so this one isn't a huge deal.
+* Every time you update Python in-place on your local machines, ALL of your
+  virtualenvs break because the Python binaries inside are linked against global
+  Python library files.  Have fun recreating them.  I hope you froze your
+  `requirements.txt` files before you updated.
+
+Using Vagrant and VMs means you can just worry about ONE operating system and its
+quirks.  It saves you a ton of time.
+
+Aside from that, there's another benefit to using Vagrant: it strongly encourages you
+to learn and use an automated provisioning system.  Support for Puppet and Chef is
+built in.  I chose Puppet, but if you prefer Chef that's cool too.
+
+Because you're developing against a VM and deploying to a VM, you can reuse 90% of
+the provisioning code across the two.
+
+When I make a new site, I do the following to initialize a new Vagrant VM:
+
+1. `vagrant up` (which runs Puppet to initialize the VM)
+2. `fab dev bootstrap`
+
+When I'm ready to go live, I do the following:
+
+1. Buy a Linode VPS.
+2. Run Puppet to initialize the VPS.
+3. Enter the Linode info in my fabfile.
+4. `fab prod bootstrap`
+
+No more screwing around with different paths, different versions of Nginx, different
+versions of Python.  When I'm developing something I can be pretty confident it will
+"just work" in production without any major surprises.
+
+[VirtualBox]: http://www.virtualbox.org/
+[bug]: https://trac.macports.org/ticket/24421
+[homebrew]: http://mxcl.github.com/homebrew/
+
+### Using Fabric to Stay Fast and Automate Everything
+
+One of the problems with this setup is that I can't just run `python manage.py
+whatever` any more because I need it to run on the VM.
+
+To get around this I've created many simple [Fabric][] tasks to automate the common
+things I need to do.  Fabric is an awesome little Python utility for scripting tasks
+(like deployments).  We use it constantly at Dumbwaiter.  Here are a few examples
+from our fabfiles.
+
+[Fabric]: http://fabfile.org/
+
+This first set is for running abitrary commands easily.
+
+`cmd` and `vcmd` will `cd` into the site directory on the VM and run a command of my
+choosing.  `vcmd` will prefix the command with the path to the virtualenv's `bin`
+directory, so I can do something like `fab dev vcmd`, `pip install markdown`.
+
+The `sdo` commands do the same thing, but `sudo`'ed.
+
+    :::python
+    def cmd(cmd=""):
+        '''Run a command in the site directory.  Usable from other commands or the CLI.'''
+        require('site_path')
+
+
+        if not cmd:
+            sys.stdout.write(_cyan("Command to run: "))
+            cmd = raw_input().strip()
+
+        if cmd:
+            with cd(env.site_path):
+                run(cmd)
+
+    def sdo(cmd=""):
+        '''Sudo a command in the site directory.  Usable from other commands or the CLI.'''
+        require('site_path')
+
+        if not cmd:
+            sys.stdout.write(_cyan("Command to run: sudo "))
+            cmd = raw_input().strip()
+
+        if cmd:
+            with cd(env.site_path):
+                sudo(cmd)
+
+    def vcmd(cmd=""):
+        '''Run a virtualenv-based command in the site directory.  Usable from other commands or the CLI.'''
+        require('site_path')
+        require('venv_path')
+
+        if not cmd:
+            sys.stdout.write(_cyan("Command to run: %s/bin/" % env.venv_path.rstrip('/')))
+            cmd = raw_input().strip()
+
+        if cmd:
+            with cd(env.site_path):
+                run(env.venv_path.rstrip('/') + '/bin/' + cmd)
+
+    def vsdo(cmd=""):
+        '''Sudo a virtualenv-based command in the site directory.  Usable from other commands or the CLI.'''
+        require('site_path')
+        require('venv_path')
+
+        if not cmd:
+            sys.stdout.write(_cyan("Command to run: sudo %s/bin/" % env.venv_path.rstrip('/')))
+            cmd = raw_input().strip()
+
+        if cmd:
+            with cd(env.site_path):
+                sudo(env.venv_path.rstrip('/') + '/bin/' + cmd)
+
+This next set is just some common commands that I need to run often.
+
+    :::python
+    def syncdb():
+        '''Run syncdb.'''
+        require('site_path')
+        require('venv_path')
+
+        with cd(env.site_path):
+            run(_python('manage.py syncdb --noinput'))
+
+    def collectstatic():
+        '''Collect static media.'''
+        require('site_path')
+        require('venv_path')
+
+        with cd(env.site_path):
+            sudo(_python('manage.py collectstatic --noinput'))
+
+    def rebuild_index():
+        '''Rebuild the search index.'''
+        require('site_path')
+        require('venv_path')
+        require('process_owner')
+
+        with cd(env.site_path):
+            sudo(_python('manage.py rebuild_index'))
+            sudo('chown -R %s .xapian' % env.process_owner)
+
+    def update_index():
+        '''Update the search index.'''
+        require('site_path')
+        require('venv_path')
+        require('process_owner')
+
+        with cd(env.site_path):
+            sudo(_python('manage.py update_index'))
+            sudo('chown -R %s .xapian' % env.process_owner)
+
+We also use Fabric to automate some of the more complex things we need to do.
+
+This task `curl`'s the site's home page to make sure we haven't completely borked
+things.  We use it in lots of other tasks as a sanity check.
+
+    :::python
+    def check():
+        '''Check that the home page of the site returns an HTTP 200.'''
+        require('site_url')
+
+        print('Checking site status...')
+
+        if not '200 OK' in local('curl --silent -I "%s"' % env.site_url, capture=True):
+            _sad()
+        else:
+            _happy()
+
+The `_happy` and `_sad` functions just print out some simple messages to get our
+attention:
+
+    :::python
+    from fabric.colors import red, green
+
+    def _happy():
+        print(green('\nLooks good from here!\n'))
+
+    def _sad():
+        print(red(r'''
+              ___           ___
+             /  /\         /__/\
+            /  /::\        \  \:\
+           /  /:/\:\        \__\:\
+          /  /:/  \:\   ___ /  /::\
+         /__/:/ \__\:\ /__/\  /:/\:\
+         \  \:\ /  /:/ \  \:\/:/__\/
+          \  \:\  /:/   \  \::/
+           \  \:\/:/     \  \:\
+            \  \::/       \  \:\
+             \__\/         \__\/
+              ___           ___           ___           ___
+             /__/\         /  /\         /  /\         /  /\     ___
+             \  \:\       /  /::\       /  /:/_       /  /:/_   /__/\
+              \  \:\     /  /:/\:\     /  /:/ /\     /  /:/ /\  \  \:\
+          _____\__\:\   /  /:/  \:\   /  /:/ /:/_   /  /:/ /::\  \  \:\
+         /__/::::::::\ /__/:/ \__\:\ /__/:/ /:/ /\ /__/:/ /:/\:\  \  \:\
+         \  \:\~~\~~\/ \  \:\ /  /:/ \  \:\/:/ /:/ \  \:\/:/~/:/   \  \:\
+          \  \:\  ~~~   \  \:\  /:/   \  \::/ /:/   \  \::/ /:/     \__\/
+           \  \:\        \  \:\/:/     \  \:\/:/     \__\/ /:/          __
+            \  \:\        \  \::/       \  \::/        /__/:/          /__/\
+             \__\/         \__\/         \__\/         \__\/           \__\/
+
+
+             Something seems to have gone wrong!
+             You should probably take a look at that.
+        '''))
+
+This one is for when `python manage.py reset APP` is broken because you've changed
+some `db_column` names and Django chokes because of some constraits and you just want
+to **reset the fucking app**.
+
+It's the "NUKE IT FROM ORBIT!!" option.
+
+    :::python
+    def KILL_IT_WITH_FIRE(app):
+        require('site_path')
+        require('venv_path')
+
+        with cd(env.site_path):
+            # Generate and download the reset SQL.
+            sudo(_python('manage.py sqlreset %s > reset.orig.sql' % app))
+            get('reset.orig.sql')
+
+            with open('reset.sql', 'w') as f:
+                with open('reset.orig.sql') as orig:
+                    # Step through the first chunk of the file (the "drop" part).
+                    line = orig.readline()
+                    while not line.startswith('CREATE'):
+                        if 'CONSTRAINT' in line:
+                            # Don't write out CONSTRAINT lines.
+                            # They're a problem when you change db_colum names.
+                            pass
+                        elif 'DROP TABLE' in line:
+                            # Cascade drops.
+                            # Hence with "with fire" part of this task's name.
+                            line = line[:-2] + ' CASCADE;\n'
+                            f.write(line)
+                        else:
+                            # Write other lines through untoched.
+                            f.write(line)
+                        line = orig.readline()
+
+                    # Write out the rest of the file untouched.
+                    f.write(line)
+                    f.write(orig.read())
+
+        # Upload the processed SQL file.
+        put('reset.sql', os.path.join(env.site_path, 'reset.ready.sql'), use_sudo=True)
+
+        with cd(env.site_path):
+            # Use the SQL to reset the app, and fake a migration.
+            run(_python('manage.py dbshell < reset.ready.sql'))
+            sudo(_python('manage.py migrate --fake --delete-ghost-migrations ' + app))
+
+This task uses Mercurial's local tags to add a `production` or `staging` tag in your local
+repository, so you can easy see where the production/staging servers are at
+compared to your local repo.
+
+    :::python
+    def retag():
+        '''Check which revision the site is at and update the local tag.
+
+        Useful if someone else has deployed (which makes your production/staging local
+        tag incorrect.
+        '''
+        require('site_path', provided_by=['prod', 'stag'])
+        require('env_name', provided_by=['prod', 'stag'])
+
+        with cd(env.site_path):
+            current = run('hg id --rev . --quiet').strip(' \n+')
+
+        local('hg tag --local --force %s --rev %s' % (env.env_name, current))
+
+This task tails the Gunicorn logs on the server so you can quickly find out what's
+happening when things blow up.
+
+    :::python
+    def tailgun(follow=''):
+        """Tail the Gunicorn log file."""
+        require('site_path')
+
+        with cd(env.site_path):
+            if follow:
+                run('tail -f .gunicorn.log')
+            else:
+                run('tail .gunicorn.log')
+
+We've got a lot of other tasks but they're pretty specific to our setup.
+
+Wrangling Databases with South
+------------------------------
+
+If you're not using [South][], you need to start.  Now.
+
+No, really, I'll wait.  Take 30 minutes, try the [tutorial][Southtut], wrap your head
+around it and come back.  It's far more important than this blog post.
+
+[South]: http://south.aeracode.org/
+[Southtut]: http://south.aeracode.org/docs/tutorial/index.html
+
+### Useful Fabric Tasks
+
+South is awesome but its commands are very long-winded.  Here's the set of fabric
+tasks I use to save quite a bit of typing:
+
+    :::python
+    def migrate(args=''):
+        '''Run any needed migrations.'''
+        require('site_path')
+        require('venv_path')
+
+        with cd(env.site_path):
+            sudo(_python('manage.py migrate ' + args))
+
+    def migrate_fake(args=''):
+        '''Run any needed migrations with --fake.'''
+        require('site_path')
+        require('venv_path')
+
+        with cd(env.site_path):
+            sudo(_python('manage.py migrate --fake ' + args))
+
+    def migrate_reset(args=''):
+        '''Run any needed migrations with --fake.  No, seriously.'''
+        require('site_path')
+        require('venv_path')
+
+        with cd(env.site_path):
+            sudo(_python('manage.py migrate --fake --delete-ghost-migrations ' + args))
+
+Remember that running a migration without specifying an app will migrate everything,
+so a simple `fab dev migrate` will do the trick.
+
+Watching for Changes
+---------------------
+
+When developing locally you'll want to make a change to your code and have the server
+reload that code automatically.  The Django development server does this, and we can
+hack it into our Vagrant/Gunicorn setup too.
+
+First, add a `monitor.py` file at the root of your project (I believe I found this
+code [here][monitor], but I may be wrong):
+
+    :::python
+    import os
+    import sys
+    import time
+    import signal
+    import threading
+    import atexit
+    import Queue
+
+    _interval = 1.0
+    _times = {}
+    _files = []
+
+    _running = False
+    _queue = Queue.Queue()
+    _lock = threading.Lock()
+
+    def _restart(path):
+        _queue.put(True)
+        prefix = 'monitor (pid=%d):' % os.getpid()
+        print >> sys.stderr, '%s Change detected to \'%s\'.' % (prefix, path)
+        print >> sys.stderr, '%s Triggering process restart.' % prefix
+        os.kill(os.getpid(), signal.SIGINT)
+
+    def _modified(path):
+        try:
+            # If path doesn't denote a file and were previously
+            # tracking it, then it has been removed or the file type
+            # has changed so force a restart. If not previously
+            # tracking the file then we can ignore it as probably
+            # pseudo reference such as when file extracted from a
+            # collection of modules contained in a zip file.
+
+            if not os.path.isfile(path):
+                return path in _times
+
+            # Check for when file last modified.
+
+            mtime = os.stat(path).st_mtime
+            if path not in _times:
+                _times[path] = mtime
+
+            # Force restart when modification time has changed, even
+            # if time now older, as that could indicate older file
+            # has been restored.
+
+            if mtime != _times[path]:
+                return True
+        except:
+            # If any exception occured, likely that file has been
+            # been removed just before stat(), so force a restart.
+
+            return True
+
+        return False
+
+    def _monitor():
+        while 1:
+            # Check modification times on all files in sys.modules.
+
+            for module in sys.modules.values():
+                if not hasattr(module, '__file__'):
+                    continue
+                path = getattr(module, '__file__')
+                if not path:
+                    continue
+                if os.path.splitext(path)[1] in ['.pyc', '.pyo', '.pyd']:
+                    path = path[:-1]
+                if _modified(path):
+                    return _restart(path)
+
+            # Check modification times on files which have
+            # specifically been registered for monitoring.
+
+            for path in _files:
+                if _modified(path):
+                    return _restart(path)
+
+            # Go to sleep for specified interval.
+
+            try:
+                return _queue.get(timeout=_interval)
+            except:
+                pass
+
+    _thread = threading.Thread(target=_monitor)
+    _thread.setDaemon(True)
+
+    def _exiting():
+        try:
+            _queue.put(True)
+        except:
+            pass
+        _thread.join()
+
+    atexit.register(_exiting)
+
+    def track(path):
+        if not path in _files:
+            _files.append(path)
+
+    def start(interval=1.0):
+        global _interval
+        if interval < _interval:
+            _interval = interval
+
+        global _running
+        _lock.acquire()
+        if not _running:
+            prefix = 'monitor (pid=%d):' % os.getpid()
+            print >> sys.stderr, '%s Starting change monitor.' % prefix
+            _running = True
+            _thread.start()
+        _lock.release()
+
+Next add a `post_fork` hook to your Gunicorn config file that uses the monitor to
+watch for changes:
+
+    :::python
+    def post_fork(server, worker):
+        import monitor
+        import local_settings
+        if local_settings.DEBUG:
+            server.log.info("Starting change monitor.")
+            monitor.start(interval=1.0)
+
+Now the Gunicorn server will automatically restart whenever code is changed.  Use
+whatever method for determining debug status that you like.  We use
+`local_settings.py` files which all have `DEBUG` variables, so that works for us.
+
+It will *not* restart when you add new code (e.g. when you install a new app), so
+you'll need to handle that manually with `fab dev restart`, but that's not too bad!
+
+[monitor]: http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode
+
+### Using the Werkzeug Debugger with Gunicorn
+
+The final piece of the puzzle is being able to use the fantastic [Werkzeug
+Debugger][debug] while running on the development VM with Gunicorn.
+
+To do this, create a `debug_wsgi.py` file at the root of your project:
+
+    :::python
+    import os
+    import sys
+    import site
+
+    parent = os.path.dirname
+    site_dir = parent(os.path.abspath(__file__))
+    project_dir = parent(parent(os.path.abspath(__file__)))
+
+    sys.path.insert(0, project_dir)
+    sys.path.insert(0, site_dir)
+
+    site.addsitedir('VIRTUALENV_SITE_PACKAGES')
+
+    from django.core.management import setup_environ
+    import settings
+    setup_environ(settings)
+
+    import django.core.handlers.wsgi
+    application = django.core.handlers.wsgi.WSGIHandler()
+
+    from werkzeug.debug import DebuggedApplication
+    application = DebuggedApplication(application, evalex=True)
+
+    def null_technical_500_response(request, exc_type, exc_value, tb):
+        raise exc_type, exc_value, tb
+    from django.views import debug
+    debug.technical_500_response = null_technical_500_response
+
+Have Gunicorn use this file to run your development server with `gunicorn
+debug_wsgi:application`.
+
+Make sure to replace `'VIRTUALENV_SITE_PACKAGES'` with the _full_ path to your
+virtualenv's `site_packages` directory.  You might want to make this a setting in
+a machine-specific settings file.
+
+[debug]: http://werkzeug.pocoo.org/docs/debug/
+
+### Pulling Uploads
+
+Once you give a client access to a site they'll probably be uploading images (through
+Django's built-in file uploading features or with [django-filebrowser][]).
+
+When you're making changes locally it's often useful to have these uploaded files on
+your local VM, otherwise you end up with a bunch of broken images.
+
+Here's a simple Fabric task that will pull down all the uploads from the server:
+
+    :::python
+    def pull_uploads():
+        '''Copy the uploads from the site to your local machine.'''
+        require('uploads_path')
+
+        sudo('chmod -R a+r "%s"' % env.uploads_path)
+
+        rsync_command = r"""rsync -av -e 'ssh -p %s' %s@%s:%s %s""" % (
+            env.port,
+            env.user, env.host,
+            env.uploads_path.rstrip('/') + '/',
+            'media/uploads'
+        )
+        print local(rsync_command, capture=False)
+
+You might be wondering about the line that strips `/` characters and then adds them
+back in.  `rsync` does different things depending on whether you end a path with
+a `/`, so this is actually pretty important.
+
+In your host task you'll need to set the `uploads_path` variable to something like
+this:
+
+    :::python
+    import os
+    env.site_path = os.path.join('var', 'www', 'myproject')
+    env.uploads_path = os.path.join(env.site_path, 'media', 'uploads')
+
+Now you can run `fab production pull_uploads` to pull down all the files people have
+uploaded to the production server.
+
+[django-filebrowser]: http://code.google.com/p/django-filebrowser/
+
+### Preventing Accidents
+
+Deploying to test and staging servers should be quick and easy. Deploying to
+production servers should be harder to prevent people from accidentally doing it.
+
+I've created a little function that I call before deploying to production servers.
+It forces me to type in random words from the system word list before proceeding to
+make sure I *really* know what I'm doing:
+
+    :::python
+    import os, random
+
+    from fabric.api import *
+    from fabric.operations import prompt
+    from fabric.utils import abort
+
+    WORDLIST_PATHS = [os.path.join('/', 'usr', 'share', 'dict', 'words')]
+    DEFAULT_MESSAGE = "Are you sure you want to do this?"
+    WORD_PROMPT = '  [%d/%d] Type "%s" to continue (^C quits): '
+
+    def prevent_horrible_accidents(msg=DEFAULT_MESSAGE, horror_rating=1):
+        """Prompt the user to enter random words to prevent doing something stupid."""
+
+        valid_wordlist_paths = [wp for wp in WORDLIST_PATHS if os.path.exists(wp)]
+
+        if not valid_wordlist_paths:
+            abort('No wordlists found!')
+
+        with open(valid_wordlist_paths[0]) as wordlist_file:
+            words = wordlist_file.readlines()
+
+        print msg
+
+        for i in range(int(horror_rating)):
+            word = words[random.randint(0, len(words))].strip()
+            p_msg = WORD_PROMPT % (i+1, horror_rating, word)
+            answer = prompt(p_msg, validate=r'^%s$' % word)
+
+You may need to adjust `WORDLIST_PATHS` if you're not on OS X.
+
+Working with Third-Party Apps
+-----------------------------
+
+One of the best parts about working with Django is that many problems have already
+been solved and the solutions have been released as open-source applications.
+
+We use quite a few open-source apps, and there are a couple of tricks I've learned to
+make working with them easier.
+
+### Installing Apps from Repositories
+
+If I'm going to use an open-source Django app in a project I'll almost always install
+it as an editable repository on the VM with `pip install -e`.
+
+Others may disagree with me on this, but I think it's the best way to work.
+
+Often I'll find a bug that I think may be in one of the third-party apps I'm using.
+Installing the apps as repositories makes it easy to read their source and figure out
+if the bug is really in the app.
+
+If the bug *is* in the third-party app having the app installed as a repository makes
+it simple to fix the bug, fork the project on BitBucket or GitHub, send a pull
+request, and get back to work.
+
+### Mirroring Repositories
+
+One problem we've run into at Dumbwaiter is that the repos for third-party apps we
+use are scattered across GitHub, BitBucket, Google Code, and other servers.  If any
+one of these services goes down we're stuck waiting for it to come back up.
+
+A while ago I took half a day and consolidated all of these repos onto one of the
+servers that we control.  The basic process went like this:
+
+* Use [hg-git][] and [hgsubversion][] to convert the git and SVN repos to Mercurial
+  repos.
+* Set up a master `mirror` Mercurial repo with all the app repos as subrepos.
+* Push the master repo and all the subrepos up to one of our Linodes.
+
+Now we can use `-e ssh://hg@OUR_LINODE/mirror/APP@REV_THAT_WORKS#egg=APP` in our
+`requirements.txt` files to install apps from our mirror.  When we want to update our
+dependencies we can simply pull from the upstream repos and commit in the mirror
+repo.
+
+If our mirror goes down it's not a big deal, because we have far bigger problems to
+worry about than new projects.
+
+I wrote a few scripts to automate updating apps and such, but they're extremely hacky
+so I don't want ot post them here.  Take half a day and write your own set -- it's
+definitely worth it to have your own mirror of your specific dependencies.
+
+[hg-git]: http://hg-git.github.com/
+[hgsubversion]: https://bitbucket.org/durin42/hgsubversion/wiki/Home
+
+### Using BCVI to Edit Files
+
+I said that when I find a bug that I think is in a third-party app I'll poke around
+with the app and try to figure it out.  But since all the apps are installed in
+a virtualenv on the Vagrant VM it might seem like it's a pain in the ass to edit
+those files!
+
+Luckily [BCVI][] exists.  It's a utility that opens a "back channel" to your local
+machine when you SSH and lets you run `vi FILE` to open that file in
+Vim/MacVim/GVim/etc on your *local* machine.
+
+It can be a bit tricky to set up, but it's worth it.  Trust me.
+
+[BCVI]: http://sshmenu.sourceforge.net/articles/bcvi/
+
+Improving the Admin Interface
+-----------------------------
+
+I'm going to be honest: Django's admin interface is the main reason I'm still using
+it.  Other frameworks like [Flask][] are great, but Django's admin saves me
+*ridiculous* amounts of time when I'm making simple CRUD sites for clients.
+
+That said, the Django admin isn't the prettiest thing around, but we can give it
+a facelift.
+
+[Flask]: http://flask.pocoo.org/
+
+### Enter Grappelli
+
+[Grappelli][] is a Django app that reskins the admin interface beautifully.  It also
+adds some functionality like drag-and-drop reordering of inlines, and allows you to
+customize the dashboard to your liking.  *Every* Django site I work on uses Grappelli
+-- it's just that good.
+
+The downside of Grappelli is that it changes quite a lot and breaks backwards
+compatibility at the drop of a hat.
+
+If you're going to use Grappelli you *must* freeze your requirements.txt files and
+work with a single version at a time.  Trying to always work from the trunk will make
+you drink.
+
+[Grappelli]: http://django-grappelli.readthedocs.org/
+
+### An Ugly Hack to Show Usable Foreign Key Fields
+
+A limitation of both Grappelli and the stock Django admin is that it seems like you
+can't easily show fields from related models in the admin list view.
+
+For example, if you're new to Django you might expect this to work:
+
+    :::python
+    class BlogEntryAdmin(admin.ModelAdmin):
+        list_display = ('title', 'author__name')
+
+Unfortunately Django chokes on the `author__name` lookup.  You can *display* the name
+without too much fuss:
+
+    :::python
+    class BlogEntryAdmin(admin.ModelAdmin):
+        list_display = ('title', 'author_name')
+
+        def author_name(self, obj):
+            return o.name
+
+That will display the name just fine.  However, it won't be a fully-fledged column in
+the Django admin because you can't sort on it.
+
+It may seem like this is the end -- if it could be a fully-functional field, why
+wouldn't Django just let you use `author__name`?  Luckily we can add one more line to
+fix the problem:
+
+    :::python
+    class BlogEntryAdmin(admin.ModelAdmin):
+        list_display = ('title', 'author_name')
+
+        def author_name(self, obj):
+            return o.name
+        author_name.admin_order_field = 'author__name'
+
+Now the author name has all the functionality of a real `list_display` entry.
+
+Using Django-Annoying
+---------------------
+
+If you haven't heard of [django-annoying][] you should definitely check it out.  It's
+got a bunch of miscellaneous functions that fix some common, annoying parts of
+Django.
+
+My two personal favorites from the package are a pair of decorators that help make
+your views much, much cleaner.
+
+[django-annoying]: https://bitbucket.org/offline/django-annoying/wiki/Home
+
+### The render\_to Decorator
+
+The decorator is called `render_to` and it eliminates the ugly `render_to_response`
+calls that Django normally forces you to use in every single view.
+
+Normally you'd use something like this:
+
+    :::python
+    def videos(request):
+        videos = Video.objects.all()
+        return render_to_response('video_list.html', { 'videos': videos },
+                                  context_instance=RequestContext(request))
+
+With `render_to` your view gets much cleaner:
+
+    :::python
+    @render_to('video_list.html')
+    def videos(request):
+        videos = Video.objects.all()
+        return { 'videos': videos }
+
+Less typing `context_instance=...` over and over, and less syntax to remember.
+
+### The ajax\_request Decorator
+
+The `ajax_request` decorator is like `render_to` for AJAX requests.  You simply
+return a Python dictionary from your view and the decorator handles the JSON encoding
+and such:
+
+    :::python
+    @ajax_request
+    def ajax_get_entries(request):
+        blog_entries = BlogEntry.objects.all()
+        return { 'entries': [(entry.title, entry.get_absolute_url())
+                             for entry in entries]}
+
+Templating Tricks
+-----------------
+
+I'm not a frontend developer, but I've done my share of HTML hacking at Dumbwaiter.
+Here are a few of the tricks I've learned.
+
+### Null Checks and Fallbacks
+
+A common pattern I see in Django templates looks like this:
+
+    :::text
+    {% templatetag openblock %} if business.title {% templatetag closeblock %}
+        {% templatetag openvariable %} business.title {% templatetag closevariable %}
+    {% templatetag openblock %} else {% templatetag closeblock %}
+        {% templatetag openvariable %} business.short_title {% templatetag closevariable %}
+    {% templatetag openblock %} endif {% templatetag closeblock %}
+
+Here's a simpler way to do that:
+
+    :::text
+    {% templatetag openblock %} firstof business.title business.short_title {% templatetag closeblock %}
+
+`firstof` will return the first non-Falsy item in its arguments.
+
+### Manipulating Query Strings
+
+Query strings are normally not a big deal, but every once in a while you'll have
+a model listing page where you need to filter by category, and number of spaces, and
+tags, etc all at once.
+
+If you're trying to manage GET queries manually it can get pretty hairy very fast.
+
+[This Django snippet][qstring] makes working with query strings in templates
+a breeze.
+
+[qstring]: http://djangosnippets.org/snippets/2237/
+
+### Satisfying Your Designer with Typogrify
+
+If you haven't heard of [Typogrify][] you should take a look at it.  It makes it easy
+to add all the typographic goodness your designers are looking for.
+
+[Typogrify]: http://code.google.com/p/typogrify/
+
+The Flat Page Trainwreck
+------------------------
+
+Creating a site for a client is very different than creating a site for yourself.
+For pretty much every client we've dealt with we've heard: "can't we just create
+a new page at /drink-special/ for this special deal we're running?"
+
+Having clients go through you to make new pages is simply too much overhead.  We
+needed a way to let clients create new pages (like `/drink-special/`) on the fly,
+without our intervention.
+
+Django has a "flatpages" app that solves this problem.  Kind of.
+
+When using flat pages clients need to do two things that are often too much for
+non-technical people:
+
+* Manage URLs manually.
+* Write all content as raw HTML in a single text field.
+
+We've tried a lot of Django CMS apps at Dumbwaiter, and none of them made us happy.
+They all seemed to have some or all of the following problems:
+
+* They take over your site and make you write a "Django-WhateverCMS site" instead of
+  a "Django site".
+* They're extremely feature-rich and complicated with features like
+  internationalization, redirects, versions, and many others.  This is great if you
+  need the flexibility, but bad if your clients just need to create a couple of
+  pages.
+* They break `APPEND_TRAILING_SLASH` and make you clutter your `urls.py` files with
+  a bunch of extra code ot handle this.
+
+I finally got fed up and wrote my own Django CMS app: [Stoat][].  Stoat is designed
+to be sleek, with only the features that our clients need.
+
+It's not officially version 1.0 yet, but we're using it for a few clients and it's
+working well.  Check it out if you're looking for a more lightweight CMS app.
+
+[Stoat]: http://stoat.rtfd.org/
+
+Editing with Vim
+----------------
+
+I [use Vim][vimpost] to edit everything.  Naturally I've found a bunch of plugins,
+mappings and other tricks that make it even better when working on Django projects.
+
+[vimpost]: /blog/2010/09/coming-home-to-vim/
+
+### Vim for Django
+
+There are a lot of ways to make Vim work with Django.  I won't go into all of them in
+this post, but a good place to start is [this Django wiki page][vimdjango].
+
+[vimdjango]: https://code.djangoproject.com/wiki/UsingVimWithDjango
+
+### Filetype Mappings
+
+Most files in a Django project have one of two extensions: `.py` and `.html`.
+Unfortunately these extensions aren't unique to Django, so Vim doesn't automatically
+set the correct `filetype` when you open one.
+
+I've added a few mappings to my `.vimrc` to make it quick and easy to set the correct
+`filetype`:
+
+    :::text
+    nnoremap _dt :set ft=htmldjango<CR>
+    nnoremap _pd :set ft=python.django<CR>
+
+I also have a few autocommands that set the filetype for me when I'm editing a file
+whose name "sounds like" a Django file:
+
+    :::text
+    au BufNewFile,BufRead admin.py     setlocal filetype=python.django
+    au BufNewFile,BufRead urls.py      setlocal filetype=python.django
+    au BufNewFile,BufRead models.py    setlocal filetype=python.django
+    au BufNewFile,BufRead views.py     setlocal filetype=python.django
+    au BufNewFile,BufRead settings.py  setlocal filetype=python.django
+    au BufNewFile,BufRead forms.py     setlocal filetype=python.django
+
+### Python Sanity Checking
+
+Lets be honest here: it takes a lot of work to turn Vim into an "IDE", and even then
+it doesn't reach the level of something like Eclipse for Java.  Anyone who claims it
+has the same levels of integration and functionality is simply lying.
+
+With that said I'll make an opinionated statement that is going to piss some of you
+off.
+
+**I am a programmer, not an IDE operator.**
+
+I know Python.
+
+I know Django.
+
+I don't need to hit Cmd+Space twice for every line of code I write.
+
+When someone asks me "how do you run your site" I do **not** answer: "click the green
+triangle in Eclipse".
+
+However, I am human.  I do stupid things like forgetting a colon or forgetting an
+import.  To help me with those problems I've turned to Kevin Watters' [Pyflakes
+plugin][] for Vim.
+
+Pyflakes doesn't have IDE-level integration with your code.  It doesn't check that
+whatever libraries you `import` actually exist.  It simply checks that your files are
+probably-valid Python, and tells you when they're not.
+
+This is enough for me.  It catches the stupid mistakes I make.  The less-stupid,
+more-subtle mistakes slip by it, but to be fair many of them would have slipped by an
+"IDE" as well.
+
+[Pyflakes plugin]: http://www.vim.org/scripts/script.php?script_id=2441
+
+### Javascript Sanity Checking and Folding
+
+If you haven't used [Syntastic][], you definitely need to check it out.  It's a Vim
+plugin that adds on-the-fly syntax-checking for many different file formats: one of
+which is JavaScript.  It's not perfect but it *will* catch things like using trailing
+commas in object literals.
+
+Some people like using CTags to get an overview of their code.  I take a more
+low-tech approach and am in love with code folding.  When I fold my code
+I automatically get an overview of everything in each file.
+
+By default Vim doesn't fold Javascript files, but you can add some basic, perfectly
+serviceable folding with these two lines in your .vimrc:
+
+    :::text
+    au FileType javascript setlocal foldmethod=marker
+    au FileType javascript setlocal foldmarker={,}
+
+[Syntastic]: http://www.vim.org/scripts/script.php?script_id=2736
+
+
+### Django Autocommands
+
+I *rarely* work with raw HTML files any more.  Whenever I open a file ending in
+`.html` it's almost always a Django template (or a [Jinja][] template, which has
+a very similar syntax).  I've added an autocommand to automatically set the correct
+filetyle whenever I open a `.html` file:
+
+[Jinja]: http://jinja.pocoo.org/
+
+    :::text
+    au BufNewFile,BufRead *.html setlocal filetype=htmldjango
+
+I also have some autocommands that tweak how a few specific files are handled:
+
+    :::text
+    au BufNewFile,BufRead urls.py      setlocal nowrap
+    au BufNewFile,BufRead settings.py  normal! zR
+    au BufNewFile,BufRead dashboard.py normal! zR
+
+This automatically unfolds `urls.py`, `dashboard.py` and `settings.py` (I prefer
+seeing those unfolded) and unsets line wrapping for `urls.py` (lines in a `urls.py`
+file can get long and are hard to read when wrapped).
+
+Conclusion
+----------
+
+I hope that this longer-than-expected blog entry has given you at least one or two
+things to think about.
+
+I've learned a lot while working with Django for Dumbwaiter every day, but I'm sure
+there's still a lot I've missed.  If you see something I could be doing better please
+let me know!
+
+{% endblock %}