# HG changeset patch # User Steve Losh # Date 1343591436 14400 # Node ID 0230ec523afdf63b4bd61164e49e135c106ff4c2 # Parent 739cc98ac6892b089bc8aa9735ba5c3d921022b6 Moar. diff -r 739cc98ac689 -r 0230ec523afd content/blog/2012/07/caves-of-clojure-05.html --- a/content/blog/2012/07/caves-of-clojure-05.html Sat Jul 14 17:10:27 2012 -0400 +++ b/content/blog/2012/07/caves-of-clojure-05.html Sun Jul 29 15:50:36 2012 -0400 @@ -12,7 +12,7 @@ This post is part of an ongoing series. If you haven't already done so, you should probably start at [the beginning][]. -This entry corresponds to [post four in Trystan's tutorial][trystan-tut]. +This entry corresponds to [post five in Trystan's tutorial][trystan-tut]. If you want to follow along, the code for the series is [on Bitbucket][bb] and [on GitHub][gh]. Update to the `entry-05` tag to see the code as it stands diff -r 739cc98ac689 -r 0230ec523afd content/blog/2012/07/caves-of-clojure-06.html --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/content/blog/2012/07/caves-of-clojure-06.html Sun Jul 29 15:50:36 2012 -0400 @@ -0,0 +1,389 @@ + {% extends "_post.html" %} + + {% hyde + title: "The Caves of Clojure: Part 6" + snip: "Real combat and messages." + created: 2012-07-30 10:15:00 + flattr: true + %} + +{% block article %} + +This post is part of an ongoing series. If you haven't already done so, you +should probably start at [the beginning][]. + +This entry corresponds to [post six in Trystan's tutorial][trystan-tut]. + +If you want to follow along, the code for the series is [on Bitbucket][bb] and +[on GitHub][gh]. Update to the `entry-06` tag to see the code as it stands +after this post. + +Sorry for the long wait for this entry. I've been working on lots of other +stuff and haven't had a lot of time to write. Hopefully I can get back to it +more often! + +[the beginning]: /blog/2012/07/caves-of-clojure-01/ +[trystan-tut]: http://trystans.blogspot.com/2011/09/roguelike-tutorial-06-hitpoints-combat.html +[bb]: http://bitbucket.org/sjl/caves/ +[gh]: http://github.com/sjl/caves/ + +[TOC] + +Summary +------- + +In Trystan's sixth post he adds a combat system and messaging infrastructure. +Once again I'm following his lead and implementing the same things, but in the +entity/aspect way of doing things. + +As usual I ended up refactoring a few things, which I'll briefly cover first. + +Refactoring +----------- + +So far, the functions entities implement to fulfill aspects have looked like +this: + + :::clojure + (defaspect Digger + (dig [this world dest] + ...) + (can-dig? [this world dest] + ...)) + +The entity has to be the first argument, because that's how protocols work. +I don't have any flexibility there. I originally made the world always be the +second argument, but it turns out that it's more convenient to make the world +the *last* argument. + +To see why, imagine we want to allow players to dig and move at the same time, +instead of forcing them to be separate actions. Updating the world might look +like this: + + :::clojure + (let [new-world (dig player world dest) + new-world (move player world dest)] + new-world) + +You could make this one specific case a bit prettier, but in general chaining +together world-modifying actions is going to be a pain. If I change the aspect +functions to take the player, then other args, and *then* the world, I can use +`->>` to chain actions: + + :::clojure + (->> world + (dig player dest) + (move player dest)) + +Much cleaner! I went ahead and switched all the aspect functions to use this +new scheme. + +I also did some other minor refactoring. You can look through the changesets if +you're really curious. + +Attacking and Defending +----------------------- + +Instead of simply killing everything in one hit, I'm now going to give some +creatures a bit of hp. I also added a `:max-hp` attribute, since I'll likely +need that in the future. Here's a sample of what the `Bunny` creation function +looks like: + + :::clojure + (defn make-bunny [location] + (map->Bunny {:id (get-id) + :name "bunny" + :glyph "v" + :color :yellow + :location location + :hp 4 + :max-hp 4})) + +I started using the `map->Foo` record constructors because the `->Foo` versions +that relied on positional arguments started getting hard to read. + +Bunnies have 4 hp. It'd be trivial to randomize this in the future, but for now +I'll stick with a simple number. + +Trystan uses a simple attack and defense system. I toyed with the idea of using +a different one (like Brogue's) but figured I should stick to his tutorial when +there's no clear reason not to. + +Trystan's system needs attack and defense values, so I added functions to the +`Attacker` and `Destructible` aspects to retrieve these: + + :::clojure + (defaspect Attacker + (attack [this target world] + ...) + (attack-value [this world] + (get this :attack 1))) + + (defaspect Destructible + (take-damage [this damage world] + ...) + (defense-value [this world] + (get this :defense 0))) + +The default attack value is `1`. If an entity has an `:attack` attribute that +will be used instead. Or the entity could provide a completely custom version +of `attack-value` (e.g.: werewolves could have a larger attack if there's a full +moon in the game or something). Defense values work the same way. + +Now that I'm starting to actually use HP I'll display it in the bottom row of +info on the screen: + + :::clojure + (defn draw-hud [screen game] + (let [hud-row (dec (second (s/get-size screen))) + player (get-in game [:world :entities :player]) + {:keys [location hp max-hp]} player + [x y] location + info (str "hp [" hp "/" max-hp "]") + info (str info " loc: [" x "-" y "]")] + (s/put-string screen 0 hud-row info))) + +The bottom row of the screen now looks like: "hp [20/20] loc: [82-103]". I'll +probably get rid of the loc soon, but for now it won't hurt to keep it onscreen. + +Now for the damage calculation. I added a little helper function to take care +of this in `attacker.clj`: + + :::clojure + (defn get-damage [attacker target world] + (let [attack (attack-value attacker world) + defense (defense-value target world) + max-damage (max 0 (- attack defense)) + damage (inc (rand-int max-damage))] + damage)) + +This matches what Trystan does. In a nutshell, the damage done is: "If defense +is higher than attack, then 1. Otherwise, a random number between 1 and (attack +- defense)." + +I kept it separate from the `attack` function so that an entity can override +attack without having to reimplement this logic. + +The `attack` default implementation needs to use this new damage calculator: + + :::clojure + (defaspect Attacker + (attack [this target world] + {:pre [(satisfies? Destructible target)]} + (let [damage (get-damage this target world)] + (take-damage target damage))) + (attack-value [this world] + (get this :attack 1))) + +`Destructible` already handles reducing HP appropriately. The only thing left +is to give my entities some non-1 attack, defense, and/or hp values. For now +I used the following values: + +* Bunnies have 4 HP, default defense. +* Lichens have 6 HP, default defense. +* Silverfish have 15 HP, default defense. +* Players have 40 HP, 10 attack, default defense. + +These are really just placeholder numbers until I add the ability for monsters +to attack back. Once I do that I'll be able to play the game a bit and +determine if it's too easy or hard. + +Messaging +--------- + +Now the player can attack things and it may take a few swings to kill them. The +problem is that there's no feedback while this is going on, so it's hard to tell +that you're actually doing damage until the monster dies. + +A messaging system will let me display informational messages to give the player +some feedback. I decided to implement this like everything else: as an aspect. + +An entity that implements the `Receiver` protocol will be able to receive +messages. Here's `entities/aspects/receiver.clj`: + + :::clojure + (ns caves.entities.aspects.receiver + (:use [caves.entities.core :only [defaspect]] + [caves.world :only [get-entities-around]])) + + (defaspect Receiver + (receive-message [this message world] + (update-in world [:entities (:id this) :messages] conj message))) + + (defn send-message [entity message args world] + (if (satisfies? Receiver entity) + (receive-message entity (apply format message args) world) + world)) + +I've got a helper function `send-message` which is what entities will use to +send messages, instead of performing the `(satisfies? Receiver entity)` check +themselves every time. If the entity they're sending the message to isn't +a `Receiver` it will simply drop the message on the floor by returning the world +unchanged. It also handles formatting the message string for them. + +The default `receive-message` simply appends the message to a `:messages` +attribute in the entity. + +I have a lot of ideas about extending this system in the future, but for now +I'll just keep it simple to match Trystan's. + +Now I need to send some messages. The most obvious place to do this is when +something attacks something else, so I updated the `Attacker` aspect once more: + + + :::clojure + (defaspect Attacker + (attack [this target world] + {:pre [(satisfies? Destructible target)]} + (let [damage (get-damage this target world)] + (->> world + (take-damage target damage) + (send-message this "You strike the %s for %d damage!" + [(:name target) damage]) + (send-message target "The %s strikes you for %d damage!" + [(:name this) damage])))) + (attack-value [this world] + (get this :attack 1))) + +This is starting to get a little crowded. If I need to do much more in here +I'll refactor some stuff out into helper functions. But for now it's still +readable. + +Here you can see how making world-altering functions take the world as the last +argument pays off by letting me use `->>` to chain together actions. + +I also added a `:name` attribute to entities so I can say "You strike the bunny" +instead of "You strike the v". Nothing too special there. + +Finally, I need a way to notify nearby entities when something happens. Here's +what I came up with: + + :::clojure + (defn send-message-nearby [coord message world] + (let [entities (get-entities-around world coord 7) + sm (fn [world entity] + (send-message entity message [] world))] + (reduce sm world entities))) + +First I grab all the entities within 7 squares of the message coordinate. Then +I create a little helper function called `sm` that wraps `send-message`. It +will take a world and an entity and return the modified world. I use `reduce` +here to iterate over the entities and send the message to each one. It's +a pretty way of handling that looping. + +The `get-entities-around` function is new: + + :::clojure + (defn get-entities-around + ([world coord] (get-entities-around world coord 1)) + ([world coord radius] + (filter #(<= (radial-distance coord (:location %)) + radius) + (vals (:entities world))))) + +It looks through all the entities in the world and returns a sequence of those +whose "radial" distance is less than or equal to the given radius. The "radial +distance" (also called the "king's move" distance by some) looks like this: + + :::text + 3333333 + 3222223 + 3211123 + 3210123 + 3211123 + 3222223 + 3333333 + +And the function: + + :::clojure + (defn radial-distance + "Return the radial distance between two points." + [[x1 y1] [x2 y2]] + (max (abs (- x1 x2)) + (abs (- y1 y2)))) + +I may end up needing to modify this `send-message-nearby` function to be a bit +more powerful in the future. Trystan's version modifies the verbs and such. +For now this is good enough for me. + +Now I can make lichens notify nearby creatures when they grow: + + :::clojure + (defn grow [{:keys [location]} world] + (if-let [target (find-empty-neighbor world location)] + (let [new-lichen (make-lichen target) + world (assoc-in world [:entities (:id new-lichen)] new-lichen) + world (send-message-nearby location "The lichen grows." world)] + world) + world)) + +The last step is to actually display these messages to the player. I added +a `draw-messages` function to the `ui/drawing.clj` file: + + :::clojure + (defn draw-messages [screen messages] + (doseq [[i msg] (enumerate messages)] + (s/put-string screen 0 i msg {:fg :black :bg :white}))) + +And then I modified the main `draw-ui` function for `:play` UIs to draw the +messages on top of the map: + + :::clojure + (defmethod draw-ui :play [ui game screen] + (let [world (:world game) + {:keys [tiles entities]} world + player (:player entities) + [cols rows] (s/get-size screen) + vcols cols + vrows (dec rows) + origin (get-viewport-coords game (:location player) vcols vrows)] + (draw-world screen vrows vcols origin tiles) + (doseq [entity (vals entities)] + (draw-entity screen origin vrows vcols entity)) + (draw-hud screen game) + (draw-messages screen (:messages player)) + (highlight-player screen origin player))) + +This does mean that messages will cover a bit of the screen. For now I'll live +with that, but in the future it's something to fix. + +Finally we need to clear the message queue out periodically, otherwise it'll +grow until it covers the entire screen! I modified the main game loop in +`core.clj`: + + :::clojure + (defn clear-messages [game] + (assoc-in game [:world :entities :player :messages] nil)) + + (defn run-game [game screen] + (loop [{:keys [input uis] :as game} game] + (when (seq uis) + (if (nil? input) + (let [game (update-in game [:world] tick-all) + _ (draw-game game screen) + game (clear-messages game)] + (recur (get-input game screen))) + (recur (process-input (dissoc game :input) input)))))) + +Results +------- + +After fixing a few other bugs (you can read the changelog if you're interested) +I've now got a working combat system, and a messaging system so I can tell +what's going on: + +![Screenshot](/media/images{{ parent_url }}/caves-06-01.png) + +It's actually starting to feel like a real game now, instead of just a sandbox +where you can break things. + +You can view the code [on GitHub][result-code] if you want to see the end +result. + +[result-code]: https://github.com/sjl/caves/tree/entry-06/src/caves + +The next article will move on to Trystan's seventh post, which adds multiple +z-levels to the caves. + +{% endblock article %} diff -r 739cc98ac689 -r 0230ec523afd content/blog/2012/07/the-homely-mutt.html --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/content/blog/2012/07/the-homely-mutt.html Sun Jul 29 15:50:36 2012 -0400 @@ -0,0 +1,885 @@ + {% extends "_post.html" %} + + {% hyde + title: "The Homely Mutt" + snip: "Sparrow's dead? Why not try Mutt?" + created: 2012-07-23 10:00:00 + flattr: true + %} + +{% block article %} + +Now that [Sparrow][] is [effectively dead][sparrow-dead] many of its users will +be looking for a new email client. If you're not afraid of the terminal you may +want to give [Mutt][] a try. + +Mutt certainly isn't the prettiest email client around, and its +setup/configuration process is one of the ugliest out there. But once you get +it set up it's got a lot of advantages over many other email clients. + +In this post I'll show you how to set up Mutt on OS X like I do. + +[Sparrow]: http://sparrowmailapp.com/ +[sparrow-dead]: http://www.theverge.com/2012/7/20/3172222/google-buys-sparrow-mail +[Mutt]: http://www.mutt.org/ + +[TOC] + +How I Use Email +--------------- + +This setup is going to be specific to the way I work with email. Notably: + +* I have a Google Apps account that provides my steve@stevelosh.com email address. +* I have a lot of other email addresses, but they all simply forward to my main one. +* All mail I send comes from steve@stevelosh.com. +* I store my contacts in the OS X address book. +* All email comes into my inbox (or to a folder for a specific mailing list). +* Once I'm done with an email, I remove it from my inbox and it lives in the + "All Mail" archive. I don't sort email into folders. +* Sometimes I write email without an internet connection and send it once I get + connected again. + +My email setup is tailored around those requirements, so that's what it does +best. Mutt is very configurable though, so if you work differently you can +probably bend it to make it work like you want. + +In particular, extending this setup to work with multiple email accounts +wouldn't be too much trouble. I used to work with two separate accounts until +I said "screw it, I'll just use the one". + +Other Guides and Resources +-------------------------- + +I've used a lot of other guides to figure out how to get this giant Rube +Goldberg machine of an email client working. Here are a few of them: + +* +* +* +* +* +* +* +* +* + +Overview +-------- + +I'm going to give it to you straight: getting this whole contraption set up is +going to take at least an hour from start to finish, not counting the time it'll +take to download all of your email and install stuff. + +It's an investment, and you might not want to make it. If not, go use +Thunderbird, er, Sparrow, er, I don't know, the Gmail web interface or +something. + +Mutt on its own doesn't do very much, so we're going to combine it with a few +other things to get the job done. Here's a bird's eye view of what it'll look +like when we're done: + +![Diagram](/media/images{{ parent_url }}/what-the-mutt.png) + +If this diagram doesn't make you run screaming, you might just be masochistic +enough to make it through the initial setup of Mutt. If you do, you'll be +rewarded with email bliss that won't go away when Google or Facebook decide to +toss some money around. + +Getting Email +------------- + +First thing's first: we're going to pull down our email from Gmail to our local +machine. All of it. It'll take a while the first time you sync, but has a few +benefits. + +### Why Local Email? + +Having a local copy of all of your email means you've always got access to it, +no matter where you are. Looking for that one person's address they emailed you +six years ago when you're trying to find their house and you don't have an +internet connection? No problem, it's on your hard drive. + +This also acts as a backup in case Google ever decides to kill your Gmail +account. It'll be stored in a common format that a lot of programs can read, so +you've got a safety net. And the email is stored as normal files, so if you use +something like Time Machine or [Backblaze][] that's yet another backup. + +In this setup all of your email is stored as plain text. If you want it +encrypted just use OS X's full-disk encryption and you're set. + +I use [offlineimap][] to pull email down from Gmail and get it on my hard drive. +Offlineimap will also sync any changes you make to this local copy back up to +Gmail. + +[offlineimap]: http://offlineimap.org/ +[Backblaze]: http://www.backblaze.com/partner/af3574 + +### Installing offlineimap + +I've gone through a number of laptops in the past few years, and each time +I spend a painful half hour or so screwing around with the lastest version of +offlineimap's backwards-incompatible changes. + +If you're determined to run the latest version of offlineimap, you can install +it with pip or something. If you just want to download your fucking email and +get on with your life, you can follow the instructions I've laid out for you +here: + +* `git clone git://github.com/spaetz/offlineimap.git` +* `cd offlineimap` +* `git checkout 679c491c56c981961e18aa43b31955900491d7a3` +* `python setup.py install` + +That's the version I'm using. It works. You can use a newer one if you want, +but expect to spend some time figuring out how to fix the configuration in this +post to work with whatever breaking changes have been made since then. The last +time I tried this I got to rewrite all my nametrans stuff. That was fun. + +### Configuring offlineimap + +Once you've got offlineimap installed, you'll need to create +a `~/.offlineimaprc` file. You can keep it in your dotfiles repo and symlink it +into place if you want. Here's a sample to get you started: + + [general] + ui = TTY.TTYUI + accounts = SteveLosh + pythonfile=~/.mutt/offlineimap.py + fsync = False + + [Account SteveLosh] + localrepository = SteveLosh-Local + remoterepository = SteveLosh-Remote + status_backend = sqlite + postsynchook = notmuch new + + [Repository SteveLosh-Local] + type = Maildir + localfolders = ~/.mail/steve-stevelosh.com + nametrans = lambda folder: {'drafts': '[Gmail]/Drafts', + 'sent': '[Gmail]/Sent Mail', + 'flagged': '[Gmail]/Starred', + 'trash': '[Gmail]/Trash', + 'archive': '[Gmail]/All Mail', + }.get(folder, folder) + + [Repository SteveLosh-Remote] + maxconnections = 1 + type = Gmail + remoteuser = steve@stevelosh.com + remotepasseval = get_keychain_pass(account="steve@stevelosh.com", server="imap.gmail.com") + realdelete = no + nametrans = lambda folder: {'[Gmail]/Drafts': 'drafts', + '[Gmail]/Sent Mail': 'sent', + '[Gmail]/Starred': 'flagged', + '[Gmail]/Trash': 'trash', + '[Gmail]/All Mail': 'archive', + }.get(folder, folder) + folderfilter = lambda folder: folder not in ['[Gmail]/Trash', + 'Nagios', + 'Django', + 'Flask', + '[Gmail]/Important', + '[Gmail]/Spam', + ] + +It's kind of a beast, so let's go through it line by line and see what's going +on. + + [general] + ui = TTY.TTYUI + accounts = SteveLosh + pythonfile=~/.mutt/offlineimap.py + fsync = False + +First we tell offlineimap to use the `TTY.TTYUI` ui. Yes, this program that +syncs yoiur email has multiple user interfaces. I guess if you can't decide +what color the bikeshed should be you can just build a whole bunch of bikesheds +instead. + +Then we specify the accounts. There's only one, because as I said before: +I only use a single email account that all my addresses forward to. If you +wanted to have many, you'd change this line. + +The `pythonfile` is just a file that offlineimap will parse (as Python) before +loading the rest of the config, so you can define custom helper functions more +easily. We'll see more of this later. + +We're also telling offlineimap that it doesn't need to fsync after every single +operation. This will speed things up, and since it's just a local copy it's +typically not a big deal if we lose an email here and there from a crash (it'll +just be synced the next time anyway). + + [Account SteveLosh] + localrepository = SteveLosh-Local + remoterepository = SteveLosh-Remote + status_backend = sqlite + +This next section hooks up a few things. First, it tells offlineimap which +local and remote repositories to use for the account. Manual configuration +instead of sane defaults is a recurring theme we'll see throughout this process. + +Hey, I titled the entry "The *Homely* Mutt" for a reason. + +We're also going to use a SQLite-based cache for this account. If you don't +already have SQLite you'll want to get it with `brew install sqlite`. + + [Repository SteveLosh-Local] + type = Maildir + localfolders = ~/.mail/steve-stevelosh.com + nametrans = lambda folder: {'drafts': '[Gmail]/Drafts', + 'sent': '[Gmail]/Sent Mail', + 'flagged': '[Gmail]/Starred', + 'trash': '[Gmail]/Trash', + 'archive': '[Gmail]/All Mail', + }.get(folder, folder) + +Now we're getting to the meat of the configuration. This "local repository" is +going to be the mail as it sites on our hard drive. We're going to use the +[Maildir format][maildir] because it plays nicely with Mutt (and tons of other +stuff). + +Then we specify the path where we're going to keep the mail. This is going to +take a lot of space if you've got a lot of mail. Attachments are downloaded +too. When I said you're getting an offline copy of your email I meant all of +it. + +I think offlineimap needs the `~/.mail` directory created for it. It's been +a while since I did this, so I might be wrong, but if it complains about not +being able to access the mail folders just go ahead and `mkdir ~/.mail`. + +Next we have the craziest part of the offlineimap configuration: name +translation. + +Here's the issue: offlineimap needs to know how to translate the names of +folders on the IMAP server to folder names on your hard drive. + +Also, Gmail doesn't actually use *folders* but its own concept called "labels". +But since the IMAP protocol doesn't know about labels, it fakes them by making +them appear to be folders. + +User-created labels in Gmail (like "Mercurial" or "Clients") will appear as +folders with those names through IMAP. + +Built-in, special Gmail folders have names that start with `[Gmail]/`. We need +to turn those into something sane for our hard drive, so that's what this +nametrans setting is for. It's a Python function that takes the remote folder +name and returns the name that should be used on your local hard drive. + +Yes, you read that right. This is Python code embedded in the right hand side +of an INI file's setting assignment. I am not fucking with you, this is +seriously how you do it. Go ahead and crack open that beer now. + +So the "Sent Mail" folder in your Gmail account will be synced to +`~/.mail/steve-stevelosh.com/sent`. Cool. + +(No, I don't know what would happen if you created a label called `[Gmail]/All +Mail` in Gmail. If you try, let me know, but I take no responsibility if it +ends with all your email deleted.) + +[maildir]: https://en.wikipedia.org/wiki/Maildir + + [Repository SteveLosh-Remote] + maxconnections = 1 + type = Gmail + remoteuser = steve@stevelosh.com + remotepasseval = get_keychain_pass(account="steve@stevelosh.com", server="imap.gmail.com") + realdelete = no + nametrans = lambda folder: {'[Gmail]/Drafts': 'drafts', + '[Gmail]/Sent Mail': 'sent', + '[Gmail]/Starred': 'flagged', + '[Gmail]/Trash': 'trash', + '[Gmail]/All Mail': 'archive', + }.get(folder, folder) + folderfilter = lambda folder: folder not in ['[Gmail]/Trash', + 'Nagios', + 'Django', + 'Flask', + '[Gmail]/Important', + '[Gmail]/Spam', + ] + +Finally, the home stretch. The last section described the folder on our local +hard drive, and this one describes our Gmail account. + +First, we tell offlineimap to only ever use a single connection at a time. You +can try increasing this number for better performance, but in my experience +Google is not stingy with its rate limits and would cut me off fairly often when +I tried that. Just leave it at one if you want to be safe. + +Next is the type. Luckily offlineimap provides a `Gmail` type that handles +a lot of the craziness that is Gmail's IMAP setup. Nice. + +Then we have the username. Nothing special here, except that if you have +a non-apps account (i.e.: an actual vanilla Gmail account) you may or may not +need to include the `@gmail.com` in the username. I don't know. If one doesn't +work, just try the other. + +Next we have `remotepasseval`. This is a bit of Python code (drink!) that +should return the password for the account. + +What is this `get_keychain_pass` function? Well, remember when we saw the +`pythonfile` setting back in the general section? It's a function defined in +there. I'll talk about that in the next section, for now just accept that it +works. + +Next we set `realdelete` to no. If this is set to yes, then deleting an email +in your inbox would actually delete it entirely. When you set it to no, then +deleting an email from your inbox (or any label's folder) will leave it in +Gmail's All Mail. + +If you want to really delete an email, you'll need to delete it from All Mail +(which is named archive on our local filesystem, remember?). I feel like this +is a good compromise. I rarely care about actually deleting mail, given that +I have many unused gigabytes available on Gmail. + +Next we have another nametrans setting. This is a Python function (drink!) just +like the one for the local repository, except it goes in the other direction. +It takes the name of a local folder and returns the name of the folder on the +IMAP server. Knowing this, it should be easy to understand this setting. + +Finally, we have `folderfilter`. This is a Python function (drink!) that takes +a **remote** folder name and returns `True` if that folder should be synced, or +`False` if it should *not* be synced. I've chosen to skip syncing my Spam and +Trash folders, as well as a few mailing list labels I don't check all that +often. Customize this to your own taste. + +### Retrieving Passwords + +We're almost ready, but there's one more thing we need to do, and that's +implement a secure way for offlineimap to get access to our Gmail password. + +If you don't care too much about security, you *can* configure offlineimap with +a plaintext password right in the config file. But don't do that. It'll only +take a minute to do this securely. + +First, you need to add your Gmail password into your OS X keychain. Open the +Keychain Access app and press the `+` button: + +![Keychain 1](/media/images{{ parent_url }}/keychain-1.png) + +Then fill out the form. The "Keychain Item Name" should be +`http://imap.gmail.com`. The "Account Name" should be your email address. The +password should be your password: + +![Keychain 2](/media/images{{ parent_url }}/keychain-2.png) + +Press "Add". Now repeat the process for the SMTP server. The "Keychain Item +Name" should be `smtp://smtp.gmail.com`. The "Account Name" should be your +email address. The password should be your password: + +![Keychain 3](/media/images{{ parent_url }}/keychain-3.png) + +Now we need to create the `offlineimap.py` file we pointed offlineimap to +earlier. It needs to contain the `get_keychain_pass` function, which takes an +`account` and `server` and return the password. Here's the file I'm using: + + :::python + #!/usr/bin/python + import re, subprocess + def get_keychain_pass(account=None, server=None): + params = { + 'security': '/usr/bin/security', + 'command': 'find-internet-password', + 'account': account, + 'server': server, + 'keychain': '/Users/sjl/Library/Keychains/login.keychain', + } + command = "sudo -u sjl %(security)s -v %(command)s -g -a %(account)s -s %(server)s %(keychain)s" % params + output = subprocess.check_output(command, shell=True, stderr=subprocess.STDOUT) + outtext = [l for l in output.splitlines() + if l.startswith('password: ')][0] + + return re.match(r'password: "(.*)"', outtext).group(1) + +In a nutshell, it uses `/usr/bin/security` to retrieve the password. Read +through the code if you're curious. + +This is not *completely* secure, but it's better than having your password in +a plaintext file in your home directory. + +Whew! Time to actually run this thing and pull down our email! + +### Running offlineimap + +Assuming everything is in place, open a terminal and run offlineimap: + + offlineimap + +Go read a book, because this is going to pull down all the email (with +attachments) in any folders you didn't exclude in the config file. + +**If there's an error, stop and figure out what went wrong**. Remember, +offlineimap is a *two-way* sync, so there's always the possibility it'll eat +your email if you seriously mess something up! I wish it had a +`--dont-touch-remote` option you could use as a safety net for the original +sync, but it doesn't, so be careful! + +In the future you can use `offlineimap -q` to run it in "quick mode". It'll +perform fewer checks but will generally be much faster. + +If you want to set up offlineimap to run every 5 minutes or so, you can use +launchd. `cron` does not work for some reason. I'm not entirely sure why. + +Personally I actually *like* having to press a key to fetch new mail. It's less +of a distraction than having new mail rolling in all the time. I can get new +email when I'm ready to actually look at it, rather than having it nagging me +all the time. + +Mutt! +----- + +Now that you've got your email on your computer, it's finally time to start +using Mutt itself! + +### Installing + +Mutt can be installed in a bunch of different ways, but the easiest is through +Homebrew: + + brew install mutt --sidebar-patch + +The sidebar patch is a third-party patch that adds a sidebar to Mutt. I don't +know why it's not in core Mutt because it's insanely useful. Oh well, at least +Homebrew makes it simple to get. + +That's pretty much it for installation, but don't get too relaxed because you're +far from done. + +### Configuring + +Mutt is *very* configurable. This is great once you've become a power user and +want to mold it to your will, but terrible when you're just getting started. + +Mutt settings are kept in a `~/.muttrc` file. If this file doesn't exist Mutt +will look for `~/.mutt/muttrc` (note the lack of a dot in the filename), so you +can put it there if you prefer. + +Here's a basic `~/.muttrc` to get you started (a lot of which was taken from +[this article][pris]). Once you've got a bit of Mutt under your belt you'll +want to read [the documentation][muttdoc] for these settings, but for now just +use them to keep things sane: + +[pris]: http://pbrisbin.com/posts/two_accounts_in_mutt +[muttdoc]: http://www.mutt.org/doc/manual/manual-6.html + + # Paths + set alias_file = ~/.mutt/alias # where to store aliases + set header_cache = ~/.mutt/cache/headers # where to store headers + set message_cachedir = ~/.mutt/cache/bodies # where to store bodies + set certificate_file = ~/.mutt/certificates # where to store certs + set tmpdir = ~/.mutt/temp # where to keep temp files + set signature = ~/.mutt/sig # signature file + + # Use Vim to compose email, with a few default options. + set editor = "vim -c 'normal! }' -c 'redraw'" + + # Colors! + source ~/.vim/bundle/badwolf/contrib/badwolf.muttrc + + # Basic Options + set wait_key = no # shut up, mutt + set mbox_type = Maildir # mailbox type + set folder = ~/.mail # mailbox location + set timeout = 3 # idle time before scanning + set mail_check = 0 # minimum time between scans + unset move # gmail does that + set delete # don't ask, just do + unset confirmappend # don't ask, just do! + set quit # don't ask, just do!! + unset mark_old # read/new is good enough for me + set beep_new # bell on new mails + set pipe_decode # strip headers and eval mimes when piping + set thorough_search # strip headers and eval mimes before searching + + # Sidebar Patch + set sidebar_delim = ' │' + set sidebar_visible = yes + set sidebar_width = 24 + color sidebar_new color221 color233 + bind index,pager sidebar-next + bind index,pager sidebar-prev + bind index,pager sidebar-open + + # Status Bar + set status_chars = " *%A" + set status_format = "───[ Folder: %f ]───[%r%m messages%?n? (%n new)?%?d? (%d to delete)?%?t? (%t tagged)? ]───%>─%?p?( %p postponed )?───" + + # Index View + set date_format = "%m/%d" + set index_format = "[%Z] %D %-20.20F %s" + set sort = threads # like gmail + set sort_aux = reverse-last-date-received # like gmail + set uncollapse_jump # don't collapse on an unread message + set sort_re # thread based on regex + set reply_regexp = "^(([Rr][Ee]?(\[[0-9]+\])?: *)?(\[[^]]+\] *)?)*" + + # Pager View + set pager_index_lines = 10 # number of index lines to show + set pager_context = 3 # number of context lines to show + set pager_stop # don't go to next message automatically + set menu_scroll # scroll in menus + set tilde # show tildes like in vim + unset markers # no ugly plus signs + + set quote_regexp = "^( {0,4}[>|:#%]| {0,4}[a-z0-9]+[>|]+)+" + alternative_order text/plain text/enriched text/html + + # Compose View + set realname = "Steve Losh" # who am i? + set envelope_from # which from? + set sig_dashes # dashes before sig + set edit_headers # show headers when composing + set fast_reply # skip to compose when replying + set askcc # ask for CC: + set fcc_attach # save attachments with the body + unset mime_forward # forward attachments as part of body + set forward_format = "Fwd: %s" # format of subject when forwarding + set forward_decode # decode when forwarding + set attribution = "On %d, %n wrote:" # format of quoting header + set reply_to # reply to Reply to: field + set reverse_name # reply as whomever it was to + set include # include message in replies + set forward_quote # include message in forwards + + # Headers + ignore * # ignore all headers + unignore from: to: cc: date: subject: # show only these + hdr_order from: to: cc: date: subject: # and in this order + + # steve@stevelosh.com {{{ + + # Default inbox. + set spoolfile = "+steve-stevelosh.com/INBOX" + + # Alternate email addresses. + alternates sjl@pculture.org still\.?life@gmail.com steve@ladyluckblues.com steve@pculture.org + + # Mailboxes to show in the sidebar. + mailboxes +steve-stevelosh.com/INBOX \ + +steve-stevelosh.com/vim \ + +steve-stevelosh.com/clojure \ + +steve-stevelosh.com/python \ + +steve-stevelosh.com/mercurial \ + +steve-stevelosh.com/archive \ + +steve-stevelosh.com/sent \ + +steve-stevelosh.com/drafts \ + + # Other special folders. + set mbox = "+steve-stevelosh.com/archive" + set postponed = "+steve-stevelosh.com/drafts" + + # Sending email. + set from = "steve@stevelosh.com" + set sendmail = "/usr/local/bin/msmtp -a stevelosh" + set sendmail_wait = 0 # no please don't silently fail, email is important + unset record + + # }}} + # Account Hooks {{{ + + # folder-hook steve-stevelosh.com/* source ~/.mutt/steve-stevelosh.com.muttrc + + # }}} + # Key Bindings {{{ + + # Unbind Stupid Keys {{{ + + bind index,pager \# noop + bind index i noop + bind index w noop + + # }}} + # Pager {{{ + + bind pager i exit + bind pager / search + bind pager k previous-line + bind pager j next-line + bind pager gg top + bind pager G bottom + bind pager R group-reply + + macro pager \Cu "|urlview" "call urlview to open links" + macro pager s "cat > ~/Desktop/" "save message as" + + # }}} + # Index {{{ + + bind index R group-reply + bind index sync-mailbox + bind index k previous-entry + bind index j next-entry + bind index gg first-entry + bind index G last-entry + bind index p recall-message + bind index collapse-thread + macro index s "cat > ~/Desktop/" "save message as" + + # Mark all as read + macro index \Cr "T~UN." "mark all messages as read" + + # Quickly change date formats + macro index f ":set date_format = \"%m/%d\"" "short date format" + macro index F ":set date_format = \"%m/%d at %I:%M %P\"" "long date format" + + # Sync email + macro index O "offlineimap -q" "run offlineimap to sync mail in the foreground" + macro index o "offlineimap -q >/dev/null 2>&1 &" "run offlineimap to sync mail in the background" + + # Saner copy/move dialogs + macro index C "?" "copy a message to a mailbox" + macro index M "?" "move a message to a mailbox" + + # Quickly change mailboxes + macro index \' "+steve-stevelosh.com/INBOX" "go to stevelosh/INBOX" + macro index \" "+steve-stevelosh.com/archive" "go to stevelosh/archive" + + # Just use notmuch for everything + macro index / "unset wait_keyread -p 'notmuch query: ' x; echo \$x >~/.cache/mutt_terms~i \"\`notmuch search --output=messages \$(cat ~/.cache/mutt_terms) | head -n 600 | tr '+' '.' | perl -le '@a=<>;chomp@a;s/\^id:// for@a;$,=\"|\";print@a'\`\"" "show only messages matching a notmuch pattern" + + # Unlimit aka show [a]ll + macro index a "all\n" "show all messages (undo limit)" + + # }}} + # Compose {{{ + + bind compose p postpone-message + + # }}} + # Attachment {{{ + + # View, god dammit! + bind attach view-mailcap + + # }}} + # "Open in Vim" {{{ + + macro index,pager V "|vim -c 'setlocal ft=mail' -c 'setlocal buftype=nofile' -" "open in vim" + macro index,pager M "|mvim -c 'setlocal ft=mail' -c 'setlocal buftype=nofile' - >/dev/null" "open in macvim" + + # }}} + + # }}} +### Running + +Now that you've got Mutt configured you can run it: + + mutt + +I like to always be in my `~/Desktop` folder when in Mutt, so that when I save +emails or attachments they go there by default. I have a little shell function +set up that cd's there for be before running Mutt. + +If you run the [new fish shell][fish], this is going to cause problems later +(long story, but it's related to the `read` builtin). Do yourself a favor and +head those confusing issues off at the pass with a fish function: + + :::text + function mutt + bash -c 'cd ~/Desktop; /usr/local/bin/mutt' $argv; + end + +[fish]: http://ridiculousfish.com/shell/ + +Reading Email +------------- + +Reading email is pretty straightforward in Mutt. You select a message in the +index and pretty return, and Mutt will display the "pager" with the contents of +the email: + +Let's add a few settings to our `~/.muttrc` to make reading email a bit +smoother. + + set pager_index_lines = 10 + +This sets the number of lines of the index (the top "pane") to show while +reading email. I like to have 10 so I can tell where I'm at in the list of +email. + + set pager_context = 3 + +This tells Mutt how far it should scroll when you "page down" in the pager with +space. I have it set to three, so when I press space Mutt scrolls down far +enough that the last three lines on the screen become the first three lines. +It's there to help you avoid losing your place when reading. + + set pager_stop + +Prevents Mutt from automatically going to the next message when you page down +when already at the end of a message. I'll move to the next message when I'm +good and ready, thank you. + + set tilde + +Shows tildes at the end of the message, like Vim does after the end of the file. +This is personal preference, but as a Vim user I like it. + + unset markers + +By default, when Mutt wraps long lines of text in the pager it will display +a `+` and the beginning of the wrapped lines. That's kind of ugly, so this +setting turns it off. + + set quote_regexp = "^( {0,4}[>|:#%]| {0,4}[a-z0-9]+[>|]+)+" + +This defines how Mutt finds "quoted text" in emails. Mutt will highlight quoted +text differently: + +![Quote Highlighting](/media/images{{ parent_url }}/mutt-quotes-1.png) + +This regex will tell Mutt to look for lines prefixed with `>` characters as +quoted text. Each `>` is one level of quoting. This is pretty standard. + +Writing Email +------------- + +Sending Email +------------- + +Mutt does have (some) built-in SMTP support, but we're going to use a separate +program to do our sending for a few reasons. + +First, Mutt's SMTP support was considered "experimental" the last time +I checked. Sending email is kind of important, so we'll stick with something +tried and true. + +Second, we want a method that won't require our password in a plaintext config +file. + +Go ahead and install the `msmtp` program through Homebrew: + + brew install msmtp + +Next we're going to need to create a `~/.msmtprc` file with the following +contents: + + account stevelosh + host smtp.gmail.com + port 587 + protocol smtp + auth on + from steve@stevelosh.com + user steve@stevelosh.com + tls on + tls_trust_file ~/.mutt/Equifax_Secure_CA.cert + + account default : stevelosh + +`msmtp` will look in your keychain for your SMTP password, which we added +earlier. No plaintext passwords! + +The other "interesting" bit here is the `tls_trust_file`. We're going to be +connecting to Gmail's SMTP server over SSL, and `msmtp` needs to know if it can +trust the certificate that the server on the other end is sending back. + +Copy the following and paste it into the path `tls_trust_file` is set to: + + :::text + -----BEGIN CERTIFICATE----- + MIIDIDCCAomgAwIBAgIENd70zzANBgkqhkiG9w0BAQUFADBOMQswCQYDVQQGEwJVUzEQMA4GA1UE + ChMHRXF1aWZheDEtMCsGA1UECxMkRXF1aWZheCBTZWN1cmUgQ2VydGlmaWNhdGUgQXV0aG9yaXR5 + MB4XDTk4MDgyMjE2NDE1MVoXDTE4MDgyMjE2NDE1MVowTjELMAkGA1UEBhMCVVMxEDAOBgNVBAoT + B0VxdWlmYXgxLTArBgNVBAsTJEVxdWlmYXggU2VjdXJlIENlcnRpZmljYXRlIEF1dGhvcml0eTCB + nzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAwV2xWGcIYu6gmi0fCG2RFGiYCh7+2gRvE4RiIcPR + fM6fBeC4AfBONOziipUEZKzxa1NfBbPLZ4C/QgKO/t0BCezhABRP/PvwDN1Dulsr4R+AcJkVV5MW + 8Q+XarfCaCMczE1ZMKxRHjuvK9buY0V7xdlfUNLjUA86iOe/FP3gx7kCAwEAAaOCAQkwggEFMHAG + A1UdHwRpMGcwZaBjoGGkXzBdMQswCQYDVQQGEwJVUzEQMA4GA1UEChMHRXF1aWZheDEtMCsGA1UE + CxMkRXF1aWZheCBTZWN1cmUgQ2VydGlmaWNhdGUgQXV0aG9yaXR5MQ0wCwYDVQQDEwRDUkwxMBoG + A1UdEAQTMBGBDzIwMTgwODIyMTY0MTUxWjALBgNVHQ8EBAMCAQYwHwYDVR0jBBgwFoAUSOZo+SvS + spXXR9gjIBBPM5iQn9QwHQYDVR0OBBYEFEjmaPkr0rKV10fYIyAQTzOYkJ/UMAwGA1UdEwQFMAMB + Af8wGgYJKoZIhvZ9B0EABA0wCxsFVjMuMGMDAgbAMA0GCSqGSIb3DQEBBQUAA4GBAFjOKer89961 + zgK5F7WF0bnj4JXMJTENAKaSbn+2kmOeUJXRmm/kEd5jhW6Y7qj/WsjTVbJmcVfewCHrPSqnI0kB + BIZCe/zuf6IWUrVnZ9NA2zsmWLIodz2uFHdh1voqZiegDfqnc1zqcPGUIWVEX/r87yloqaKHee95 + 70+sB3c4 + -----END CERTIFICATE----- + +If you're paranoid and don't trust that I'm giving you the right cert (or that +someone has hacked my site and changed it), you can generate it yourself. I'll +leave that as an exercise for the reader. + +Now we need to tell Mutt to use msmtp. Add the following to your `~/.muttrc` +file: + + set from = "steve@stevelosh.com" + set sendmail = "/usr/local/bin/msmtp -a stevelosh" + set sendmail_wait = 0 + unset record + +The `-a stevelosh` will need to change to whatever you named your account in the +msmtp config. + +The `unset record` line tells Mutt to not append a copy of every email you send +to a file on your hard drive. Gmail will save the emails you send in the sent +folder, so you'll get the the next time you sync with offlineimap anyway. + +The `sendmail_wait` line tells Mutt to wait for the msmtp program to finish +sending the mail before returning control, instead of running it in the +background. This makes it obvious if there's a problem sending a message, which +I prefer to silent, backgrounded failures. + +Now you can send email! Awesome! + +Once you've composed a test email and saved it you'll be presented with a screen +like this: + +![Sending Screen](/media/images{{ parent_url }}/mutt-send-1.png) + +The keys you need are listed along the top. Pressing `y` now will invoke msmtp +and send your email! + +You'll see "Sending message..." at the bottom of the screen while msmtp is +working. If there's a problem, Mutt will tell you the error. Figure it out +before moving on. + +Contacts +-------- + +Next we'll want to get Mutt to autocomplete our contacts from the OS X address +book. Unfortunately I've got some bad news for you: + +You're going to need to install XCode. + +No, not the command-line developer tools. The full XCode. I'm sorry, but trust +me when I say it's going to save you a lot of pain, so just grumble to yourself +a bit and do it. + +Okay, now that you've got XCode you can install the `contacts` program through +Homebrew: + + brew install contacts + +`contacts` is a command-line program that you can use to query your address +book. To tell Mutt how to use it add the following lines to your +`~/.mutt/muttrc`: + + set query_command = "contacts -Sf '%eTOKEN%n' '%s' | sed -e 's/TOKEN/\t/g'" + bind editor complete-query + bind editor ^T complete + +Now when you're filling out an email address field you can type a few characters +and hit Tab to get a screen like this: + +![Contacts](/media/images{{ parent_url }}/mutt-contacts-1.png) + +You can use `j` and `k` to select an item, press return to complete it. Press +`q` if you've changed your mind and want to cancel the completion. Look at the +top of the screen for more handy little keys you can use here. + +If there's only one item in the list Mutt won't bother showing you this screen +and will just complete it right away. + +This completion searches more than just the email address. It'll also search +the names and possibly other fields from the address book entries as well. + +Searching Email +--------------- + +notmuch +muttrc + + + +{% endblock article %} diff -r 739cc98ac689 -r 0230ec523afd media/images/blog/2012/07/caves-06-01.png Binary file media/images/blog/2012/07/caves-06-01.png has changed diff -r 739cc98ac689 -r 0230ec523afd media/images/blog/2012/07/keychain-1.png Binary file media/images/blog/2012/07/keychain-1.png has changed diff -r 739cc98ac689 -r 0230ec523afd media/images/blog/2012/07/keychain-2.png Binary file media/images/blog/2012/07/keychain-2.png has changed diff -r 739cc98ac689 -r 0230ec523afd media/images/blog/2012/07/keychain-3.png Binary file media/images/blog/2012/07/keychain-3.png has changed diff -r 739cc98ac689 -r 0230ec523afd media/images/blog/2012/07/mutt-contacts-1.png Binary file media/images/blog/2012/07/mutt-contacts-1.png has changed diff -r 739cc98ac689 -r 0230ec523afd media/images/blog/2012/07/mutt-quotes-1.png Binary file media/images/blog/2012/07/mutt-quotes-1.png has changed diff -r 739cc98ac689 -r 0230ec523afd media/images/blog/2012/07/mutt-send-1.png Binary file media/images/blog/2012/07/mutt-send-1.png has changed diff -r 739cc98ac689 -r 0230ec523afd media/images/blog/2012/07/what-the-mutt.png Binary file media/images/blog/2012/07/what-the-mutt.png has changed