• I like Unicorn because it’s Unix

• The Secret To 10 Million Concurrent Connections -The Kernel Is The Problem, Not The Solution

The first is about how good Unix is at scaling the scheduling and distribution of work among processes. The second is about how Unix is the problem when it comes to the scheduling and distribution of work at scale.

The question, of course, is “What scale?”. Like the difference between a cure and a poison is sometimes the dosage.

Identification with another is addictive: some of my life’s most profound, memorable experiences have come when something bridged the gap between me and another human. Because I’m a reader, this can occur across the distance of space and time. It’s happened with minor Chekov characters, and at the end of Kate Mansfield stories. It happens again and again with Norman Rush and George Saunders. The author has pushed a character through the page and connected with me on a deep level: identification.

Identification happens with computer programming, too.

I say this as a reader, writer, and programmer: I experience identification when reading and programming, and I strive to create it when writing and programming.

Though they deal with the messiness of reality differently, several techniques common to both disciplines enable them to achieve this mental intimacy: navigating complexity; avoiding pitfalls that inhibit communication; choosing structure wisely; harnessing expressive power; and inhabiting other minds. The Requests library, a work of computer programming by Kenneth Reitz, illustrates this.

Navigating Complexity

As humans, we’ve evolved to unconsciously filter the staggering possibilities available to us at each moment. To manage that complexity, we have rules. Humans come with some built in rules (eat when hungry), and easily learn more (don’t touch hot stoves; don’t lie, especially to those you love). When I’m writing, my stories might examine how these rules come into conflict with each other: a government doctor is ordered to a remote, neglected village, and comes to value its people more than his duty to the government.

Computers don’t come with built-in rules. When I’m programming, if I make a mistake, the computer marches into it head-on. Computers do precisely and only what you tell them (frustrations with Microsoft Word notwithstanding), and because there are a staggering number of things they can do, programmers constantly battle complexity, sometimes spelling out in excruciating detail exactly what it is we want them to do. Programming is an effort to build up sets of rules that direct the computer to do something helpful.

Staggering possibilities managed by rules: that’s also a description of natural language. The difference is that natural language has been with our species for a long time. Our brains have evolved pockets that just do languagey stuff. When I write, I benefit from this history.

When it comes to programming, we’re just getting started. Our brains need help.

Avoiding Pitfalls

Misunderstandings and garbled messages kill identification before it can begin. Grammar, punctuation, and even typing mistakes fatally divert attention from what the writer is trying to say. Layout and space matter, too: linebreaks are essential to poetry, and visual flow to longer narrative. Communication can happen without them, but immersion comes more easily when nothing gets in the way.

The stuff that programmers write, called code, can likewise be correct or incorrect, beautiful or ugly. Requests is written in a programming language called Python ¹ that’s known for being easy to read. Unusually for programming languages, Python requires a certain amount of space between bits of code, and its style guide encourages the use of even more space than required. Built-in elements of the language that programmers cannot change have clear and consistent names that further aid comprehension–not the case in every language. In Requests, Reitz embraces Python’s style.

Structure

Options for structure and sequence abound: writers order narrative chronologically, in media res, or in more complex ways. Chapters can be short, long, or omitted. From Volumes and Parts to paragraphs and sentences, structure matters.

Requests’s dual structure is typical of programming libraries: the code itself, and a document that explains how to use the code. Within the code, there are structures that are roughly analogous in scale to sentences, paragraphs, and chapters, while the documentation proceeds narratively, starting with easy matters and advancing to difficult material.

This bundle of code plus documentation is known as an application programming interface, or API.

APIs are what make it possible to post your Instagram photos to your Facebook timeline, check-in on Foursquare from your iPhone, book a flight on United from Kayak, etc. Without APIs, without a way to share units of functionality with each other, each programmer or programming team would be on its own. Everything would take much, much longer; each programmer would have to reinvent the wheel.

Here’s an interesting bit of jargon: programmers say that somebody is "exposing an API." It’s an act of confidence, but also vulnerability.

Expressive Power

The number of books to read is huge. Why would you choose one over another? Why might you read a literary novel over, say, a detective novel, or an author you love over one you don’t? One reason might be that while a thoughtful reader can eke meaning from even the flimsiest of genre novels (sometimes with great satisfaction and enjoyment), we tend to find more significance in novels and poetry that pack their sentences and stanzas with meaning, allusion, emotion, and impact. We might call this expressive power.

The collection of programming libraries is also huge. To show how they vary in expressive power, I need to introduce a bit of programming code:

>>> import urllib2
>>> req = urllib2.Request('http://www.goodreads.com')
>>> response = urllib2.urlopen(req)
>>> response.read()

This snippet ² is the standard way to retrieve a webpage (specifically, Goodreads’ homepage) using a Python library called urllib2. Requests does the same thing this way:

>>> import requests
>>> response = requests.get("http://www.goodreads.com")

Requests’ code is two lines instead of four. That might not seem like a big difference from urllib2, but note it, and bear with me just a bit longer. Both these snippets store the webpage they’ve retrieved in a variable called response. When I’m programming, I’ll want to do something useful with response: save it to a file, show it to a user, parse it to see what books are popular. If I ask Python to describe the response returned by urllib2, I get something like ³ this:

>>> response
<addinfourl at 4338521656 whose fp = <socket._fileobject object at 0x10297ce50>>

The response that Requests gives me, on the other hand, looks like:

>>> response
<Response [200]>

Again, Requests is smaller, and it turns out its two lines of code give me something much more useable than urllib2. Response [200] is meaningful; one of the first things every web programmer learns is that, in HTTP, the language of the internet, 200 means, simply, "OK". Requests lets me know my code succeeded.

The thing that urllib2 gave me, <addinfourl at 4338521656 whose fp = <socket._fileobject object at 0x10297ce50>>, is clear as mud. I can make some guesses about it, but to really understand it, I’d have to spend time rooting around in documentation. I just wanted to get a web page.

This difference between the two libraries–their expressive power–plays out over and over with the rest of their functionality. Fetching a webpage is just scratching the surface, and the efficiency penalty imposed by the gap in expressive power compounds as you do more complicated, hence interesting, things.

requests.get is Hemingway, or Strunk and White. Declarative. Terse. Say what you mean as plainly as possible. Behind the scenes, its two lines are doing more work than they seem, the way a good writer’s sentences accomplish multiple things at once: moving plot forward, imparting character, setting tone, painting a scene.

urllib2’s jumble is verbose jargon, like legalese: heretofore, whereas, the party of the first part. Its API is a menagerie of abstractions with names such as OpenerDirector and HTTPPasswordMgrWithDefaultRealm ⁴ that must be looked up to be understood, and are often underwhelming in their capabilities.

Greater expressive power matches more closely the workings of my mind, where thought flows in a continuous stream. It eases the transit of thought across the boundary between internal and external. With a good novel or a good API, the pace of comprehension accelerates, syncing with my inner monologue, setting the stage for identification.

Dealing with Reality

Programmers strive to be subtext-free. The code is tricky enough. Miss a comma or closing brace, and spend hours trying to find your mistake. The computer forgives nothing. Good programmers deal with this by making reality as clean as possible.

For example, within an app that tracks the books you’ve read, you’ve either read a book or not, or perhaps you’re currently reading it. There’s no "I read a third of it, then put it down for a while," or "I stole it from a friend and almost finished it but now Vanessa’s borrowing it, I think." A system that tried to encode every possible state of any person’s relationship to their books would never be finished. This is how programmers deal with reality: by cleaning it up.

What I said above about Requests being like Hemingway isn’t entirely accurate: his clean prose belies a messy reality that is always present; meaning is often unstated. Likewise, when Austen begins, "It is a truth universally acknowledged, that a single man in possession of a good fortune must be in want of a wife," she is not simply stating a universal truth that she has identified. The sentence comes with subtext: the truth is universal in the minds of some people, but not others, an oxymoron, emphasized by the sentence’s lack of an active subject. There is also commentary on the relative power of men and women in her time. Here is a richness that demands and rewards repeated reading, offering interpretations as varied as the number of people who read it.

When I write a story, I thrive on subtext, on creating shades of meaning and multiple readings. Ambiguity and elision and irony are my techniques for dealing with the messiness of reality. But when I type:

I aim to banish ambiguity. I name my variables clearly, according to what data they hold. I structure the code to indicate the flow of execution. Requests helps me here: the last two lines that create a new session correspond to how I intuitively think of creating a session. The equivalent in urllib2 would be messier and more verbose.

I want future readers–myself and others–to understand exactly what that code is doing. Future readers are as sure a thing with code as with literature. ⁵ I’ll be re-reading my code in six months when I add a new feature in another part of the program and it suddenly breaks something here and I have to figure out how they are connected.

What it’s like to work with a bad API

You get annoyed. You say of whoever created it, What were they thinking? You feel bogged down. Nothing comes easy. You have to keep taking breaks. You feel forgetful. You wonder if it’s your fault, if you’re missing something everyone else sees. You get a headache. You curse. You may bite or click your nails, or maniacally tap your foot. You want to be doing something else.

Grok

Programmers put so much value on understanding something with intimate and exhaustive knowledge that traditional programmer slang has its own word for it: grok ⁶. It’s from the language of the Martians in Robert Heinlein’s Stranger in a Strange Land, in which it means literally "to drink" and metaphorically "to be one with."

In a novel or poem, you’ve been led to understanding and impression by a succession of images and literary devices and experiences and revelations. In programming, you are led to understanding by encountering problems, ways of thinking about those problems, organizing them, and finally grokking them well enough to devise a solution.

Reitz groks Python and the internet well, better than I ever will, as Austen grokked relationships and power dynamics better than I ever will. Through exposure to their works, I benefit from their experience of the world. In my own work of programming and writing, I strive to match the understanding that they achieved.

Theory of Mind

Writers and programmers inhabit other minds.

Writers inhabit the minds of their characters and of readers of their work.

Programmers inhabit the minds of users. In Reitz’s case, these are other programmers (as opposed to, say, the programmers of Google Chrome, which is used by non-programmers). Programmers might also be said to inhabit mind of the computer itself.

This habitation of minds outside my own is part of what draws me to both these pursuits. It is a challenge. It broadens my world. Thinking of others, as others think, anticipating their needs and wants and questions, helps me transcend myself and gain perspective. It’s invigorating!

What it’s like to work with a great API

It’s more than if Lori Moore or John Ashbery published a notebook of exercises and prompts; it’s as if they published part of their brain, so that you too can run your thoughts through it, and have them upgraded. As you figure out how to do what you set out to do, you realize other things that would also be cool to do, and you find that the API has ways to do them, too! You think the way someone else thought, and understand their thinking on a deep level. You have a sense that we are all in this together. We’re not so different.

Identification

Identification is addictive. I seek it out, and I have an urge to spark it in others. The difficulties of navigating complexity while avoid communication pitfalls, of harnessing expressive power, and of inhabiting other minds make it difficult to find and create. If I can ever achieve it in my own work–whether writing or programming–it will be through studying–grokking–works like Pride and Prejudice, and Requests.

Footnotes

(Update: here are the slides from my talk.)

In other news, I’ve redone this site using Octopress with the oct2-orange theme. Sweet!

SpanishDict recently deployed a new text-to-speech service powered by Node. This service can generate audio files on the fly for arbitrary Spanish and English texts with rapid response times. The presentation will walk through the design, development, testing, monitoring, and deployment process for the new application. We will cover topics like how to structure an Express app, testing and debugging, learning to think in streams and pipes, writing a Chef cookbook to deploy to AWS, and monitoring the application for high performance. The lead engineer on the project, William Bert, will also talk about his experiences transitioning from a Python background to Node and some of the key insights he had about writing in Node while developing the application.

Update: here are the slides from the talk.

gensim has an excellent tutorial, and this does not replace reading and understanding it. Nonetheless, this may be helpful for those interested in doing some quick experimentation and getting their hands dirty fast. It takes you from training corpus to index and queries in about 100 lines of code, much of which is documentation.

Note that this code will not work out of the box. To train the models, you need to provide your own background corpus (a collection of documents, where a document can range from one sentence up to multiple pages of text). Choosing a good corpus is an art; generally, you want tens of thousands of documents that are representative of your problem domain. Like the gensim tutorial, this code also shows how to build a corpus from Wikipedia for experimentation, though note that doing so require a lot of computing time. You could potentially save hours by installing accelerated BLAS on your system.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104

import logging, sys, pprint

logging.basicConfig(stream=sys.stdout, level=logging.INFO)

### Generating a training/background corpus from your own source of documents
from gensim.corpora import TextCorpus, MmCorpus, Dictionary

# gensim docs: "Provide a filename or a file-like object as input and TextCorpus will be initialized with a
# dictionary in `self.dictionary`and will support the `iter` corpus method. For other kinds of corpora, you only
# need to override `get_texts` and provide your own implementation."
background_corpus = TextCorpus(input=YOUR_CORPUS)

# Important -- save the dictionary generated by the corpus, or future operations will not be able to map results
# back to original words.
background_corpus.dictionary.save(
    "my_dict.dict")

MmCorpus.serialize("background_corpus.mm",
    background_corpus)  #  Uses numpy to persist wiki corpus in Matrix Market format. File will be several GBs.

### Generating a large training/background corpus using Wikipedia
from gensim.corpora import WikiCorpus, wikicorpus

articles = "enwiki-latest-pages-articles.xml.bz2"  # available from http://en.wikipedia.org/wiki/Wikipedia:Database_download

# This will take many hours! Output is Wikipedia in bucket-of-words (BOW) sparse matrix.
wiki_corpus = WikiCorpus(articles)
wiki_corpus.dictionary.save("wiki_dict.dict")

MmCorpus.serialize("wiki_corpus.mm", wiki_corpus)  #  File will be several GBs.

### Working with persisted corpus and dictionary
bow_corpus = MmCorpus("wiki_corpus.mm")  # Revive a corpus

dictionary = Dictionary.load("wiki_dict.dict")  # Load a dictionary

### Transformations among vector spaces
from gensim.models import LsiModel, LogEntropyModel

logent_transformation = LogEntropyModel(wiki_corpus,
    id2word=dictionary)  # Log Entropy weights frequencies of all document features in the corpus

tokenize_func = wikicorpus.tokenize  # The tokenizer used to create the Wikipedia corpus
document = "Some text to be transformed."
# First, tokenize document using the same tokenization as was used on the background corpus, and then convert it to
# BOW representation using the dictionary created when generating the background corpus.
bow_document = dictionary.doc2bow(tokenize_func(
    document))
# converts a single document to log entropy representation. document must be in the same vector space as corpus.
logent_document = logent_transformation[[
    bow_document]]

# Transform arbitrary documents by getting them into the same BOW vector space created by your training corpus
documents = ["Some iterable", "containing multiple", "documents", "..."]
bow_documents = (dictionary.doc2bow(
    tokenize_func(document)) for document in documents)  # use a generator expression because...
logent_documents = logent_transformation[
                   bow_documents]  # ...transformation is done during iteration of documents using generators, so this uses constant memory

### Chained transformations
# This builds a new corpus from iterating over documents of bow_corpus as transformed to log entropy representation.
# Will also take many hours if bow_corpus is the Wikipedia corpus created above.
logent_corpus = MmCorpus(corpus=logent_transformation[bow_corpus])

# Creates LSI transformation model from log entropy corpus representation. Takes several hours with Wikipedia corpus.
lsi_transformation = LsiModel(corpus=logent_corpus, id2word=dictionary,
    num_features=400)

# Alternative way of performing same operation as above, but with implicit chaining
# lsi_transformation = LsiModel(corpus=logent_transformation[bow_corpus], id2word=dictionary,
#    num_features=400)

# Can persist transformation models, too.
logent_transformation.save("logent.model")
lsi_transformation.save("lsi.model")

### Similarities (the best part)
from gensim.similarities import Similarity

# This index corpus consists of what you want to compare future queries against
index_documents = ["A bear walked in the dark forest.",
             "Tall trees have many more leaves than short bushes.",
             "A starship may someday travel across vast reaches of space to other stars.",
             "Difference is the concept of how two or more entities are not the same."]
# A corpus can be anything, as long as iterating over it produces a representation of the corpus documents as vectors.
corpus = (dictionary.doc2bow(tokenize_func(document)) for document in index_documents)

index = Similarity(corpus=lsi_transformation[logent_transformation[corpus]], num_features=400, output_prefix="shard")

print "Index corpus:"
pprint.pprint(documents)

print "Similarities of index corpus documents to one another:"
pprint.pprint([s for s in index])

query = "In the face of ambiguity, refuse the temptation to guess."
sims_to_query = index[lsi_transformation[logent_transformation[dictionary.doc2bow(tokenize_func(query))]]]
print "Similarities of index corpus documents to '%s'" % query
pprint.pprint(sims_to_query)

best_score = max(sims_to_query)
index = sims_to_query.tolist().index(best_score)
most_similar_doc = documents[index]
print "The document most similar to the query is '%s' with a score of %.2f." % (most_similar_doc, best_score)

The presentation slides are available here. I also wrote some quick gensim example code that walks through creating a corpus, generating and transforming models, and using models to do semantic similarity. The code and slides are both also available on my github account.

Finally, I also developed a demo app to visualize semantic similarity queries. It’s a Flask web app, with gensim generating data on the backend that is clustered by scipy and scikit-learn and visualized by d3.js as agglomerative and hierarchical clusters as well as a simple table and dendrogram. To make it all work in realtime, I used threading and hookbox. I call it Visularity, and it’s available on github. You need to provide your own model and dictionary data to use–check out my presentation and visit radimrehurek.com/gensim/ to learn how. Comments and feedback welcome!

WB: Why did you create gensim?

RR: Consulting gig for a digital library project (Czech Digital Mathematics Library, dml.cz), some 3 years ago. It started off as a few loosely connected Python scripts to support the “show similar articles” functionality. We wanted to use some of the statistical methods, like latent semantic analysis. Originally, gensim only contained wrappers around existing Fortran libraries for SVD, like Propack and Svdpack.

But there were issues with that, and it scaled badly (all documents in RAM), so I started looking for more scalable, online algorithms. Running these popular methods shouldn’t be so hard, I thought!

In the end, I developed new algorithms for these methods for gensim. The theoretical part of this research later turned into a part of my PhD thesis.

Who is using gensim (as far as you know)–academics, hobbyists, commercial entities, a mixture? Any particularly interesting uses?

Yes, I’ve heard from many academic as well as commercial organizations, both on the mailing list and off. Off the top of my head: ravn.co.uk, roistr.com, sportsauthority.com, larkc.eu; TU of Denmark, U of Stuttgart, Masaryk U, U of Ghent, some people used it in the Yahoo! KD cup competition… But what they all did with gensim, or whether they still use it, I don’t know. The gensim license (LGPL) is pretty liberal in that respect.

Unfortunately, all this use rarely translates into any feedback or contributions. I guess I’m just not very good at the bring-new-developers-and-grow-open-source stuff :(

Roughly how much of the current codebase was written by you, and how much by contributors?

Almost everything by me, but I am very grateful for bug fixes and patches. I try to put every contribution from other people into the changelog: https://github.com/piskvorky/gensim/blob/develop/CHANGELOG.txt . I made some wiki pages to make contributing easier: https://github.com/piskvorky/gensim/wiki . I also try to answer general questions on the mailing list.

What are your favorite features, or parts of the code that you’re most proud of?

I don’t have emotional attachments to parts of the code – if it’s bad, it needs to go. I guess the most proven parts are the ones that had been around for the longest – LSA etc. Things that were contributed recently by other people, like the new HDP (hierarchical dirichlet process) code, or the gensim.parsing subpackage, are the most rough around the edges.

The best feature is the memory independence for sure. Most implementations of the statistical semantics methods assume the training data resides in RAM, which limits their use to small/medium corpora. Also they work in batch mode, needing a full re-train when new training data arrives. The LSA/LDA algos are online though (can be updated with new data, incrementally).

What’s your background? Academic, software engineering, both?

I finished my PhD, but I feel more like a software engineer than a pure researcher. Even during my academic years, I was working in IT commerce. I wouldn’t like to stay in academia professionally.

What are you working on next for gensim? What about outside of gensim?

Small things like adding the “hashing trick” etc: https://github.com/piskvorky/gensim/issues . Basically things that gensim users have been asking for. Some issues keep coming back on the mailing list, and while not technically bugs, they hint at minor redesigns and improvements.

One big thing that is missing is a basic visual style for gensim. I have no clue how to do that and it’s really pathetic gensim doesn’t even have a logo yet!

Outside of gensim, I am busy doing consulting (scaling up text processing: fulltext search, semantic search, ad targeting etc – backend stuff). I’m planning to do a startup that offers semantic search and similarity as a service. A kind of easy-to-use black box tool, something like searchify or myrrix. But it’s hard to find good people to work with… and hard to give up/interrupt a well-paying career :) I applied for YC last month, alone, but they turned me down.

We coded up our prototype, but when a user selects a Task, Ext JS throws this error: Uncaught TypeError: Cannot read property 'internalId' of undefined. Hmm. Everything seems to be working. Our models are loading the correct data. No obvious bugs. A lot of inspecting and googling and reading documentation later, I discover this thread. The key quote:

It doesn’t matter if the [model] ids are unique within the JSON [or any data]. It must be unique within the tree.

If you add the first json to the tree with for example the id 4_1 and you add the second json with again a node 4_1 then you have two nodes with the same id.

In other words, TreeStore doesn’t distinguish the types of roots and their children (or children’s children, etc). To TreeStore, they are all nodes, and ids must be unique across all nodes. If you have an instance of a Task model with id=1 and it has a foreign key to a Goal that also has id=1, TreeStore has a problem with that. Apparently it doesn’t introspect the objects enough to see that, say, one is a Task and its children are Goals, despite the Task model having a hasMany field that defines its relation to the Goal model. That seems counterintuitive to me, maybe even misleading. Perhaps that’s why we’re not the only ones who’ve had this problem.

My quick fix was to write a stringify_id() function to wrap ids passed to the TreeStore with a prefix unique to each type, so the id of Task id=1 becomes “task-1”. destringify_id() unwraps the ids that come back through the proxy.

TreeStore’s docs do not mention this restriction, as far as I can tell. Maybe if you purchase Ext JS, you get better docs, I’m not sure. We may be doing just that, so I could have a chance to find out. One of the complaints you sometimes hear about open source is that the docs aren’t that great, so I’m curious to see how a for-profit company’s docs stack up against the documentation culture of the communities I’m most familiar with (Python and Django), which tend to be pretty solid.

Here’s mine:

Steve Souryal is a group of 15 small islets and rocks in the central equatorial Atlantic Ocean. He lies in the Intertropical Convergence Zone, a region of severe storms. Steve exposes serpentinized abyssal mantle peridotite and kaersutite-bearing ultramafic mylonite on the top of the second-largest megamullion in the world (after the Parece Vela megamullion under Okinotoshima in the Pacific). He is the only location in the Atlantic Ocean where the abyssal mantle is exposed above sea level! In 1986, Steve was designated an environmentally protected area, and since 1998, the Danish Navy has maintained a permanently manned research facility in him. His main economic activity is tuna fishing, and we are incredibly lucky to have him with us tonight.

Apologies to Wikipedia. But somehow, it just feels right.

Some mathematically intense operations that use Numpy/Scipy can run faster with accelerated basic linear algebra subroutine (BLAS) libraries installed on your system (e.g., gensim’s corpus processing).

To see what BLAS libraries you are using, do:

1

python -c 'import numpy; numpy.show_config()'

If none of them are installed, you probably want to install one or more. ATLAS is always a good bet, since it’s portable and self-optimizing. There are others out there targeted at particular CPU architectures.

Unfortunately, the Scipy docs are out of date regarding installing accelerated BLAS libraries on Ubuntu. The instructions I have written below work for Ubuntu 10.04, the current LTS (long-term support) version, and though I haven’t tried to run them on a more recent version, it’s possible they work with those as well.

Prereqs

On Ubuntu 10.04, and possibly other versions, you need liblapack-dev and gfortran (yes, fortran):

1
2

$ sudo apt-get install liblapack-dev
$ sudo apt-get install gfortran

Instructions

Install the accelerated linear algebra libraries (ATLAS/LAPACK) in your virtualenv on Ubutu:

1
2
3
4
5
6
7

#!/bin/bash
workon [envname]
pip uninstall numpy ## only if numpy is already installed
pip uninstall scipy ## only if scipy is already installed
export LAPACK=/usr/lib/liblapack.so
export ATLAS=/usr/lib/libatlas.so
export BLAS=/usr/lib/libblas.so

Now you can install numpy and scipy into the same virtualenv and be confident they will perform operations using the accelerated BLAS routines:

1
2

$ pip install numpy
$ pip install scipy

So much for traditions; recent novelties include starting a new job, about which more another time, but basically, I love it; and getting a lesson plan published in Don’t Forget to Write, the second volume of lesson plans from 826. The lesson plan, “Busted,” aims to make storytellers out of middle schoolers by having them write about a time they got caught doing something they shouldn’t have been doing–a theme first cooked up by the folks who led the Get Used to the Seats book project. I wrote about leading the workshops that ultimately became the “Busted” lesson plan more than a year ago–right around the previous Friendsgiving. Hard to believe it’s been that long!

When I spoke with a female intern this summer, she recounted how, in 2006, the GNOME Project, a free and open source software project, received almost 200 Google Summer of Code applicants. All of them were male. When GNOME advertised an identical program for women, emphasizing opportunities for learning and mentorship instead of tough competition, they received more than 100 highly qualified female applicants for the three spots they were able to fund. What amazed me even more was when she suggested that our own company slogan — “We Help the World’s Best Developers Make Better Software” — might alienate prospective female candidates. That had never occurred to me. But according to our intern, in the world of computer science, “when you hear the phrase ‘the world’s best developers,’ you see a guy.”

From When computer programming was ‘women’s work’.

Sorting Through the Choices

There are a number of reusable Django apps out there to help with registration/login from social media sites. I found this Review of 4 Django Social Auth apps very helpful in sorting out the options. After reading it, I was left to choose between django-social-auth (I originally linked to the wrong app here, but this link is correct) and django-allauth. In the end, I went with django-social-auth (not to be confused with django-socialauth) because a friend had recommended it and because I’d already installed it before I read this article. However, the article’s conclusion that django-allauth is best out of the box also seems valid.

Installation

The instructions in django-social-auth’s docs are helpful in walking you through available settings and options.

I also found the included example app useful. To use this app, I cloned django-social-auth’s git repo, created a virtualenv called django-social-auth, ran pip install -r requirements.txt inside this virtualenv to install all the required apps, ran manage.py syncdb, and finally ran manage.py runserver. Voila, example app is up and running at 127.0.0.1, showing a simple screen with options to login through about a dozen different different services.

API Keys

The first service I tested was Twitter. I use it more than any others, and I already had the API keys for it. I threw my API key and secret key into the example local_settings.py file provided with django-social-auth and tried to log in via the example app. Boom: 401 Unauthorized. I double-checked all my settings and installation and whatnot. Seemed fine.

I turned my attention to the API keys. The ones I had were generated for Readsr, i.e., I entered readsrs.com as the domain when I generated them at dev.twitter.com. But now I was running on localhost, 127.0.0.1, so I suspected the readsrs.com keys wouldn’t be valid. I wasn’t sure whether Twitter would hand over a new consumer key for 127.0.0.1, or baulk at the request. (It seemed like it should do so, but I hadn’t seen any instructions anywhere that said to get a key for your development machine.) Turns out Twitter will happily give you a key for 127.0.0.1. Once I plugged the new keys in, I was able to log in with my Twitter credentials, and just as it should, django-social-auth automatically created an auth.user for this account.

Integrating with Readsr

I followed the instructions again to config my own app, Readsr. To add a login option using Twitter credentials, I put a link to the reversed view that begins the django-social-auth login process for twitter, i.e., {% url socialauth_begin "twitter" %}, to my login template. And it worked.

I still need to fix a few oddities. For example, Twitter returns my first and last names together in first_name (or else django-social-auth is concatenating them into that column), and doesn’t supply any email address. But the basic functionality is there, and was relatively easy to achieve.

Postscript

The author of the article I linked above had an error using OpenID when using django-social-auth, which is why he preferred django-authall. He filed a bug for the error he got, and I notice that it was closed 15 hours ago (though if you read the comments, it seems it was actually fixed back in mid-July). Good timing.

For Readsr, I need to track events that recur on a particular day of the week (e.g., first Sunday of the month, third Friday of the month). I created a DayOfWeek model to store any particular event’s day of the week. It contains a method next_day_of_week() to return a datetime.date object set to the next occurrence of whatever weekday a given event instance is set to (this helps with figuring out when the next occurrence of an event is).

It’s easier to show through an example. On Sunday 7/3/2011:

• For an object with DayOfWeek set to Sunday, next_day_of_week() would return 7/3/2011 (current day).
• For DayOfWeek set to Monday, it would return 7/4/2011 (first subsequent Monday).
• For DayOfWeek set to Saturday, it would return 7/9/2011 (first subsequent Saturday).

Sounds simple enough. It seemed like this would be a good place to do my first unit tests.

Unit Testing

To do unit testing, the typical method is to first write test cases and then write code. In this case, I’d already written my code, so I went back and wrote test cases, trying to forget how my code worked.

To write test cases, you have to detail requirements for each method you want to test: input and expected (correct) output. The list of examples for next_day_of_week() I wrote above works for this purpose. But there’s a catch: next_day_of_week() calculates the next day of the week relative to the current date, by calling datetime.date.today(). So if I write expected output for 7/3/2011, it will no longer be the correct output on 7/4/2011 or any following day. I needed a way to make datetime.date.today() always spit out my input date when I run tests, yet still continue to function normally outside of testing. Enter mocking.

Mocking

The solution was to mock out the method–to replace the real datetime.date.today() with a fake one that produces the same output no matter what day it is. To accomplish this, I used the powerful Mock library. Specifically, I needed to use the patch decorator. This decorator makes it really easy to replace on particular object within the scope of a particular method.

Before I could patch the today() method, I needed to create my own fake method. It would look like this:

1
2

def faketoday()
    return date(2011, 7, 4)

There’s a problem, though, when I try to patch (or mock out) the method:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

>>> import mock
>>> def faketoday():
...     return datetime.date(2011, 7, 4)
...
>>> @mock.patch("datetime.date.today", faketoday)
... def testfunc():
...     return datetime.date.today()
...
>>> testfunc()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Users/wbert/.virtualenvs/readsr_env/lib/python2.6/site-packages/mock.py", line 561, in patched
    arg = patching.__enter__()
  File "/Users/wbert/.virtualenvs/readsr_env/lib/python2.6/site-packages/mock.py", line 623, in __enter__
    setattr(self.target, self.attribute, new_attr)
TypeError: can't set attributes of built-in/extension type 'datetime.date'
>>>

datetime.date is considered a Python built-in and can’t be modified.

Modifying a Class That Can’t Be Modified

The trick is to write a child class that can be modified, and thus faked:

1
2
3
4

class FakeDate(date):
  "A fake replacement for date that can be mocked for testing."
  def __new__(cls, *args, **kwargs):
      return date.__new__(date, *args, **kwargs)

All this does is create a class whose constructor returns an instance of its parent’s class, date. Usually, this would be pointless, but it’s useful here because the new class isn’t a built-in and thus can be mocked.

To use it, we simply decorate any test method that calls datetime.date.today() with a patch to replace datetime.date with FakeDate, and we also provide FakeDate a fake today() method that returns only and always the particular we are going to use for testing:

1
2
3
4
5
6
7
8
9
10

class TestDayOfWeek(TestCase):
  """Test the day of the week functions."""

@mock.patch('series.models.date', FakeDate)
def test_valid_my_next_day_of_week_sameday(self):
  from datetime import date
  FakeDate.today = classmethod(lambda cls: date(2011, 7, 3)) # July 3, 2011 is a Sunday
  new_day_of_week = DayOfWeek.objects.create()
  new_day_of_week.day = "SU"
  self.assertEquals(new_day_of_week.my_next_day_of_week(), date(2011, 7, 3))

A couple things to note: the patch only applies within this particular method, so each method to use a patch must be decorated. Also, the real datetime.date is imported in the method so we can use it inside the fake today() method. We could put this fake today() method inside FakeClass, but making it a lambda (anonymous) method assigned inside the test case gives us the flexibility to set a particular date for each test case.

Namespacing

You may be wondering why the patch decorator takes “series.models.date” as the method to replace instead of “datetime.date”. That was how I tried it at first, and I was confused when it didn’t work. It seemed as if the patch hadn’t taken effect.

Well, it hadn’t. That’s because within the module being tested (models.py in the series app, or series.models in Python dotted notation), date has been imported like so:

1

from datetime import date

This means that within series.models, date is now available as series.models.date, so that’s the name that needs to be mocked out. For more on namespacing when mocking, checked out Mock’s Where to patch documentation.

Now we can supply out unit tests with any date we want, ensuring that we know what the results should be and can test against them.

References

Learning how to do this stuff, I posted my first question at Stackoverflow (I ended up answering it myself). I also learned about using a fake class from this question. Finally, the Mock documentation was very helpful as well.

I’ve been teaching myself Django by writing a web app that tracks reading series in cities: Readsr. A reading series is a kind of recurring event. It defines a time, location, and recurrence rule such as first Monday of the month. Writing a list view to display upcoming readings was easy, but I also wanted to create a calendar view, similar to what Google Calendar provides. That took a little more work.

Existing Solutions

First I searched for existing Django calendar solutions. I found several. Swingtime and django-agenda seemed very well thought out and comprehensive, but were also perhaps overkill for what I needed. django-gencal doesn’t appear to be maintained and I had trouble understanding the documentation (though you may have better results, as I am slow).

A Way Forward

I found that Python’s calendar module has a built template called HTMLCalendar, which sounded promising. Then I found a couple examples of people inheriting from HTMLCalendar to add data to the calendar display. This sounded right on, so I adapted this code for my reading events.

Problem: Presentation and Content Mixed

I noticed a problem in the code I was adapting. The view was producing HTML. That seemed to violate separation of content and presentation. Shouldn’t the HTML be generated in the template? And since the templating language itself isn’t powerful enough to generate an HTML table from a list of objects, that meant I needed to write my own template tag. Yikes.

Writing My First Template Tag

Django’s documentation made it easy, though. A template tag consists of a few parts. Below is the code; it goes into a subdirectory of the django app called “templatetags.”

The first part follows directly from the Django docs: get a register object.

Then I define a function to parse the template tag arguments and return the node (the HTML code from which the page is eventually build). The template syntax is defined here.

Then I define the node itself, which is made thread-safe by storing and retrieving the variables passed through the template tag from a context (again, this is straight from the Django docs).

Then I inherit from HTMLCalendar and redefine the format methods to add the particular reading event data. You could adapt this class to any kind of event that has an associated date/time by changing the groupby lambda function to use whatever field your event object uses to store its date and time (my reading object simply calls it “date_and_time”).

Finally, I register this template tag so it is available to templates.

Here’s the code.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92

from calendar import HTMLCalendar
from django import template
from datetime import date
from itertools import groupby

from django.utils.html import conditional_escape as esc

register = template.Library()

def do_reading_calendar(parser, token):
  """
 The template tag's syntax is { % reading_calendar year month reading_list %}
 """

  try:
      tag_name, year, month, reading_list = token.split_contents()
  except ValueError:
      raise template.TemplateSyntaxError, "%r tag requires three arguments" % token.contents.split()[0]
  return ReadingCalendarNode(year, month, reading_list)
  

class ReadingCalendarNode(template.Node):
  """
 Process a particular node in the template. Fail silently.
 """
  
  def __init__(self, year, month, reading_list):
      try:
          self.year = template.Variable(year)
          self.month = template.Variable(month)
          self.reading_list = template.Variable(reading_list)
      except ValueError:
          raise template.TemplateSyntaxError
      
  def render(self, context):
      try:
          # Get the variables from the context so the method is thread-safe.
          my_reading_list = self.reading_list.resolve(context)
          my_year = self.year.resolve(context)
          my_month = self.month.resolve(context)
          cal = ReadingCalendar(my_reading_list)
          return cal.formatmonth(int(my_year), int(my_month))
      except ValueError:
          return           
      except template.VariableDoesNotExist:
          return


class ReadingCalendar(HTMLCalendar):
  """
 Overload Python's calendar.HTMLCalendar to add the appropriate reading events to
 each day's table cell.
 """
  
  def __init__(self, readings):
      super(ReadingCalendar, self).__init__()
      self.readings = self.group_by_day(readings)

  def formatday(self, day, weekday):
      if day != 0:
          cssclass = self.cssclasses[weekday]
          if date.today() == date(self.year, self.month, day):
              cssclass += ' today'
          if day in self.readings:
              cssclass += ' filled'
              body = ['<ul>']
              for reading in self.readings[day]:
                  body.append('<li>')
                  body.append('<a href="%s">' % reading.get_absolute_url())
                  body.append(esc(reading.series.primary_name))
                  body.append('</a></li>')
              body.append('</ul>')
              return self.day_cell(cssclass, '<span class="dayNumber">%d</span> %s' % (day, ''.join(body)))
          return self.day_cell(cssclass, '<span class="dayNumberNoReadings">%d</span>' % (day))
      return self.day_cell('noday', '&nbsp;')

  def formatmonth(self, year, month):
      self.year, self.month = year, month
      return super(ReadingCalendar, self).formatmonth(year, month)

  def group_by_day(self, readings):
      field = lambda reading: reading.date_and_time.day
      return dict(
          [(day, list(items)) for day, items in groupby(readings, field)]
      )

  def day_cell(self, cssclass, body):
      return '<td class="%s">%s</td>' % (cssclass, body)

# Register the template tag so it is available to templates
register.tag("reading_calendar", do_reading_calendar)

Then, here’s the view (and a couple helper functions) that gets called with the arguments from the URL, including the year, month, and series to display events for.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

def named_month(month_number):

  """
 Return the name of the month, given the number.
 """
  return date(1900, month_number, 1).strftime("%B")
  
def this_month(request):
  """
 Show calendar of readings this month.
 """
  today = datetime.now()
  return calendar(request, today.year, today.month)
  
  
def calendar(request, year, month, series_id=None):
  """
 Show calendar of readings for a given month of a given year.
 ``series_id``
 The reading series to show. None shows all reading series.
 """
  
  my_year = int(year)
  my_month = int(month)
  my_calendar_from_month = datetime(my_year, my_month, 1)
  my_calendar_to_month = datetime(my_year, my_month, monthrange(my_year, my_month)[1])

  my_reading_events = Reading.objects.filter(date_and_time__gte=my_calendar_from_month).filter(date_and_time__lte=my_calendar_to_month)
  if series_id:
      my_reading_events = my_reading_events.filter(series=series_id)

  # Calculate values for the calendar controls. 1-indexed (Jan = 1)
  my_previous_year = my_year
  my_previous_month = my_month - 1
  if my_previous_month == 0:
      my_previous_year = my_year - 1
      my_previous_month = 12
  my_next_year = my_year
  my_next_month = my_month + 1
  if my_next_month == 13:
      my_next_year = my_year + 1
      my_next_month = 1
  my_year_after_this = my_year + 1
  my_year_before_this = my_year - 1
  return render_to_response("cal_template.html", { 'readings_list': my_reading_events,
                                                      'month': my_month,
                                                      'month_name': named_month(my_month),
                                                      'year': my_year,
                                                      'previous_month': my_previous_month,
                                                      'previous_month_name': named_month(my_previous_month),
                                                      'previous_year': my_previous_year,
                                                      'next_month': my_next_month,
                                                      'next_month_name': named_month(my_next_month),
                                                      'next_year': my_next_year,
                                                      'year_before_this': my_year_before_this,
                                                      'year_after_this': my_year_after_this,
  }, context_instance=RequestContext(request))

And finally, here’s the template where we load the template tag and employ it, passing the year, month, and list from the view (you would also want to write some control elements that use previous_year, previous_month, etc. to allow the user to change what the calendar displays, but because I want to wrap this up I’ll forgo writing that out):

1
2
3
4
5

{% load reading_tags %}

<div id="calendar">
  {% reading_calendar year month reading_list %}
</div>

Hopefully that makes sense. Enjoy!

You can also see this code (made generic for any kind of event) on django snippets.

The students came to us talented and hard-working, and we just helped them along with workshops on poetic forms, poetic sound, and metaphor (this is the one I’m awkwardly teaching in the photo above), and by giving plenty of exercises and time to generate new material. We even had a guest visit from the amazing Carolyn Forche.

All this culminates today in a workshop at the White House, hosted by the Obamas, where the students will work with the likes of Billy Collins, Common, Rita Dove, Aimee Mann, Jill Scott and several other amazing poets to hone their craft even further. Very, very cool. Congrats to 826DC and especially to the students!

Craig Nova is the resident author for this session at DBI, and I’m looking forward to reading his novel The Good Son, starting on the commute home this afternoon. It’ll be good to dive back into reading and writing, which have taken a back seat so far this season. I did have the honor of getting tapped for a Larry Neal award, 2nd place for adult fiction  (the non-euphemistic kind), for a story I began back in Arizona but kept revisiting up until last fall. That just made me want to sit down and get back to writing, drafting, and revising even more! I don’t have any illusions that late spring and summer will be slow, but few things suck up time more than moving, and with a good solid week to get jump-started, I’m feeling optimistic.

The issue includes work from many better writers than me, including a poem and two translations from my friend Naomi Ayala. It was Naomi who suggested I submit to Anomalous. In this way and many others, she’s been a great friend and supporter of my writing, and I hope someday to be able to pay her back. You can catch another poem of hers when Call + Response: Textures opens on April 16. (More on that tomorrow.)

Anomalous Press #1 is available for download as a PDF, but only if you pay with a tweet. Neat concept. Attention as currency. And from a technical standpoint, I’m interested in how they integrated with Twitter using OAuth—I want to do a similar thing with the reading series web app (Readsr/Readthing/whatever) I’m working on: tweet whenever a reading series is updated. So I’ll be looking into this soon.

First, though, I plan to deploy an alpha version of the site this weekend. I’ve been reading up on virtualenv, pip, fabric, and git, trying to figure out how to deploy the right way–with automated version control and dependency isolation. This part, the admin stuff, is as complicated as any coding problem I’ve faced. Looking forward to tackling it in a marathon session on Saturday!