gensim has an excellent tutorial, and this does not replace reading and understanding it. Nonetheless, this may be helpful for those interested in doing some quick experimentation and getting their hands dirty fast. It takes you from training corpus to index and queries in about 100 lines of code, much of which is documentation.

Note that this code will not work out of the box. To train the models, you need to provide your own background corpus (a collection of documents, where a document can range from one sentence up to multiple pages of text). Choosing a good corpus is an art; generally, you want tens of thousands of documents that are representative of your problem domain. Like the gensim tutorial, this code also shows how to build a corpus from Wikipedia for experimentation, though note that doing so require a lot of computing time. You could potentially save hours by installing accelerated BLAS on your system.

import logging, sys, pprint

logging.basicConfig(stream=sys.stdout, level=logging.INFO)

### Generating a training/background corpus from your own source of documents
from gensim.corpora import TextCorpus, MmCorpus, Dictionary

# gensim docs: "Provide a filename or a file-like object as input and TextCorpus will be initialized with a
# dictionary in `self.dictionary`and will support the `iter` corpus method. For other kinds of corpora, you only
# need to override `get_texts` and provide your own implementation."
background_corpus = TextCorpus(input=YOUR_CORPUS)

# Important -- save the dictionary generated by the corpus, or future operations will not be able to map results
# back to original words.
background_corpus.dictionary.save(
    "my_dict.dict")

MmCorpus.serialize("background_corpus.mm",
    background_corpus)  #  Uses numpy to persist wiki corpus in Matrix Market format. File will be several GBs.

### Generating a large training/background corpus using Wikipedia
from gensim.corpora import WikiCorpus, wikicorpus

articles = "enwiki-latest-pages-articles.xml.bz2"  # available from http://en.wikipedia.org/wiki/Wikipedia:Database_download

# This will take many hours! Output is Wikipedia in bucket-of-words (BOW) sparse matrix.
wiki_corpus = WikiCorpus(articles)
wiki_corpus.dictionary.save("wiki_dict.dict")

MmCorpus.serialize("wiki_corpus.mm", wiki_corpus)  #  File will be several GBs.

### Working with persisted corpus and dictionary
bow_corpus = MmCorpus("wiki_corpus.mm")  # Revive a corpus

dictionary = Dictionary.load("wiki_dict.dict")  # Load a dictionary

### Transformations among vector spaces
from gensim.models import LsiModel, LogEntropyModel

logent_transformation = LogEntropyModel(wiki_corpus,
    id2word=dictionary)  # Log Entropy weights frequencies of all document features in the corpus

tokenize_func = wikicorpus.tokenize  # The tokenizer used to create the Wikipedia corpus
document = "Some text to be transformed."
# First, tokenize document using the same tokenization as was used on the background corpus, and then convert it to
# BOW representation using the dictionary created when generating the background corpus.
bow_document = dictionary.doc2bow(tokenize_func(
    document))
# converts a single document to log entropy representation. document must be in the same vector space as corpus.
logent_document = logent_transformation[[
    bow_document]]

# Transform arbitrary documents by getting them into the same BOW vector space created by your training corpus
documents = ["Some iterable", "containing multiple", "documents", "..."]
bow_documents = (dictionary.doc2bow(
    tokenize_func(document)) for document in documents)  # use a generator expression because...
logent_documents = logent_transformation[
                   bow_documents]  # ...transformation is done during iteration of documents using generators, so this uses constant memory

### Chained transformations
# This builds a new corpus from iterating over documents of bow_corpus as transformed to log entropy representation.
# Will also take many hours if bow_corpus is the Wikipedia corpus created above.
logent_corpus = MmCorpus(corpus=logent_transformation[bow_corpus],
    id2word=dictionary)

# Creates LSI transformation model from log entropy corpus representation. Takes several hours with Wikipedia corpus.
lsi_transformation = LsiModel(corpus=logent_corpus, id2word=dictionary,
    num_features=400)

# Alternative way of performing same operation as above, but with implicit chaining
# lsi_transformation = LsiModel(corpus=logent_transformation[bow_corpus], id2word=dictionary,
#    num_features=400)

# Can persist transformation models, too.
logent_transformation.save("logent.model")
lsi_transformation.save("lsi.model")

### Similarities (the best part)
from gensim.similarities import Similarity

# This index corpus consists of what you want to compare future queries against
index_documents = ["A bear walked in the dark forest.",
             "Tall trees have many more leaves than short bushes.",
             "A starship may someday travel across vast reaches of space to other stars.",
             "Difference is the concept of how two or more entities are not the same."]
# A corpus can be anything, as long as iterating over it produces a representation of the corpus documents as vectors.
corpus = (dictionary.doc2bow(tokenize_func(document)) for document in index_documents)

index = Similarity(corpus=lsi_transformation[logent_transformation[corpus]], num_features=400, output_prefix="shard")

print "Index corpus:"
pprint.pprint(documents)

print "Similarities of index corpus documents to one another:"
pprint.pprint([s for s in index])

query = "In the face of ambiguity, refuse the temptation to guess."
sims_to_query = index[lsi_transformation[logent_transformation[dictionary.doc2bow(tokenize_func(query))]]]
print "Similarities of index corpus documents to '%s'" % query
pprint.pprint(sims_to_query)

best_score = max(sims_to_query)
index = sims_to_query.tolist().index(best_score)
most_similar_doc = documents[index]
print "The document most similar to the query is '%s' with a score of %.2f." % (most_similar_doc, best_score)

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105

import logging, sys, pprint

logging.basicConfig(stream=sys.stdout, level=logging.INFO)

### Generating a training/background corpus from your own source of documents
from gensim.corpora import TextCorpus, MmCorpus, Dictionary

# gensim docs: "Provide a filename or a file-like object as input and TextCorpus will be initialized with a
# dictionary in `self.dictionary`and will support the `iter` corpus method. For other kinds of corpora, you only
# need to override `get_texts` and provide your own implementation."
background_corpus = TextCorpus(input=YOUR_CORPUS)

# Important -- save the dictionary generated by the corpus, or future operations will not be able to map results
# back to original words.
background_corpus.dictionary.save(
    "my_dict.dict")

MmCorpus.serialize("background_corpus.mm",
    background_corpus)  #  Uses numpy to persist wiki corpus in Matrix Market format. File will be several GBs.

### Generating a large training/background corpus using Wikipedia
from gensim.corpora import WikiCorpus, wikicorpus

articles = "enwiki-latest-pages-articles.xml.bz2"  # available from http://en.wikipedia.org/wiki/Wikipedia:Database_download

# This will take many hours! Output is Wikipedia in bucket-of-words (BOW) sparse matrix.
wiki_corpus = WikiCorpus(articles)
wiki_corpus.dictionary.save("wiki_dict.dict")

MmCorpus.serialize("wiki_corpus.mm", wiki_corpus)  #  File will be several GBs.

### Working with persisted corpus and dictionary
bow_corpus = MmCorpus("wiki_corpus.mm")  # Revive a corpus

dictionary = Dictionary.load("wiki_dict.dict")  # Load a dictionary

### Transformations among vector spaces
from gensim.models import LsiModel, LogEntropyModel

logent_transformation = LogEntropyModel(wiki_corpus,
    id2word=dictionary)  # Log Entropy weights frequencies of all document features in the corpus

tokenize_func = wikicorpus.tokenize  # The tokenizer used to create the Wikipedia corpus
document = "Some text to be transformed."
# First, tokenize document using the same tokenization as was used on the background corpus, and then convert it to
# BOW representation using the dictionary created when generating the background corpus.
bow_document = dictionary.doc2bow(tokenize_func(
    document))
# converts a single document to log entropy representation. document must be in the same vector space as corpus.
logent_document = logent_transformation[[
    bow_document]]

# Transform arbitrary documents by getting them into the same BOW vector space created by your training corpus
documents = ["Some iterable", "containing multiple", "documents", "..."]
bow_documents = (dictionary.doc2bow(
    tokenize_func(document)) for document in documents)  # use a generator expression because...
logent_documents = logent_transformation[
                   bow_documents]  # ...transformation is done during iteration of documents using generators, so this uses constant memory

### Chained transformations
# This builds a new corpus from iterating over documents of bow_corpus as transformed to log entropy representation.
# Will also take many hours if bow_corpus is the Wikipedia corpus created above.
logent_corpus = MmCorpus(corpus=logent_transformation[bow_corpus],
    id2word=dictionary)

# Creates LSI transformation model from log entropy corpus representation. Takes several hours with Wikipedia corpus.
lsi_transformation = LsiModel(corpus=logent_corpus, id2word=dictionary,
    num_features=400)

# Alternative way of performing same operation as above, but with implicit chaining
# lsi_transformation = LsiModel(corpus=logent_transformation[bow_corpus], id2word=dictionary,
#    num_features=400)

# Can persist transformation models, too.
logent_transformation.save("logent.model")
lsi_transformation.save("lsi.model")

### Similarities (the best part)
from gensim.similarities import Similarity

# This index corpus consists of what you want to compare future queries against
index_documents = ["A bear walked in the dark forest.",
             "Tall trees have many more leaves than short bushes.",
             "A starship may someday travel across vast reaches of space to other stars.",
             "Difference is the concept of how two or more entities are not the same."]
# A corpus can be anything, as long as iterating over it produces a representation of the corpus documents as vectors.
corpus = (dictionary.doc2bow(tokenize_func(document)) for document in index_documents)

index = Similarity(corpus=lsi_transformation[logent_transformation[corpus]], num_features=400, output_prefix="shard")

print "Index corpus:"
pprint.pprint(documents)

print "Similarities of index corpus documents to one another:"
pprint.pprint([s for s in index])

query = "In the face of ambiguity, refuse the temptation to guess."
sims_to_query = index[lsi_transformation[logent_transformation[dictionary.doc2bow(tokenize_func(query))]]]
print "Similarities of index corpus documents to '%s'" % query
pprint.pprint(sims_to_query)

best_score = max(sims_to_query)
index = sims_to_query.tolist().index(best_score)
most_similar_doc = documents[index]
print "The document most similar to the query is '%s' with a score of %.2f." % (most_similar_doc, best_score)

The presentation slides are available here. I also wrote some quick gensim example code that walks through creating a corpus, generating and transforming models, and using models to do semantic similarity. The code and slides are both also available on my github account.

Finally, I also developed a demo app to visualize semantic similarity queries. It’s a Flask web app, with gensim generating data on the backend that is clustered by scipy and scikit-learn and visualized by d3.js as agglomerative and hierarchical clusters as well as a simple table and dendrogram. To make it all work in realtime, I used threading and hookbox. I call it Visularity, and it’s available on github. You need to provide your own model and dictionary data to use–check out my presentation and visit radimrehurek.com/gensim/ to learn how. Comments and feedback welcome!

WB: Why did you create gensim?

RR: Consulting gig for a digital library project (Czech Digital Mathematics Library, dml.cz), some 3 years ago. It started off as a few loosely connected Python scripts to support the “show similar articles” functionality. We wanted to use some of the statistical methods, like latent semantic analysis. Originally, gensim only contained wrappers around existing Fortran libraries for SVD, like Propack and Svdpack.

But there were issues with that, and it scaled badly (all documents in RAM), so I started looking for more scalable, online algorithms. Running these popular methods shouldn’t be so hard, I thought!

In the end, I developed new algorithms for these methods for gensim. The theoretical part of this research later turned into a part of my PhD thesis.

Who is using gensim (as far as you know)–academics, hobbyists, commercial entities, a mixture? Any particularly interesting uses?

Yes, I’ve heard from many academic as well as commercial organizations, both on the mailing list and off. Off the top of my head: ravn.co.uk, roistr.com, sportsauthority.com, larkc.eu; TU of Denmark, U of Stuttgart, Masaryk U, U of Ghent, some people used it in the Yahoo! KD cup competition… But what they all did with gensim, or whether they still use it, I don’t know. The gensim license (LGPL) is pretty liberal in that respect.

Unfortunately, all this use rarely translates into any feedback or contributions. I guess I’m just not very good at the bring-new-developers-and-grow-open-source stuff

Roughly how much of the current codebase was written by you, and how much by contributors?

Almost everything by me, but I am very grateful for bug fixes and patches. I try to put every contribution from other people into the changelog: https://github.com/piskvorky/gensim/blob/develop/CHANGELOG.txt . I made some wiki pages to make contributing easier: https://github.com/piskvorky/gensim/wiki . I also try to answer general questions on the mailing list.

What are your favorite features, or parts of the code that you’re most proud of?

I don’t have emotional attachments to parts of the code — if it’s bad, it needs to go. I guess the most proven parts are the ones that had been around for the longest — LSA etc. Things that were contributed recently by other people, like the new HDP (hierarchical dirichlet process) code, or the gensim.parsing subpackage, are the most rough around the edges.

The best feature is the memory independence for sure. Most implementations of the statistical semantics methods assume the training data resides in RAM, which limits their use to small/medium corpora. Also they work in batch mode, needing a full re-train when new training data arrives. The LSA/LDA algos are online though (can be updated with new data, incrementally).

What’s your background? Academic, software engineering, both?

I finished my PhD, but I feel more like a software engineer than a pure researcher. Even during my academic years, I was working in IT commerce. I wouldn’t like to stay in academia professionally.

What are you working on next for gensim? What about outside of gensim?

Small things like adding the “hashing trick” etc: https://github.com/piskvorky/gensim/issues . Basically things that gensim users have been asking for. Some issues keep coming back on the mailing list, and while not technically bugs, they hint at minor redesigns and improvements.

One big thing that is missing is a basic visual style for gensim. I have no clue how to do that and it’s really pathetic gensim doesn’t even have a logo yet!

Outside of gensim, I am busy doing consulting (scaling up text processing: fulltext search, semantic search, ad targeting etc – backend stuff). I’m planning to do a startup that offers semantic search and similarity as a service. A kind of easy-to-use black box tool, something like searchify or myrrix. But it’s hard to find good people to work with… and hard to give up/interrupt a well-paying career I applied for YC last month, alone, but they turned me down.

We coded up our prototype, but when a user selects a Task, Ext JS throws this error: Uncaught TypeError: Cannot read property 'internalId' of undefined. Hmm. Everything seems to be working. Our models are loading the correct data. No obvious bugs. A lot of inspecting and googling and reading documentation later, I discover this thread. The key quote:

It doesn’t matter if the [model] ids are unique within the JSON [or any data]. It must be unique within the tree. If you add the first json to the tree with for example the id 4_1 and you add the second json with again a node 4_1 then you have two nodes with the same id.

In other words, TreeStore doesn’t distinguish the types of roots and their children (or children’s children, etc). To TreeStore, they are all nodes, and ids must be unique across all nodes. If you have an instance of a Task model with id=1 and it has a foreign key to a Goal that also has id=1, TreeStore has a problem with that. Apparently it doesn’t introspect the objects enough to see that, say, one is a Task and its children are Goals, despite the Task model having a hasMany field that defines its relation to the Goal model. That seems counterintuitive to me, maybe even misleading. Perhaps that’s why we’re not the only ones who’ve had this problem.

My quick fix was to write a stringify_id() function to wrap ids passed to the TreeStore with a prefix unique to each type, so the id of Task id=1 becomes “task-1″. destringify_id() unwraps the ids that come back through the proxy.

TreeStore’s docs do not mention this restriction, as far as I can tell. Maybe if you purchase Ext JS, you get better docs, I’m not sure. We may be doing just that, so I could have a chance to find out. One of the complaints you sometimes hear about open source is that the docs aren’t that great, so I’m curious to see how a for-profit company’s docs stack up against the documentation culture of the communities I’m most familiar with (Python and Django), which tend to be pretty solid.

Here’s mine:

Steve Souryal is a group of 15 small islets and rocks in the central equatorial Atlantic Ocean. He lies in the Intertropical Convergence Zone, a region of severe storms. Steve exposes serpentinized abyssal mantle peridotite and kaersutite-bearing ultramafic mylonite on the top of the second-largest megamullion in the world (after the Parece Vela megamullion under Okinotoshima in the Pacific). He is the only location in the Atlantic Ocean where the abyssal mantle is exposed above sea level! In 1986, Steve was designated an environmentally protected area, and since 1998, the Danish Navy has maintained a permanently manned research facility in him. His main economic activity is tuna fishing, and we are incredibly lucky to have him with us tonight.

Apologies to Wikipedia. But somehow, it just feels right.

Some mathematically intense operations that use Numpy/Scipy can run faster with accelerated basic linear algebra subroutine (BLAS) libraries installed on your system (e.g., gensim’s corpus processing).

To see what BLAS libraries you are using, do:

python -c 'import numpy; numpy.show_config()'

If none of them are installed, you probably want to install one or more. ATLAS is always a good bet, since it’s portable and self-optimizing. There are others out there targeted at particular CPU architectures.

Unfortunately, the Scipy docs are out of date regarding installing accelerated BLAS libraries on Ubuntu. The instructions I have written below work for Ubuntu 10.04, the current LTS (long-term support) version, and though I haven’t tried to run them on a more recent version, it’s possible they work with those as well.

Prereqs

On Ubuntu 10.04, and possibly other versions, you need liblapack-dev and gfortran (yes, fortran):

sudo apt-get install liblapack-dev
sudo apt-get install gfortran

Instructions

Install the accelerated linear algebra libraries (ATLAS/LAPACK) in your virtualenv on Ubutu:

#!/bin/bash
workon [envname]
pip uninstall numpy # only if numpy is already installed
pip uninstall scipy # only if scipy is already installed
export LAPACK=/usr/lib/liblapack.so
export ATLAS=/usr/lib/libatlas.so
export BLAS=/usr/lib/libblas.so

Now you can install numpy and scipy into the same virtualenv and be confident they will perform operations using the accelerated BLAS routines:

pip install numpy
pip install scipy

So much for traditions; recent novelties include starting a new job, about which more another time, but basically, I love it; and getting a lesson plan published in Don’t Forget to Write, the second volume of lesson plans from 826. The lesson plan, “Busted,” aims to make storytellers out of middle schoolers by having them write about a time they got caught doing something they shouldn’t have been doing–a theme first cooked up by the folks who led the Get Used to the Seats book project. I wrote about leading the workshops that ultimately became the “Busted” lesson plan more than a year ago–right around the previous Friendsgiving. Hard to believe it’s been that long!

> When I spoke with a female intern this summer, she recounted how, in 2006, the > GNOME Project, a free and open source software project, received almost 200 Google > Summer of Code applicants. All of them were male. When GNOME advertised an > identical program for women, emphasizing opportunities for learning and mentorship > instead of tough competition, they received more than 100 highly qualified female > applicants for the three spots they were able to fund. What amazed me even more was > when she suggested that our own company slogan — “We Help the World’s Best > Developers Make Better Software” — might alienate prospective female candidates. > That had never occurred to me. But according to our intern, in the world of > computer science, “when you hear the phrase ‘the world’s best developers,’ you see > a guy.”

From When computer programming was ‘women’s work’.

Sorting Through the Choices

There are a number of reusable Django apps out there to help with registration/login from social media sites. I found this Review of 4 Django Social Auth apps very helpful in sorting out the options. After reading it, I was left to choose between django-social-auth (I originally linked to the wrong app here, but this link is correct) and django-allauth. In the end, I went with django-social-auth (not to be confused with django-socialauth) because a friend had recommended it and because I’d already installed it before I read this article. However, the article’s conclusion that django-allauth is best out of the box also seems valid.

Installation

The instructions in django-social-auth‘s docs are helpful in walking you through available settings and options.

I also found the included example app useful. To use this app, I cloned django-social-auth‘s git repo, created a virtualenv called django-social-auth, ran pip install -r requirements.txt inside this virtualenv to install all the required apps, ran manage.py syncdb, and finally ran manage.py runserver. Voila, example app is up and running at 127.0.0.1, showing a simple screen with options to login through about a dozen different different services.

API Keys

The first service I tested was Twitter. I use it more than any others, and I already had the API keys for it. I threw my API key and secret key into the example local_settings.py file provided with django-social-auth and tried to log in via the example app. Boom: 401 Unauthorized. I double-checked all my settings and installation and whatnot. Seemed fine.

I turned my attention to the API keys. The ones I had were generated for Readsr, i.e., I entered readsrs.com as the domain when I generated them at dev.twitter.com. But now I was running on localhost, 127.0.0.1, so I suspected the readsrs.com keys wouldn’t be valid. I wasn’t sure whether Twitter would hand over a new consumer key for 127.0.0.1, or baulk at the request. (It seemed like it should do so, but I hadn’t seen any instructions anywhere that said to get a key for your development machine.) Turns out Twitter will happily give you a key for 127.0.0.1. Once I plugged the new keys in, I was able to log in with my Twitter credentials, and just as it should, django-social-auth automatically created an auth.user for this account.

Integrating with Readsr

I followed the instructions again to config my own app, Readsr. To add a login option using Twitter credentials, I put a link to the reversed view that begins the django-social-auth login process for twitter, i.e., {% url socialauth_begin "twitter" %}, to my login template. And it worked.

I still need to fix a few oddities. For example, Twitter returns my first and last names together in first_name (or else django-social-auth is concatenating them into that column), and doesn’t supply any email address. But the basic functionality is there, and was relatively easy to achieve.

Postscript

The author of the article I linked above had an error using OpenID when using django-social-auth, which is why he preferred django-authall. He filed a bug for the error he got, and I notice that it was closed 15 hours ago (though if you read the comments, it seems it was actually fixed back in mid-July). Good timing.

For Readsr, I need to track events that recur on a particular day of the week (e.g., first Sunday of the month, third Friday of the month). I created a DayOfWeek model to store any particular event’s day of the week. It contains a method next_day_of_week() to return a datetime.date object set to the next occurrence of whatever weekday a given event instance is set to (this helps with figuring out when the next occurrence of an event is).

It’s easier to show through an example. On Sunday 7/3/2011:

• For an object with DayOfWeek set to Sunday, next_day_of_week() would return 7/3/2011 (current day).
• For DayOfWeek set to Monday, it would return 7/4/2011 (first subsequent Monday).
• For DayOfWeek set to Saturday, it would return 7/9/2011 (first subsequent Saturday).

Sounds simple enough. It seemed like this would be a good place to do my first unit tests.

Unit Testing

To do unit testing, the typical method is to first write test cases and then write code. In this case, I’d already written my code, so I went back and wrote test cases, trying to forget how my code worked.

To write test cases, you have to detail requirements for each method you want to test: input and expected (correct) output. The list of examples for next_day_of_week() I wrote above works for this purpose. But there’s a catch: next_day_of_week() calculates the next day of the week relative to the current date, by calling datetime.date.today(). So if I write expected output for 7/3/2011, it will no longer be the correct output on 7/4/2011 or any following day. I needed a way to make datetime.date.today() always spit out my input date when I run tests, yet still continue to function normally outside of testing. Enter mocking.

Mocking

The solution was to mock out the method—to replace the real datetime.date.today() with a fake one that produces the same output no matter what day it is. To accomplish this, I used the powerful Mock library. Specifically, I needed to use the patch decorator. This decorator makes it really easy to replace on particular object within the scope of a particular method.

Before I could patch the today() method, I needed to create my own fake method. It would look like this:

1
2

def faketoday()
    return date(2011, 7, 4)

There’s a problem, though, when I try to patch (or mock out) the method:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

>>> import mock
>>> def faketoday():
...     return datetime.date(2011, 7, 4)
... 
>>> @mock.patch("datetime.date.today", faketoday)
... def testfunc():
...     return datetime.date.today()
... 
>>> testfunc()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Users/wbert/.virtualenvs/readsr_env/lib/python2.6/site-packages/mock.py", line 561, in patched
    arg = patching.__enter__()
  File "/Users/wbert/.virtualenvs/readsr_env/lib/python2.6/site-packages/mock.py", line 623, in __enter__
    setattr(self.target, self.attribute, new_attr)
TypeError: can't set attributes of built-in/extension type 'datetime.date'
>>>

datetime.date is considered a Python built-in and can’t be modified.

Modifying a Class That Can’t Be Modified

The trick is to write a child class that can be modified, and thus faked:

1
2
3
4

class FakeDate(date):
	"A fake replacement for date that can be mocked for testing."
	def __new__(cls, *args, **kwargs):
		return date.__new__(date, *args, **kwargs)

All this does is create a class whose constructor returns an instance of its parent’s class, date. Usually, this would be pointless, but it’s useful here because the new class isn’t a built-in and thus can be mocked.

To use it, we simply decorate any test method that calls datetime.date.today() with a patch to replace datetime.date with FakeDate, and we also provide FakeDate a fake today() method that returns only and always the particular we are going to use for testing:

1
2
3
4
5
6
7
8
9
10

class TestDayOfWeek(TestCase):
	"""Test the day of the week functions."""
 
	@mock.patch('series.models.date', FakeDate)
	def test_valid_my_next_day_of_week_sameday(self):
		from datetime import date
		FakeDate.today = classmethod(lambda cls: date(2011, 7, 3)) # July 3, 2011 is a Sunday
		new_day_of_week = DayOfWeek.objects.create()
		new_day_of_week.day = "SU"
		self.assertEquals(new_day_of_week.my_next_day_of_week(), date(2011, 7, 3))

A couple things to note: the patch only applies within this particular method, so each method to use a patch must be decorated. Also, the real datetime.date is imported in the method so we can use it inside the fake today() method. We could put this fake today() method inside FakeClass, but making it a lambda (anonymous) method assigned inside the test case gives us the flexibility to set a particular date for each test case.

Namespacing

You may be wondering why the patch decorator takes “series.models.date” as the method to replace instead of “datetime.date”. That was how I tried it at first, and I was confused when it didn’t work. It seemed as if the patch hadn’t taken effect.

Well, it hadn’t. That’s because within the module being tested (models.py in the series app, or series.models in Python dotted notation), date has been imported like so:

1

from datetime import date

This means that within series.models, date is now available as series.models.date, so that’s the name that needs to be mocked out. For more on namespacing when mocking, checked out Mock’s Where to patch documentation.

Now we can supply out unit tests with any date we want, ensuring that we know what the results should be and can test against them.

References

Learning how to do this stuff, I posted my first question at Stackoverflow (I ended up answering it myself). I also learned about using a fake class from this question. Finally, the Mock documentation was very helpful as well.