Bill Mill

Visualizing Permutations

Thu, 29 Apr 2010 21:10:00 +0000

At some point a few years ago, I got interested in permutation algorithms and implemented a few that were in Knuth as well as gathered some that were floating around the internet. I never did anything with them, until I saw Aldo Cortesi's excellent sorting visualizations which reminded me of a figure from Knuth, and inspired me to create some visualizations of my own using Aldo's code.

Lexicographic Permutations

Informally, the permutations of a set are all possible orderings of its members. The permutations of the set {1,2,3}, are:

[1,2,3], [1,3,2], [2,1,3], [2,3,1], [3,1,2], [3,2,1]

This particular ordering of permutations is in lexicographic order; it's listed in the same order as it would be in the dictionary. There are many well-known algorithms for generating permutations in lexicographic order. Here is my version of one such algorithm; go read Knuth if you're at all interested in learning more.

Here's what a lexicographic permutation of four elements looks like:

It's easy to see that each element gets its turn at the top of the list, and that each time a new element goes to the top the remainder of the list is sorted.

Single-transposition Permutations

It is an interesting and non-obvious fact that there's a way to permute any given set by only switching the positions of one pair of elements per iteration. This permutation is deeply related to the Gray code, which if you haven't heard of, I highly recommend you go read about. The Knuth paper I mentioned aleady has a superb bit on the Gray code.

This image demonstrates clearly that at each step, there is exactly one crossing. My implementation of this algorithm is also on github.

CLP Permutation

I know very little about this algorithm, except that I got it from a message to comp.lang.python where it's attributed to Alex Martelli, Duncan Smith, and somebody named Anton (Anton Vredegoor?). Despite the crazy number of switches, and the fact that it reorders the list it's passed, it's actually crazy fast.

I'd love to hear from anyone with more info on this algorithm; my slightly modified version is here.

Odds and Ends

Well, that's it, just wanted to post some fun pictures of permutations, I hope you enjoyed it. The code I used to generate the pictures is derived from Aldo Cortesi's wonderful sortvis, and all my modifications to it are available on github as well.

If you want bonus points, I never got around to implementing Knuth's algorithm E (it's given towards the end of this), and I'd love for somebody else to do my work for me. If you're tough enough, that is.

The Bill Mill NCAA Bracket Randomizer, 2010 edition

Sun, 14 Mar 2010 20:15:00 +0000

I've updated my NCAA bracket randomizer for the 2010 field; check it out here. As of now, there are no improvements, but those may come in the next few days; feel free to suggest ideas.

Multi-Line Lambdas in Python Using the With Statement

Thu, 20 Aug 2009 22:55:00 +0000

Python does not have multi-line lambdas because Guido dislikes them aesthetically. However, with just a bit of introspection, code like this is possible:

>>> with each([12, 14, 16]):
...   def _(x):
...     print x
...     print x+1
...  
12
13
14
15
16
17

I'll say a bit about my motivation for creating code like this, show how easy it is to write, and then I'll argue that code like this is both pythonic and aesthetically appealing in some circumstances.

A Mystery Solved (By Holmes Himself!)

When I first saw code using the with statement, my hope was that it would be able to be used somewhat like Haskell's Where clause or Ruby's blocks. When I dug into the spec, I was disappointed to discover that if it was possible, it wasn't easy, and I pushed the thought aside.

That was a couple years ago, and I didn't give it a moment's thought until I saw a blog post by Richard Jones that uses a with statement in exactly the way I had considered impossible up to now. I spent a few hours trying to figure it out, but I was stumped, so I put up a question on Stack Overflow to see if somebody could show me how he did it.

Within a few hours, Alex Martelli himself chimed in with a wonderful solution. The gist of the answer is that you can use the inspect module to access the context manager's calling scope, and figure out what variables have been defined between its __enter__ and __exit__ functions. I'm glad I asked aloud, because even if I had stumbled close to the solution, I surely wouldn't have come up with one as complete as his.

The How

Once I had Alex's proof of concept code in hand, I went to work making it do what I'd had in my head so long ago. In about an hour, I was able to write code that looks like this:

@accepts_block
def each(iterable, block):
  for i in iterable:
      block(i)

with each(["twelve", "fourteen", "sixteen"]):
  def _(x):
    print x

@accepts_block
def bmap(arr, block):
  return map(block, arr)

with bmap([1,2,3]) as foo:
  def _(x):
    return (float(x) + 1) / 2

print foo # [1.0, 1.5, 2.0]

What you see above are two functions which use a decorator giving them access to the function defined within the with block. The decorator passes the block to the function as its last argument just like in Ruby.

To understand how this happens, you need to know how context managers work. Context managers consist of a class with __enter__ and __exit__ methods which are called upon entering the with block and upon exiting, just as you'd expect.

Alex's solution involves scanning the scope of the calling function from the __enter__ and __exit__ methods, and pulling out the differences between them. These differences will be all the variables that were defined in the with block. A sketch:

class FindInteresting(object):
  def __enter__(self):
    f = inspect.currentframe(1)
    self.already_defined = dict(f.f_locals)

  def __exit__(self):
    f = inspect.currentframe(1)
    #pick out the differences between f.f_locals and self.already_defined

When we pick out the differences between the two, we need to be careful to check for names that have been redefined so that we don't miss out on new functions that reuse old names.

def __exit__(self):
  f = inspect.currentframe(1)
  interesting = {}
  for n in f.f_locals:
    newf = f.f_locals[n]
    if n not in self.already_defined:
      interesting[n] = newf
      continue
    anf = self.already_defined[n]
    if id(newf) != id(anf):
      interesting[n] = newf

After this function has run, interesting is a dictionary which (probably) contains all the names and values of the variables that have been redefined in the with block.

Because we have to use the id check to determine if a name has been redefined, and Python sometimes caches objects in memory, our function can be fooled. In this case, interesting will not detect x because it's being redefined and cpython caches the low integers, so id(x) will be the same for both xs.

x = 1
with FindInteresting:
  x = 1

In general, the cpython runtime is not aggressive about caching, but you should know that this possibility exists. If you use this technique, I recommend being strict about checking only newly defined functions, since there's no way to be sure if you missed any redefined names.

To make the teaser code at the top of the article work, I just wrapped Alex's code into a decorator that returned a context manager, then called the function being decorated with the definitions that we found in the interesting dictionary. The context manager's __call__ function gets overridden to allow you to pass in arguments for the function being decorated.

def accepts_block(f):
  class BlockContextManager(object):
    def __call__(self, *args, **kwargs):
      self.thefunction = functools.partial(f, *args, **kwargs)
      return self

    def __enter__(self):
      #do Alex's magic, just as above
    
    def __exit__(self):
      #make the interesting dictionary, just as above

      if len(interesting) == 1:
        block = list(interesting.itervalues())[0]
        assert isinstance(block, type(lambda:None))
        self.thefunction(block)

  return BlockContextManager()

It looks complicated and nested, but all it's doing is saving the function and all its arguments, grabbing the definitions from the with block, making sure there's only one definition and it's a function, then tacking it onto the end of the arguments list for the function and calling it. Phew.

The code above handles the case where you don't need to store the result of the function being decorated:

@accepts_block
def each(iterable, block):
  for i in iterable:
    block(i)

But what if we want to store the result? Turns out, we can further abuse the with block by hijacking its as clause. Because a variable defined in the as clause gets detected by Alex's code, we can use the inspect module to change that variable so that after the with block it reflects the result of our computation.

First we check to see if we probably have a block and a variable in the as statement, then we reach in and store our result there if we are in that case:

def __exit__(self):
  #exactly as before; frame = inspect.currentframe(1)

  if len(interesting) == 1:
    #exactly the same as before
  elif len(interesting) == 2:
    block = None
    savename = None
    for n,v in interesting.iteritems():
      if isinstance(v, type(lambda:None)): block = v
      else: savename = n

    assert savename and isinstance(block, type(lambda:None))

    frame.f_locals[savename] = self.thefunction(block)

Which lets us do this:

@accepts_block
def bmap(iterable, block):
  return map(block, iterable)
  
with bmap([1,2,3]) as result:
  def _(x):
    return x**2

print result #[1,4,9]

This time, we're really taking a leap by assuming that if we find a callable and any other variable, that the variable is where we want to store our results. This can lead to somewhat unexpected results:

with bmap([1,2,3]):
  not_a_result = 12
  def _(x):
    return x**2

print not_a_result # [1,4,9] instead of 12

That's my extremely long-winded description of how to abuse the with operator. If you want to see the full function and the super-lame test code I wrote, you can head on over to github and check it out.

Aesthetics

It should be clear from all of the disclaimers I've had to put into this article that this technique is of limited use in Python as it stands today. I'd like to make an argument that it suggests some nice syntactic sugar for python to support someday, while remaining totally ignorant of the actual difficulties of putting it into the language.

To do so, I'll start by posting the motivating example for decorators from the relevant PEP. It argues that this code:

def foo(cls):
  pass
foo = synchronized(lock)(foo)
foo = classmethod(foo)

is not nearly as readable as this code:

@classmethod
@synchronized(lock)
def foo(cls):
  pass

The main problem with the readability of the first snippet is not that it requires 2 redefinitions and 4 repetitions of foo. Rather, the main problem is that it places the cart before the horse by putting the function body ahead of the declarations that are required to understand it.

Similarly, when we define callback functions before we use them, we're required to define the function body before the place where it will be actually used. Often, we see:

def handle_click(self):
  foo = self.flim()
  if foo:
    self.flam(foo)

onClick(handle_click)

When it would be clearer to write:

with onClick():
  def _(self):
    foo = self.flim()
    if foo:
      self.flam()

Which I find much more appealing.

Conclusion

I expect that there's no way that syntax like this could be officially supported by Python, both because of syntactic constraints and the BDFL's aesthetic concerns. I do think it is a neat exercise in pushing the Python interpreter and syntax past where they want to go, and I hope that it gives some food for thought on an interesting Python syntax.

I'm excited to see where Richard Jones goes with his project, and the design choices that he makes in it, since he's pushing the boundaries of Python design. Many thanks to him and Alex Martelli for sending me down quite an enjoyable path.

Finally, in case you missed it above, go ahead and take a look at the code on github.

If you want to leave a comment, I suggest leaving it on reddit.

Update:

Someone has taken this technique a bit farther, using some bytecode hackery.

What Gladwell is Missing: Institutional Memory

Sun, 31 May 2009 16:50:00 +0000

Malcolm Gladwell's recent article, ostensibly about how Davids can beat Goliaths, featured great praise for the full-court press as a method for less-skilled teams to beat more-skilled ones. He writes:

In the world of basketball, there is one story after another like this about legendary games where David used the full-court press to beat Goliath. Yet the puzzle of the press is that it has never become popular. People look at upsets like Fordham over UMass and call them flukes. Basketball sages point out that the press can be beaten by a well-coached team with adept ball handlers and astute passers—and that is true... Playing insurgent basketball did not guarantee victory. It was simply the best chance an underdog had of beating Goliath.

After his piece was rightly criticized all over the web, ESPN published a long and interesting email conversation between him and Bill Simmons that addressed the criticism. In it, Gladwell defended his column:

After my piece ran in The New Yorker, one of the most common responses I got was people saying, well, the reason more people don't use the press is that it can be beaten with a well-coached team and a good point guard. That is (A) absolutely true and (B) beside the point. The press doesn't guarantee victory. It simply represents the underdog's best chance of victory… I went to see a Lakers-Warriors game earlier this season, and it was abundantly clear after five minutes that the Warriors' chances of winning were, oh, no better than 10 percent. Why wouldn't you have a special squad of trained pressers come in for five minutes a half and press Kobe and Fisher?… Best case is that you rattle the Lakers and force a half-dozen extra turnovers that turn out to be crucial. And if you lose, so what? You were going to lose anyway.

Although lots of people have responded to this rebuttal, I haven't seen anyone mention what I consider to be an important reason that many teams have chosen not to implement full-court presses: Institutional Memory.

Let's imagine that the Warriors had in fact, put on the press, and that it had worked. The Warriors won against the vaunted Lakers! How would the team respond? They would press more. When that worked to win some more games, they would probably trade some of their players who were less well suited to the press to make room for some younger, faster, fitter players. They would win a few more games than they had been before. Everything's going good, right?

Then they hit a snag; No team has won the NBA Championship by pressing . The Warriors could optimize like crazy for the press, but they'd be training and working and trading and drafting to be, at best, a pretty good team. But no NBA team wants to be a pretty good team—they all want to win championships.

The mental exercise reveals what Gladwell has missed: the Warriors aren't playing to win that game, or even the most games that season. They're playing to try and win an NBA championship, and to do so, they need to spend years trying to build up the institutional memory of how to win games in the traditional way so that they can eventually beat every team, not just to win more games than they did playing traditionally.

All of this leads to a simple admonishment when analyzing organizations: don't expect that the organization is optimizing for what you think they're optimizing for. And when an organization seems to be acting in ways that are surprising to you, look for metrics that may be more important to them than the ones you expect.

The Bill Mill NCAA Bracket Randomizer

Wed, 18 Mar 2009 16:00:00 +0000

Each year, when the NCAA basketball tournament comes around, I end up in four or five pools, with a separate bracket filled out for each. I love the games, and I love having teams to root for, but I really hate the process of guessing to fill out my brackets. I inevitably pick too many upsets, just because I want to have fun rooting for underdogs; instead I end up bored after the first two rounds.

This year, I thought I could write some software to help me pick out my brackets. If I let the computer pick reasonably but randomly for each pool, I figure that I stand a better chance of having one decent bracket instead of the assortment of crappy ones I normally end up with.

So the last two nights, I wrote myself a bracket randomizer; just push the "randomize" button at the top and watch it go.

In order to pick what team will win a given game, it first calculates the chance each team will win by plugging Ken Pomeroy's ratings into the log5 formula. Then it picks a random number and compares it to the probability of the favorite winning; if the number is lower than that, it advances the favorite. Otherwise, it advances the underdog. Rinse and repeat, and you should have a reasonable random bracket for the whole tournament.

The Output

Next to each team in the bracket, you'll see three numbers in parentheses. These numbers represent, respectively, the team's Pythagorean rating, adjusted offensive efficiency, and adjusted defensive efficiency.

If that's Greek to you (groan), go check out Ken's explanation of what that means.

The color of each team, once you've randomized, represents their odds of winning. Brighter green is more of a favorite, deeper red more of an underdog. It should update the colors if you manually change the teams, but it won't; I just didn't have time to get everything done that I wanted to. Similarly, it won't update future games if you change the winner of an early one.

The Code

The surprisingly difficult part of this project was creating a simple HTML bracket that looked reasonable and allowed you to click to advance a team. I didn't get everything into the page that I wanted to, simply because I spent so much time just getting that done. (Keep in mind we're talking about a 2-night hack here).

The code to generate the bracket is contained in one super-ugly python file.

If you've got ideas for stuff to add, or want to generate a cooler looking bracket, or just check out the code, you can go get it at github. Feel free to fork and enjoy!