Python lambda expressions unleashed


Carl Kadie, Ph.D., is a research developer in Microsoft Research/TnR working on Genomics.

Launch Notebook Now!

Lambda expressions provide a way to pass functionality into a function. Sadly, Python puts two annoying restrictions on lambda expressions. First, lambdas can only contain an expression, not statements. Second, lambdas can't be serialized to disk. This blog shows how we can work around these restrictions and unleash the full power of lambdas.

So what are lambda's good for? Suppose, you have a list of words from a string.

In [1]:
"This is a test string from Carl".split()
Out[1]:
['This', 'is', 'a', 'test', 'string', 'from', 'Carl']

You can sort the words with sorted().

In [2]:
sorted("This is a test string from Carl".split())
Out[2]:
['Carl', 'This', 'a', 'from', 'is', 'string', 'test']

Notice, however, that all the capitalized words, sort before all the lower-case words. This can be fixed by passing a lambda expression as the key argument to the sorted() function.

In [3]:
sorted("This is a test string from Carl".split(), key=lambda word: word.lower())  # `key=str.lower` also works.
Out[3]:
['a', 'Carl', 'from', 'is', 'string', 'test', 'This']

Lambda can be more complicated. Suppose we want to sort the words based on their (lower-case) back-to-front letters? As a reminder, here is a Python way to reverse the lower-case letters of a word:

In [4]:
str.lower("Hello")[::-1]
Out[4]:
'olleh'

And here is how to pass this functionality to sorted() using a lambda:

In [5]:
sorted("This is a test string from Carl".split(), key=lambda word: word.lower()[::-1])
Out[5]:
['a', 'string', 'Carl', 'from', 'is', 'This', 'test']

But what if you want even more complex functionality? For example, functionality that requires if statements and multiple lines with unique scoping? Sadly, Python restricts lambdas to expressions only. But there is a workaround!

Define a function that

  • defines an inner function and ...
  • returns that inner function.

Note that the inner function can refer to variables in the outer function, giving you that private scoping.

In this example lower_sorted() is the outer function. It has an argument called back_to_front. Inside lower_sorted, we define and return an inner function called inner_lower_sorted(). That inner function has multiple lines including an if statement that references back_to_front.

In [6]:
 def lower_sorted(back_to_front=False):
    def inner_lower_sorted(word):
        result = word.lower()
        if back_to_front: #The inner function can refer to outside variables
            result = result[::-1]
        return result
    return inner_lower_sorted

print(sorted("This is a test string from Carl".split(), key=lower_sorted()))
print(sorted("This is a test string from Carl".split(), key=lower_sorted(back_to_front=True)))
['a', 'Carl', 'from', 'is', 'string', 'test', 'This']
['a', 'string', 'Carl', 'from', 'is', 'This', 'test']

You may find lambdas and these inner functions handy enough that you'd like to serialize one to disk for use later. Sadly, if you try to seralize with pickle module, you'll get an error message like "TypeError: can't pickle function objects".

A nice workaround is to use the dill project in place of pickle. The dill project is a third-party package that is now included in the standard Anaconda distribution. Here is an example:

In [7]:
!pip install dill
Requirement already satisfied (use --upgrade to upgrade): dill in /home/nbcommon/anaconda3_23/lib/python3.4/site-packages
You are using pip version 8.1.1, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
In [8]:
import dill as pickle

with open("temp.p",mode="wb") as f:
    pickle.dump(lower_sorted(back_to_front=True), f)

with open("temp.p", mode="rb") as f:
    some_functionality = pickle.load(f)

sorted("This is a test string from Carl".split(), key=some_functionality)
Out[8]:
['a', 'string', 'Carl', 'from', 'is', 'This', 'test']

Serialization of lambdas and these inner functions opens exciting possibilities. For example, we use it in one of our libraries to run work in different processes and even on different machines in a cluster.

We've seen that lambdas are a handy way to pass functionality into a function. Python's implementation of lambdas has two restrictions, but each restriction has a workaround.

  • Multiple lines not allowed.
    • Workaround: Define a function that defines and returns an inner function. The inner function can use variables outside itself.
  • Can't pickle lambdas or inner functions.
    • Workaround: Replace pickle with dill.

Python offers features such as list comprehensions that makes lambdas less used that in other languages. When you do need lambdas, however, they will now be unleashed.

Comments (8)
  1. T says:

    In line # 3, does it make sense to pass a function to convert the string to lower case, when it doesn't actually convert the string to lower case?

    sorted("This is a test string from Carl".split(), key=lambda word: word.lower())

    1. CarlKadie says:

      T, Thanks for your note. Good question. The trick is that Python normally sorts all words that start with an upper-case letter before all words that that start with a lower-case letter. Like this:
      ['Carl', 'This', 'a', 'from', 'is', 'string', 'test']
      Passing this lambda to the "sorted" function is a way to get it to sort without regard to case like this:
      [‘a’, ‘Carl’, ‘from’, ‘is’, ‘string’, ‘test’, ‘This’]
      As you observe, using the lambda here doesn't change the case of the result. It just changes how it is sorted. For details, the Python documentation includes a "How To" section that inspired my example: https://docs.python.org/2/howto/sorting.html#sortinghowto

  2. TC says:

    Why bother using an example the misleads the user into believing it will
    change the words to lower case?? ->key=lambda word: word.lower())

    Two of these words still have upper case letters
    ['a', 'Carl', 'from', 'is', 'string', 'test', 'This']

    1. CarlKadie says:

      TC, Thanks for your question. Please see my reply to T, above.

  3. Igor says:

    Oh. My. God... This is a fantastic example of why bad programmers don't like Python -- because Python restricts their ability to write terrible code!
    1) Why would you ever want to put your functions to disk in a pickle file when you can make a module?! To keep your code in a unreadable non-text format?! This is crazy! O_O
    2) "here is a Python way to reverse the lower-case letters of a word: str.lower("Hello")[::-1]". What??? The actual "python way" is to use reversed() built-in function!
    Lamdas are hard to read so it's very good that they have this restriction. Everything above could be rewritten in a much more readable idiomatic way. Well, actually it explains a lot if such programmers work at Microsoft! xD

    1. CarlKadie says:

      Igor, Good question about the motivation for using Dill on a lambda, closure or function. We use it in the open-source FaST-LMM project (https://github.com/MicrosoftGenomics/FaST-LMM) to distribute -- on-the-fly -- little bits of computation to HPC, Hadoop, and Azure clusters. I believe other Python distributed computation libraries do the same.

      With respect to reversing the letters of a string, surprisingly, "reversed" doesn't work. See this stackoverflow discussion to learn more: http://stackoverflow.com/questions/931092/reverse-a-string-in-python, Yours, Carl

  4. Tom Booth says:

    The IPython notebook link back to this article is broken.

    1. Carl says:

      It seems to be working now. Is anyone else having problems?

Comments are closed.

Skip to main content