A Quick Jump Into Twisted

I recently had to deal with twisted code, everything was already set up since I was writing a plugin. I never delt with asynchronous programming or twisted before, so my first problem was understanding what all those addCallback where doing.

First of all addCallback is a method of defer.Deferred objects that expect as argument a callable (a function). A must to read is the official Deferred Reference. As introduction, a Deffered is an object that “hold” the result until is ready and when it will be the added callback function will be called with that result as argument, this function will return a new result that will be hold for a while and when it will be available it will be passed to the next added callback and so on. This is a chain of callbacks.

The consequences are that you’ll have to think in a more functional way and you’ll baiscally have two kinds of functions, the “twisted aware” ones and the normal ones. Here’s an example that might help explain:

from twisted.internet import reactor, defer

def double(x):
    # usually here there's something expensive
    # that's going to take some time
    return x*x

def print_triple(x):
    # we need to think in a functional way, so we need a function
    # to print a result
    print 'the triple of {} is {}'.format(x, 3*x)

def main():
    d = defer.Deferred()
    d.addCallback(double)
    d.addCallback(print_triple)
    
    # let's start the callback chain, inserting the first value
    d.callback(10)


main()

# manually set up the end of the process by asking the reactor to
# stop itself in 1 seconds time
reactor.callLater(1, reactor.stop)

# start up the Twisted reactor (event loop handler) manually
reactor.run()

Those reactor run and (dalyed) stop are just to start and stop the twisted engine. The main point here is that one has to think functionally for everything, because all the values/results are available only to functions through the callback system.

Install python package in its own virtualenv

This will be a practical example with flexget, but with any other package the drill is the same.

Reasons for why you may want to do this:

  • Flexget comes with a lot dependecies.
  • Some of this depending libraries are requiring a specific version. For instance SQLAlchemy==0.7.4 while the latest version is 0.7.5 (as for now).
  • Avoid conflicts between the flexget depending libraries and the ones that are already installed (or will be in future).

How to do it:

  1. Install virtualenv:

    ~$ sudo pip install virtualenv
    or
    ~$ sudo pip-3.2 install virtualenv

    If you want the default python version to be used inside virtualenv to be 3.2 (I choose that one because most of the code I wrote is in 3.2). Anyway flexget requires python2.x, so I’ll use the --python option later, check out the virtualenv help:
    ~$ virtualenv -h
    [...]
    -p PYTHON_EXE, --python=PYTHON_EXE
    The Python interpreter to use, e.g.,
    --python=python2.5 will use the python2.5 interpreter
    to create the new environment. The default is the
    interpreter that virtualenv was installed with
    (/usr/bin/python3)

  2. Choose where to place the virtual envairoment. /usr/local/lib will be a good choice:
    ~$ cd /usr/local/lib

  3. Creating the virtual envairoment.
    With options:
    --no-site-packages
    --distribute
    -p /usr/bin/python2.7

    /usr/local/lib$ sudo virtualenv flexget_virtual_env --no-site-packages --distribute -p /usr/bin/python2.7
    Running virtualenv with interpreter /usr/bin/python2.7
    The --no-site-packages flag is deprecated; it is now the default behavior.
    New python executable in flexget_virtual_env/bin/python2.7
    Also creating executable in flexget_virtual_env/bin/python
    Installing distribute...........................................................
    ................................................................................
    ..................................................done.
    Installing pip...............done.

  4. With the virtualenv-pip install flexget:
    /usr/local/lib$ cd flexget_virtual_env/bin
    /usr/local/lib/flexget_virtual_env/bin$ ls
    activate activate.fish easy_install pip python
    activate.csh activate_this.py easy_install-2.7 pip-2.7 python2.7
    /usr/local/lib/flexget_virtual_env/bin$ sudo ./pip install flexget
    Downloading/unpacking flexget
    Downloading FlexGet-1.0r2724.tar.gz (699Kb): 699Kb downloaded
    Running setup.py egg_info for package flexget

    [...]

    Successfully installed flexget FeedParser SQLAlchemy PyYAML BeautifulSoup html5lib PyRSS2Gen pynzb progressbar jinja2 flask cherrypy requests Werkzeug certifi chardet
    Cleaning up...

    Well, that looks good.

  5. Copying the flexget script into /usr/bin/:
    /usr/local/lib/flexget_virtual_env/bin$ sudo cp ./flexget /usr/bin/

  6. Let’s see if it works:
    /usr/local/lib/flexget_virtual_env/bin$ cd
    ~$ flexget --version
    1.0r2724

Done!

How to handle integer and string from input

It’s not very long since I joined the SO community, but even in my short experience I could see that from time to time ir comes a guy trying to build some kind of dungeon/room-based super game in Python without any Python skills.

Some time ago I gave an asnwer to someone who seemed to coming straight off php. His Python script was a simple “print a menu and ask at the user what to do”. It wasn’t a game, but the core it’s similar and maybe it could of interest to somebody.

He was having problems handling both integers and strings (that’s where the title came from).
And this was my answer (ported in python-3.x):

Read more of this post