Python Variables

Following up on my previous post, in this one I will writing mostly on variables, scopes, modules

Variables

  • Variables are name, not containers
    • Everything is an object
    • Everything is a reference
    • Variables are neither
    • They are not allocated memory that holds data, they are neither objects or references they are only labels
  • Everything concrete is an object
    • Whenever you hold an object, whenever you store it anywhere you are storing a reference, you dont own an object, you just own their reference.
  • Everything that holds anything, holds references
  • Variables refers to objects
    • namespaces maps names to objects

Scopes

  • 3 Scope rule
    • names are either local or global or built ins
  • Local names defined by assignment unless ‘global’ definition says otherwise
    • local means existing in current class, function, global means existing in current module and not existing everywhere
    • builtins is just a special namespace used by builtin functions
    • if you have a global of the same name as a builtin the compiler will find the global instead of the builtin

Modules

  • They have their own namespace, keeps the rest of the modules tidy
  • Executed on first import, namespace becomes attributes
  • Then cached in sys.modules
  • import is syntactic sugar, it calls in the following function
    • mod = __import__(“mod”)
  • sys.modules doesn’t have to contain only modules
    • it is just a mapping of names to whatever object you want import to return
  • If you map None to a name then it will raise import error

Python Design

I will add a series of these posts which are just a bunch of notes that I am going to create for learning Python in depth from one of the Google talks on Python.

  • Everything is run time* (compile time* too is runtime)
    • somethings are cached (.pyc files)
  • Execution happens in namespace
    • modules, functions and classes all have their own namespace.
  • Modules are executed top to bottom
    • just like a script
  • def and class statements are runtime

Function creation

  • def statement creates a function
  • Code is compiled at compile time
    • into a code object
    • The code for the function will be compiled into a code object, which is a separate object from whatever code the rest of the program is compiled into
  • Wrapped in a function object at def time
    • When the def statement is executed the code object is wrapped into a function object along with whatever arguments it takes and any defaults to those arguments
  • Arguments defaults evaluated then too
  • def is primarily an assignment statement
    • essentially if you look at the bytecode then it is the same bytecode as the assignment operation

def func(arg1, arg2=func2()):
    print "entering func1"
    def inner_func(arg3=arg2):
        print "entering inner_func"
        return arg1, arg3
    arg1 = arg2 = None
    return inner_func

When the def statement is reached that whole line gets evaluated including the default arguments and the rest of the program which is


    print "entering func1"
    def inner_func(arg3=arg2):
        print "entering inner_func"
        return arg1, arg3
    arg1 = arg2 = None
    return inner_func

has been compiled because the compile time is in the past but is not being executed until you call the function. When func() is executed the inner func is evaluated till the def statement along with the defaults but the rest is compiled and also the defaults are evaluated at this time so whatever arg2 was is assigned to arg3 and not arg3 will not get that value of arg2 when you call inner_func so if the value of arg2 changes later on the value of arg3 will still be that value of arg2 when the def statement was reached.

 

Class Creation

Pretty similar to functions but a little different as well

  • Class statements executes suite
    • in a separate namespace
    • Class statement is followed by a block of code that is executed when the class statement is reached.
  • Class object is created out of the dict
    • name space becomes attribute space
  • Methods are just functions inside a class
    • Special wrappings happens automatically at attribute level
    • self is passed implicitly but received explicitly
  • Class body is normal code block.

* There is a pretty clear answer on stackoverflow regarding this concept which I would like to add here.

Run time and compile time are different stages in the running of a computer program. It is quite a difficult concept to learn and it is best approachable if you ask it in terms of these questions.

  1. What invariants the program satisfy.
  2. What can go wrong in this phase.
  3. If the phase succeeds what are the post conditions.
  4. What are inputs and outputs if any.

Compile time

  1. The program need not satisfy any invariant, in fact it need not be well formed at all.
  2. Things that can go wrong here are
    1. Syntax errors
    2. Type checking errors
    3. Compiler crashes
  3. If the compiler succeeds we know that the program is well formed and it is possible to run the program.
  4. Inputs and Outputs?
    1. Inputs are the program itself along with the headers or any other libraries to be imported
    2. Output is the assembly code or relocatable object code or executable. If the program has errors then outputs are the bunch of those error messages.

Run time

  1. We know nothing of program invariants, they can be anything that the programmer specifies
  2. Things that can go wrong here are
    1. Division by zero
    2. Dereferencing a null pointer
    3. opening a file which is not already there
  3. If run time succeeds the program finished without crashing
  4. Inputs and outputs are entirely on the programmer.

Scoping in Python – The LEGB Rule

Actually, a concise rule for Python Scope resolution, from Learning Python, 3rd. Ed.. (These rules are specific to variable names, not attributes. If you reference it without a period, these rules apply)

LEGB Rule.

L. Local. (Names assigned in any way within a function (def or lambda), and not declared global in that function.

E. Enclosing function locals. (Name in the local scope of any and all enclosing functions (def orlambda), form inner to outer.

G. Global (module). Names assigned at the top-level of a module file, or declared global in a def within the file.

B. Built-in (Python). Names preassigned in the built-in names module : open,range,SyntaxError,…

So, in the case of

code1
class Foo:
   code2
   def spam.....
      code3
      for code4..:
       code5
       x()

The for loop does not have it’s own namespace. It would look in the LEGB order,

L : local, in the current def.

E : Enclosed function, any enclosing functions(if def spam was in another def)

G : Global. Were there any declared globally in the module?

B : Any builtin x() in Python.

 

 

source- stackoverflow

A quirky take on programming languages

OK, this is going to be a little bit un-PC, but what the hell.

PHP is your teenage sweetheart, the girl you first awkwardly fumbled around with that one summer. Just don’t try and start a more serious relationship – this girl has serious issues.

Perl is PHP’s older sister. She might be a bit old for you, but she was pretty popular back in the 90s. She’s looking pretty ugly and scarred now, so you don’t hear from her too much.

Ruby is the cool kid of the scripting family. When you first saw her, she took your breath away with her beauty. She was fun, too. At the time she seemed a bit slow and ditzy – though she’s matured a lot in the last few years.

Python is Ruby’s sensible (and slightly more boring) sister.

Java is a successful career woman. What she lacks in raw intelligence she makes up for in dress – always turned out in immaculateCamelCase, sure to impress the enterprise customers. You might feel like she’s the sensible type you should settle down with. Just prepare for years of “NO THAT DOESNT GO THERE GOD YOU ALWAYS USE THE WRONG TYPE AND YOU MISSED A SEMICOLON” nagging.

C++ is Java’s cousin. Similar to Java in many ways, the main difference being she grew up in a more innocent time and doesn’t believe in condoms/automatic memory management, so be cautious.

is C++’s mom. Mention her name to some old grey beard hackers and they’re sure to reminisce with a twinkle in their eye.

Objective C is another member of the C family. She joined that weird church a while back, and won’t date anyone outside of it.

Haskell, Clojure, Scheme and their friends are those hipster, artsy, intellectual girls you probably spent a blissful college summer with a few years ago. The first girls who really challenged you. Of course, it could never have become something more serious (you tell yourself). Though you’ll always be left asking “what if?”

You might be put off C# due to her family’s reputation. But they’ve gone legit, the last few years, they tell you. Once you’re one of us, you’re one of us, you hear? You need a database? Her brother MSSQL will hook you up. Need a place to stay? Heck, her daddy will even buy you your own mansion on Azure avenue. What’s that, you’re having second thoughts about all these overly friendly relatives? No, you can never leave. You’re part of the family, now, ya hear?

Javascript – hey, wasn’t that the girl you first kissed, way before even PHP came on the scene? I wonder what she’s doing now. I hear her career’s really taken off in the last few years. Would be cool to catch up some time, if only for old time’s sake… (You catch sight of her, dressed head to toe in designer jQuery)… wow,somebody grew into a beautiful swan…

 

 

***Originally posted on quora by Isaac Lewis (http://thectonetwork.com/blog)***

ssh git@github.com Agent admitted failure to sign using the key. Permission denied (publickey)

Now this error was something that made me bang my head on the table. I had tried almost everything— looking up in google, stackover flow, adding my ssh key to the github account but nothing seem to work.

the error–

$ Permission denied (publickey).
$ fatal: The remote end hung up unexpectedly

these are some of the solutions on stackoverflow  that I had tried but none seemed to work.

http://stackoverflow.com/questions/7065257/permission-denied-publickey-fatal-the-remote-end-hung-up-unexpectedly-while

http://stackoverflow.com/questions/7065257/permission-denied-publickey-fatal-the-remote-end-hung-up-unexpectedly-while

and finally what did work was the most simplest solution of all

you just have to add these lines

$ ssh-add ~/.ssh/id_rsa

Problems While Deploying Ruby on Rails App on Heroku

Ok this was the second day and it was no less cumbersome than the previous one, tried deploying the app on Heroku and that turned out to be a big headache. so it turns out that heroku doesn’t supports sqlite3. One way of solving this problem is to put sqlite3 in development mode in your Gemfile==> bundle install(again)==> add Gemfile using git and then run git push heroku master.

so if you are facing the following error

Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.
/usr/local/bin/ruby extconf.rb
checking for sqlite3.h... no
sqlite3.h is missing. Try 'port install sqlite3 +universal'
or 'yum install sqlite-devel' and check your shared library search path (the
location where your sqlite3 shared library is located).
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers. Check the mkmf.log file for more
details. You may need configuration options.
Provided configuration options:
--with-opt-dir
--without-opt-dir
--with-opt-include
--without-opt-include=${opt-dir}/include
--with-opt-lib
--without-opt-lib=${opt-dir}/lib
--with-make-prog
--without-make-prog
--srcdir=.
--curdir
--ruby=/usr/local/bin/ruby
--with-sqlite3-dir
--without-sqlite3-dir
--with-sqlite3-include
--without-sqlite3-include=${sqlite3-dir}/include
--with-sqlite3-lib
--without-sqlite3-lib=${sqlite3-dir}/lib
--enable-local
--disable-local
Gem files will remain installed in /tmp/build_26udn5p5vbb5b/vendor/bundle/ruby/1.9.1/gems/sqlite3-1.3.6 for inspection.
Results logged to /tmp/build_26udn5p5vbb5b/vendor/bundle/ruby/1.9.1/gems/sqlite3-1.3.6/ext/sqlite3/gem_make.out
An error occurred while installing sqlite3 (1.3.6), and Bundler cannot continue.
Make sure that `gem install sqlite3 -v '1.3.6'` succeeds before bundling.
!
! Failed to install gems via Bundler.
!
! Detected sqlite3 gem which is not supported on Heroku.
! http://devcenter.heroku.com/articles/how-do-i-use-sqlite3-for-development
!
! Heroku push rejected, failed to compile Ruby/rails app To git@heroku.com:xxxxxxxxx-xxxxxx-4612.git! [remote rejected] master -> master (pre-receive hook declined)error: failed to push some refs to 'git@heroku.com:powerful-garden-4612.git'

 

 

 

 

source 'https://rubygems.org'

gem 'rails', '3.2.7'

group :development do
 gem 'sqlite3', '1.3.5'
end
# Gems used only for assets and not required
# in production environments by default.
group :assets do
 gem 'sass-rails', '3.2.4'
 gem 'coffee-rails', '3.2.2'

gem 'uglifier', '1.2.3'
end

gem 'jquery-rails', '2.0.2'

*courtesy Ruby on Rails Tutorial Book by M. Hartl 

copy the above source code from listing 1.5 in the book and probably you will have to change your rails version and sqlite3 version to your current installed version for me it was rails-3.2.8 and sqlite3-1.3.6. After changing this do the following


$ bundle install

$ git add Gemfile

$ git commit -m "Put Sqlite3 into development in the Gemfile"

$ git push heroku master

*courtesy http://stackoverflow.com/questions/7963561/heroku-stack-cedar-cannot-run-git-push-heroku-master

And I believe this will finally solve all your problem

Problems and Fixes While Installing Ruby on Rails on Ubuntu 12.04

So yesterday I finally decided to give Ruby on Rails a try, the installation of which was a major headache and kept me up all night. Some of the major problems that one will face will installing Ruby on rails using Ruby Version Manager especially on Ubuntu are as follows.

ZLIB not installed

The dreaded zlib error

ERROR:  Loading command: update (LoadError)
    no such file to load -- zlib
ERROR:  While executing gem ... (NameError)
    uninitialized constant Gem::Commands::UpdateCommand

It seems that there are some of the libraries that are not pre installed in Ubuntu and because of that ‘gem‘ will not install or update rails. One way to get out of this mess is to install these libraries beforehand, otherwise there will be a problem while installing rails, or while running rails server.

so the 1st thing to do is to install all these libraries with the following command

$ sudo apt-get install zlib1g-dev libssl-dev libreadline6-dev libxml2-dev libsqlite3-dev

and then install ruby if you haven’t installed it yet

rvm install 1.9.3

or reinstall it if you have already installed it

rvm reinstall 1.9.3

Javascript runtime error

Could not find a JavaScript runtime. See https://github.com/sstephenson/execjs for a list of available runtimes.
/usr/local/lib/ruby/gems/1.9.1/gems/execjs-1.3.0/lib/execjs/runtimes.rb:50:in `autodetect'
/usr/local/lib/ruby/gems/1.9.1/gems/execjs-1.3.0/lib/execjs.rb:5:in `<module:ExecJS>'
/usr/local/lib/ruby/gems/1.9.1/gems/execjs-1.3.0/lib/execjs.rb:4:in `<top (required)>'
/usr/local/lib/ruby/gems/1.9.1/gems/bundler-1.0.21/lib/bundler/runtime.rb:68:in `require'
/usr/local/lib/ruby/gems/1.9.1/gems/bundler-1.0.21/lib/bundler/runtime.rb:68:in `block (2 levels) in require'
/usr/local/lib/ruby/gems/1.9.1/gems/bundler-1.0.21/lib/bundler/runtime.rb:66:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/bundler-1.0.21/lib/bundler/runtime.rb:66:in `block in require'
/usr/local/lib/ruby/gems/1.9.1/gems/bundler-1.0.21/lib/bundler/runtime.rb:55:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/bundler-1.0.21/lib/bundler/runtime.rb:55:in `require'
/usr/local/lib/ruby/gems/1.9.1/gems/bundler-1.0.21/lib/bundler.rb:122:in `require'
/home/vitalarthur/rails/316-private-pub/chatter-after/config/application.rb:7:in `<top (required)>'
/usr/local/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
/usr/local/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
/home/vitalarthur/rails/316-private-pub/chatter-after/Rakefile:5:in `<top (required)>'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/rake_module.rb:25:in `load'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/rake_module.rb:25:in `load_rakefile'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:501:in `raw_load_rakefile'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:82:in `block in load_rakefile'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:133:in `standard_exception_handling'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:81:in `load_rakefile'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:65:in `block in run'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:133:in `standard_exception_handling'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:63:in `run'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/bin/rake:33:in `<top (required)>'
/usr/local/bin/rake:19:in `load'
/usr/local/bin/rake:19:in `<main>'

The catch here is that Windows has a builtin JavaScript engine which execjs can locate. On Linux there is not a builtin although there are several available that one can install. therubyracer is one of them. Others are listed in the execjs.

the trick here is to include

gem 'execjs'
gem 'therubyracer'

in your Gemfile which can be found in your first_app directory (the directory which is installed when you run the following command ‘rails new first_app’). So I believe this will take care of some of your problems

and here are a few links which might be useful

http://stackoverflow.com/questions/9202324/execjs-could-not-find-a-javascript-runtime-but-execjs-and-therubyracer-are-in

http://ubuntuforums.org/showthread.php?t=1494997

The Theory of Metaclass

Classes as objects

Before understanding metaclasses, you need to master classes in Python. And Python has a very peculiar idea of what classes are, borrowed from the Smalltalk language.

In most languages, classes are just pieces of code that describe how to produce an object. That’s kinda true in Python too:

  >>> class ObjectCreator(object):
  ...       pass
  ... 

  >>> my_object = ObjectCreator()
  >>> print my_object
  <__main__.ObjectCreator object at 0x8974f2c>

But classes are more than that in Python. Classes are objects too.

Yes, objects.

As soon as you use the keyword class, Python executes it and creates an OBJECT. The instruction

  >>> class ObjectCreator(object):
  ...       pass
  ... 

creates in memory an object with the name ObjectCreator.

This object (the class) is itself capable of creating objects (the instances), and this is why it’s a class.

But still, it’s an object, and therefore:

  • you can assign it to a variable
  • you can copy it
  • you can add attributes to it
  • you can pass it as a function parameter

e.g.:

  >>> print ObjectCreator # you can print a class because it's an object
  <class '__main__.ObjectCreator'>
  >>> def echo(o):
  ...       print o
  ... 
  >>> echo(ObjectCreator) # you can pass a class as a parameter
  <class '__main__.ObjectCreator'>
  >>> print hasattr(ObjectCreator, 'new_attribute')
  False
  >>> ObjectCreator.new_attribute = 'foo' # you can add attributes to a class
  >>> print hasattr(ObjectCreator, 'new_attribute')
  True
  >>> print ObjectCreator.new_attribute
  foo
  >>> ObjectCreatorMirror = ObjectCreator # you can assign a class to a variable
  >>> print ObjectCreatorMirror.new_attribute
  foo
  >>> print ObjectCreatorMirror()
  <__main__.ObjectCreator object at 0x8997b4c>

Creating classes dynamically

Since classes are objects, you can create them on the fly, like any object.

First, you can create a class in a function using class:

  >>> def choose_class(name):
  ...     if name == 'foo':
  ...         class Foo(object):
  ...             pass
  ...         return Foo # return the class, not an instance
  ...     else:
  ...         class Bar(object):
  ...             pass
  ...         return Bar
  ...     
  >>> MyClass = choose_class('foo') 
  >>> print MyClass # the function returns a class, not an instance
  <class '__main__.Foo'>
  >>> print MyClass() # you can create an object from this class
  <__main__.Foo object at 0x89c6d4c>

But it’s not so dynamic, since you still have to write the whole class yourself.

Since classes are objects, they must be generated by something.

When you use the class keyword, Python creates this object automatically. But as with most things in Python, it gives you a way to do it manually.

Remember the function type? The good old function that lets you know what type an object is:

>>> print type(1)
<type 'int'>
>>> print type("1")
<type 'str'>
>>> print type(ObjectCreator)
<type 'type'>
>>> print type(ObjectCreator())
<class '__main__.ObjectCreator'>

Well, type has a completely different ability, it can also create classes on the fly. type can take the description of a class as parameters, and return a class.

(I know, it’s silly that the same function can have two completely different uses according to the parameters you pass to it. It’s an issue due to backwards compatibility in Python)

type works this way:

  type(name of the class, 
       tuple of the parent class (for inheritance, can be empty), 
       dictionary containing attributes names and values)

e.g.:

>>> class MyShinyClass(object):
...       pass

can be created manually this way:

  >>> MyShinyClass = type('MyShinyClass', (), {}) # returns a class object
  >>> print MyShinyClass
  <class '__main__.MyShinyClass'>
  >>> print MyShinyClass() # create an instance with the class
  <__main__.MyShinyClass object at 0x8997cec>

You’ll notice that we use “MyShinyClass” as the name of the class and as the variable to hold the class reference. They can be different, but there is no reason to complicate things.

type accepts a dictionary to define the attributes of the class. So:

>>> class Foo(object):
...       bar = True

Can be translated to:

  >>> Foo = type('Foo', (), {'bar':True})

And used as a normal class:

  >>> print Foo
  <class '__main__.Foo'>
  >>> print Foo.bar
  True
  >>> f = Foo()
  >>> print f
  <__main__.Foo object at 0x8a9b84c>
  >>> print f.bar
  True

And of course, you can inherit from it, so:

  >>>   class FooChild(Foo):
  ...         pass

would be:

  >>> FooChild = type('FooChild', (Foo,), {})
  >>> print FooChild
  <class '__main__.FooChild'>
  >>> print FooChild.bar # bar is inherited from Foo
  True

Eventually you’ll want to add methods to your class. Just define a function with the proper signature and assign it as an attribute.

>>> def echo_bar(self):
...       print self.bar
... 
>>> FooChild = type('FooChild', (Foo,), {'echo_bar': echo_bar})
>>> hasattr(Foo, 'echo_bar')
>>> hasattr(FooChild, 'echo_bar')
True
>>> my_foo = FooChild()
>>> my_foo.echo_bar()
True

You see where we are going: in Python, classes are objects, and you can create a class on the fly, dynamically.

This is what Python does when you use the keyword class, and it does so by using a metaclass.

What are metaclasses (finally)

Metaclasses are the ‘stuff’ that creates classes.

You define classes in order to create objects, right?

But we learned that Python classes are objects.

Well, metaclasses are what create these objects. There are the classes’ classes, you can picture them this way:

  MyClass = MetaClass()
  MyObject = MyClass()

You’ve seen that type lets you do something like this:

  MyClass = type('MyClass', (), {})

It’s because the function type is in fact a metaclass. type is the metaclass Python uses to create all classes behind the scenes.

Now you wonder why the heck is it written in lowercase, and not Type?

Well, I guess it’s a matter of consistency with str, the class that creates strings objects, and int the class that creates integer objects. type is just the class that creates class objects.

You see that by checking the __class__ attribute.

Everything, and I mean everything, is an object in Python. That includes ints, strings, functions and classes. All of them are objects. And all of them have been created from a class:

  >>> age = 35
  >>> age.__class__
  <type 'int'>
  >>> name = 'bob'
  >>> name.__class__
  <type 'str'>
  >>> def foo(): pass
  >>> foo.__class__
  <type 'function'>
  >>> class Bar(object): pass
  >>> b = Bar()
  >>> b.__class__
  <class '__main__.Bar'>

Now, what is the __class__ of any __class__ ?

  >>> a.__class__.__class__
  <type 'type'>
  >>> age.__class__.__class__
  <type 'type'>
  >>> foo.__class__.__class__
  <type 'type'>
  >>> b.__class__.__class__
  <type 'type'>

So, a metaclass is just the stuff that creates class objects.

You can call it a ‘class factory’ if you wish.

type is the built-in metaclass Python uses, but of course, you can create your own metaclass.

The __metaclass__ attribute

You can add a __metaclass__ attribute when you write a class:

class Foo(object):
  __metaclass__ = something...
  [...]

If you do so, Python will use the metaclass to create the class Foo.

Careful, it’s tricky.

You write class Foo(object) first, but the class object Foo is not created in memory yet.

Python will look for __metaclass__ in the class definition. If it finds it, if will use it to create the object class Foo. If it doesn’t, it will use type to create the class.

Read that several times.

When you do:

class Foo(Bar):
  pass

Python does the following:

Is there a __metaclass__ attribute in Foo?

If yes, create in memory a class object (I said a class object, stay with me here), with the name Foo by using what is in __metaclass__.

If Python can’t find __metaclass__, it will look for a __metaclass__ in Bar (the parent class), and try to do the same.

If Python can’t find __metaclass__ in any parent, it will look for a __metaclass__ at the MODULE level, and try to do the same.

Then if it can’t find any __metaclass__ at all, it will use type to create the class object.

Now the big question is, what can you put in __metaclass__ ?

The answer is: something that can create a class.

And what can create a class? type, or anything that subclasses or uses it.

Custom metaclasses

The main purpose of a metaclass is to change the class automatically, when it’s created.

You usually do this for APIs, where you want to create classes matching the current context.

Imagine a stupid example, where you decide that all classes in your module should have their attributes written in uppercase. There are several ways to do this, but one way is to set __metaclass__ at the module level.

This way, all classes of this module will be created using this metaclass, and we just have to tell the metaclass to turn all attributes to uppercase.

Luckily, __metaclass__ can actually be any callable, it doesn’t need to be a formal class (I know, something with ‘class’ in its name doesn’t need to be a class, go figure… but it’s helpful).

So we will start with a simple example, by using a function.

# the metaclass will automatically get passed the same argument
# that you usually pass to `type`
def upper_attr(future_class_name, future_class_parents, future_class_attr):
  """
    Return a class object, with the list of its attribute turned 
    into uppercase.
  """

  # pick up any attribute that doesn't start with '__'
  attrs = ((name, value) for name, value in future_class_attr.items() if not name.startswith('__'))
  # turn them into uppercase
  uppercase_attr = dict((name.upper(), value) for name, value in attrs)

  # let `type` do the class creation
  return type(future_class_name, future_class_parents, uppercase_attr)

__metaclass__ = upper_attr # this will affect all classes in the module

class Foo(object): 
  # we can define __metaclass__ here instead to affect only this class
  bar = 'bip'

print hasattr(Foo, 'bar')
# Out: False
print hasattr(Foo, 'BAR')
# Out: True

f = Foo()
print f.BAR
# Out: 'bip'

Now, let’s do exactly the same, but using a real class for a metaclass:

# remember that `type` is actually a class like `str` and `int`
# so you can inherit from it
class UpperAttrMetaclass(type): 
    # __new__ is the method called before __init__
    # it's the method that creates the object and returns it
    # while __init__ just initializes the object passed as parameter
    # you rarely use __new__, except when you want to control how the object
    # is created.
    # here the created object is the class, and we want to customize it
    # so we override __new__
    # you can do some stuff in __init__ too if you wish
    # some advanced use involves overriding __call__ as well, but we won't
    # see this
    def __new__(upperattr_metaclass, future_class_name, 
                future_class_parents, future_class_attr):

        attrs = ((name, value) for name, value in future_class_attr.items() if not name.startswith('__'))
        uppercase_attr = dict((name.upper(), value) for name, value in attrs)

        return type(future_class_name, future_class_parents, uppercase_attr)

But this is not really OOP. We call type directly and we don’t override call the parent __new__. Let’s do it:

class UpperAttrMetaclass(type): 

    def __new__(upperattr_metaclass, future_class_name, 
                future_class_parents, future_class_attr):

        attrs = ((name, value) for name, value in future_class_attr.items() if not name.startswith('__'))
        uppercase_attr = dict((name.upper(), value) for name, value in attrs)

        # reuse the type.__new__ method
        # this is basic OOP, nothing magic in there
        return type.__new__(upperattr_metaclass, future_class_name, 
                            future_class_parents, uppercase_attr)

You may have noticed the extra argument upperattr_metaclass. There is nothing special about it: a method always receives the current instance as first parameter. Just like you have self for ordinary methods.

Of course, the names I used here are long for the sake of clarity, but like for self, all the arguments have conventional names. So a real production metaclass would look like this:

class UpperAttrMetaclass(type): 

    def __new__(cls, name, bases, dct):

        attrs = ((name, value) for name, value in dct.items() if not name.startswith('__'))
        uppercase_attr = dict((name.upper(), value) for name, value in attrs)

        return type.__new__(cls, name, bases, uppercase_attr)

We can make it even cleaner by using super, which will ease inheritance (because yes, you can have metaclasses, inheriting from metaclasses, inheriting from type):

class UpperAttrMetaclass(type): 

    def __new__(cls, name, bases, dct):

        attrs = ((name, value) for name, value in dct.items() if not name.startswith('__'))
        uppercase_attr = dict((name.upper(), value) for name, value in attrs)

        return super(UpperAttrMetaclass, cls).__new__(cls, name, bases, uppercase_attr)

That’s it. There is really nothing more about metaclasses.

The reason behind the complexity of the code using metaclasses is not because of metaclasses, it’s because you usually use metaclasses to do twisted stuff relying on introspection, manipulating inheritance, vars such as __dict__, etc.

Indeed, metaclasses are especially useful to do black magic, and therefore complicated stuff. But by themselves, they are simple:

  • intercept a class creation
  • modify the class
  • return the modified class

Why would you use metaclasses classes instead of functions?

Since __metaclass__ can accept any callable, why would you use a class since it’s obviously more complicated?

There are several reasons to do so:

  • The intention is clear. When you read UpperAttrMetaclass(type), you know what’s going to follow
  • You can use OOP. Metaclass can inherit from metaclass, override parent methods. Metaclasses can even use metaclasses.
  • You can structure your code better. You never use metaclasses for something as trivial as the above example. It’s usually for something complicated. Having the ability to make several methods and group them in one class is very useful to make the code easier to read.
  • You can hook on __new____init__ and __call__. Which will allow you to do different stuff. Even if usually you can do it all in __new__, some people are just more comfortable using __init__.
  • These are called metaclasses, damn it! It must mean something!

Why the hell would you use metaclasses?

Now the big question. Why would you use some obscure error prone feature?

Well, usually you don’t:

Metaclasses are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don’t (the people who actually need them know with certainty that they need them, and don’t need an explanation about why).

Python Guru Tim Peters

The main use case for a metaclass is creating an API. A typical example of this is the Django ORM.

It allows you to define something like this:

  class Person(models.Model):
    name = models.CharField(max_length=30)
    age = models.IntegerField()

But if you do this:

  guy = Person(name='bob', age='35')
  print guy.age

It won’t return an IntegerField object. It will return an int, and can even take it directly from the database.

This is possible because models.Model defines __metaclass__ and it uses some magic that will turn the Person you just defined with simple statements into a complex hook to a database field.

Django makes something complex look simple by exposing a simple API and using metaclasses, recreating code from this API to do the real job behind the scenes.

The last word

First, you know that classes are objects that can create instances.

Well in fact, classes are themselves instances. Of metaclasses.

  >>> class Foo(object): pass
  >>> id(Foo)
  142630324

Everything is an object in Python, and they are all either instances of classes or instances of metaclasses.

Except for type.

type is actually its own metaclass. This is not something you could reproduce in pure Python, and is done by cheating a little bit at the implementation level.

Secondly, metaclasses are complicated. You may not want to use them for very simple class alterations. You can change classes by using two different techniques:

  • monkey patching
  • class decorators

99% of the time you need class alteration, you are better off using these.

But 99% of the time, you don’t need class alteration at all 🙂

Iterables

Iterables

To understand what yield does, you must understand what generators are. And before generators come iterables. When you create a list, you can read its items one by one, and it’s called iteration :

>>> mylist = [1, 2, 3]
>>> for i in mylist :
...    print(i)
1
2
3

Mylist is an iterable. When you use a comprehension list, you create a list, and so an iterable :

>>> mylist = [x*x for x in range(3)]
>>> for i in mylist :
...    print(i)
0
1
4

Everything you can use “for… in…” on is an iterable : lists, strings, files… These iterables are handy because you can read them as much as you wish, but you store all the values in memory and it’s not always what you want when you have a lot of values.

Generators

Generators are iterables, but you can only read them once. It’s because they do not store all the values in memory, they generate the values on the fly :

>>> mygenerator = (x*x for x in range(3))
>>> for i in mygenerator :
...    print(i)
0
1
4

It just the same except you used () instead of []. BUT, you can not perform for i in mygenerator a second time since generators can only be used once: they calculate 0, then forget about it and calculate 1 and ends calculating 4, one by one.

Yield

Yield is a keyword that is used like return, except the function will return a generator.

>>> def createGenerator() :
...    mylist = range(3)
...    for i in mylist :
...        yield i*i
...
>>> mygenerator = createGenerator() # create a generator
>>> print(mygenerator) # mygenerator is an object!
<generator object createGenerator at 0xb7555c34>
>>> for i in mygenerator:
...     print(i)
0
1
4

Here it’s a useless example, but it’s handy when you know your function will return a huge set of values that you will only need to read once.

To master yield, you must understand that when you call the function, the code you have written in the function body does not run. The function only returns the generator object, this is bit tricky 🙂

Then, your code will be run each time the for uses the generator.

Now the hard part :

The first time your function will run, it will run from the beginning until it hits yield, then it’ll return the first value of the loop. Then, each other call will run the loop you have written in the function one more time, and return the next value, until there is no value to return.

The generator is considered empty once the function runs but does not hit yield anymore. It can be because the loop had come to ends, or because you do not satisfy a “if/else” anymore.

Your code explained

Generator:

# Here you create the method of the node object that will return the generator
def node._get_child_candidates(self, distance, min_dist, max_dist):

  # Here is the code that will be called each time you use the generator object :

  # If there is still a child of the node object on its left
  # AND if distance is ok, return the next child
  if self._leftchild and distance - max_dist < self._median:
                yield self._leftchild

  # If there is still a child of the node object on its right
  # AND if distance is ok, return the next child
  if self._rightchild and distance + max_dist >= self._median:
                yield self._rightchild

  # If the function arrives here, the generator will be considered empty
  # there is no more than two values : the left and the right children

Caller:

# Create an empty list and a list with the current object reference
result, candidates = list(), [self]

# Loop on candidates (they contain only one element at the beginning) 
while candidates:

    # Get the last candidate and remove it from the list
    node = candidates.pop()

    # Get the distance between obj and the candidate
    distance = node._get_dist(obj)

    # If distance is ok, then you can fill the result
    if distance <= max_dist and distance >= min_dist:
        result.extend(node._values)

    # Add the children of the candidate in the candidates list 
    # so the loop will keep running until it will have looked
    # at all the children of the children of the children, etc. of the candidate
    candidates.extend(node._get_child_candidates(distance, min_dist, max_dist))

return result

This code contains several smart parts :

  • The loop iterates on a list but the list expands while the loop is being iterated 🙂 It’s a concise way to go through all these nested data even if it’s a bit dangerous since you can end up with an infinite loop. In this case, candidates.extend(node._get_child_candidates(distance, min_dist, max_dist)) exhausts all the values of the generator, but while keep creating new generators object which will produce different values from the previous ones since it’s not applied on the same node.
  • The extend() method is a list object method that expects an iterable and adds its values to the list.

Usually we pass a list to it :

>>> a = [1, 2]
>>> b = [3, 4]
>>> a.extend(b)
>>> print(a)
[1, 2, 3, 4]

But in your code it gets a generator, which is good because :

  1. You don’t need to read the values twice.
  2. You can have a lot of children and you don’t want them all stored in memory.

And it works because Python does not care if the argument of a method is a list or not. Python expects iterables so it will work with strings, lists, tuples and generators ! This is called duck typing and is one of the reason why Python is so cool. But this is another story, for another question…

You can stop here, or read a little bit to see a advanced use of generator :

Controlling a generator exhaustion

>>> class Bank(): # let's create a bank, building ATMs
...    crisis = False
...    def create_atm(self) :
...        while not self.crisis :
...            yield "$100"
>>> hsbc = Bank() # when everything's ok the ATM gives you as much as you want
>>> corner_street_atm = hsbc.create_atm()
>>> print(corner_street_atm.next())
$100
>>> print(corner_street_atm.next())
$100
>>> print([corner_street_atm.next() for cash in range(5)])
['$100', '$100', '$100', '$100', '$100']
>>> hsbc.crisis = True # crisis is coming, no more money!
>>> print(corner_street_atm.next())
<type 'exceptions.StopIteration'>
>>> wall_street_atm = hsbc.create_atm() # it's even true for new ATMs
>>> print(wall_street_atm.next())
<type 'exceptions.StopIteration'>
>>> hsbc.crisis = False # trouble is, even post-crisis the ATM remains empty
>>> print(corner_street_atm.next())
<type 'exceptions.StopIteration'>
>>> brand_new_atm = hsbc.create_atm() # build a new one to get back in business
>>> for cash in brand_new_atm :
...    print cash
$100
$100
$100
$100
$100
$100
$100
$100
$100
...

It can be useful for various things like controlling access to a resource.

Itertools, your best friend

The itertools module contains special functions to manipulate iterables. Ever wish to duplicate a generator ? Chain a two generators ? Groups values in nested list with a one liner ? Map / Zip without creating another list ?

Then just import itertools.

An example ? Let’s see the possible orders of arrival for a 4 horses race:

>>> horses = [1, 2, 3, 4]
>>> races = itertools.permutations(horses)
>>> print(races)
<itertools.permutations object at 0xb754f1dc>
>>> print(list(itertools.permutations(horses)))
[(1, 2, 3, 4),
 (1, 2, 4, 3),
 (1, 3, 2, 4),
 (1, 3, 4, 2),
 (1, 4, 2, 3),
 (1, 4, 3, 2),
 (2, 1, 3, 4),
 (2, 1, 4, 3),
 (2, 3, 1, 4),
 (2, 3, 4, 1),
 (2, 4, 1, 3),
 (2, 4, 3, 1),
 (3, 1, 2, 4),
 (3, 1, 4, 2),
 (3, 2, 1, 4),
 (3, 2, 4, 1),
 (3, 4, 1, 2),
 (3, 4, 2, 1),
 (4, 1, 2, 3),
 (4, 1, 3, 2),
 (4, 2, 1, 3),
 (4, 2, 3, 1),
 (4, 3, 1, 2),
 (4, 3, 2, 1)]

Understanding the inner mechanisms of iteration

Iteration is a process implying iterables (implementing the __iter__() method) and iterators (implementing the __next__() method). Iterables are any objects you can get an iterator from. Iterators are objects that let you iterate on iterables.

More about it in in this article about how does the for loop work.