Note
Module addresses similar needs to shared. New code is encouraged to use shared variables.
Now that we’re familiar with the basics, we introduce Theano’s more advanced interface, Module. This interface allows you to define Theano “files” which can have variables and methods sharing those variables. The Module system simplifies the way to define complex systems such as a neural network. It also lets you load and save these complex systems using Python’s pickle mechanism.
Let’s use Module to re-implement the example using state.
>>> m = Module()
>>> m.state = T.dscalar()
>>> m.inc = T.dscalar('inc')
>>> m.new_state = m.state + m.inc
>>> m.add = Method(m.inc, m.new_state, {m.state: m.new_state})
>>> m.sub = Method(m.inc, None, {m.state: m.state - m.inc})
>>> acc = m.make(state = 0)
>>> acc.state, acc.inc
(array(0.0), None)
>>> acc.add(2)
array(2.0)
>>> acc.state, acc.inc
(array(2.0), None)
>>> acc.state = 39.99
>>> acc.add(0.01)
array(40.0)
>>> acc.state
array(40.0)
>>> acc.sub(20)
>>> acc.state
array(20.0)
This deserves to be broken up a bit...
>>> m = Module()
Here we instantiate an empty Module. If you can imagine that Theano is a way of generating code (expression graphs), then a Module() is like a fresh blank file.
>>> m.state = T.dscalar()
>>> m.inc = T.dscalar('inc')
Then we declare Variables for use in our Module. Since we assign these input Variables as attributes of the Module, they will be member Variables of the Module. Member Variables are special in a few ways, which we will see shortly.
Note
There is no need to name the Variable explicitly here. m.state will be given the name 'state' automatically, because it is being assigned to the attribute named 'state'.
Note
Since we made it a member of m, the acc object will have an attribute called inc. This attribute will keep its default value of None throughout the example.
>>> m.new_state = m.state + m.inc
This line creates a Variable corresponding to some symbolic computation. Although this line also assigns a Variable to a Module attribute, it does not become a member Variable like state and inc because it represents an expression result.
>>> m.add = Method(m.inc, m.new_state, {m.state: m.new_state})
Here we declare a Method. The three arguments are as follow:
>>> m.sub = Method(m.inc, None, {m.state: m.state - m.inc})
We declare another Method, that has no outputs.
>>> acc = m.make(state = 0)
This line is what does the magic (well, compilation). The m object contains symbolic things such as Variables and Methods. Calling make on m creates an object that can do real computation and whose attributes contain values such as numbers and numpy ndarrays.
At this point something special happens for our member Variables too. In the acc object, make allocates room to store numbers for m‘s member Variables. By using the string 'state' as a keyword argument, we tell Theano to store the number 0 for the member Variable called state. By not mentioning the inc variable, we associate None to the inc Variable.
>>> acc.state, acc.inc
(array(0.0), None)
Since state was declared as a member Variable of m, we can access it’s value in the acc object by the same attribute. Ditto for inc.
Note
Members can also be accessed using a dictionary-like notation. The syntax acc['state'] is equivalent to acc.state.
>>> acc.add(2)
array(2.0)
>>> acc.state, acc.inc
(array(2.0), None)
When we call the acc.add method, the value 2 is used for the symbolic m.inc. The first line evaluates the output and all the updates given in the updates argument of the call to Method that declared acc. We only had one update which mapped state to new_state and you can see that it works as intended, adding the argument to the internal state.
Note also that acc.inc is still None after our call. Since m.inc was listed as an input in the call to Method that created m.add, when acc.add was created by the call to m.make, a private storage container was allocated to hold the first parameter. If we had left m.inc out of the Method input list, then acc.add would have used acc.inc instead.
>>> acc.state = 39.99
The state can also be set. When we manually set the value of a member attribute like this, then subsequent calls to the methods of our module will use the new value.
>>> acc.add(0.01)
array(40.0)
>>> acc.state
array(40.0)
>>> acc.sub(20)
>>> acc.state
array(20.0)
Here, note that acc.add and acc.sub share access to the same state value but update it in different ways.
A friendlier way to use Module is to implement your functionality as a subclass of Module:
from theano.compile import Module, Method
import theano.tensor as T
class Accumulator(Module):
def __init__(self):
super(Accumulator, self).__init__() # don't forget this
self.inc = T.dscalar()
self.state = T.dscalar()
self.new_state = self.inc + self.state
self.add = Method(inputs = self.inc,
outputs = self.new_state,
updates = {self.state: self.new_state})
self.sub = Method(inputs = self.inc,
outputs = None,
updates = {self.state: self.state - self.inc})
if __name__ == '__main__':
m = Accumulator()
acc = m.make(state = 0)
This is just like the previous example except slightly fancier.
Warning
Do not forget to call the constructor of the parent class! (That’s the call to super().__init__ in the previous code block.)
If you forget it, you’ll get strange behavior :(
Let’s say we want to add a method to our accumulator to print out the state and we want to call it print_state. There are two mechanisms to do this: let’s call them _instance_method and InstanceType.
This is the preferred way of adding a few instance methods with a minimum of boilerplate code.
All we need to do to use this mechanism is to give a method called _instance_print_state to our Module class.
Any method called like _instance_XXX will cause the object obtained through a call to make to have a method called XXX. Note that when we define _instance_print_state there are two “self” arguments: self which is symbolic and obj which is the compiled object (the one that contains values).
Hint: self.state is the symbolic state variable and prints out as “state”, whereas obj.state is the state’s actual value in the accumulator and prints out as “0.0”.
If a number of instance methods are going to be defined, and especially if you will want to inherit from the kind of class that gets instantiated by make, you might prefer to consider using the InstanceType mechanism.
As was said in the previous section, you can add functionality with _instance_XXX methods. One of these methods is actually special: _instance_initialize will be called with whatever arguments you give to make. There is a default behavior which we have used, where we give the states’ initial values with keyword arguments (acc.make(state = 0)). If you want more personalized behavior, you can override the default with your own method, which has to be called _instance_initialize.
Probably the most powerful feature of Theano’s Modules is that one can be included as an attribute to another so that the storage of each is available to both.
As you read through examples of Theano code, you will probably see many instances of Modules being nested in this way.
Classes implementing Theano’s Module system.
For design notes, see doc/advanced/module.txt
Exception raised when a Variable has no associated storage.
Base class for the various kinds of components which are not structural but may be meaningfully used in structures (Member, Method, etc.)
Populates the memo dictionary with gof.Variable -> io.In pairings. The value field of the In instance should contain a gof.Container instance. The memo dictionary is meant to tell the build method of Components where the values associated to certain variables are stored and how they should behave if they are implicit inputs to a Method (needed to compute its output(s) but not in the inputs or updates lists).
Makes an instance of this Component using the mode provided and taking the containers in the memo dictionary.
A Component which builds nothing, such as External, may return None.
The return value of this function will show up in the Module graph produced by make().
Allocates the necessary containers using allocate() and uses build() to make an instance which will be returned. The initialize() method of the instance will be called with the arguments and the keyword arguments. If ‘mode’ is in the keyword arguments it will be passed to build().
Allocates the necessary containers using allocate() and uses build() with the provided mode to make an instance which will be returned. The initialize() method of the instance will not be called.
Returns a pretty representation of this Component, suitable for reading.
ComponentDictInstance is meant to be instantiated by ComponentDict.
Component Instance that allows new items to be added
ComponentList represents a sequence of Component. It builds a ComponentListInstance.
Composite represents a structure that contains Components.
Does allocation for each component in the composite.
Returns all components.
Returns (key, value) pairs corresponding to each component.
Generator that yields each component in a flattened hierarchy of composites and components. If include_self is True, the list will include the Composite instances, else it will only yield the list of leaves.
Generator that yields (path, component) pairs in a flattened hierarchy of composites and components, where path is a sequence of keys such that:
component is self[path[0]][path[1]]...
If include_self is True, the list will include the Composite instances, else it will only yield the list of leaves.
Get the Component associated to the key.
Set the Component associated to the key.
Generic type which various Composite subclasses are intended to build.
External represents a Variable which comes from somewhere else (another module) or is a temporary calculation.
Builds nothing.
alias of ModuleInstance
Member represents a Variable which is a state of a Composite. That Variable will be accessible from a built Composite and it is possible to do updates on Members.
Member builds a gof.Container.
If the memo does not have a Container associated to this Member’s Variable, instantiates one and sets it in the memo.
Returns the Container associated to this Member’s Variable.
Method is a declaration of a function. It contains inputs, outputs and updates. If the Method is part of a Composite which holds references to Members, the Method may use them without declaring them in the inputs, outputs or updates list.
inputs, outputs or updates may be strings. In that case, they will be resolved in the Composite which is the parent of this Method.
Method builds a Function (same structure as a call to theano.function)
Method allocates nothing.
Compile a function for this Method.
Parameters: | allocate_all – if True, storage will be allocated for all needed Variables even if there is no associated storage for them in the memo. If allocate_all is False, storage will only be allocated for Variables that are reachable from the inputs list. |
---|---|
Returns: | a function that implements this method |
Return type: | Function instance |
function inputs (see compile.function)
If Module members are named explicitly in this list, then they will not use shared storage. Storage must be provided either via an io.In value argument, or at the point of the function call.
This will override the Module compilation mode for this Method
function outputs (see compile.function)
Convert all inputs, outputs, and updates specified as strings to Variables.
This works by searching the attribute list of the Module to which this Method is bound.
update expressions for module members
If this method should update the shared storage value for a Module member, then the update expression must be given in this dictionary.
Keys in this dictionary must be members of the module graph–variables for which this Method will use the shared storage.
The value associated with each key should be a Variable (or a string that can be resolved to a Variable) representing the computation of a new value for this shared storage after each function call.
WRITEME
You should inherit from Module with the members will be other Modules or Components. To make more specialized elements of a Module graph, consider inheriting from Component directly.
alias of ModuleInstance
Allocates the necessary containers using allocate() and uses build() to make an instance which will be returned. The initialize() method of the instance will be called with the arguments and the keyword arguments. If ‘mode’ is in the keyword arguments it will be passed to build().
Module’s __setattr__ method hides all members under local_attr. This method iterates over those elements and wraps them so they can be used in a computation graph. The “wrapped” members are then set as object attributes accessible through the dotted notation syntax (<module_name> <dot> <member_name>). Submodules are handled recursively.
WRITEME
Note : | ModuleInstance is meant to be instantiated by Module. This differs from ComponentDictInstance on a key point, which is that getattr does a similar thing to getitem. |
---|---|
Note : | ModuleInstance is compatible for use as ComponentDict.InstanceType. |
Creates a dummy module, with external member variables for the input parameters required by the function f, and a member output defined as:
output <= f(**kwinit)
Parameters: |
|
---|
Wraps x in a Component. Wrappers can be registered using register_wrapper to allow wrapping more types.
It is necessary for Module attributes to be wrappable. A Module with an attribute that is not wrappable as a Component, will cause Component.make to fail.
Returns a wrapper function appropriate for x Returns None if not appropriate wrapper is found