Dynamical properties of recurrent neural networks

Sept 30th, 2015

Martin Huelse



Content

  1. Source code
  2. General remarks
  3. Schmitt trigger
  4. Low-pass filter
  5. Adaptive Schmitt trigger
  6. Oscillator
  7. Literature



Source code

The examples are implemented in standard C. They should compile on a standard unix-like operating system, such as ubuntu etc. The C source code archive sources.tar.gz is ready for download.

Assuming the classic software development environment - a terminal running on a unix-, linux-like operating system, please unpack and compile the source files as follows:

  tar -xvf sources.tar.gz
  cd sources
  make

There are four example programms which can be executed like

  ./rnnST
  ./rnnAST
  ./rnnLowPass
  ./rnnPG

Without any arguments the programs run with default parameter values. Each prints a series of numeric values on the standard output. This output can be redirected in a file, like this:

  ./rnnST >t.txt

Now the data are written into file t.txt and can be plotted by other applications. You might use gnuplot

  gnuplot
  gnuplot>
  gnuplot> plot  [-0.1:1.1][-0.1:1.1] "t.txt" notitle
  gnuplot>

When executing the command gnuplot the interpreter environment is activated where you can execute plot-commands as follows

  gnuplot>
  gnuplot> plot  [-0.1:1.1][-0.1:1.1] "t.txt" notitle
  gnuplot>

When executing the command above the following window should be displayed.

Picture


General remarks

The simple time discrete dynamics is applied for these artifical neural network implementations. Thus, the neuron activity of a single neuron presented by variable a_1 is calcuated in each time step by calling the following function

  void updateNeuronActivities(double s){
    a_1 = B + R*sigmoide(a_1) + s*W;
  }

where function paramter s represents the input signal.

Considering a network with more neurons (e.g. the oscillator without input singnal) then the update function is defined as follows:

  void updateNeuronActivities(){
    a_1 = a*sigmoide(a_1) + b*sigmoide(a_2) - 0.5*(a+b);
    a_2 = a*sigmoide(a_2) - b*sigmoide(a_1) + 0.5*(b-a);
    a_3 = W*sigmoide(a_2) + P;
  }

where neuron activities of three neurons (a_1, a_2 and a_3) are calculated accordingly.

For all the examples presented here the same transfer function sigmoide(x) is applied. It is defined as follows:

  double sigmoide(double x){
    double y;
    y = 1.0 / (1 + exp(-x));
    return y;
  }

According to this definition return value is always large than zero and smaller than one. For this reason all the values serving as input signals for the networks are also in the range of 0.0 to 1.0.


Schmitt trigger

Picture

Example 1 - corresponding command:

>
>./rnnST >t.txt 
>gnuplot  -e "plot [-0.1:1.1][-0.1:1.1] 't.txt' using 1:2 notitle " -persist
>

Example 1 - execution with network and simulation parameters B, R, W and nmbSteps:

>
>./rnnST help
Wrong number of command line parameters! Please use in the following to ways:
1.) using default values (B=25, R=12, W=-50.0, nmbSteps=5000):  ./rnnST    
2.) specifing free numeric parameters B, R, W and nmbSteps:  ./rnnST  <B> <R> <W> <nmbSteps> 
>
>
>./rnnST 25 12 -50 5000 >t.txt 
>gnuplot  -e "plot [-0.1:1.1][-0.1:1.1] 't.txt' using 1:2 notitle " -persist
>

Low-pass filter

Picture

Example 2 - corresponding command:

>
>./rnnLowPass >t.txt 
>gnuplot  -e "plot [:][-0.1:1.1] 't.txt' using 1 with lines  notitle " -persist
>
>gnuplot  -e "plot [:][-0.1:1.1] 't.txt' using 2 with lines  notitle " -persist
>
>

Example 2 - execution with network and simulation parameters B, R, W and nmbSteps:

>
>./rnnLowPass help
Wrong number of command line parameters! Please use in the following to ways:
1.) using default values (B=-2.67, R=5.0, W=0.35, nmbSteps=30000):  ./rnnLowPass    
2.) specifing free numeric parameters B, R, W and nmbSteps:  ./rnnLowPass   <B>  <R>  <W>  <nmbSteps>   
>
>./rnnLowPass -2.67 5.0 0.35 30000
>
>gnuplot  -e "plot 't.txt' using 1 with lines notitle " -persist
>
>gnuplot  -e "plot 't.txt' using 2 with lines notitle " -persist
>

Picture

Example 3 - corresponding execution with network and simulation parameters B, R, W and nmbSteps:

>
>./rnnLowPass help
Wrong number of command line parameters! Please use in the following to ways:
1.) using default values (B=-2.67, R=5.0, W=0.35, nmbSteps=30000):  ./rnnLowPass    
2.) specifing free numeric parameters B, R, W and nmbSteps:  ./rnnLowPass   <B>  <R>  <W>  <nmbSteps>   
>
>./rnnLowPass -2.67 5.0 0.35 30000
>
>gnuplot  -e "plot 't.txt' using 1 with lines notitle " -persist
>
>gnuplot  -e "plot 't.txt' using 3 with lines notitle " -persist
>

Adaptive Schmitt trigger

Picture

Example 4 - corresponding command:

>
>./rnnAST >t.txt 
>gnuplot  -e "plot [-0.1:1.1][-0.1:1.1] 't.txt' using 1:4  notitle " -persist
>
>

Example 4 - execution with network and simulation parameters S and nmbSteps:

>
> ./rnnAST help
Wrong number of command line parameters! Please use in the following to ways:
 1.) using default values (S=0.6, nmbSteps=5000):  ./rnnAST    
 2.) specifing free numeric parameters S and nmbSteps:  ./rnnAST  <S> <nmbSteps> 
>
>./rnnAST 0.1 5000 >t.txt
>
>gnuplot  -e "plot 't.txt' using 1:4 notitle " -persist
>
>
>./rnnAST 0.4 5000 >t.txt
>
>gnuplot  -e "plot 't.txt' using 1:4 notitle " -persist
>

Oscillator

Picture

Example 5 - corresponding command:

>
>./rnnPG >t.txt 
>gnuplot  -e "plot [0:400]'t.txt' using 2 notitle, 't.txt' using 3 notitle " -persist
>
>

Example 5 - execution with network and simulation parameters E, F, B, W and nmbSteps:

>
> 
./rnnPG help
Wrong number of command line parameters! Please use in the following to ways:
1.) using default values (E=0.08, F=0.1, P=-10.0, W=20.0; nmbSteps=5000):  ./rnnPG    
2.) specifing free numeric parameters E, F, B, W and nmbSteps:  ./rnnPG  <E> <F> <P> <W> <nmbSteps>  
>
>./rnnPG 0.08 0.1 -10.0 20 5000 >t.txt
>
>gnuplot  -e "plot [0:400]'t.txt' using 2 notitle, 't.txt' using 3 notitle " -persist
>
>

Picture

Example 5 - execution with network and simulation parameters E, F, B, W and nmbSteps:
>
> 
./rnnPG help
Wrong number of command line parameters! Please use in the following to ways:
1.) using default values (E=0.08, F=0.1, P=-10.0, W=20.0; nmbSteps=5000):  ./rnnPG    
2.) specifing free numeric parameters E, F, B, W and nmbSteps:  ./rnnPG  <E> <F> <P> <W> <nmbSteps>  
>
>./rnnPG 0.08 0.1 -10.0 20 5000 >t.txt
>
>gnuplot  -e "plot [0:400]'t.txt' using 3 notitle, 't.txt' using 4 notitle " -persist
>
>

Literature

[1] Presentation (pdf)
[2] M. Hülse, Multifunktionalität rekurrenter neuronaler Netze. DISKI 306, 2007.(pdf)
[3] Hülse, M., Pasemann, F.: Dynamical Neural Schmitt Trigger for Robot Control. J. R. Dorronsoro(ed.): ICANN 2002, LNCS 2415, Springer, 783-788, 2002.(pdf)
[4] Pasemann, F., Hild, M., und Zahedi, K.: SO(2)-Networks as Neural Oscillators. In Proceedings of International Work-Conference on Artificial and Natural Neural Networks (IWANN) (2003). (pdf)
[5] Manoonpong, P., Pasemann, F., Fischer, J., Roth, H. (2005). Neural processing of auditory signals and modular neural control for sound tropism of walking machines. International Journal of Advanced Robotic Systems (ARS), ISSN: 1729-5506, vol. 2, no. 3, pp. 223-234. (pdf)
[6] Wischmann, S., Hülse, M., Knabe, J., Pasemann, F.: Synchronization of internal neural rhythms in multi-robotic systems. Adaptive Behavior 14(2): 117-127, 2006. (pdf)
[7] Hülse, M., Wischmann, S., and Pasemann F.: Structure and function of evolved neuro- controllers for autonomous robots. Connection Science, 16(4): 249-266, 2004.(pdf)
[8] http://manoonpong.com
[9] http://www.neurorobotik.de
[10] Neurocybernetics
[11] http://wischmann.ws