SYN@PTICS®
Neural and Adaptive Systems

---

 DISTRIBUTED NEURAL OBJECTS
WITH JAVA AND CORBA
Luca Marchese

While anything's living around Internet world is very popular, some promising technologies like neural networks stay often closed in reserch centers and universities. In this article I try to celebrate a marriage between a software implementation of an artificial neural network and two Internet strongly related technologies like Java and CORBA. While I have not space to explain the details of the neural learning algorithm I can divide it in functional blocks and explain the general behavior at this level.
I like to put more attention on the Java implementation aspects of the algorithm than on mathematical ones: you can find anything about them on neural networks specialized books. This project is diveded in two subtasks:

a) a Java 1.02 compliant program that enables the neural network to learn from examples files and save synaptic files (don't warry you'll understand later!)for specific tasks.

b) a Java/CORBA client-server application where the server object is a neural engine that can load a task-related synaptic file (see the task a) and the client is an application/applet that can invoke his services through the Object Request Broker.
 
 

An artificial neural network is a structural semplified simulation of brain that is composed of single units called neurons. The biological neuron is mathematically view as a device with more inputs(x[n]) and one output(y) and a transfer function:
y = f(x[1] + x[2] + ... + x[n])
(see also fig.1).

In a brain and in a nervous system there are a lot of neurons interconnected between themselves(output -> input) and these connections are called synapses. These connections have a "conductivity" value that influences the signal transfer between neurons.
Artificial neural networks are semplified models in witch neurons can works. One tipical model is called feed-forward due to the fact that signals flow in a single direction from the global input to the global output(see fig.2).

The network is composed of four layers of neurons where the outputs of every n-layer neuron is connected to an input of every (n+1)-layer neuron.
You can see this model like a black-box with n inputs and m outputs. The behavior of the network when it answers to a question is described follow:

- inputs values are applied - signals propagate to next layers through synaptic connections - output values are got as response to input stimulus

The output response values are determined by the input stimuli and the values of the synaptic connections. In order to determine the outputs you must calculate the neurons activation for any layer proceding from input to output. For neuron j of layer m, supposing that layer m-1 has 10 neurons, I can write:
 

A[j][m] = sum(from k=1 to k=10) {y[k][m-1] * w([j][m])([k][m-1])}
   where A[j][m] is the activation of the neuron
   w([j][m])([k][m-1]) is the value of the connection
                                     between the neuron j of the layer m
                                    and the neuron k of the layer m-1

The output of the neuron is simply a function of his activation:
        y[j][m] = f(A[j][m])

where f() is normally a sigmoid function:

 O = 1/(1 + exp(-A))

where O = output of the neuron
            A = activation of the neuron
The values of the synaptic connections are the know-how of the network, such as the ability to produce correct responses to stimuli. These values are not programmed but "learned" from database of known correct associations of stimulus-response. This process called "supervidsed learning" is commonly performed on a feed-forward neural network, using the "backpropagation" algorithm. More generally the algorithm cicles on the examples (input-output associations) and for any input calculates the output with the actual synaptic values and, after, the difference between the output and the desired output of the example. Now the synaptic values are modified in order to obtain a lower error for the example. This process is repeated for any example a lot of times called "epochs", waiting the global error (the maximum error considering all examples) reachs a desired value commonly called "target"(see fig.3).

I must underline the fact that neural networks need inputs normalized in the range 0.0 -> 1.0 and the outputs are in the same range. It means that you must map the ranges of the input and output variables of the task in this range:
this operation is usually called normalization. In order to obtain the output variables(the answers) in the original range you must perform the inverse operation(denormalization). I have no space here to explain the details of the learning algorithm but I want remark some aspect of neural neural networks that make them so useful.
While the programmed systems can answer only to precise questions (think to a database), neural networks can answer to fuzzy or noisy questions. It is like to say they extract the important features of the answer-questions associations used in learning and they can answer sufficiently correctly to questions not available in the learning database. This characteristic enables neural networks to perform well in tasks like forecast, classification, pattern recognition. I can speak about credit-risk task for banks as an example for all. In this task, some characterics of the person that requires the credit are the inputs, while the risk is the output. This is a tipical task where a client-server solution would be useful.
 
 

In this article you'll see the development of a neural network software simulation in Java and a CORBA-applet based client server solution for neural services. The first program is composed of two classes contained in the files backprop.java and training.java. The class backprop is the core of the neural network and has all the methods to perform the learning process reading from a file containing the examples related to a particular task. The examples file is in ascii format and the examples are inserted like follow:
0.01   //input 1 of example 1
0.40   //input 2 of example 1
0.30   //output 1 of example 1
0.90   //output 2 of example 1
0.51   //input 1 of example 2
0.10   //input 2 of example 2
0.10   //output 1 of example 2
0.40   //output 2 of example 2
...and so on for any example.
And now I can go inside the class "backprop"(listing 1). The constructor build the network with specified(by the parameters) number of inputs, outputs and neurons in the two hidden layers. It builds also the synaptics matrix for any couple of layers(3 matrixes). The learning process is running on a separate thread and can be started/stopped/paused/resumed by other objects using the public interface methods startLearn(), stopLearn(), pauseLearn(), resumeLearn(). You can see inside the startLearn() the "start()" method of the "learningThread" and inside the run() method(that's the body of learningThread) the call to the learn() method that is the core of the learning process. I want you note the "implements Runnable" at the declaration of the class and the "throws InterruptedException" at the learn() method declaration. The Runnable interface is necessary in order to have multithread capability inside the object and the code invoking the learn() method must be informad to manage this possible exception. Because of Java 1.1 thread specification is not strongly defined and in order to guarantee correct execution in more environments, I use not-preemptive multithread managment. You can see that the threads leaves cpu to other threads using "sleep(time)" and I substitute the "pause()" method with a "while(condition){sleep(time)}" code. The "resume()" method is substituted simply by a method that updates the condition flag. Some private methods invoked by "learn()" complete the learning process: exec() executes the network for any examples and backpropagation() updates the synaptic values in function of the difference between obtained output and desired output. The "startLearn()" public method must be invoked with the parameters specifing the examples file, the number of examples to consider, the target error and a learning coefficient "epsilon".These parameters will be passed to the "learn()" method. When the target is reached, the network has his know-how in the synaptic values that can be saved on file(saveNet(filename)) in order to be reloaded after(loadNet(filename)).This feature can be useful for interrupt and restart a long learning process from a middle point instead from the random initialization of the synaptic values.You'll see it'll be used, also, by the CORBA server in the next section. If you want reload a saved network you must use the alternative constructor that loads a synaptic file with the same name of the network(the only one parameter passed). If you want continue a learning process on the loaded network you must disable the random initialization of synapses performed by "weightStarter()", setting to false the "randomInit" flag. The "training" class(listing 2) is the multiframe/multinetwork learning consolle with GUI. It enables you to the following actions:

1) Create/destroy a network (an instance of backprop.class). Necessary textfields are number of inputs, outputs, neurons in the hidden layers and an identifier name of the network. If the "LOADNET" Checkbox is setted these fields are not necessary but it's required only to set the name of the network that is used by the second constructor of "backprop" class to load a network from a file with the same name. The name of the network is labeled as "TASKID".

2) Start/pause/resume/stop a learning process. To start a learning process it is necessary to fill the textfields related with the name of the examples-file, the target error, the learning coefficient(epsilon), the number of the examples (EXAMPLES) and the number of the epochs before a pause(STEP). If you start a learning on a loaded network the error will start from the error previously reached by it.

3) Enable/disable the neural network to save itself automatically every an epochs number step. The "AUTOSAVE" Checkbox enables/disables this feature and "SAVESTEP" textfield must contain the number of epochs between any saving. The network will be saved on a file with the same name of the network, such as "TASKID".

4) Create a new consolle-frame (an instance of training.class) in order to manage simultaneusly more neural networks learning processes.

5) Perform the execution of the network at the actual status of learning, reading input from and writing output to the files indicated in the "<-IN OUT->" fields. This action is performed pressing the "TEST" button.

The "training" class implements the Runnable interface in order to have a thread(body of run()) dedicated to monitoring the status of the "learningThread" in the "backprop" instance. The event managing is in Java1.02 style and performed by the "action()" method. The textfields related to net creation and learning start are setted to "not-editable" when these actions are respectively performed and performing, in order to avoid incongruent data on the frame. To start the program is sufficient to run the training.class that is a main class having a main() method. This program thanks to the not-preemptive multithread managing is running well also in the Java1.02 virtual machine of ADK beta3 from IBM under Windows 3.1.

CORBA is the Common Object Request Broker Architecture developed by the OMG (Object Management Group) in order to have a new client server environment in witch objects distributed on different computers in a network can interact. In this dream environment objects(read also instances if you like) written in any OOP language can remotely invoke methods on objects residing on an other computer in the network, passing them parameters and obtaining results. This is really possible because the request of a service is not directly object to object performed, but is transported by a "virtual bus" called ORB or Object Request Broker (fig.4).

In a CORBA distributed objects environment there are server objects and clients performing method invokation on them. It is possible also that servers and clients invert temporaneusly their roles through the use of "callbacks" but I'll not speak about them in this article. In order to obtain an open environment where objects written in different languages can interact, CORBA defines an Interface Description Language(IDL) that enables you to describe the external interface of your objects indipendently by the language you have written them. When you build a server object you must define in a file his IDL interface and after generate from it two "pseudo objects" called "Skeleton" and "Stub" using the tool provided by the CORBA development environments available. The Skeleton must stay with the server object on a server computer while the Stub must stay with the client program in the client computer. Skeleton is a Java class that unmarshals the arguments for the server object and brings together the Java and CORBA object models. Skeleton can make it because it is a Java class that implements the CORBA.Object interface(the root CORBA object). The Stub is a Java class that provides marshaling functions for the argument passed by the client.
 
 

In the next section I describe the realization of a CORBA server object that implement a neural network with only execution capability(not learning). This object loads a synaptic file related to a task and previously created with the "training" program before descripted. Then the execution of the neural network can be invoked on the object, located in an HTTP server, as a method by an applet client. You can see an application of a similar project in an Intranet of a Credit Institute where the CORBA server object resides on the HTTP server and browsers on distributed PC can request his services in a applet. The updating of the neural network know-how in order to satisfy new statistical data, can be made rebuilding the synaptic file only on the server.This operation can be preformed by a neural-networks expert with specific task experience. The task-example that I describe in this article is a toy task where the neural network is trained to recognize profiles of 5 values, classifing them on 5 classes:
mountain, valley, positive slope, negative slope, plane.
ex: the profile 0.3, 0.6, 0.8, 0.5, 0.2 is a hill.
while the profile 0.1, 0.4, 0.8, 0.3, 0.1 is a mountain.
 
 

In this section you can see the developing steps of a CORBA client-server application entirely written in Java. I must write the following source files:

1) The IDL interface "neur.idl" containing the CORBA interface of the server object "ann.class".

2) The server object "ann.java" containing the method that performs the execution of the neural network

3) The main server "annserv.java" instantiating the server object "ann.class" registering it to the ORB.

4) The client applet "nnclient.java" invoking the method on the remote server object "ann.class". This client has also code to be an application: it is sufficient to comment/decomment some lines/methods in order to have an applet or an application.

5) The HTML file "ann.html" containing the applet (this is necessary only for the applet mode).
 

When I'll arrive to the end of the development process there will be more than five files involved in the client-server application behavior, but the other ones will be created by the development tools provided by the CORBA development kit. In this phase of the project I use the JDK 1.1 from Sun as Java development environment and Visibroker 3.1 from Visigenic as CORBA development kit under Windows95. Now I can explain the development process step by step:

1) Write the IDL file "neur.idl" with the interface of the server object(listing 3). This define only one method that perform the execution of the neural network.

2) Run the Visibroker program that creates some important Java classes from the IDL file:

prompt> idl2java "neur.idl" -no_comments -no_tie
 

This program will create the following files:
The Skeleton : "neurpack.neurImplBase.java" (server)
The Stub: "neurpack._st_neur.java" (client)
The Interface: "neurpack.neur.java" (server)
The Holder: "neurpack.neurHolder.java" (client)
The Helper: "neurpack.neurHelper.java" (client)
The Example: "neurpack._example_neur.java" (server example)
 

3) Write the server object "ann.java". This class must extend the Skeleton class "neurpack.neurImplBase.class" that implements the Interface "neurpack.neur.class" and must contain the body of the method defined in it(listing 4).
The contructor of this class is the same that one present in the "backprop.class" for load the neural network from file. The constructor loads from a file the number of inputs,outputs, hidden neurons of the two hidden levels and the value of any synaptic connection.
So this class is a part of the "backprop.class" with only the execution function adapted in the input/output. The parameter of the "perform()" method is the input array of float. In Java arrays are passed by reference also if they are arrays of primitive types (passed by value). So the client that invoked the method on this remote object, has the float outputs array attribute updated concordly to the response of the neural network.

4) Write the main server class that does the following(listing 5):
a) initialize the ORB
b) initialize the BOA
c) instantiate an "ann" object
d) export the "ann" object to the ORB
e) wait for requests from clients
The CORBA.ORB class is a run time part of VisiBroker implementing in Java the functions specified by OMG in the CORBA::ORB interface. The CORBA.BOA is the Basic Object Adapter providing services that the "ann" server-object can access. This pseudo-object is directly created by the ORB and his service "obj_is_ready(server-object)" exports the server-object to the ORB.

5) Write the applet/application "nnclient.java" that can invokes the method "perform()" on the server object "ann.class". This class must simply extend the Applet(or Frame)class(listing 6). The CORBA related operations are the ORB initialization and the binding to the server objectusing the bind() method of the "neurpack.neurHelper" object.
The CORBA interface object returned by the bind() method is then used to invoke the perform() method on the remote server-object. In order to have an application instead of an applet I've inserted the "main()" method and the "nnclient()" constructor. The source code is explained on the applet/application tranforming.

6) Write the HTML file "nnclient.html" that simply contain the applet (listing 7).

7) Compile all the files (the global procedure is shown in fig.5)

You can test well the application also in a single computer. You must do the following actions:

1) be sure your client is compiled as an application.
2) copy the server related files on a directory and the client related files on an other(or simply copy all the files on the same). Be sure at this step that a copy of neurpack directory is under both the server and the client directory(really you could remove some files in anyone because of server OR client related).
3) start the Visibroker OSAgent: this program provides fault-tolerant location services and must be running on at least one machine in the LAN environment.
4) go to the server directory and start the server:
prompt> vbj annserv
5) go to the client directory and start the client:
prompt> vbj nnclient
6) the frame of the client will appear.
7) fill the 5 input fields with float values in the range 0.0 1.0 so that they describe a shape like hill, valley, negative pendence or positive pendence
8) press the "RUN" button and see the result on the 5 output fields that have the following means:
mountain - valley - positive_slope - negative_slope - plane
 
 

In order to test the client(applet)/server system on a real HTTP connection a single machine you need to do the following:

1) be sure your client is compiled as an applet.
2) install the CORBA client-server classes and the "neurpack" package on the HTTP server in a directory.
3) install the "nnclient.html" in the same directory.
4) start the Visibroker OSAgent.
5) start the Visibroker "IIOP GateKeeper" that behaves like a proxy enabling the applet to invoke an object server on a host other than the one from witch the applet originated.In this case it behaves also like an HTTP server:
prompt> gatekeeper -ORBbackCompat true
(start in the same directory where is the html file).
The "-ORBbackCompat true" option is necessary to have compatibility between gatekeeper 3.0 and previous versions of Visibroker(ex: Visibroker 2.5 on Netscape Communictor 4.0)
6) from the server dirctory start the main server program:
prompt> vbj annserv
7) from the client directory run the browser and load the URL of the Web page "ann.html":
http://YOUR_MACHINE:15000/ann.htm (15000 is the default port of gatekeeper)
8) fill the 5 input fields with float values in the range 0.0 1.0 so that they describe an altimetric profile.
9) press the "RUN" button and see the result on the 5 output fields that have the following means:
mountain - valley - positive_slope - negative_slope - plane

While this program is not trivial it locates in the toys-programs class due to many choices made in order to semplify the description of the global behavior and put the attention to the topic arguments: a neural engine in Java that shows how Java can match well sophisticated tasks and a CORBA client-server solution that shows a beat of the good marriage Java-CORBA.

 NOTE: the listing are not actually published but may be them will be available for downlaod with compiled classes in future.