Message Protocol
We wanted to set a goal that the messages, that would be sent between the
client and server, had an as transparent protocol as possible. We wanted to
obtain a framework that allows us to write one protocol definition
that would be used both client- and serverside, in order to avoid protocol
dependancies between these sides. Therefore the determination of the type of the message,
the marshalling and the unmarshalling should be done by this framework itself.
We decided to use Sun's ObjectOutputStream/ObjectInputStream
for marshalling and unmarshalling the messages, and we created
{@link quest.global.net.Message} as the base class for the network
messages. This way only objects of type Message
could be sent,
and the Message
implementation of the toString
method
would facilitate debugging. Combined with creation of inner classes
of the subclass that override Message, a message hierarchy can be constructed that
defines the type of a certain object message is. This means you can recognize the
message by looking at it's Java type.
The advantages are clear: creating a new subclass of message (e.g. for a new
process in the server, such as a AI part) requires only the extension of
Message
to define a new subtype. By creating all the required messages
as inner classes in this new subclass, you can create a new protocol that can still
be handled by the server in the same way as the other protocols, since the server
only knows about receiving this kind of Message
objects.
Combined with the "Daemons and processes"
approach it results in a rubust message passing environment.
Daemons and processes
To handle the incoming messages at the client- and serverside, we had to come
up with a flexible structure that handles the messages. Since we were able to
determine the type of a message, we came up with a structure in which processes
could register themself at a deamon by specifying the type of messages they
wanted to receive. This corresponds to an extended version of the Observer
pattern, since you're not only registring you want to receive updates
(incoming messages), but also what kind of updates (the type of the messages).
The final framework is constructed as follows.
Network layer objectstreams
For the basic network layer messagestream we use the standard
{@link java.io.ObjectInputStream} and {@link java.io.ObjectOutputStream}.
This provides us with the ability to send and receive normal Java objects.
This however needs every object to be serializable, what, with e.g.
the Observable class, turned out the be a bit of a problem (see
the section encountered problems for more
information about this).
Connections
For the connection between the server and the clients, both sides have a
single daemon who handles this. The {@link quest.client.net.ClientDaemon} and
the {@link quest.server.net.ServerDaemon} maintain a permanent connection with
an input and output connection on top of that. Both these deamons are
Singletons.
The server- and clientdaemon both run their own messagelistener.
Clientside it listens constantly for messages of the server and forwards
them to the appropriate service. Serverside it checks every connection with
a client whether data is available. If data is available the message is received
and distributed to the appropriate service.
This way the network layer offers a flexible messagestream which every service
can use for communication.
Processes
Now that the basic communication is set up, additional services can be added.
These services are called processes, and are defined in {@link quest.global.net.Proces}.
Every service, like a chatsystem or a gamesystem, must register
itself as a proces both server- and clientside. This way it registers the
messagetype it uses for communication between server and clients and the
server- and clientlisteners it offers for the handling of the messages.
The server- and clientdaemon now know that when they receive a message of
the registered type, they should call the appropriate listener for it.
Every new service can be built on top of the network layer by simply constructing a
Proces
object, with a message type that extends
{@link quest.global.net.Message} and a listener which implements
{@link quest.global.net.MessageServerListener} or {@link quest.global.net.MessageClientListener}.
This process should register itself at the server- and clientdaemon and the
service is ready.
A process can be registerred as USER_BASED or GLOBAL_BASED.
In short this means that when it's USER_BASED every client needs to be
registerred seperately to be allowed to communicate with that service. When
it's GLOBAL_BASED registration is not needed, so that all users who can communicate
with the main service can also communicate with the subservices. This last
possibility is mainly used for subservices like the three added services which
help the chatserver perform his tasks.
Listeners
Every process needs to offer a listener for the message it has registered for
communication. The interface for these listeners are defined by
{@link quest.global.net.MessageServerListener} and
{@link quest.global.net.MessageClientListener}.
Each of these listeners is then supposed to handle the messages it
receives from the daemons. When it is finished with handling and has constructed
the reply messages it hands these to the daemons, together with a list of usernames
whom the messages should be sent to, who take care of the
communication.
This way the network layer is transparent for the services and
the services are able to run on top of the network layer
as long as it follows the guidelines.
Authentication
When a client first connects it must follow the authentication guidelines
described in {@link quest.server.login}. This way the serverdaemon receives an
indication that the user is authenticated and is now allowed to communicate with
the services running on top of the network layer. Until the login procedure
is finished the client isn't allowed access to the services.
Because the serverdaemon knows nothing about the services, except how to call
them the services know a little about eachother for the added security.
This way the services can inform the serverdaemon when the client needs to
talk to another service.
For example, the services included in Amazing Quest do the following:
- The loginmanager informs the serverdaemon when the authentication was
succesful, that the client is now allowed to communicate with the chatserver.
This is done on a user-based level.
- The chatserver informs the serverdaemon that all clients who are allowed to
communicate with the chatserver, can also communicate with the channelserver, rankingserver and searchserver. This is done because these are servers which
help the chatserver in performing his task, so it's up to the chatserver to
register these processes and the give the client the ability to communicate
with them.
This is done on a global-based level.
- The chatserver informs the serverdaemon when a game is started, that the clients who participate in the game, are now allowed to communicate with the gameserver.
- The chatserver informs the serverdaemon when a game is finished, that the clients who participated in the game, should now be denied access to the gameserver.
- The loginmanager informs the serverdaemon when a client logged out, that the client should now be denied access to all services.
This way access to the services is strictly controlled for added security.
Databases
Both the server- and clientdaemon maintain small tables of the processes
registered at their side. The serverdaemon also maintains a table of online users,
including the messagestream setup with them and the services they are allowed to
communicate with.
The loginmanager uses a connection with the quest database on a SQL server
through {@link quest.server.util.QuestDatabase} and {@link quest.server.util.MYSQLCommunicator}.
In the SQL database all user information like username, password, number of played games and
ranking are stored.
Logging
All services are allowed to write to the logfile. This way they can inform the
administrators of the program about there status and errors.
All that needs to be done is to get an instance of the {@link quest.server.log.LogManager} and with this the service can write entries to the logfile.
The logmanager makes sure that every logentry is in a standard format including
the date and time of the entry, for easy reading.
Technical problems
Observable
was not serializable
Java has two standard classes to implement the Observer pattern:
Observer
and Observable
. Observers can register
themself to an Observable
, which notifies all registered
observers when it has changed some internal data. By inheriting
from this Observable
class, a user can create observable
software components.
The registration of the observers is internally done by Observable
in a private Vector
. The problem is that Sun for some
reason didn't make Observable
serializable. This means you
can't create a subclass of Observable
that can be serialized,
since the private Vector
that contains all the observers
is lost in the serialization process.
This posed a big problem, since the Observer pattern was used extensivily
in the package {@link quest.global.game}, which implements the logic
of an Amazing Quest game. The root object that defines a complete game
({@link quest.global.game.GameLogic}) is constructed at the game server,
and then sent to all the clients. Because all the registered observers
inside that object are lost during the serialization and deserialization
of the observable parts, clients received a crippled version of the
object.
We made a workaround, {@link quest.global.util.SerializableObservable},
that extends Observable
by making it serializable. This is
done by providing the same functionality as the base class, but saving
the data (the registered observers) also in the derived class. This redundant
data is correctly serialized. During the deserialization process, the
SerializableObservable
restores the data in the base class
with its redundant information. This way the registred observers were
not lost, and the code could still use the Java Observer
class.
ObjectInputStream
issues
We had to make a design decission of how to internally support multiple concurrent
connections with the server. The decission we made afterwards posed that it was not
possible to detect client timeouts.
The default Java (Sun) paradigm is to use a single thread per listening client.
At that time we found that paradigm outrageous because the server would have to
support possibly hundreds to thousands of clients, and that would produce much
threading overhead. This threading overhead could of course be minimalized if
the host operating system's scheduler is sophisticated enough. Because we didn't
want to depend on the operating system that much in order to let the server
function reasonably fast, we chose to use a single thread for receiving the
messages, which spawns seperate threads for handling the responses.
This way we had to rely on the available
method of
ObjectInputStream
to return the amount of bytes that could be
read from the input stream. The problem was that this method
didn't return non-zero when the other side had (wanted to) sent an object over this connection.
Possible because ObjectInputstream
and ObjectOutputstream
classes use a rendez-vous scheme that has to be hidden from the user. We circumvented
this problem by using the SocketInputstream the ObjectInputStream was based on. Here the
available
method worked reliably.
Also it occurred to us that it wasn't possible to determine whether a connection
was terminated, using the available
method. This was a serious problem
because the termination could only be determined by sending or trying to receive
a message or object. We found it strange that in the available
method the
IOException
exception was declared in the API, but never thrown. The
documentation wasn't clear about when this exception would be raised. A glance at
Sun's Java forum at java.sun.com made clear out of numerous developer discussions that
Sun didn't (want to) actively support the single-thread solution. Because the only solution
to the single-thread problem would be a hack, Sun will probably succeed in
enforcing their thread-per-client solution.
We generated the following solutions to this problem:
- Sending a heartbeat between the connections.
- Only block for a small amount of time(time spent on this operation is unreliable).
- Using some sort of leases: sending a keepalive packet after a period of inactivity.
- Using the thread-per-client method anyway (takes lot of time).
- Ignore it until later.
We chose to lower the priority of this problem. The perceived hinder would not
crash the clients, hence it wasn't crucial.
Blending textures in Java3D
The 3D maze that makes up the game board contains a number of treasures
that are identified by their color and symbol. This is implemented
by using a black-and-white texture for each symbol, and mapping this
onto a 3D box (which represents a treasure) with a certain color.
By blending the texture with the color of the box, we needed only
one texture for each symbol, instead of needing a seperate texture
for each combination of a color and a symbol.
The problem is that the same blending behaviors differently on various
execution platforms. In Windows98, the blending results in black boxes
with a colored symbol on it. In Windows NT, the same code results in
colored boxed with a black symbol on it. So much for platform-independency.
Since this doesn't hinder the game from being played (you can still recognize
the treasures) we just accept this behavior of Java3D.
Starting the applet in Netscape
Loading of the applet in netscape under SunOS resulted in a ModifyThreadGroup
exception when the classpath included the path where the classes resided.
Typically when including '.' in the classpath and starting netscape in the root
of the quest directory, Java produced a couple of SecurityExceptions.
The simple solution was to just start netscape in another directory,
the real reason why netscape did value the CLASSPATH environment setting over
the Java plugin settings couldn't be determined.
Development environment
Java3D was only runnable on one machine for a long time
To run Java3D programs it has to be installed, of course. The Windows NT
machines didn't have a decent Java installation at all until the start
of the 2001/2002 college year, so we had to test everything on UNIX
machines. Getting the JavaPlugin to work under UNIX Netscape was an
exercise on it's own, by the way.
The problem was that Java3D program only ran on hardware
OpenGL accelerated X-servers, since the installed OpenGL software libraries
where too old to be used by Java3D. There was only one machine available
for students with a special 3D graphics card (the one the student assistant room).
Since the game was meant to run with multiple client, testing the behaviour
of the program with multiple clients became almost impossible.
We discovered that by using Exceed on the Windows NT machines (which are
all installed with a 3D graphics card) we could use the Java installation
of UNIX and the OpenGL capabilities of these computers. This method was
still not very satisfactory, since the Exceed-layer in between made the system
very slow, and a lot of fonts and keybindings weren't available this way.
This crippled the program and the testing of it a lot.
At the start of the new college year, all the Windows NT machines were installed
with the latest JDK and Java3D version, so finally we got a decent testing
environment. Much of the testing of multiple clients is therefore done in the last
week before the deadline, so we weren't able remove all the bugs involving
multiple clients.
JavaPlugin+Java3D is not really stable
We experienced a lot of crashes during the testing. Sometimes the JavaPlugin
just freezes, and the runtime system of Java3D also crashes sometimes.
It even made Windows NT crash a number of times, which isn't done very easily.
Since it is still being developed, the relativily unstable nature of Java3D
is understandable, but irritating sometimes. Especially the testing of
multiple clients was hindered by it, since once a while a client just crashed.
The database/webserver was pretty slow
The only available webserver we could get shell permission on to run a server
for our game was the raderboot. We needed a shell account to run the quest server
on because of a java applet sandbox restriction: the applet can only open
connections with the server where the codebase resides. The machine on which the
webserver runs is a SPARCstation 5 with a 70Mhz processor and 64Mb of physical RAM.
Furthermore a practical course 'Inleiding Gegevensverwerking'(IGV) was
being held in spring 2001 on the same machine, on which participants used the
MySQL server also operating there. On busy days it took for ages to retrieve the
applet over the webserver and to start the server. Using an external server (out
of reach of the NFS we used) would introduce file synchronization problems and
an upload delay. Because of this we chose to use the raderboot nevertheless as
webserver.
Learning curve
Java3D
The interface of the game, which uses Java3D, was made by Mathijs den Burger:
"Understanding how to work with Java3D took some time, but the online
manual [Java3Dman] was very helpful. The first
challenge was to show some 3D object at all. You have to create a
standard scene graph part which defines how the user views the scene
before Java3D will display your 3D world. The demo's that come
with Java3D mostly use the default SimpleUniverseManager to handle that
part automatically. This class hides a lot of things I wanted to
control myself (such as placing the viewer at a certain position in the
3D world), so it took a while before I understood how to do this part
myself and was able to show simple 3D objects in a window.
The theory of the scene graph was pretty clear, but using it was a
different story. When I understood this part, building the static
scene was possible. The next difficult part were the animations:
most examples that come with Java3D perform a continues animation (like spinning a cube),
but the program would have to perform a different animation each time
(e.g. to let the user walk through the maze). This was solved by
generating the desired behaviour at runtime, and adding it to the
scene graph.
The last problem was to select objects in the 3D scene. The basic
selection was possible by using the classes about picking in Java3D.
This I used in a little framework for selecting 3D objects, by defining
PickableObjects
about which a PickBehavior
could dispatch PickEvents
to interested PickListeners
.
This provided a very useful way of handling the selection of 3D objects.
A lot of Java3D's functionality isn't used in this game. Especially the methods to
load complex 3D models and display all kinds of fancy 3D objects
aren't used. But this practical work wasn't about 3D modeling of course."
Swing & building GUIs
The interface of the login procedure and chat environment was made
by Sander Brandenburg:
"Because I've worked with Swing in the practical course
"Software Engineering" there wasn't much new to me. Though this time I used
much of the nifty 'Document' features that visualize panels on which you can
draw colored or styled text etc. The biggest problem was how to structure the
funcionality of the GUI in such a way that it was easy for the components
to communicate with eachother. Though the
search, help and ranking overviews are quite distinct features, the chat,
globalchat and channeloverview had to communicate with eachother in order to
decide whether to enable or disable certain actions (buttons). The Singleton
design pattern aided me in solving this, because all windows had to be
instantiated only once.
Most of the GUI work was done using GridBagLayout as the layoutmanager, and
emacs as editing tool. Because GridBagLayout offers lots of flexibility over
bunches of variables, it was hard to design the GUI in text mode and evaluate
them in Java, which takes a while to load. Still, sometimes a frame was layout
differently (read: strangely) in CDE X, KDE2 X, Exceed under Windows NT and
plain Windows NT, which gave me a hard time."
Handling multiple threads
The central server was made by Merijn Evertse:
"Because the goal was to design a server application that would be able to handle a
virtually unlimited amount of clients it soon was clear that it had to be multi-threaded.
You want to make sure that clients do not have to wait substantially for eachother.
So I started out by running every message receiving, sending and handling action in a
seperate thread. At first this seemed to work but already with two clients problems started to
occur and the order of the messages was sometimes total chaos. Different race conditions occured
for instance in the message handling.
This was solved by making the handling of each client-to-server message inside the services, and
the access point at the ServerDaemon for sending messages synchronous
.
After that I put a lot of time in defining the transparent border between the services and the
ServerDaemon, by using the different pattern. This way the several services were cleanly seperated
which made it easier to distanciate the threads from eachother, which for instance made it possible to
construct the synchronous
access points".
References