Serialization in Storm

This tutorial illustrates how the serialization library works by using two examples. The first example uses the serialization library to save settings for a program. The second example uses the serialization library to implement a simple but robust communication protocol.

The code presented in this tutorial is available in the directory root/tutorials/serialization in the Storm release. There are a few different entry points that can be run from the Basic Storm interactive top-loop:

Saving Settings

In the first part of the tutorial we will use the serialization library to save settings for a fictional program to disk and load them back into memory. We will also explore the abilities of the serialization library to load data saved from earlier versions of a program.

Setup

For this part of the tutorial we will work in a single file named settings.bs. As a start, add the following contents to the file: 

void save() {
    print("TODO");
}

Then open a terminal and change to the directory where you saved the file. You should be able to run the code by typing: 

storm settings.bs -f settings.save

If done successfully, Storm should print TODO and exit.

Data

The next step is to define some data that we can serialize. For the purposes of this tutorial, we create a class Settings that we imagine contains all the settings that our imaginary program wishes to save. As a part of the settings, we also create a class User that stores basic information about the user. We implement the classes as follows: 

class Settings {
    User user = User("Filip", 30);
    Str country = "Sweden";

    void toS(StrBuf to) : override {
        to << "{ user: " << user << ", country: " << country << " }";
    }
}

class User {
    Str name;
    Nat age;

    init(Str name, Nat age) {
        init { name = name; age = age; }
    }

    void toS(StrBuf to) : override {
        to << "{ name: " << name << ", age: " << age << " }";
    }
}

In the Settings class, we use default values for the data members to make it easier to create instances of the Settings object with interesting contents. This may or may not be suitable in real applications.

Serialization

To save the contents of the two classes, we first need to make them serializable. Typically, this is done by adding the serializable decorator to the affected classes. The logic for this decorator resides in the package util:serialize, so that package needs to be included as well: 

use util:serialize;

class Settings : serializable {
    // ...
}

class User : serializable {
    // ...
}

After that, we can turn our attention to the actual serialization logic. To serialize objects, we need to use the ObjOStream class (for object output stream). As with the text streams in the previous tutorial, the ObjOStream accepts an OStream as a parameter to its constructor that specifies where data should be stored.

After creating an ObjOStream, we can write objects to them by calling the member function write in the object we wish to store. This function is generated automatically by the serializable decorator we added earlier.

All in all, we can implement the serialization in the save function as follows: 

use core:io; // Add to the top of the file

Url settingsUrl() {
    return cwdUrl / "settings.dat";
}

void save() {
    Settings settings; // Object we wish to save.

    Url saveTo = settingsUrl();
    ObjOStream out(saveTo.write()); // Create an ObjOStream
    settings.write(out);            // Write the object
    out.close();                    // Close the stream.

    print("Saved to: ${saveTo}");
}

We can run the program by typing storm settings.bs -f settings.save in the terminal. This serializes the settings object into the file settings.dat in the current working directory (the program outputs the path as well).

Inspecting the Serialized Data

If you open the settings.dat file you serialized previously, you will see that it uses a binary format. We do, however, see some strings in the file that contains names of the types that were stored. In fact, the serialized data contains enough information to deserialize the serialized data without access to the original class definitions. It is possible to use this data to convert the binary stream into a textual, human-readable format using the class TextObjStream that is implemented in the util.serialize package.

We can use the stream to read our serialized data as follows: 

void text() {
    Url loadFrom = settingsUrl();
    TextObjStream in(loadFrom.read());

    Str textRepr = in.read();
    print("Settings as text:");
    print(textRepr);
}

If we run the program by typing storm settings.bs -f settings.text in the terminal, we will get the following output (note that the class names are different if you use the code in the Storm release): 

settings.Settings (instance 0) {
    user: settings.User (instance 1) {
        name: "Filip"
        age: 30n
    }
    country: "Sweden"
}

Objects are represented by a class name followed by a pair of curly braces that denotes the contents of the object. For class types, we also see (instance N) that indicates which instance of the class is being defined. This is sometimes used by the textual representation to refer back to previously stored objects, in case the object graph contains multiple references to the same object.

Deserialization

Now that we know that our data was serialized properly, we can deserialize the data into a Settings object again. We do this using the ObjIStream object together with the static read function that was generated by the serializable decorator. Similarly to when serializing, we first open the file as a plain IStream, which we then pass to the constructor of the ObjIStream: 

void load() {
    Url loadFrom = settingsUrl();
    ObjIStream in(loadFrom.read());

    print("Loading settings from: ${loadFrom}...");

    try {
        Settings settings = Settings:read(in);
        print("Success: ${settings}");
    } catch (SerializationError e) {
        print("Failed: ${e.message}");
    }
}

If we run the program by typing storm settings.bs -f settings.load in the terminal, we will get the following output: 

Loading settings from: /home/storm/settings.dat...
Success: { user: { name: Filip, age: 30 }, country: Sweden }

Changing the Settings Class

The serialization library is able to support changes to classes to a certain degree. This means that it is possible to load serialized data from an old version of the system, even if changes were made to the serialized classes.

To illustrate this, let's assume that we have developed our imaginary program further and we wish to add a new setting for the user's preferred language. We can add the setting by modifying the Settings class as follows: 

class Settings : serializable {
    User user = User("Filip", 30);
    Str country = "Sweden";
    Str language;

    void toS(StrBuf to) : override {
        to << "{ user: " << user << ", country: " << country << ", language: " << language << " }";
    }
}

If we run the load function at this time, deserialization will fail with the following message: 

Failed: The member language, required for type Settings, is not present in the
stream and has no default value in the source code.

As indicated by the message, the serialization library has realized that a new member was added to the Settings class compared to the time when the data file was serialized. The message also gives a hint that it is possible to solve the issue by specifying a default value. Let's try to specify a default value for language as follows: 

Str language = "Swedish";

If we run the load function again at this point, the serialization library uses the default value to fill in the missing value from the serialized representation, and we get the following output: 

Loading settings from: /home/storm/settings.dat...
Success: { user: { name: Filip, age: 30 }, country: Sweden, language: Swedish }

It is possible to remove members from classes as well. It is, however, worthwile to be a bit careful with renaming data members. The serialization library uses names of member variables to match data members, and renaming a data member is thus treated as the removal of the old name and the addition of the new name. This means that the user will lose the value of the renamed setting.

Complex Data

The serialization library takes care to preserve the structure of the object graph during serialization. In particular, if multiple variables in the serialized representation refer to the same instance of an object, the object will not be duplicated, and the deserialized representation will preserve this property.

To illustrate this, let's add an alternate user identity to the Settings class and make the two refer to the same User instance. We also modify the toS function to indicate if user and alternate refer to the same object or not. 

class Settings : serializable {
    User user;
    User alternate;
    Str country = "Sweden";

    init() {
        User tmp("Filip", 30);
        init {
            user = tmp;
            alternate = tmp;
        }
    }

    void toS(StrBuf to) : override {
        to << "{ user: " << user << ", alternate: ";
        if (user is alternate) {
            to << "(same as user)";
        } else {
            to << alternate;
        }
        to << ", country: " << country << " }";
    }
}

We can now serialize the new object using storm settings.bs -f settings.save, and then inspect the textual representation of the serialized data using storm settings.bs -f settings.text. The textual representation shows the fact that user and alternate refer to the same object as follows: 

settings.Settings (instance 0) {
    user: settings.User (instance 1) {
        name: "Filip"
        age: 30n
    }
    alternate: <link to instance 1>
    country: "Sweden"
}

As we can see, the textual representation shows the value of alternate as <link to instance 1>. If we look at the value for user, we can see that it has been labeled as (instance 1). This shows that the fact that they are the same instance has been preserved in the serialized representation as well.

Of course we can deserialize the data again to verify that the deserialization logic behaves in the same way. If we run storm settings.bs -f settings.load, we will see the following output that verifies that it is indeed the case since it prints (same as user) as the value for alternate: 

Success: { user: { name: Filip, age: 30 }, alternate: (same as user), country: Sweden }

Communication Protocol

As mentioned in the top of this page, it is also possible to use the serialization library to conveniently implement a network protocol. In this part of the tutorial, we will use the protocol within a single process, but the same idea can be extended to communication between different machines.

Setup

For this part of the tutorial we will work in a single file named protocol.bs. As a start, we add the function that we will use as the entry-point to the file: 

use core:io;
use util:serialize;

void main() {
    print("TODO");
}

Then open a terminal and change to the directory where you saved the file. You should be able to run it by typing storm protocol.bs. If done successfully, it should print TODO and then exit.

Requests and Responses

This protocol will be a simple request-response protocol. The client sends a request to the server, and assumes that each request will eventually produce a response of some type. It is, of course, possible to extend the idea presented here to a more complex scheme as well. It does, however, introduce more complexity in the implementation.

We model requests using a class that we call Request. To make it easy for the server to determine how to respond to each request, we define an abstract function execute in the Request class that implements the behavior that the server should execute. We let the execute function receive a reference to a Server object as well, so that it may access the state that is present in the server: 

class Request : serializable {
    Response execute(Server server) : abstract;
}

Since responses belong to messages, we do not need the same dispatch logic there. As such, we represent responses as an empty class that we can then inherit from to add additional data in the responses: 

class Response : serializable {
}

The Server

Now that we have an idea of how we will model requests and responses, we can start implementing the server. We will represent the server as a class that contains the state managed by the server. In our case, we implement a server that simply stores a list of strings. It implements two requests, one for adding an element to the list, and one for retrieving the current contents of the list.

With this in mind, we can start implementing the server class as follows: 

class Server {
    Str[] data;
}

Now that we know how the data is stored inside the server, we can implement the request that adds a string to the list as follows. Note that we need to store any data that should be sent from the client to the server as members inside the class: 

class AddRequest : extends Request, serializable {
    Str toAdd;

    init(Str toAdd) {
        init { toAdd = toAdd; }
    }

    Response execute(Server server) : override {
        server.data.push(toAdd);
        return Response();
    }
}

As we can see, we can implement the logic for how to handle the request in the execute function. This will make it possible to implement the logic in the Server class by simply calling execute on the received Request. This also makes it easy to extend the server with new messages in the future, without having to modify the Server class itself.

In a similar way, we can implement the request for retrieving the list as follows. Since this request needs to return some data in its response, we also need to define a subclass to Response that stores the data we wish to return: 

class GetRequest : extends Request, serializable {
    Response execute(Server server) {
        return GetResponse(server.data);
    }
}

class GetResponse : extends Response, serializable {
    Str[] data;

    init(Str[] data) {
        init { data = data; }
    }
}

Since the actual logic for handling each request is implemented in the corresponding Request class, all that remains is the logic for deserializing requests and serializing the responses. We implement this as a loop in the Server class as follows: 

class Server {
    Str[] data;

    void run(IStream read, OStream write) {
        ObjIStream input(read);
        ObjOStream output(BufferedOStream(write));

        try {
            do {
                Request request = Request:read(input);
                Response response = request.execute(this);
                response.write(output);
                output.flush();
            }
        } catch (EndOfStream e) {
            // End of stream reached, this is normal.
        } catch (SerializationError e) {
            print("Serialization error: ${e.message}");
        }

        input.close();
        output.close();
    }
}

As we can see, the central parts of the server logic is inside the do loop inside the try block. This loop simply reads a message by calling Request:read(input), executes the logic associated with the message by calling request.execute(this), and finally sending the response back to the client using response.write(output).

When the client closes their end of the input stream, deserialization will fail with the EndOfStream exception. As such, the code above catches the exception and allows execution to continue normally in this case. Note that EndOfStream is a subclass of SerializationError, that is used to report other forms of serialization errors. This means that the order of the catch clauses are important in the code above.

A detail worth mentioning in the code above is the use of a BufferedOStream when creating the ObjOStream. The BufferedOStream acts as a layer between the ObjOStream and the OStream that collects the small writes that are performed by the ObjOStream and only writes them when the internal buffer is full or when flush is called. This is technically not necessary in this example, since we will send data between two threads in the same system. However, this buffering is often very beneficial when sending data over the network, especially if encryption is used, since there is much more overhead involved in network transmissions.

The Client

Now that we have a server, we are ready to implement the client. We implement the client as a class that contains two object streams, one for input and one for output. The class then contains a function called request that sends a request to the server and waits for a response. We can achieve this as follows: 

class Client {
    private ObjIStream input;
    private ObjOStream output;

    init(IStream read, OStream write) {
        init {
            input(read);
            output(BufferedOStream(write));
        }
    }

    Response request(Request request) {
        request.write(output);
        output.flush();
        return Response:read(input);
    }

    void close() {
        input.close();
        output.close();
    }
}

As with the server, we use a BufferedOStream to buffer writes to the output stream. We can also see that the request function is fairly simple. It starts by writing the request object to the output stream, flushes the stream (to inform the BufferedOStream that it needs to send any buffered data immediately), and reads a response from the input stream.

We can use the Client class to interact with the server as follows: 

void runClient(IStream read, OStream write) {
    Client client(read, write);

    client.request(AddRequest("a"));
    client.request(AddRequest("b"));

    if (response = client.request(GetRequest()) as GetResponse) {
        print("Final data: ${response.data}");
    } else {
        print("Invalid response.");
    }

    client.close();
}

The code above sends two AddRequests to add two elements to the array in the server. After that it asks the server for the contents of the array with a GetRequest. At this point we are interested in inspecting the result from the request function. As such, we first cast it into a GetResponse using a weak cast, and then print the data member of the response.

Connecting the Server and the Client

Finally, we wish to connect the server and the client we have written. In this tutorial we simulate the network connection using the Pipe class. We create two pipes, one for moving data from the client to the server, and another for moving data from the server to the client. We then spawn two user threads, one for the client and one for the server, and wait for them to finish: 

void main() {
    Pipe a;
    Pipe b;
    var server = {
        Server server;
        spawn server.run(a.input, b.output);
    };
    var client = {
        spawn runClient(b.input, a.output);
    };

    // Wait for both to finish.
    server.result();
    client.result();
}

Now, we can run the program by typing main protocol.bs in the terminal. If everything is done correctly, it should produce: 

Final data: [a, b]

If the program appears to freeze, the reason is likely that some part of the code failed to compile, but the exception containing the error message is caught in the future for the client, but is not shown since the program is waiting for the server to complete. To see the error, you can ask Storm to compile the program before launching the server or the client by adding the following line at the start of main, and add use lang:bs:macro; to the top of the file: 

named{}.compile();

Security Considerations

Since the program above sends entire classes from the client to the server, it might appear that it is possible to make the server execute arbitrary code by simply sending it a new class with the desired code in the execute function. This is however not possible. The serialized data only contains the names of classes and their data members. No code is transmitted in the serialized representation. As such, if the client were to send a serialized version of a class that is not present in the server, serialization will simply fail with an exception.

One consideration when working with potentially untrusted data from the network is, however, what happens in cases where the client (either intentionally or unintentionally) sends large data structures to the server. This can cause the server to exhaust its memory, which could cause it to start performing poorly. To guard against this type of behaviors, it is useful to set maxReadSize and maxArraySize in the ObjIStream to sensible values in order to limit the memory used by the deserialized objects. The default values of these variables are large in order to not cause issues when working with trusted data. When working with potentially untrusted data it is, however, reasonable to set them to a couple of megabytes instead.