3D Rendering

The UI library provides the ability to draw hardware-accelerated 3D graphics. As for 2D rendering, this happens on a separate thread (ui.Render) to avoid blocking the ui.Ui thread.

Since a large part of the 3D library (particularly the language extensions) is implemented in Basic Storm, it is located in its own sub-package: ui.s3d so that applications that do not require 3D rendering do not need to load it. s3d is short for "Storm 3D".

The following people have greatly helped developing the 3D rendering in the UI library: Viggo Ingmarsson, Mattis Olevall, Rasmus Ring, Max Ryttersten, Alfred Lindström, Emil Jönsson, Elsa Annell

Painters and Graphics

3D rendering interacts with the windowing system very similarly to the 2D rendering system. This is perhaps not surprising since they share much of the infrastructure for hardware-accelerated rendering. As such, to render 3D graphics in a window, it is necessary to subclass the ui.Painter3D class, override the render member, and attach the painter to a window by calling the void painter(core.Maybe<ui.BasicPainter> to) member in ui.Window. This will cause the UI library to call the render member whenever a new frame should be rendered. Either at a steady framerate, or whenever repaint is called.

The ui.Painter3D class has the following members:

As can be seen from above, the ui.Painter3D class provides a ui.Graphics3D instance that can be used for 3D rendering. In contrast to ui.Graphics used for 2D rendering, the ui.Graphics3D class provides a relatively sparse public interface:

Pipelines

To render 3D geometry through a ui.Graphics3D object it is necessary to define a pipeline. This is done as follows in the language extension that the ui.s3d library provides to the Basic Storm language: 

use ui:s3d;

pipeline ExamplePipeline {
    // Describe the data flow through the pipeline:
    data (Input input, uniform Float time) -> Intermediate -> Output;

    // Other members may be defined here. They become uniforms.
    Transform camera;

    // It is also possible to define arbitrary Basic Storm members. They
    // are only accessible to Storm, not to shaders.
}

As can be seen above, defining a pipeline uses the keyword pipeline followed by a name. This intentionally resembles a class definition, since a pipeline behaves much like a class. The exception is that it contains a data definition, the data members become accessible as uniforms in the shaders. Any member functions are only accessible from Storm, not from shaders.

The first line inside a pipeline is special. It defines the data flow through the pipeline. The data keyword is followed by a set of parentheses that specify which parameters should be passed as inputs to the parameter. The syntax resembles the definition of function parameters. This is also intentional. As we shall see, the types passed here will become parameters to the render function of the pipeline.

There are two types of parameters. Vector inputs and uniforms. The first parameter (Input input) is a vector input. This means that we will give the render function an array of vertices of the type Input stored in a VertexBuffer. The rendering pipeline will then apply the shaders on each element individually (you can think of it as the system adding a for loop for you).

Uniform do not work this way. They are set once, and have the same value across all invocations of the shaders. As such, the system does not expect an array of Floats for the time parameter for example. Rather, it expects a single Float that will be used for all vertices. It is also possible to define uniforms as data members inside the pipeline. As we shall see, they work a little differently to uniforms defined in the parameter list.

After the input definition, there is an arrow -> followed by another vertex type (Intermediate). This specifies the output type of the vertex shader and subsequently the input type of the pixel shader. Finally, there is another arrow -> and another vertex type (Output). This type specifies the output vertex type for the pixel shader.

A pipeline is defined as a class that (indirectly) inherits from ui.Pipeline, and inherits some members from those classes. However, what is relevant is typically the following members generated by the pipeline definitions:

Vertices

The pipeline requires that all inputs, outputs, and uniforms are vertices. In essence, a vertex type is a type that can be manipulated on the GPU. The name stems from the fact that they are typically used to represent vertex data in a 3D mesh (e.g. position, normal, etc.).

The following built-in types are considered primitive vertex types:

It is also possible to define custom composite vertex types using the vertex keyword. As can be seen below, vertex types are similar to class and value definitions. However, there are some limitations, most importantly that all members need to be vertex types themselves.

For example, the input type from the pipeline above may be defined as follows: 

use ui:s3d;

vertex Input {
    Vector pos;
    Color color;
}

At the time of writing, it is not possible to define member functions in vertex types. This may be added in the future. However, as the shader language follows the same lookup rules as Basic Storm, it is possible to call free functions as if they are member functions.

Vertex types automatically generate a set of constructors, both for the shader language and for Basic Storm. These constructors simply accept one parameter for each member of the vertex type and initialize members one by one. There are also constructors with fewer parameters than members. These simply initialize the remaining members to zero.

For example, the expression Input(Vector(1.0, 1.0, 1.0), Color(0.0, 1.0, 0.0)) initializes both pos and color. It is also possible to write Input(Vector(1.0, 1.0, 1.0)) to initialize pos as specified and color to zero (i.e. transparency).

Vertex types also allow marking members based on how they should be used by the rendering pipeline (DirectX call these "semantics"). This is used to (1) use the output from the vertex shader to determine which pixels need to be rendered, (2) determine how to interpolate vertices from the vertex shader as inputs to the pixel shader, and (3) determine which output from the pixel shader represents the color on the screen.

To fullfill (1) and (2) above, the vertex type that is used as an output from the vertex shader has the following requirements:

The following interpolation modes are supported:

For example, the intermediate vertex can be defined as follows: 

use ui:s3d;

vertex Intermediate {
    Vector4 pos : position, perspective;
    Color color : perspective;
}

As can be seen above, pos is marked as both position and perspective for clarity. The color member is only marked as perspective for clarity. Again, this is optional as perspective is the default interpolation.

To fullfill (3) above, the output from the pixel shader needs to have at least one graphics.Color member marked color or output to let the rendering pipeline know which member to paint onto the screen. It is also possible to modify the depth value of the pixel by annotating a member with the depth flag (a core.Float).

As such, the Output vertex type from above can be defined as below: 

use ui:s3d;

vertex Output {
    Color out : color;
}

Shaders

Shaders are defined using a syntax that is similar to ordinary functions in Basic Storm. The main difference is that shaders need to know which pipeline they belong to, so that the system knows which uniforms are available (i.e. shaders are similar to member functions, except for the fact that they are typically defined outside of the pieline itself). Shaders also need to know if they are used as a vertex shader or a pixel shader to check that inputs and outputs match the specification in the pipeline.

All of this is done using a set of keywords before a normal function definition. Vertex shaders start with vertex shader for <pipeline>: and pixel shaders start with pixel shader for <pipeline>:. In both cases, <pipeline> is the name of a pipeline.

The remainder of the shader definition follows the semantics of a Basic Storm function definition. That is: <output> <name>(<parameters>), where <output> is the name of the output type, <name> is the name of the shader, and <parameters> is a comma-separated list of parameters to the shader.

Note that the output and parameter list need to match the data declaration in the associated pipeline. For vertex shaders, this means that the parameter list needs to match the types (but not names) in the parentheses after data. However, any uniform parameters should not be listed to the shader. They are available anyway by virtue of being uniforms. The output type must match the type after the first arrow (->).

For the pixel shader, the input must match the type after the first arrow (->). There may be additional inputs that correspond to quantities that are generated by the system (e.g. current Z depth, but this is not yet implemented). The output type must match the type after the last arrow (->) in the pipeline.

The code in the body of the shader is written in a reduced version of Basic Storm. In general, operations that can be used with the vertex types above are supported, basic control flow (if, for, while), and return, but not much else. The shader code is then compiled to GLSL or HLSL depending on which rendering system is used on the target platform.

Below are two example shader functions that can be used with the ExamplePipeline defined earlier. Note that the uniforms (e.g. camera) can be accessed as if they are global variables. 

use ui:s3d;

vertex shader for ExamplePipeline: Intermediate vertexShader(Input input) {
    Vector4 pos = Vector4(input.pos) * camera;
    Intermediate(pos, input.color);
}

pixel shader for ExamplePipeline: Output pixelShader(Intermediate point) {
    Output(point.color);
}

Shader functions then become available in the name tree. They are exposed as functions that simply return a corresponding shader object. As such, they can be used to create a pipeline as shown below. 

SimplePipeline pipeline = SimplePipeline:create(vertexShader, pixelShader);

It is also possible to define functions that are usable by shaders (shader functions). These differ from vertex and pixel shaders in that they are not associated with any particular pipeline. They are therefore usable from any shader from any pipeline. However, this also means that it is not possible to access uniforms from within shader functions, unless they are passed as parameters to the shader function.

Shader functions are declared by prepending the keyword shader before a Basic Storm function definition. For example, a function that mixes two colors together may be defined as follows: 

use ui:s3d;

shader Color lerp(Color a, Color b, Float t) {
    return a * t + b * (1 - t);
}

Note: it is currently not possible to call neither shaders nor shader functions directly from Basic Storm. This may be possible in the future to simplify testing.

Textures

Textures in the 3D rendering process are represented by the ui.Texture class. Instances of this class represent a texture that is accessible to the GPU, and the associated sampler state (e.g. how to handle wrapping, mip-mapping, etc.).

Even though textures are not vertex types, it is possible to define a texture variable in a pipeline. The pipeline is aware of the ui.Texture class and handles it appropriately.

The ui.Texture class has the following members:

In particular, note that Texture objects are uploaded to the GPU on demand. Replacing the image inside a texture always causes the new image to be uploaded to the GPU. As such, if you are using one pipeline to render multiple textures in a single frame, it is better to create multiple Texture objects and simply assign different ones to the shader rather than replacing the image in the texture.

The graphics.Color sample(core.geometry.Point coords) function can be called both on the CPU and the GPU to sample textures. As such, applying a texture to geometry can be done with a pixel shader that looks like below (assumes that Intermediate has a tex member that stores texture coordinates, and that the pipeline has a texture named texture). 

use ui:s3d;

pixel shader for TexturePipeline: Output pixelShader(Intermediate point) {
    Color texColor = texture.sample(point.tex);
    Output(texColor);
}

Models

The 3D library contains the class ui.s3d.Model, which represents some 3D model that can be rendered. In essence, it wraps an index buffer and a vertex buffer together to provide a standardized representation for exchanging geometry and some rudimentary operations.

The ui.s3d.Model class has the following members:

It also provides the following static member functions to create some standard 3D shapes:

It is also possible to load 3D models from .obj files using the loadObj function:

The model uses the vertex type ui.s3d.Vertex, which has the following members:

Example usage

The tutorials section of the documentation contains a few examples of how to use the 3D part of the UI library to render some simple geometry and apply some simple effects.