Make your own free website on Tripod.com

8 Design

This section describes the software design for the simulation of visual impairments and is a development of the specification in Section 6. The design includes:

8.1 The Simulation Window

8.1.1 Window Management

The simulation used a single window that was created using the GLUT library functions described in Section 7.7.1. A full screen window was requested using glutFullScreen(). The window used double buffering (Section 7.1.4) and the RGBA colour mode (Section 7.2.1). The display callback function registered with the GLUT event loop (Section 7.7.2) was used to paint the contents of the window.

8.1.2 Setting up Two-Dimensional Viewing

Orthographic (parallel) projection (Section 7.1.3) was used to set up two-dimensional viewing. The viewport was set to the size of the window. The projection matrix and viewport were re-specified each time the window was resized:

// Specify a 2D-parallel projection.
glMatrixMode(GL_PROJECTION);
glLoadIdentity();
gluOrtho2D(0.0, WindowWidth, 0.0, WindowHeight);
glMatrixMode(GL_MODELVIEW);
glViewport(0, 0, WindowWidth, WindowHeight);

8.2 Rendering the Window Background

The rendering of the window background was common to all the impairments that were simulated. The following sub-sections describe the options that were considered:

8.2.1 Transparent Window

In order to show the displayed output of other applications running below the simulation window, it would have been desirable to make the window transparent and then draw the impairments within the window. However, it is not possible to render an OpenGL window with a transparent background.

8.2.2 Background Image

A bitmap image could have been loaded by the simulation and used as the background for drawing the impairments. Different images such as landscapes, portraits and text would have allowed the impairments to be simulated at different focal distances (Req 2.2). This is the technique used by existing simulations (Section 2.16).

8.2.3 OpenGL Rendered Scene

The effects of the visual impairments could have been applied after modelling a scene using OpenGL. A three-dimensional scene would have allowed the effects of distance to be simulated.

8.2.4 Screen Capture

When an OpenGL window is created, a copy of the pixels below the window are placed in the frame buffer so that the window initially appears transparent. The screen image can be captured by copying the contents of the frame buffer to a pixel array using glReadPixels() before any rendering takes place within the window. The array of pixels can then be copied back to the frame buffer using glDrawPixels() each time the window is re-painted.

8.2.5 Chosen Background

Two modes of operation were provided for the creation of the background image. A transparent window effect was created by capturing an image of the screen and then applying the simulated impairments. To simulate the use of a computer with a visual impairment (Req 4), a simple drawing program implemented in OpenGL was also used as the background for simulating the impairments. The operation of the drawing program is described in Section 8.11. In both cases, the simulated visual impairments were applied to a two-dimensional image with no available depth information, so varying the focal length of the vision (Req 3.4) was not simulated.

8.3 Myopia / Hyperopia

The simulations of myopia and hyperopia (Req 1.1 and Req 1.2) were both simulated by blurring the background image and were grouped together as a single impairment in the simulation.

8.3.1 Creating the Blur Effect

A convolution filter was applied to the image to achieve a blur effect (Section 7.4). The following 3x3 kernel was applied to the image to place a higher emphasis on the pixels immediately above, below and to the right and left of the centre pixel [36]:

1

2

1
2 1 2
1 2 1

Figure 8.1 - 3x3 Convolution Kernel for a Blur Effect

The convolution operation was performed using the accumulation buffer. Three stages were required:

1. The accumulation buffer was first cleared using glClearAccum(0.0, 0.0, 0.0, 1.0) and glClear(GL_ACCUM_BUFFER_BIT).

2. The image was then accumulated in the buffer. For each kernel entry (i, j), the input image was translated by (-i, -j) from its original position and redrawn in the colour buffer. The contents of the colour buffer were then added to the accumulation buffer using glAccum(GL_ACCUM, (BlurFilter[i][j])/13).

The resulting image was then copied back to the colour buffer for display with glAccum(GL_RETURN, 1.0).

The accumulation buffer has a limited precision. Colour values are clamped in the range 0.0 to 1.0. To avoid colour saturation each time the image was added to the buffer, the kernel values in Figure 8.1 were modified by dividing each one by the total of all the kernel values [36].

8.3.2 Increasing the Severity

To achieve a greater effect, the 3x3 kernel was applied multiple times in successive passes. The number of passes for each severity is shown in Table 8.1

Severity  Number of Passes
Very Low 1
Low 2
Medium 3
High 4
Very High 5

Table 8.1 - Number of Passes of the Blur Filter

8.4 Glaucoma / Retinitis Pigmentosa

Glaucoma and Retinitis Pigmentosa both result in tunnel vision (Req 1.3 and Req 1.4) and were grouped together as a single impairment in the simulation. The tunnel effect was simulated by obscuring part of the screen. The unobscured area of the display was restricted to a circle centred on the centre of vision.

8.4.1 Initial Design

Each pixel of the display was associated with an x and y value. The position of the centre of vision was stored using the same coordinate system. The distance of each pixel from the centre of vision was calculated using Pythagoras' Theorem:

Centre of Vision

Figure 8.2 - Calculating the Distance from the Centre of Vision

To create the 'tunnel', the background was first rendered, then each pixel of the display was either coloured black, or left unchanged depending whether the distance from the centre of vision exceeded the radius of the tunnel. The result was a circle with a sharp edge (Figure 8.3.a). To achieve a more realistic effect, the edges of the tunnel were progressively blended into the background. (Figure 8.3.b).

Tunnel with no blendingTunnel with blending

Figure 8.3 (a) Tunnel with no Blending (b) Blended Tunnel

For each pixel, an alpha colour component (Section 7.2.5) was calculated based on the distance from the centre of vision. The region outside the circular tunnel was given an alpha value of 1.0 and was completely opaque. A second, inner circle was given an alpha value of 0.0 and was completely transparent. Each pixel in the region between the two circles had an alpha component which was calculated using Equation 8.1:

alpha = (distance from centre - inner radius) / (outer radius - inner radius);

Equation 8.1 - Calculating the Alpha Component

8.4.2 Improved Design

The circular tunnel was bounded by a square. To improve performance when updating the display, only the region within this bounding square needed to be drawn. Rendering was restricted to the bounding square by using a scissor test (Section 7.1.5). Black was used to clear the colour buffer before performing the scissor test. Pixels in the area outside the scissor box remained black when the background image was rendered to give the impression that the image was obscured.

Glaucoma Initial Design

Figure 8.4 - Simulating glaucoma / Retinitis Pigmentosa
Initial Design

8.4.3 Alternative Design

The area to be rendered was restricted to a circular area of the display by using a stencil test instead of the scissor test. A stencil pattern was created in the stencil buffer (Section 7.1.5) by drawing an approximation of a filled circle using a polygon (Section 7.1.2). The coordinates of each vertex of the circle (x, y) were calculated using the trigonometric equations:

x = radius * sin (angle);
y = radius * cos (angle);

Equation 8.2 - Approximating a Circle Using a Polygon

The value of the angle was incremented by 0.1 radians for each vertex of the circle. The blended region of the tunnel was drawn as a black ring using a quad strip with blending and smooth shading enabled (Section 7.2.3). Each segment of the ring had two vertices (x1, y1) and (x2, y2). The outer vertex was drawn black with an alpha value of 1.0. The inner vertex was drawn black with an alpha value of 0.0. The OpenGL shading model produced a smooth transition between the inner and outer vertices from opaque to transparent. The region inside the ring was unobscured.

8.4.4 Final Design

The stencil test had the extra overhead of creating the stencil pattern in the stencil buffer each time the display was updated. The blended region of the tunnel was drawn as described in Section 8.4.3. The scissor test was used instead of the stencil test to restrict the rendering to a rectangle bounding the quad strip. The region between the outer edge of the quad strip and the edge of the scissor box was rendered black to obscure the background image. A second black ring was drawn outside the first ring using another quad strip.

Glaucoma Final Design

Figure 8.5 - Simulating Glaucoma / Retinitis Pigmentosa
Final Design

8.4.5 Dimensions

The size of the unobscured area of the screen was decreased as the severity was increased. To account for varying screen sizes, the inner and outer radii of the blended region of the tunnel were calculated from the window dimensions. The dimensions of the scissor box were defined by the diameter of the tunnel. Table 8.2 shows values which gave a suitable range of tunnel sizes for the five severities used in the simulation.

Severity Outer Radius (% of Window Height) Inner Radius (% of Window Height)
Very Low 100 80
Low 80 64
Medium 60 48
High 40 32
Very High 20 16

Table 8.2 - Glaucoma / Retinitis Pigmentosa Dimensions

8.5 Macular Degeneration

The simulation of macular degeneration was required to obscure the display by drawing a black 'patch' centred on the centre of vision (Req 1.5).

8.5.1 Drawing the Black Patch

The black patch was drawn by approximating a circle using a polygon. The result was a black patch with sharp edges. To create a more realistic effect, the edges of the black patch were progressively blended into the background. This was achieved by drawing a black ring around the polygon using a quad strip with blending and smooth shading. Each inner vertex was drawn black with an alpha value of 1.0. Each outer vertex was drawn black with an alpha value of 0.0. The OpenGL shading model produced a smooth transition between the inner and outer vertices from opaque to transparent.

Macular Degeneration Design

Figure 8.6 - Simulation Macular Degeneration

8.5.2 Dimensions

The radii of the inner and outer circles were determined by the severity of the impairment. The size of the obscured area of the screen was increased with the severity. To account for varying screen sizes, the radii of the circles were calculated from the window dimensions. Table 8.3 shows values which gave a suitable range of circle sizes for the five severities used in the simulation.

Severity Outer Radius (% of Window Height) Inner Radius (% of Outer Radius)
Very Low 20 10
Low 40 20
Medium 60 30
High 80 40
Very High 100 50

Table 8.3 - Macular Degeneration Dimensions

8.6 Diabetic Retinopathy

The simulation of diabetic retinopathy required 'floaters' or 'blobs' to obscure the field of vision (Req 1.6).

The display was obscured by drawing three black patches at positions relative to the centre of vision. Each black patch was drawn in the same way as for macular degeneration (Section 8.5.1) by approximating a circle using a polygon and a quad strip. Table 8.4 shows values which gave a suitable range of circle sizes for each of the five severities.

Severity Outer Radius (% of Window Height) Inner Radius (% of Outer Radius)
Very Low 10 10
Low 20 20
Medium 30 30
High 40 40
Very High 50 50

Table 8.4 - Diabetic Retinopathy Dimensions

8.7 Hemianopia

The simulation of hemianopia required one side of the field of vision to be obscured (Req 1.7). The macula at the centre of the field of vision was either 'involved' or 'spared' from the obscured region.

8.7.1 Hemianopia with Macular Involvement

The simulation of hemianopia with macular involvement required the right side of the visual field to be completely obscured:

Hemianopia Design 1

Figure 8.7 Simulation of Hemianopia with Macular Involvement

A scissor test was used to restrict rendering to the region of the screen to the left of the centre of vision. Black was used to clear the colour buffer before performing the scissor test. The background was then rendered with the scissor test enabled. Pixels in the area outside the scissor box remained black when the background image was rendered to give the impression that the image was obscured.

The edge of the obscured region was progressively blended into the background by drawing a rectangle with blending and smooth shading enabled. The two left-most vertices were drawn in black with an alpha value of 0.0. The two right-most vertices were drawn in black with an alpha value of 1.0. The OpenGL shading model produced a smooth transition between the left and right vertices from transparent to opaque.

8.7.2 Hemianopia with Macular Sparing

The simulation of hemianopia with macular sparing required that the right side of the visual field to be obscured except for a semi-circular region around the centre of vision which was left unobscured:

Hemianopia Design 2

Figure 8.8 Simulation of Hemianopia with Macular Sparing

The obscured part of the display was drawn using a black quad strip:

Hemianopia Design 3

Figure 8.9 - Creating the Obscured Region for the
Simulation of Hemianopia with Macular Sparing

The curved section was approximated by splitting a region around the centre of vision into 60 segments and using an equation based on x = cos (alpha) over the range 0..2 pi radians. Each segment of the quad strip was defined by a pair of vertices, (x1, y) and (x2, y) which were calculated using:

for (n = 0; n < number of segments; n++)
{
alpha = n * 2*pi / number of segments
x1[n] = centre.x - scale factor * cos (alpha);
x2 = width of window;
y = max y - n * (max y - min y) / number of segments;
}

The edge of the obscured region was progressively blended into the background by drawing a second quad strip with blending and smooth shading enabled. Each left-hand vertex was drawn in black with an alpha value of 0.0. Each right-hand vertex was drawn in black with an alpha value of 1.0. The OpenGL shading model produced a smooth transition between the left and right vertices from transparent to opaque.

8.7.3 Dimensions

In the simulation, the dimensions of the obscured region did not depend on the severity of the impairment. When the 'track to cursor' mode was enabled, the position of the mouse cursor determined the centre of vision. The region of the screen to the right of the mouse cursor was obscured.

8.8 Cataracts

The simulation was required to simulate three different symptoms of cataracts (Req 1.8) which are described in the following sub-sections:

8.8.1 Cloudy Cataracts

The simulation of cloudy cataracts required the display to be obscured by drawing a cloudy region centred on the centre of vision.

The cloudy region of the display was drawn in a similar way to the black patch for macular degeneration (Section 8.5). Blending was enabled and a smooth shading model applied. The inner circle was drawn as an approximation using a polygon and filled with a colour that was 75% grey and 95% opaque. The outer 'ring' was drawn as an approximation using a quad strip. Each inner vertex was drawn 75% grey with an alpha value of 0.95. Each outer vertex was drawn 75% grey with an alpha value of 0.0. The OpenGL shading model produced a smooth transition between the inner and outer vertices from opaque to transparent.

Cataracts Design

Figure 8.10 - Simulating Cloudy Cataracts

The radii of the inner and outer circles were determined by the severity of the impairment. The values used were the same as for macular degeneration (Table 8.3).

8.8.2 Yellow Brown Cataracts

The simulation of yellow / brown cataracts required a yellowing and darkening of the image.

After the background had been rendered, a yellow rectangle with the dimensions of the screen was blended with the background image to give the image the yellowing effect. A rectangle coloured pure yellow using glColor (1.0, 1.0, 0.0) resulted in an image that had a high intensity. To make the image more realistic, the rectangle was instead coloured with a dull yellow using glColor (0.8, 0.8, 0.0).

The severity of the impairment determined the alpha value used to perform the blending. At the lowest severity, the yellow rectangle was drawn with a low alpha value. This gave a high transparency and resulted in only a slight yellowing of the image. As the severity increased, the rectangle was drawn with a higher alpha value making it less transparent and increasing the yellowing effect of the image. The values used for each severity are shown in Table 8.5:

Severity Colour
Very Low glColor4f(0.8, 0.8, 0.0, 0.3)
Low glColor4f(0.8, 0.8, 0.0, 0.4)
Medium glColor4f(0.8, 0.8, 0.0, 0.5)
High glColor4f(0.8, 0.8, 0.0, 0.6)
Very High glColor4f(0.8, 0.8, 0.0, 0.7)

Table 8.5 - Yellow/Brown Cataracts: Colour Values

8.8.3 Double Vision Cataracts

An illusion of double vision was created using the accumulation buffer. Once the background image had been rendered, the contents of the colour buffer were loaded into the accumulation buffer using glAccum(GL_LOAD, 0.5). The image was then translated by an x and y offset and added to the accumulation buffer using glAccum(GL_ACCUM, 0.5). The result was a combination of the two images, which was then copied back to the colour buffer using glAccum(GL_RETURN, 1.0).

The double vision effect was reduced by changing the values of the second parameter in the calls to glAccum() to place a greater emphasis on either the first or second image. The values used for each severity were shown in Table 8.6:

Severity Image 1 Image 2
Very Low glAccum(GL_LOAD, 0.9) glAccum(GL_ACCUM, 0.1)
Low glAccum(GL_LOAD, 0.8)  glAccum(GL_ACCUM, 0.2)
Medium glAccum(GL_LOAD, 0.7)  glAccum(GL_ACCUM, 0.3)
High glAccum(GL_LOAD, 0.6) glAccum(GL_ACCUM, 0.4)
Very High glAccum(GL_LOAD, 0.5)  glAccum(GL_ACCUM, 0.5)

Table 8.6 - Double Vision Cataracts: Values to glAccum()

8.9 Colour Blindness

A simple simulation of colour blindness (Req 1.9) was created by converting the display to a greyscale image. A greyscale image stores only a luminance (intensity) value for each pixel. The image in the colour buffer was automatically converted to a luminance image by OpenGL by calling glReadPixels() with the image format argument set to GL_LUMINANCE. The luminance image was then written back to the colour buffer using glDrawPixels()[36].

"[W]hen OpenGL converts a color image to luminance, it simply adds the color channels together. If the three color channels add up to a value greater than 1.0, it is simply clamped to 1.0" [36]. The National Television Standards Committee standard [as cited in 36] for the conversion from RGB colour space to greyscale is:

luminance = (0.3 * red) + (0.59 * green) + (0.11 * blue)

Equation 8.3 - Conversion from RGB to Greyscale [36]

This colour scaling was performed in OpenGL for each colour component during the glReadPixels() operation [36] and was set up using:

glPixelTransferf(GL_RED_SCALE, 0.3f);
glPixelTransferf(GL_GREEN_SCALE, 0.59f);
glPixelTransferf(GL_BLUE_SCALE, 0.11f);

After reading the pixels, the colour scaling was returned to normal by setting the scaling factor for each colour component to 1.0.

8.10 User Interaction

The simulation was required to handle input from the user (Req 3). User interaction included:

Mouse and keyboard input was handled by registering callback functions with the GLUT event loop (Section 7.7). The operations of these functions is described in the following sub-sections:

8.10.1 Menus

The popup menu described in Section 6.3 was updated to reflect the grouping of impairments. Myopia and hyperopia were grouped together as a single impairment as were glaucoma and retinitis pigmentosa. The three variations of cataracts were grouped together in a submenu of the 'Change Impairment' menu. The 'Change Focal Distance' submenu was removed. The popup menu was implemented using the GLUT menu functions described in Section 7.7.6. The menu was attached to the right mouse button using glutAttachMenu(GLUT_RIGHT_BUTTON). When a selection was made from the menu to change either the impairment or the severity, the display was updated by calling glutPostRedisplay().

8.10.2 Keyboard Shortcuts

Keyboard shortcuts were provided to allow the user to change the current impairment and severity and are shown in Table 8.7. The keyboard shortcuts had the same effect as selecting the corresponding item from the menu.

Action Keyboard Shortcut
Change impairment to myopia / hyperopia F2
Change impairment to glaucoma / retinitis pigmentosa F3
Change impairment to macula degeneration F4
Change impairment to diabetic retinopathy F5
Change impairment to hemianopia with macular involvement F6
Change impairment to hemianopia with macular sparing F7
Change impairment to cataracts - yellow/ brown vision F8
Change impairment to cataracts - cloudy vision F9
Change impairment to cataracts - double vision F10
Change impairment to colour blindness F11
Change severity to very low 1
Change severity to low 2
Change severity to medium 3
Change severity to high 4
Change severity to very high 5

Table 8.7 - Keyboard Shortcuts

Additional keyboard shortcuts were used to allow the position of the centre of vision to be moved independently of the mouse cursor when the 'track to cursor' mode was disabled. The centre of vision was stored as a screen coordinate with an x and y value. These values were incremented or decremented by the key presses shown in Table 8.8:

Action  Keyboard Shortcut
Move centre of field of vision to the centre of the screen centre.x = window width / 2
centre.y = window height / 2
Home Key
Move centre of field of vision up Increment centre.y Up Arrow
Move centre of field of vision down Decrement centre.y Down Arrow
Move centre of field of vision left  Decrement centre.x Left Arrow
Move centre of field of vision right  Increment centre.x Right Arrow

Table 8.8 - Keyboard Shortcuts to Adjust the Centre of Vision

8.10.3 Feedback

The simulation was required to "keep users informed about what is going on through appropriate feedback" [35] (Req 6.2). When a change to the current impairment, severity, or centre of vision was requested, there was a short delay as the image was processed in the back buffer before being swapped to the front buffer for display. The mouse cursor was changed to the 'egg timer' cursor to indicate that the simulation was processing the image. The mouse cursor was changed back to the 'arrow' cursor when the contents of the colour buffers were swapped, and the updated image was displayed. The cursor type was selected using glutSetCursor(GLUT_CURSOR_WAIT) and glutSetCursor(GLUT_CURSOR_LEFT_ARROW).

8.10.4 Track To Cursor Mode

The 'track to cursor' mode (Section 6.3.7) was implemented by registering a callback function with the GLUT event loop to respond to passive mouse movement (movement of the mouse with no mouse buttons depressed). Each time the function was called, the screen coordinates of the mouse cursor were assigned to the coordinates of the centre of vision. The display was updated by calling glutPostRedisplay() to inform the GLUT event loop that the window was to be re-painted.

8.10.5 Hiding the Cursor

The mouse cursor was required to be visible only in the unobscured areas of the screen. As the mouse cursor was moved around the screen, the mouse coordinates returned by the passive mouse motion callback function were compared with the dimensions of the current impairment and the current centre of vision to determine the visibility of the cursor. The mouse cursor was turned on and off using glutSetCursor(GLUT_CURSOR_LEFT_ARROW) and glutSetCursor(GLUT_CURSOR_NONE).

8.11 Simple Paint Program

To simulate the use of a computer with a visual impairment (Req 5), an interactive two-dimensional drawing program was implemented using OpenGL and used as the background for drawing the visual impairments. The program was based on "Newpaint.c" by E. Angel [23]. A toolbar at the top of the screen provided buttons for selecting a drawing tool and a colour. Simple shapes were drawn using the mouse. There was no facility for loading or saving images to a file.

8.11.1 The Toolbar

Simple Paint Toolbar

Figure 8.11 - The Toolbar in the Simple Paint Program

The toolbar was drawn at the top of the window as a series of squares to represent buttons. Each square was drawn twice, first as a solid coloured square, and then again as a black line loop. The symbols on the buttons were drawn using simple geometric shapes. The characters were drawn using GLUT bitmap characters (Section 7.7.7).

8.11.2 Clicking the Buttons

The toolbar buttons were selected by single-clicking the left mouse button. Mouse input was handled by registering a callback function with the GLUT event loop (Section 7.7.4). The toolbar was located in the top left corner of the screen. The coordinates of the mouse click were relative to the top left corner of the screen and were used to determine which button, if any, had been clicked:

if (y > BtnSize) return NO_BUTTON;
else if (x < BtnSize) return BUTTON_1;
else if (x < 2*BtnSize) return BUTTON_2;
else if (x < 3*BtnSize) return BUTTON_3;
...
else if (x < n*BtnSize) return BUTTON_N;
else return NO_BUTTON;

The buttons are described in Sections 8.11.3 to 8.11.6. If no button was clicked, the mouse coordinates were used to draw with the current drawing tool as described in Sections 8.11.7 to 8.11.11.

8.11.3 The Drawing Tools

The first five buttons on the toolbar were for the line tool, rectangle tool, triangle tool, point tool and text tool. The currently selected tool was shown with a thick black border. Initially no drawing tool was selected.

8.11.4 The Fill Button

The 'fill' button was used to enable or disable the 'fill polygon' mode. When enabled the fill button was drawn with a thick black border and rectangles and triangles were drawn as filled polygons. When disabled, rectangles and triangles were drawn as line loops.

8.11.5 The Colour Buttons

The eight colour buttons allowed the foreground and background colours to be selected. The foreground colour was the colour used when drawing shapes with the drawing tools and was initially set to black. It was selected by clicking one of the colour buttons with the left mouse button. The currently selected foreground colour was shown with a thick black border. Changing the foreground colour did not change the colour of shapes already drawn. The background colour was the colour used to clear the colour buffer and was initially set to white. It was selected by clicking one of the colour buttons with the middle mouse button.

8.11.6 The Undo and Clear Buttons

The last two buttons on the toolbar were 'undo' and 'clear'. Clicking the undo button removed the last shape drawn from the display. Clicking the clear button removed all shapes from the display.

8.11.7 Drawing Lines

Lines were drawn using GL_LINES. The sequence of actions required to draw a line once the line tool had been selected was:

1. Position the mouse cursor on the screen at the point where the starting vertex of the line should be drawn.

2. Single-click the left mouse button.

3. Position the mouse cursor on the screen at the point where the end vertex of the line should be drawn.

4. Single-click the left mouse button.

After each mouse click, the coordinates of the vertex were saved. When both vertices had been specified, the line was drawn on the screen.

8.11.8 Drawing Rectangles

If the fill polygon mode was enabled, a filled rectangle was drawn using GL_QUADS, otherwise an outline of a rectangle was drawn using GL_LINE_LOOP. The sequence of actions required to draw a rectangle once the rectangle tool had been selected was:

1. Position the mouse cursor on the screen at the point where one vertex of the rectangle should be drawn.

2. Single-click the left mouse button.

3. Position the mouse cursor on the screen at the point where the vertex at the diagonally opposite corner of the rectangle should be drawn.

4. Single-click the left mouse button.

After each mouse click, the coordinates of the vertex were saved. When both vertices had been specified, the rectangle was drawn on the screen.

8.11.9 Drawing Triangles

If the fill polygon mode was enabled, a filled triangle was drawn using GL_TRIANGLES, otherwise an outline of a triangle was drawn using GL_LINE_LOOP. The sequence of actions required to draw a triangle once the triangle tool had been selected was:

For each vertex of the triangle:

1. Position the mouse cursor on the screen at the point where the vertex should be drawn.

2. Single-click the left mouse button.

After each mouse click, the coordinates of the vertex were saved. When all three vertices had been specified, the triangle was drawn on the screen.

8.11.10 Drawing Points

Points were drawn using GL_POINTS with the point size set to 4 pixels. The sequence of actions required to draw a point once the point tool had been selected was:

1. Position the mouse cursor on the screen where the point should be drawn.

2. Single-click the left mouse button.

The point was drawn at the coordinates specified by the mouse click.

8.11.11 Drawing Text

Keyboard input was handled by registering a callback function with the GLUT event loop (Section 7.7.5). Text was drawn using GLUT Bitmap characters. The sequence of actions required to draw characters once the text tool had been selected was:

1. Position the mouse cursor on the screen where the text should be drawn.

2. Single-click the left mouse button.

3. Type characters by pressing one or more keys on the keyboard.

The raster position was set to the coordinates of the mouse click. The x-coordinate was incremented by the width of a character each time a character was drawn.

8.11.12 Using Display Lists

Display lists (Section 7.1.5) were used to store the sequence of OpenGL instructions required to draw each shape on the screen so that they could be reproduced each time the display was updated. The display lists were stored as an array. A new display list was created for each shape that was drawn by the user. The undo operation removed the last display list from the array. The clear operation removed all the display lists from the array.

8.12 Suitability of GLUT

The GLUT library provides functions for window management and event handling (Section 7.7). GLUT is designed as a tool for learning purposes and has a simple interface making it easy to use. Replacing the generic interface provided by GLUT by a platform-specific interface allows an application greater control over:

This increase in functionality is at the cost of decreasing the portability of the application.

8.12.1 Changing from GLUT to the Windows API

Two versions of the simulation were maintained, one using the original GLUT interface and a second which replaced the GLUT interface by a Windows API interface to take full advantage of the features offered by the Microsoft Windows operating system. The code used to produce the simulated impairments was common between the two systems. A web browser was added to the Windows API version. The impairments were applied to the displayed output from the web browser to simulate the use of a computer with a visual impairment (Req 5).

8.13 Review of the Windows API

8.13.1 The Windows API

The Windows Software Development Kit (SDK) is the Windows programming API. It includes hundreds of functions contained in dynamic-link libraries (DLLs) that an application can call on to perform various tasks such as creating a window, drawing a line, and performing file input and output [37]. A set of functions prefixed with the letters wgl provides an interface between OpenGL and the Windows operating system [36]. The Microsoft Foundation Class library is an object-orientated wrapper around the Windows API [37].

8.13.2 The Windows Messaging Loop

The Windows API uses an event-driven model to process messages from the operating system. The message processing is at a lower level than the event handling provided by GLUT and allows more control over the management of windows and the response to input events. Messages sent to a window are placed in a message queue. Windows programs begin execution with a function called WinMain. The WinMain function contains a message loop which retrieves messages from the queue and dispatches them to a function called the Window Procedure:

// Process application messages until the application closes.
while (GetMessage(&msg, NULL, 0, 0))
{
TranslateMessage(&msg);
DispatchMessage(&msg);
}

The Window Procedure processes the messages from the message loop. It has four parameters: a handle of the window (a 32-bit identifier) to which the message is directed, an enumerated message ID that identifies the message type, and two 32-bit parameters known as wParam and lParam that contain information specific to the message type. Unprocessed messages are passed to an API function named DefWindowProc, which provides a default response. The message loop ends when a WM_QUIT message is posted on the message loop [37].

8.13.3 Window Management

An applications creates a window using the function CreateWindow() whose parameters include the type of window, its location on the screen, and its dimensions. If successful, the function returns a handle to the window and posts a WM_CREATE message to the Window Procedure.

8.13.4 Windows Rendering

Graphical output is displayed in an area of the window know as the client area which excludes the title bar, menu bar, window borders and scroll bars. A device context identifies the object to be drawn on. The device context of a window is obtained when the window is created by calling GetDC(hWnd), where hWnd is a handle to the window [36].

OpenGL rendering requires a rendering context which specifies the current rendering environment. An OpenGL application may have more than one rendering context, however only one rendering context may be active at any one time per thread. When made current, a rendering context is associated with a device context and thus with a particular window. A rendering context is created using wglCreateContext(hDC) and is made current using wglMakeCurrent(hDC, hRC), where hDC is a handle to the device context of a window, and hRC is a handle to an OpenGL rendering context [36].

The pixel format defines the properties of the device context that are required for OpenGL rendering such as whether single or double buffering is to be used and the size of the colour, depth, stencil and accumulation buffers. Only a limited number of pixel formats are available for a given window. A pixel format descriptor is a record that specifies the desired attributes:

// Select a pixel format for the given device context.
static PIXELFORMATDESCRIPTOR pfd;
pfd.nSize = sizeof(PIXELFORMATDESCRIPTOR);
pfd.dwFlags = PFD_DRAW_TO_WINDOW | PFD_DOUBLEBUFFER | PFD_SUPPORT_OPENGL;
pfd.iPixelType = PFD_TYPE_RGBA;
pfd.cColorBits = 32; // Color Buffer.
pfd.cAccumBits = 64; // Accumulation Buffer.
pfd.cDepthBits = 0; // Depth Buffer.
pfd.cStencilBits = 0; // Stencil Buffer.

The function ChoosePixelFormat() returns the pixel format that is the closest match to the pixel format descriptor. The pixel format for the window device context is then set using SetPixelFotmat() [36].

8.13.5 Mouse Events

Mouse events are posted to the message loop for the press and release of each mouse button. For the left mouse button these events are called WM_LBUTTONDOWN and WM_LBUTTONUP. The movement of the cursor is associated with the WM_MOUSEMOVE message. For these messages, the parameter wParam indicates whether any of the keyboard keys are pressed. If the mouse has a wheel, a WM_MOUSEWHEEL message is posted for each turn of the wheel. The parameter wParam indicates the distance the wheel was rotated and whether any of the mouse buttons or keyboard keys are depressed. For all of the mouse related messages, the parameter lParam contains the screen coordinates of the mouse cursor relative to the upper-left corner of the window.

8.13.6 Keyboard Events

The messages WM_KEYDOWN and WM_KEYUP are posted for the press and release of a key. The parameters wParam and lParam specify a virtual key code to indicate which key was pressed, the key repeat count, and flags which provide information about the key press. If the key is a character, an additional WM_CHAR message is posted. The virtual key code is translated to the ASCII character code for the key.

8.13.7 Menus

The Windows API supports menus via the menu bar and popup context menus. Menus are typically created using a menu-template resource file and loaded at run-time using LoadMenu(). Alternatively a popup menu can be created at run-time using the functions CreatePopupMenu() and AppendMenu(). A WM_CONTEXTMENU message is posted when the right mouse button is clicked within the client area of the window. The parameter lParam contains the screen coordinates of the mouse click. The menu is displayed using TrackPopupMenu() which takes as its parameters a handle to the window and the menu, the screen coordinates for the menu's position, a flag specifying the menu's position relative to these coordinates and a flag specifying which mouse button should be used to make selections. A WM_COMMAND message is posted to the Window Procedure when an item is selected from a menu. The parameter wParam contains the id of the selected item.

8.14 Modifying the Simulation to use the Windows API

The following sub-sections describe the changes that were required to replace the GLUT event loop with a Windows messaging loop to handle window management and input events.

8.14.1 Full Screen Rendering

To render the window in true full screen mode, the device context of the Windows desktop was queried to find the dimensions of the screen:

// Get the Window handle and Device context of the desktop.
HWND hDesktopWnd = GetDesktopWindow();
HDC hDesktopDC = GetDC(hDesktopWnd);
 
// Get the screen dimensions.
int nScreenX = GetDeviceCaps(hDesktopDC, HORZRES);
int nScreenY = GetDeviceCaps(hDesktopDC, VERTRES);
 
// Release the desktop device context.
ReleaseDC(hDesktopWnd, hDesktopDC);

A window was drawn with no title-bar or borders and the width and height set to match the dimensions of the screen [36].

8.14.2 Updating the Display

A request to update the display was made by invalidating the client area of the window using InvalidateRect(hWnd, NULL, FALSE) where hWnd is a handle to the window. The InvalidateRect() function posted a WM_PAINT message to the window's message queue. When the Window Procedure processed this message, the display callback function that was previously registered with the GLUT event loop was called to render the display to the back colour buffer. The front and back buffers were then swapped using SwapBuffers(hWnd) and the client area was validated using ValidateRect(hWnd, NULL) [36].

A WM_SIZE message was posted to the Window Procedure when the window was resized. On receiving this message, the simulation called the same Reshape() function as the GLUT implementation which contained OpenGL function calls to update the projection matrix and the viewport (Section 8.1.2).

8.14.3 The Popup Menu

The popup menu provided in the simulation used was modified to include an option to start the web browser. The same functions as the GLUT implementation (Section 8.10.1) were used change the current impairment and severity when a selection was made from the menu.

8.14.4 Keyboard Shortcuts

Keyboard shortcuts were implemented by responding to the WM_CHAR message. The character keys were used in the same way as the GLUT implementation (Section 8.10.2).

8.14.5 Track To Cursor Mode

The 'track to cursor' mode (Section 6.3.7) was implemented by responding to the WM_MOUSEMOVE message. The coordinates of the mouse cursor were assigned to the coordinates of the centre of vision and the display was updated.

8.14.6 Hiding the Cursor

The visibility of the mouse cursor as it was moved around the screen was determined from the coordinates supplied by the WM_MOUSEMOVE message in the same way as the in the GLUT implementation (Section 8.10.5). The mouse cursor was turned on and off using the function ShowCursor() which takes a Boolean value.

8.15 Modifying the Simple Paint Program

The Simple Paint program is described in Section 8.11. To make the program work with the Windows API, the callback functions that were previously registered with the GLUT event loop were called in response to messages posted to the Window Procedure. Mouse coordinates and ASCII character codes required to process input events were extracted from the parameters wParam and lParam supplied by each message.

8.16 The Web Browser

A web browser was embedded into a second window within the simulation. The main simulation window was run in the foreground in front of the browser window. Output from the browser window was captured and used as the background image when drawing the impairments in the main window.

8.16.1 The Browser Interface

The embedded browser was an OLE/COM object and used Internet Explorer's IWebBrowser2 interface. Four functions written by Jeff Glatt and posted on the Code Guru website [38] encapsulated the OLE commands required to embed the browser in a window and display a web page:

The function EmbedBrowserObject(hWnd) used OleCreate() to create the browser object and then called the DoVerb() function within the browser object to embed the browser within the window. The embedded browser object was sized to occupy the entire client area of the window using the IwebBrowser2 interface methods put_Left(), put_Right(), put_Width() and put_Height().

The function DisplayHTMLPage(hWnd, URL) used the IWebBrowser2's Navigate2() method to navigate the browser to the location identified by the URL.

The function DisplayHTMLStr(hWnd, HTML_String) used the IWebBrowser2's Navigate2() and get_Document() methods to display the web page described by the HTML string argument.

The function UnEmbedBrowserObject(hWnd) called the OLE object's Close() method to unembed the browser object from the window.

8.16.2 Interacting with the Browser Window

To interact with the browser, mouse and keyboard events directed to the main window were reproduced in the browser window and the display of the main window was updated to reflect any changes to the browser display.

8.16.3 Simulating Mouse Events

Mouse input to the main window was handled by the Window Procedure. Input events were simulated in the browser window using the function mouse_event(). To ensure that the browser window received the input events, the main window was hidden and the browser window was brought to the foreground and activated.

To avoid cascading messages being sent to the Window Procedure, a flag was set to indicate that a mouse event had been sent to the browser window. This flag ensured that the simulated event did not generate a further mouse event when it was detected by the Window Procedure. The flag was reset once the display of the main window had been updated.

The left mouse button was used to click hyperlinks, buttons, scrollbars, textboxes and other elements on a web page. On receiving a WM_LMOUSEDOWN message, a MOUSEEVENTF_LEFTDOWN event followed by an MOUSEEVENTF_LEFTUP event was generated.

8.16.4 The Browser Context Menu

Clicking the right mouse button in the browser window displayed a context menu. However, the simulation used the right mouse button to display its own context menu. Right mouse clicks in the browser window were simulated using the middle mouse button in the simulation. When the Window Procedure received a WM_MMOUSEDOWN message, a MOUSEEVENTF_RIGHTDOWN event followed by an MOUSEEVENTF_RIGHTUP event was generated.

The simulation set a flag to indicate the visibility of the browser context menu and saved the mouse coordinates of the mouse click used to display the menu. The flag was reset when the left mouse button was clicked.

The left mouse button was used to select an item from the browser context menu. The browser context menu was only visible while the browser window was active. To allow an item to be selected from the menu when a left mouse click was detected, the menu was first redisplayed by simulating a right-mouse click at the stored coordinates, and then the left mouse click was simulated.

8.16.5 Simulating Keyboard Events

Keyboard input was simulated in the browser window when the Window Procedure processed a WM_KEYDOWN message. A single key press and release was simulated using the keybd_event() function with the virtual key code supplied by the Window Procedure parameter wParam.

The Control and Shift keys could be held down at the same time as other keys are pressed. Two flags were used to record whether these keys were held down. The flags were set by comparing the virtual key code supplied by wParam for the WM_KEYDOWN and WM_KEYUP messages to the virtual key codes VK_CONTROL and VK_SHIFT. For each key press that was simulated in the browser window, the Control and Shift keys were also simulated if the flags indicated they were pressed down.

8.16.6 Updating the Main Window

After each mouse event or key press was simulated in the browser window the simulation waited for the browser window to update before recapturing the browser display and updating the main window. A timer was started with an elapse time. After this time limit expired, a WM_TIMER message was posted to the Window Procedure.

The IWebBrowser2 interface function Busy() was used to return a Boolean value to indicate the browser object's status. If the browser was still updating, this function returned TRUE and no action was taken to update the main window. Instead the simulation waited until the timer period expired again to check the browser status. If the browser had finished updating its display, the Busy() function returned FALSE. The main window was then updated and the timer was stopped by calling KillTimer().

8.16.7 Recapturing the Browser Display

The main window was hidden each time a mouse or keyboard event was simulated in the browser window using ShowWindow(hWnd, SW_HIDE), where hWnd is a handle to the window. Once the display of browser window had been updated, the main window was brought to the foreground using ShowWindow(hWnd, SW_SHOW). When the main window was redisplayed, a copy of the screen pixels below the window were placed in the window's frame buffer in the same way as when the window was created (Section 8.2.4). The browser display was captured by saving the pixels in the frame buffer immediately after returning the main window to the foreground.

8.17 Summary

The design was based on the software design specification in Section 6 and included the simulations of myopia, hyperopia, glaucoma, retinitis pigmentosa, macular degeneration, diabetic retinopathy, hemianopia, cataracts and colour blindness in OpenGL. A simple drawing program was added to simulate the use of a computer with a visual impairment. The initial design used the GLUT library for window management and event handling. This was replaced by Windows API function calls to allow a web browser to be embedded within a second window and further simulate computer use with a visual impairment.


Stephen Ratcliffe. 2005
Department of Computer Science
University of York
Heslington
York
YO10 5DD