The Creation of the Monolith Multi-Touch Surface Computer

Application Coding And Development

Tracking the finger touches for flash

Programmed in flash's Action script 3, it accepts OSC messages in the form of TUIO. The tracker used on the monolith is the CCV tracker [1] (previously TBETA) by the NUI group. NUI group is an Open source community dedicated to mastering design of various methods of multi touch tracking. Most members concentrate on the design of multi touch surface computer design or contribute to the open source tracking system (CCV). This tracker uses computer vision systems to enable the tracking of multiple finger touches on the surface of The Monolith using IR and FTIR technology (for more info see the TBETA and CCV chapter) these touches are converted to OSC in the form of TUIO which are messages containing the X and Y locations of the touch on the surface. The OSC messages need converting for use in Flash which is done via FLOSC which is a Java based program which converts OSC into XML packages. The Monolith launcher uses three external classes to handle the use of these XML packages which are:

·      TUIO.as

·         TouchEvent.as

·         Touchable.as (which is and extension of TouchEvent.as)

These are imported at the start of the script for any multi touch application designed in this manor:

Code Example 1:

Import flash.event.TouchEvent;

import app.core.action.touchable;

import flash.event.TouchEvent;

Also imported is the RotatableScalable.as Class which will be used and discussed later.

Import app.core.action.RotatableScalable;

The external classes have been developed by the NUI open source multi-touch community. These are the basis of any multi touch flash application development (other than writing your own multi touch classes). These classes are responsible for importing the XML data and creating touch events on the surface.

 TUIO.as converts the XML data into surface events placing finger touches on the screen where ever you touch. These are represented as small circles under the user’s fingers when contact is made. The TouchEvents.as and Touchable.as classes are responsible for creating event listeners for use in your action script applications. For instance if I had a button in the centre of the screen, I could add an event listener to that button to respond when a touch event occurred over that button.

Code Example2 :

Button.addEventListener(TouchEvent.MOUSE_OUT, buttonover);

Then all is needed is to create a function containing the desired result.

Code Example 3:

function buttonover (e:TouchEvent):void{

          button.alpha = 0.5;

         }

In this case the second line(the action within the function reduces the alpha of the button by half - in effect it makes it translucent but any action could be applied. These event listeners exist in the TouchEvent class which is an extension of MouseEvent. The touchable class is simply an extension to the TouchEvent class extending it and simplifying it.

Quartz 1.0

The first version of Quartz worked on the idea of loading already compiled applications (external .swf files) into the launcher. This version of the launcher would launch my own applications and the ones created by the NUI group.

The general structure for this design was to build up action script on each frame being used in the whole application. On each frame action script would be built up for adding relevant objects to that frame. For instance the code for adding a button to the centre of the screen would be:

Code Example 5

var button1 = new button() ;

button1.x = 462 ;

button1.y = 400 ;

addChild (button1) ;

After each object is added to the stage, event listeners would then be added to the objects which would need interactivity (see code example 2). This repeated process builds up the elements for each frame in the application.

The external applications themselves are added using a basic .swf loader on the individual frames where the child of the loaded application will reside. The navigation buttons are simply skipping around the frames loading the .swf files when needed.

Each page that an .swf is loaded onto, also has a home button added and  floating above the loaded .swf. This is an important part of the process, as the code to removeChild of any loaded .swf exists on the first page.  The “home button” system always makes sure that the program returns to frame 1 before the next app is called. This ensures the last added child is removed.

Code Example 6

var ldrmovingjigsaw:Loader = new Loader();

 var urlmovingjigsaw:String = "moving jigsaw.swf";

 var urlReqmovingjigsaw:URLRequest = new URLRequest(urlmovingjigsaw);

 ldrmovingjigsaw.load(urlReqmovingjigsaw);

 addChild(ldrmovingjigsaw);

Quartz 2.0

Quartz 2.0 was an attempt to launch the applications internally – that is to hard code each application within Quartz making the launcher in effect one giant program with several multi touch programs embedded.

The code worked well and was a simple task of importing the applications own action script file at the beginning of my own code.

Code Example 7: 

import app.core.canvas.RippleCanvas

The above code shows an example of importing Ripplecanvas.as – the code which compiles into the “Ripples” program. Once this .as file is imported all we then need to do is to call a child of that action script file (RippleCanvas.as) when needed.

Code Example 8:

var addripples = new RippleCanvas();   

This line states IF i request the word “addripples” then i want an instance of RippleCanvas

addripples.x = 0;                                      

These are the X/Y locations of the instance of RippleCanvas IF added

addripples.y = 0;                                      

These are the X/Y locations of the instance of RippleCanvas IF added

addChild (addripples);                             

This is now the actual line that adds the RipplesCanvas

The above two code examples will add the ripple program to the stage but this will occur as soon as you run the code. You could wrap “code example 6” within a function and have it execute after a button is pressed

In effect you are saying - when “button 1” is pressed then create an instance of “RippleCanvas”.

This process is the complete premise of quartz 2.0. the children are removed using the same “home button” system as in Quartz 1.0 and a “remove child” function on page one 

This process was duplicated for each of the applications to be launched into the launcher.

Final Build – Monolith Launcher RC

The final build of the Monolith Launcher in terms of internal code structure and programming design - goes back to a similar structure of the first version of quartz. The main aim is to load the external .swfs directly into the program at runtime and then adding them to the stage as the relevant button is touched.

The difference is the programs that launch within this launcher - are all of own design and all of them maintain the same design characteristics.

The applications to be added (loaded) are:

·         Photo app

·         Maps app

·         Moving Jigsaw

·         Drinks ordering system

The main characteristics of this version of the launcher are the control panel which is situated at the top of the screen at all times for access to other applications.

The launch app panel is a simple tween, with the majority of the panel existing off of the stage. When it’s called with the down button the animation starts to slide it on the stage. Once frame 30 of the animation is reached the time line is stopped and the fully visible panel waits for either a command to reset – in which case the animation resumes up to frame 45 and loops to a reset at the top of the page, or an application is called - the panel then jumps back to frame one waiting for the next command.

Code Example 9

Down1.addEventListener(TouchEvent

.MouseOut, down1over);

function down1over (e:TouchEvent):void{

gotoAndPlay(2);

}

up1.addEventListener(TouchEvent.MouseOut, up1over);

function up1over (e:TouchEvent):void{

gotoAndPlay(31);

}

The structure of these code segments are for each of them to exist on their own time line (the object timeline).

The main time line on the main action script layer is used to load the applications and houses listeners for the application buttons (on the launcher panel object).

For instance:

Code Example 10:

Launcherpanel1.photoappbutton.addEventListener(TouchEvent.MouseOver, photoappbuttonover);

function photoappbuttonover (e:TouchEvent):void{

gotoAndStop (3);

}

And then on frame 3

Code Example 11:

var ldrmovingjigsaw:Loader = new Loader();

 var urlmovingjigsaw:String = "moving jigsaw.swf";

 var urlReqmovingjigsaw:URLRequest = new URLRequest(urlmovingjigsaw);

 ldrmovingjigsaw.load(urlReqmovingjigsaw);

 addChild(ldrmovingjigsaw);

The  layer ”action” is used to listen to the launcher panel and extends along every frame within the app.

The “application” layer has a variation of the above code on each frame where the relevant application should exist.  When the launcher moves the main time line to the frame of the desired application, then that program is loaded and the child of that loaded application is added. Also on each “application” frame there is a function which removes the child of any other application which might be loaded – including the application which is being loaded. This is done first removing any remnants first before adding the new child.

This approach is to ensure that children of any of the loaded applications don’t keep running in the background or multiple versions don’t run layered on top of each other.

Code example 12:

function unload1 (){

}