Audio Visual i t

In Summary 

  • Technical  – Back End open source
    • ReacTIVision + Fiducial markers
    • TUIO is an open framework that defines a common platform for tangible multitouch surfaces.
    • Pure data Extended
  • Front end – User interface
  • Work in progress
  • Future development


The Brief

Create a generative and/or an interactive audio and/or visual piece


Education 23/02/15

Teachers are increasingly expected to be adept at a variety of ICT-based and other approaches for content delivery, learner support, and assessment; to collaborate with other teachers both inside and outside their schools; to routinely use digital strategies in their work with students and act as guides and mentors.

European Commission, Horizon Report Europe 2014 Schools Edition.



Front  patch and design


Chemical Reaction patch and design


Intermediate working demo 8  


General patch development


Frontend user interface  02/02/15

Demo3 Subpatch visibility

Demo2, Fiducial Marker front page function

Demo1, with send and receive  (S + R)


Backend Functionality  02/02/15

ReacTIVsion, Tuio, Puredata patch demo1


Idea development 19/01/15

in order to help rectify the problem I tried  to find a PC version of Reactivision and found it on

 Win32: A Visual Studio 2012 project as well as all the necessary libraries and headers (SDL2, videoInput) are included. The project should build right away without any additional configuration.


Idea development 16/01/15

When I went home to get my initial patch working – it didn’t!  this is because of the transition between Mac and PC. I think I’m going to have to use my laptop in class


Idea development 15/01/15

The first Tuio client patch with Fiducials 0 & 3 Identified in the ‘select’

Screen Shot 2015-01-15 at 12.00.23


Tutorial 2  08/01/15

  • Presented the work in progress to date  which included:ideausingperiodicandelementsymbolsasbasisforfiducial markers,  web cam, presented the 3mm acrylic square shapes
    • Use TUIO Pure Data with the TUIO client
    • use existing Fiducial markers initially to get the software working
    • use square shaped markers as this is the shape in the periodic table
    • look at the aesthetics later 2 – 3 layers put together
    • Decide if the marker is read from below (read above required video camera ) or above (Webcam set on tripod)
    • select 10 chemical reactions to work with
    • obtain permissions to use existing video
    • problems which may be encountered with using elements as markers – they need a minimal number of zones


Audio   –  02/01/15

There are 118 Elements on the periodic table all of which would could be both illustrated and included in this work.  For the purposes of this module I will be simulating both the video and Audio experience partly because is an absence of high quality video and audio which can be acquired for inclusion.

update form slides


Idea development  – Element design & FMarkers 05/01/15

the original plan was to utilise d-touch to desigm my own markers

Visual markers or object recognition are graphic symbols designed to be easily recognised by machines… D-touch is a visual marker recognition system which allows users to create their own markers. Designable Visual Markers, Costanza E., Huang J., Media and Design Lab

These markers control what is communicated both visually and functionally.

Screenshot (397)

Having read the paper on design rules for fiducial markers by Messrs Ross Bencina et al, the question is how do I get my maker design from A – D and still maintain the integrity of the scientific symbol?

I have decided to only mock up the marker as my coding skills will been to be much more advanced to both design a new maker and code it to respond like a fiducial at the current time


The design plan for the maker is to have one side which is read by the webcam and one side which is read by the user, as illustrated above

Mock ups – it would be easier to draw this in Illustrator as it could then be read by the etching machine, but as I do not have illustrator at the moment I will be using Powerpoint to demonstrate how it would be designed.

Screenshot (399)

This Element mock-up was intended to become a Fiducial Marker and was is designed along the lines of the standard Fiducial.  However the issue became the complexity of the element design i.e

  • too many levels of nesting i.e. the Element above has 8 regions of nesting.  For the marker to be successful ideally it should have nor more than 3 levels of nesting

So it is not currently possible to create a Fiducial using the original Element design without simplifying the element.

This is a significant issue primarily because, the predominant consideration in this maker design must have a strong

resemblance to the acknowledged element design

Bearing in mind that the project purpose of the project is

Recognition and Retention

of the elements on the periodic table, its construction and associated data, the marker design could not be simplified to suit the requirments of the software.

Following are the 3d markers


Side 1 examples of the 3d tactile markers illustrating the Fiducial markers to be read by the computer system. This side could be etched concave asn filled black to have a tactile element to resemble a domino tile.


Side 2 examples of the 3d tactile markers illustrating the Chemical elements to be read by the user.  Working backwards from the most complex to the simplest of the Element, illustrated side by side.  The Elements have definable characteristics highlighted in colour to aid Recognition and Retention

This theoretical project will be centered on traditional metals = 40 elements, numbers 21 – 112


Idea development 02/01/15

I sent an e-mail to my tutor asking if he considered the d-touch software was the way to go with this project as not all the components are available for download and I would need to contact the professor in Southampton asking permissions to use it, in addition to which I would need to ask him for a compete file of components

Screenshot (309)

Dylan recommended that I revisit a paper that I have come across previously by Messrs Ross Bencina et al.    05/01/14


Screenshot (304)Tangible Intuitive Interactive Interfaces


ReavTIVion v1.5 webcam functions

Figuring out the relevant camera functions for demonstration and use.

Screenshot (393)

Screenshot (387) F1 = full screen function, the setting for demonstrations

Screenshot (382) s = source a standard image. P = Pause is a useful function to help save processing power during operation

Screenshot (381) O = defines camera functions + H= help function whic can also be activated in the the full screen option F1.

Screenshot (380) s source image + SPACE, activates the frame equaliser

Screenshot (390) t + e function both makes the image highly pixelated, cancels out the background in oorderto make the fiducial easier to read

Screenshot (388) t= target image (highly pixelated image)

1.Screenshot (294)First stage was to download d-touch server – the result was intreging an image of me upside down and a rectange in the centre of the screen.  The rectangle is the systems attempt to {read} a marker, no maker appeared


Screenshot (302) = programme in mode which is not attempting to {read} a marker?

2a. Idea: Development ~ Assessment

In order for my idea to work even on an elementary level I will need to have a good basic understanding of existing {fiducial Makers}.  For this project to have any real use to me in future I need make some significant progress towards both self generated {marker designs} and {self generated code} in order that I can code my own marker design in future……. (yes that is what I am thinking and hope I will not be found muttering to myself in a darked corner rocking back and forth on deadline day….)

Background Research

from 1. ReaTIVison to 2. d-touch – read about Designable Visual Markers

Screenshot (292)

d-touch defined as : An open source system which allows users to create their own markers, and control their
aesthetic qualities, this is to be expressive to humans not just convey information to computers

2. Idea: Development ~ Assessment

Basic model – visually stimulating – chosen for development

I came to the next developmental idea by looking at the periodic table of elements with specific reference to electrons (Rutherford-Bohr atomic model) and atoms

4.  this is the element for Calcium illustrated as follows: Neurons and Protons in the nucleus (Ca),  with a concentric circle of Electrons set out in a unique pattern at the four levels of the shell.


Screenshot (396)

The periodic table as arranged by order of atomic number, its

EnergyLevelsTable3. (c) Greg Robson and shows each element with the name, symbol and number of electrons in each shell by actually showing the shells.

© Royal Society of Chemistry 2014 2. visual periodic table (C)  the Royal Society for chemistry- an opportunities for unique symbols?Conventional periodic table 1. conventional periodic table

 2. Idea: Development ~ Assessment

  • Hypothesis: The visual element involved in interacting and manipulating physical blocks {of information} aides in the process of absorbing complex ideas
  • Theory: Chemistry
  • Object: Acrylic blocks
  • Software: Fiducial Markers (coded markers)
  • Interactivity: Combination objects {participant} – illustrates a correct answer {video?} chemical reactions
  • Visual: Directly related to the input i.e. a know structure = a known output
  • Assessment: LIKe! ~ could be visually stimulating –  loud and very exciting- idea for development
  • Note: need to learn something about chemistry – Yikes!!


Week 9

Tutorial 1 – 18/12/14

  • Presented the initial idea  which included: Wanting to work with a process for learning, making a learning game with a set structure and therefore a set outcomeOriginally I thought of Algebra illustrated below,  is the fundamental stage in the  learning process algebra i.e. (Define what is being learnt) . the problem with this approach is it has no natural  visual interactive element.
  • Advised to:
    • rethink the basic idea


 1. Idea: Development ~ Assessment

  • Hypothesis: that visual element of interacting and manipulating physical blocks aides in the process of absorbing complex ideas
  • Theory: Algebraic
  • Object: Acrylic blocks – too many combinations
  • Software: Fiducial Markers – why?
  • Interactivity: Combination objects – illustrate a correct answer
  • Visual: Out put ??  – this would need to be abstract – e.g a picture of a cute kitten – why?
  • Assessment: Does not look like it would either be visually stimulations or a natural outcome of the sum- not for development


Week 8

introduction to ReactoiVision and TUIO (touch User imput Output)

  • ReactoiVision
  • TUIO is
  • Fiducial Markers are

example of RecatiVsion capabilities as follows

(how to stop the final images comming up)

and work with PD in the following ways

ReactoiVision + pure data video


To document a screen recording using Qucktime, select onscreen recording, select area to be recorded, start and save when finished.

upload to youtube or Vimero select the unlisted option to keep  the video semi private, imbed into wordpress and set the interface size

understanding the elements of getting the video working

Screen Shot 2014-12-04 at 12.11.09f

and adjusting the parameters

1. Gemwin at 1000 x 1000 with a video (video is playing) area of rectangle at 1 x 8 – my very first independent patch!!

Screenshot (279)

2. this shows an adjustment in size to both the gemwin, area of video (video is playing) shape

Screenshot (280)

3. using the 3d patch of week 6 with the video (video is playing) in the background frame with static image in the foreground is quite beautiful (Although the second patch is just essentially a picture frame at the moment)

Screenshot (281)

same patches and perimeters change of video

Screenshot (282)

Screenshot (283)

Week 7

Here because the [t b b b] object was connected from the left to the right it was not outputting the correct data see print file list in order to correct deleat connections to xxx

Screen Shot 2014-11-20 at 12.44.28

Week 6

Screen Shot 2014-11-13 at 10.21.21

replace an image with a movie – starting from last lesson point!

Screen Shot 2014-11-06 at 13.45.51

Screen Shot 2014-11-13 at 12.26.17 Screen Shot 2014-11-13 at 12.35.42

Screen Shot 2014-11-13 at 13.48.43

mini asignment

– see if I can get this work – Patch from week 5 try yo get the movie to play

Demonstrates the Colour converstion table for

Screen Shot 2014-11-06 at 13.25.07

Stages to connect a function the external controller

Patch which is not working!! – the problems seemed to be that there were two Gem Windows Open

Screen Shot 2014-11-06 at 13.51.11

patch which is working!

Screen Shot 2014-11-06 at 13.06.13

Sum 126 x N = 360 ( the tyotal rotation of the object)

126 is the total roation of the knob = answer 2.83464 whci goes in the number box

then connected to the  [roatation XYZ]

Screen Shot 2014-11-06 at 12.32.50Screen Shot 2014-11-06 at 12.32.36Screen Shot 2014-11-06 at 12.31.50Screen Shot 2014-11-06 at 12.31.33Screen Shot 2014-11-06 at 11.43.48Screen Shot 2014-11-06 at 11.08.40Screen Shot 2014-11-06 at 11.02.12Screen Shot 2014-11-06 at 11.04.44  Screen Shot 2014-11-06 at 11.02.12


Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s