Help with the use of the

FPMs TCTS - Multitel audio games

 

Download here

 

 

Welcome in these interactive games based on the acoustic analysis of speech. The software that you installed was developed at the TCTS laboratory (http://tcts.fpms.ac.be ) of the Faculté Polytechnique de Mons (FPMs: www.fpms.ac.be ), in collaboration with the MULTITEL research center (www.multitel.be ). 

The algorithms which constitute the heart of this software were developed by Mamadou DIALLO, within the framework of its Master’s thesis in Electrical Engineering (2004). The final design of the software was financed by the European network of excellence SIMILAR (www.similar.cc ), of which FPMs and MULTITEL ASBL are members, and who gathers at the European level 32 research laboratories working in the field of the man-machine interfaces.

 

1.     Activation of the microphone

·        To use the fast zone of launching: You may see on the desk, on the taskbar an icon indicating the volume. Double click on this icon, then go in the menu 'options'. Then, choose 'recording', and 'ok'. In the new window, enable the 'microphone'  box and approximately position the cursor at its quarter.

·        To use the menu to start: Click on ‘Start’, then go in ‘parameters', then 'configuration panel', then 'sounds and Multimedia'. Then choose the 'audio’ window, and click on 'Volume' in 'sound recording'. Enable the 'Microphone' and approximately position the cursor at its quarter.

 

2.     Calibration

For a first use of the software, it is imperative to carry out the calibration of the sound volume. The goal of this calibration is to fix the values of minimal and maximum volume in order to associate values of energy of the voice to positions of objects on the screen.

You will be able to change calibration (terminals of energy) at any time, if you are not running simultaneously one of the interfaces of game.

To check that the calibration and the choice of the microphone gain are correct, launch the game 'Music-Break' game, choose game 2 (to press on key 2 of the keyboard) and test it. If the displacement of the bar is very sensitive, decrease the microphone gain and retry.

 

3.     Description of games

v    First game : Music Break

The interface is made up of:

·        A balloon, coming from the sky and whose initial position is set randomly, falls down with a rectilinear trajectory. This balloon rebounds in contact with the side walls of the interface, with an angle equal to the angle of incidence. This balloon finishes its race in water if it is not stopped and, a few moments after its fall, another balloon appears in the sky. If the balloon is stopped, it rebounds towards the sky.

·        A box (or a bar, both are possible), of given width is placed a little above water. The displacement of this box is controlled by the energy of the voice, and this displacement is limited to horizontal movements. The goal of the box is to stop the balloon so that it rebounds.

·        A helicopter which turns while emitting a sound each time it is touched by the balloon. 

·        Two birds which steal and emit cries when they are touched.

·        A music symbol which undergoes a rotation by emitting the sound associated to him when it is touched.

·        A planet which moves away and emits a sound each time that it is touched.

·        Lastly, a witch, which narrows the box each time she is touched.

 


Figure 1 : Screen snapshot of the ‘Music Break’ game.

           

The goal of the play thus consists in moving the box with the energy of one’s voice, so as to make the balloon rebound and touch objects in the sky. 

For each object touched, the player gains 20 points; each time the balloon falls into water, the player loses 10 points. 

We propose four different ways of moving the box with voice energy:

1. Absolute displacement: The position of the box on screen is calculated according to the energy level of the voice. Thus, the position with the extreme left of the screen corresponds to the lower energy threshold (Emin), i.e. the energy corresponding to the noise in the absence of voice (as measured during calibration). The extreme right position corresponds to a high energy threshold (Emax; as measured during calibration).

There are two plays for which the displacement of the box is managed by this principle. 

·        For the first (key 2 on the keyboard), it is necessary to keep a constant energy if one wants to maintain the box with a given position of the screen, otherwise the box will start moving. 

·       For the second (key 3 on the keyboard), to fix the box at a given position, it is enough to briefly produce the required energy; the box will stay where it was positioned.

2. Relative displacement: The box has this time a continuous horizontal movement and changes direction with each collision with the edges (left and right) of the screen. If one wants to change the direction of the box, it is necessary to provide an energy level related to the energy level previously provided. The user must thus memorise the energy level constantly previously provided. 

There are also two ways of moving the box with voice energy in this mode: 

·        For the first (key 1 on the keyboard), producing a higher energy level then before moves the box to the right; a lower level moves the box to the left.

·        For the second (key 4 of the keyboard), the change of direction is the same in the previous case, but one needs to produce continuous energy variation duration about one second (in the previous case, short voice activity was sufficient).

These last two games are more difficult to play, whereas the first two are more accessible to the children because they are more intuitive.

 

v    Second game : Vowel Hunt

The interface is made up of several ` vowel-birds'. These vowel-birds cross the screen from left to right while following a random trajectory. Each time a bird leaves the screen, another one appears randomly. The number of simultaneous birds is fixed to three. Figure 2 shows the interface of this game, with only three vowels on the screen, and a box representing the position of the couple (first formant, second formant) of the sound produced by the player:


Figure 2 : Screen snapshot of the ‘Vowel Hunt’ game.

 

When the user pronounces the name of a vowel correctly, the bird is shaken. If the energy of the voice of the user at this time is lower than a fixed threshold, the bird-vowel vibrates in all directions and its displacement is slowed down. If this energy exceeds the threshold, then the bird  vibrates and its displacement is accelerated.

Each time the box touches a bird-vowel with the box (when the player produces a vowel whose formants are those of the vowel-bird), the player gains points; if not, the player looses points.

 

v    Third game : Treasure Hunt

Figure 3 shows the screen of this game, with the treasures and the box which should be moved to reach them:

           


Figure 3 : Screen snapshot of the ‘Treasure Hunt’ game.

 

Voice energy is associated to vertical movements of the box, while the pitch (fundamental frequency) is associated to horizontal movements. Low frequency sounds leave the box on the left, with the very high frequency sounds push it to the right. Silence corresponds to bottom positions of the box; high energy sounds push it to the top of the screen.

The goal of the game is to control one’s energy and fundamental frequency to bring the box to the treasure positions. The constraint of the game is that one should take a treasure only if all the treasures which precede it have already been gained. Otherwise, there is a penalty.