key points
As a summary, here are the theoretical key points of the project, as a proposed strategy to face the problem of multitouch (tabletop) gestural interfaces, stated in the thesis’ abstract (see previous post). The implemention is described in italics, between the brackets.
Note: this is taken from the slides of the thesis discussion, that I’ll eventually publish.
- Tangible objects. Integration with GUI, natural interaction. (Non-Z-ordered target areas.)
- Contextuality of gestures with respect to the current state of the interface (Access to the client’s environment plus global/multi-target gestures).
- Resolution of the conflicts. Which is the actual target of the gesture? Which is the actual meaning of the produced gesture? (Flexibility in designing gestures plus a priority mechanism.)
example of (ambiguous) gesture
In the picture above, the user producing the gesture may mean to link the objects A and C, or to remove the existing link between A and B.
thesis
It has been a hard year but finally I graduated.
I still don’t know the future of this project, nor mine!, but in the meantime here’s my thesis.
Soon I’ll upload some new code, at least to reflect the contents of the dissertation, though I’m afraid it will stay in the alpha phase for a while. More applications and tests are needed and I don’t have a multitouch table anymore.
Abstract
Surface computing is opening new horizons on human-computer interaction, in particular by the use of multitouch and tangible user interfaces (TUIs). However, the investigation into theoretic and practical aspects of the development of this kind of interfaces is still in its infancy, leading to a general tendency to adopt simple gestural paradigms or worse, to decline into the GUI and WIMP canons.
In this context, the present thesis introduces Grafiti, a Gesture Recognition mAnagement Framework for Interactive Tabletop Interfaces. This tool is meant to aid the design of rich multitouch gestural interfaces, focusing on the possibility to use tangible ob jects as targets, promoting a seamless natural interaction between the TUI and the GUI.
In form of a C# layer, Grafiti is independent of the client’s software environment and, using the Tuio protocol, it abstracts from the underlying input device. It can be extended with new gestures and recognition algorithms, offering functionalities that cover various aspects of the implementation of interfaces with different levels of complexity, and encouraging to develop structured and reusable code.
proceedings
Google summer of code has finished. The project has not however. Although it’s still in the alpha phase it’s usable and has reached a point where there are all the main functionalities.
I’m currently writing the documentation, thinking about improvements (like supporting composition of gestures) and developing a better demo in C++, with OpenGL, and multiplatform. By the way, the current C++ demo in svn works only in Windows, because it uses mixed code (C++/CLI and C#) and mono doesn’t like it. However Mono doesn’t mind if managed code use P/Invoke so that’s the way to go.
I’ll be working on the project at least until november.
building demo app
I decided to edit (again) this post instead of creating a new one, since the demo application under discussion is still under development.
- grafiti demo app
In order to show the main features of the framework, how it works and how it can be used, I’ve started to develop a demo application just for this purpose. It’s available to download via svn. However without an (upcoming) documentation it’s still hard to understand many things of the framework.
Features of the demo (in short):
- Add/move/remove tangible objects. These objects, (internally related to TuioObject so from now on I’ll refer to them simply as tuio objects) are listeners of gesture events. They’re represented by coloured squares and their round area of interaction as it is explained below.
- Visual feedback showing how fingers are grouped together. A group will be considered as producing a gesture (or more). A different color is associated to each group of fingers. The traces have a losing colored tail and their head shows the id number of the trace. The head also says if the cursor has been removed, i.e. the finger released (black circle) of not (grey circle). The fingers composing a group are linked by lines towards the centroid calculated on the present fingers and the recently removed ones. Finally, a semitransparent round area indicates where another finger can be put on the surface in order to be added to the same group of the finger.
- There are various kind of gestures that can be recognize:
a) The gesture of ‘pinching’ (like a zoom with two or more fingers). This is *local* and that means that refers to a single tuio object, which is the listener (target) of the gesture event. ‘Local gestures’ have to be produced closed to the target. A parameter in settings.xml specifies that the gesture have to remain within the radius of interaction of the object (instead of only starting in that area). In the demo, the length of the radius is connected to the pinching gesture, so that can be modified.
b) A ‘multitrace’ gesture. This is *global* and that means that the gesture can have multiple targets, not necessarily tuio objects. By dragging one or more fingers from (proximally) an object to another, these are linked together by a line (the thickness is proportional to the number of fingers used for the gesture). This is a sort of drag&drop.
c) Another global gesture is to touch (or drag over) the central point of a link with one finger to remove such link. Internally the gesture recognizer module makes use of a client’s object (the list of current links) as a resource (passed in the GR’s configurator).
d) Lazoing (sorrounding with a trace) one or more tuio objects (global gesture) will select (highlight) them.
e) Tapping (clicking with one finger) close to one tuio object will unselect it.
f) Hovering (hold fingers in a point of the surface) with at least three fingers close to a tuio object will mark it as “requesting a link on channel n” where n is the number of fingers. If two tuio objects are requesting a link on the same channel, they will be linked together.
g) Drawing a circle (the recognition algorithm is to fix) will make a GUI panel to appear/disappear.
h) Moreover the GUI panel has touch-sensible buttons (finger down/up/tap), including a set of three radio buttons which can be selected by dragging a finger over them.
- Visual feedback showing the possible local targets of the Local Gesture Recognizers for a group. The centroid of the group is linked with the central point of the possibly listening object. If a LGR succeeds in its recognition and it’s exclusive, i.e. stops the next GRs in the priority list from processing, then its relative target is focused by a thicker link. In the demo example the LGRs (recognizing the pinching gesture) have been registered with the same priority number, so they’ll work in parallel, but being set as exclusive, the priority goes to the LGR associated with the closest target. In other words, when you pinch, you’re calling one pinching GR for each close target, but only the closest target will receive the gesture event.
The demo runs terribly slow because of Windows Forms that has to render the graphics of the fingers and the objects. Pressing T will hide the tuio graphics and it works fine (on Windows!), however you’ll see nothing but the GUI panel. Mono has problems with Windows Forms. Next demo will be much better!
projects out there
Links to some multitouch frameworks and/or gesture recognition frameworks. If you think I should add anything else (with code) please tell me. They’ve just been coming out like mushrooms.
C#
Bespoke Multi-Touch Framework
Multitouch framework (requires .NET 2.0) – Microsoft Public License (Ms-PL)
http://www.bespokesoftware.org/wordpress/?page_id=41%20/
Tangibilis
WPF framework for multiple tuio input applications – Microsoft Reciprocal License (Ms-RL)
http://www.codeplex.com/Tangibilis
MultiTouchVista
WPF framework for multi-input applications – GNU General Public License (GPL) v2
http://www.codeplex.com/MultiTouchVista
multitouchframework
WPF multitouch framework – GNU General Public License (GPL) v3
http://code.google.com/p/multitouchframework/
dotnetmtf
WPF multitouch framework – New BSD License
http://code.google.com/p/dotnetmtf/
C++ [edit: and Java]
Sparsh-UI
Multiplatform, multitouch gesture recognition framework – GNU Lesser General Public License (LGPL)
http://code.google.com/p/sparsh-ui/
Java
JetMesa (was open-table)
Multiplatform, multitouch application platform – double license, MySQL-like
http://www.open-table.org/ moving to www.jetmesa.com
iGesture
Gesture recognition framework for pen-based applications – Apache 2.0 License
http://www.igesture.org/
check out
Grafiti is available to download from Sourceforge via svn.
svn co https://grafiti.svn.sourceforge.net/svnroot/grafiti/trunk grafiti
The downloaded folder will contain the source code and the Visual Studio solution, that can be opened and compiled also with Mono Develop. You can try the demo with a multitouch table + reactivision or with the newest Tuio simulator.
It’s an alpha version, and it needs to be tested more than I could do, not having a multi touch table yet. Feedbacks are welcome, expecially the negative ones. Soon I’ll publish a guide.
background theory
To begin, I’d like to report some pointers about the background of multitouch and gesture recognition interfaces, a good point from which to start when designing them.
Bill Buxton is a researcher who’s been working on multitouch since 1984. He compiled this interesting overview.
At the MTG I had the pleasure to meet Shahram Izadi, who held a conference called “Hands-on Computing” talking about the history of the user interfaces, the keys for their development, and the state of the art about multitouch in its larger extent.
“Interactive Gestures: Designing Gestural Interfaces”, is a new book by Dan Saffer. Its first chapter is publicly available as a free sample and it also features the NUI group.
Welcome
This blog is about grafiti, my Google Summer of Code project, and it’s main purpose it to show my proceedings to the NUI group, my mentoring organization, and the GSoC community.
Grafiti (with one ‘f’ only) is a, and stands for, Gesture Recognition mAnagement Framework for Interactive Tabletop Interfaces. The infrastructure, using the TUIO protocol, supports multi-touch and multi-user interactions, with particular regard to the use of tangible objects.
I’m in the Music Technology Group of Barcelona’s UPF university, supervised by Martin Kaltenbrunner and Sergi Jorda of the reactable project. From the NUI group I’m mentored by Mikael Bauer from Sweden, my official mentor, and the mentoring organization administrator Paweł Sołyga from Poland. This is also my master thesis in computer science at the University of Pisa.
