article about taxonomy and overview of multitouch frameworks
I’d like to share an interesting article I’ve just found (dated 2010, but I don’t know when exactly), called Taxonomy and Overview of Multi-touch Frameworks: Architecture, Scope and Features, by D. Kammer, M. Keck, G. Freitag and M. Wacker. Featuring various multitouch frameworks, among which Grafiti, it is valuable from both the theoretical and the practical perspectives.
off topic: my future iPhone app
During the past months, I’ve been working on a music application for iPhone and iPod Touch, oriented to live performances and improvisations. My goal was to create something actually useful for musicians and easy to use. Clearly I cared a lot about the user interaction. The development is approaching the beta testing phase.
This app will be the main topic of my new blog http://alessandrodenardi.wordpress.com/
[edit: The app, called Ricepad, is now available on the app store. The official site is www.ricecode.com]
See you there!
changing license
About a year has passed since I suspended my activity on this project. I would have liked to work a lot more, hopefully within a Ph.D. program, but unfortunately I didn’t find the opportunity.
Even if I don’t think that changing the license to this software will be of much help to live out of the theoretic interests, a permissive license is more suited for a framework. So I decided to release it under the MIT license.
I hope to come back here, but it won’t be too soon anyway… Right now I’m working on another project, and that’s the topic of my next post!
key points
As a summary, here are the theoretical key points of the project, as a proposed strategy to face the problem of multitouch (tabletop) gestural interfaces, stated in the thesis’ abstract (see previous post). The implemention is described in italics, between the brackets.
Note: this is taken from the slides of the thesis discussion, that I’ll eventually publish.
- Tangible objects. Integration with GUI, natural interaction. (Non-Z-ordered target areas.)
- Contextuality of gestures with respect to the current state of the interface (Access to the client’s environment plus global/multi-target gestures).
- Resolution of the conflicts. Which is the actual target of the gesture? Which is the actual meaning of the produced gesture? (Flexibility in designing gestures plus a priority mechanism.)
example of (ambiguous) gesture
In the picture above, the user producing the gesture may mean to link the objects A and C, or to remove the existing link between A and B.
thesis
It has been a hard year but finally I graduated.
I still don’t know the future of this project, nor mine!, but in the meantime here’s my thesis.
Soon I’ll upload some new code, at least to reflect the contents of the dissertation, though I’m afraid it will stay in the alpha phase for a while. More applications and tests are needed and I don’t have a multitouch table anymore.
Abstract
Surface computing is opening new horizons on human-computer interaction, in particular by the use of multitouch and tangible user interfaces (TUIs). However, the investigation into theoretic and practical aspects of the development of this kind of interfaces is still in its infancy, leading to a general tendency to adopt simple gestural paradigms or worse, to decline into the GUI and WIMP canons.
In this context, the present thesis introduces Grafiti, a Gesture Recognition mAnagement Framework for Interactive Tabletop Interfaces. This tool is meant to aid the design of rich multitouch gestural interfaces, focusing on the possibility to use tangible ob jects as targets, promoting a seamless natural interaction between the TUI and the GUI.
In form of a C# layer, Grafiti is independent of the client’s software environment and, using the Tuio protocol, it abstracts from the underlying input device. It can be extended with new gestures and recognition algorithms, offering functionalities that cover various aspects of the implementation of interfaces with different levels of complexity, and encouraging to develop structured and reusable code.
proceedings
Google summer of code has finished. The project has not however. Although it’s still in the alpha phase it’s usable and has reached a point where there are all the main functionalities.
I’m currently writing the documentation, thinking about improvements (like supporting composition of gestures) and developing a better demo in C++, with OpenGL, and multiplatform. By the way, the current C++ demo in svn works only in Windows, because it uses mixed code (C++/CLI and C#) and mono doesn’t like it. However Mono doesn’t mind if managed code use P/Invoke so that’s the way to go.
I’ll be working on the project at least until november.
building demo app
I decided to edit (again) this post instead of creating a new one, since the demo application under discussion is still under development.
- grafiti demo app
In order to show the main features of the framework, how it works and how it can be used, I’ve started to develop a demo application just for this purpose. It’s available to download via svn. However without an (upcoming) documentation it’s still hard to understand many things of the framework.
Features of the demo (in short):
– Add/move/remove tangible objects. These objects, (internally related to TuioObject so from now on I’ll refer to them simply as tuio objects) are listeners of gesture events. They’re represented by coloured squares and their round area of interaction as it is explained below.
– Visual feedback showing how fingers are grouped together. A group will be considered as producing a gesture (or more). A different color is associated to each group of fingers. The traces have a losing colored tail and their head shows the id number of the trace. The head also says if the cursor has been removed, i.e. the finger released (black circle) of not (grey circle). The fingers composing a group are linked by lines towards the centroid calculated on the present fingers and the recently removed ones. Finally, a semitransparent round area indicates where another finger can be put on the surface in order to be added to the same group of the finger.
– There are various kind of gestures that can be recognize:
a) The gesture of ‘pinching’ (like a zoom with two or more fingers). This is *local* and that means that refers to a single tuio object, which is the listener (target) of the gesture event. ‘Local gestures’ have to be produced closed to the target. A parameter in settings.xml specifies that the gesture have to remain within the radius of interaction of the object (instead of only starting in that area). In the demo, the length of the radius is connected to the pinching gesture, so that can be modified.
b) A ‘multitrace’ gesture. This is *global* and that means that the gesture can have multiple targets, not necessarily tuio objects. By dragging one or more fingers from (proximally) an object to another, these are linked together by a line (the thickness is proportional to the number of fingers used for the gesture). This is a sort of drag&drop.
c) Another global gesture is to touch (or drag over) the central point of a link with one finger to remove such link. Internally the gesture recognizer module makes use of a client’s object (the list of current links) as a resource (passed in the GR’s configurator).
d) Lazoing (sorrounding with a trace) one or more tuio objects (global gesture) will select (highlight) them.
e) Tapping (clicking with one finger) close to one tuio object will unselect it.
f) Hovering (hold fingers in a point of the surface) with at least three fingers close to a tuio object will mark it as “requesting a link on channel n” where n is the number of fingers. If two tuio objects are requesting a link on the same channel, they will be linked together.
g) Drawing a circle (the recognition algorithm is to fix) will make a GUI panel to appear/disappear.
h) Moreover the GUI panel has touch-sensible buttons (finger down/up/tap), including a set of three radio buttons which can be selected by dragging a finger over them.
– Visual feedback showing the possible local targets of the Local Gesture Recognizers for a group. The centroid of the group is linked with the central point of the possibly listening object. If a LGR succeeds in its recognition and it’s exclusive, i.e. stops the next GRs in the priority list from processing, then its relative target is focused by a thicker link. In the demo example the LGRs (recognizing the pinching gesture) have been registered with the same priority number, so they’ll work in parallel, but being set as exclusive, the priority goes to the LGR associated with the closest target. In other words, when you pinch, you’re calling one pinching GR for each close target, but only the closest target will receive the gesture event.
The demo runs terribly slow because of Windows Forms that has to render the graphics of the fingers and the objects. Pressing T will hide the tuio graphics and it works fine (on Windows!), however you’ll see nothing but the GUI panel. Mono has problems with Windows Forms. Next demo will be much better!
projects out there
Links to some multitouch frameworks and/or gesture recognition frameworks. If you think I should add anything else (with code) please tell me. They’ve just been coming out like mushrooms.
C#
Bespoke Multi-Touch Framework
Multitouch framework (requires .NET 2.0) – Microsoft Public License (Ms-PL)
http://www.bespokesoftware.org/wordpress/?page_id=41%20/
Tangibilis
WPF framework for multiple tuio input applications – Microsoft Reciprocal License (Ms-RL)
http://www.codeplex.com/Tangibilis
MultiTouchVista
WPF framework for multi-input applications – GNU General Public License (GPL) v2
http://www.codeplex.com/MultiTouchVista
multitouchframework
WPF multitouch framework – GNU General Public License (GPL) v3
http://code.google.com/p/multitouchframework/
dotnetmtf
WPF multitouch framework – New BSD License
http://code.google.com/p/dotnetmtf/
C++ [edit: and Java]
Sparsh-UI
Multiplatform, multitouch gesture recognition framework – GNU Lesser General Public License (LGPL)
http://code.google.com/p/sparsh-ui/
Java
JetMesa (was open-table)
Multiplatform, multitouch application platform – double license, MySQL-like
http://www.open-table.org/ moving to http://www.jetmesa.com
iGesture
Gesture recognition framework for pen-based applications – Apache 2.0 License
http://www.igesture.org/
check out
Grafiti is available to download from Sourceforge via svn.
svn co https://grafiti.svn.sourceforge.net/svnroot/grafiti/trunk grafiti
The downloaded folder will contain the source code and the Visual Studio solution, that can be opened and compiled also with Mono Develop. You can try the demo with a multitouch table + reactivision or with the newest Tuio simulator.
It’s an alpha version, and it needs to be tested more than I could do, not having a multi touch table yet. Feedbacks are welcome, expecially the negative ones. Soon I’ll publish a guide.
grafiti 