an entry from

Piotr's R&D blog

I'm getting this idea out of the way because I've recently discovered that it's (mostly) been done.

The problem

The scenario: some kind of application that supports colocated collaboration on a tabletop display. The app involves moving around elements that have a preferred orientation relative to the viewer and can be resized, but should not otherwise be deformed. (Think "Video Bench" if you've seen it, or a collaborative photo album editing app.) You immediately run into two visualization problems:

1. The users are sitting on different sides of the table, and hence have different preferred orientations for the elements. Some elements are shared between the participants and might need to be reoriented on demand. Some elements are semi-private and will always be facing their owner.
2. The tabletop surface is small and the display resolution is low. For editing, the elements must be large enough to show detail despite the low resolution. But if all the elements are large, you'll run out of space or have lots of overlap. So you need to manage element sizes.

A solution

The simplest solution to the problem is to give users direct control over the orientation and size of each element. However, unless these manipulations are actually the goal of the application (e.g. laying out a floorplan), they are pure overhead for the users and will distract them from the primary task. We'd like to make orientation and sizing automatic, while still giving the user some control.

Element positioning is probably the only generic manipulation you can't live without in this kind of app; the user simply must be able to reposition the elements as they desire. The obvious idea then is to use position to control orientation and size of the element as well. For example, if we somehow know where each user is, the elements get bigger as they get closer to the user and always rotate to face the person interacting with the element.

Great idea, but as I found out today it's been done. Read this paper before continuing.

So what's left to do? Here's my take on the limitations of the approach reported above:

1. It's specific to a circular tabletop. ("Well, duh!" mutters the audience.) It's cool, but doesn't match up with the reality that most tables and projectors are decidedly rectangular in shape. I'm not convinced the polar coordinate system would be as useful on a rectangular surface.
2. It seems to have only two automatic orientation modes: all elements face away from the center, or all elements face the same global direction. This is very limiting. In the first scheme, the orientation of elements "between" users is useless to all participants (i.e. nobody is facing them). In the second scheme, the collaborative aspect is forgotten as one participant's preference effectively takes over the whole table.
3. The orientation and sizing mode is global: the same rules apply all over the table. There is no way to create local "zones" with different rules that may be changed independently.

A better solution?

Here's my idea on a superset of the circular design. The tabletop surface is partitioned into spaces. Each space has its own local coordinate system and, for every point in the space, determines the desired orientation and scaling factor for that point. One concrete but simple implementation would be to have a "baseline" in each space, which could be any 1D shape (e.g. a straight line segment, a circle, a point, discontinuous lines, etc.). Given a point of interest, find the shortest vector between the point and the baseline. The vector's direction gives you the desired orientation (like gravity), while the vector's length is inversely proportional to the scaling factor. The circular "central focus" mode is equivalent to putting a basedot in the centre; the "black hole" mode is a circle around the table's circumferance; the "magnet" mode is a baseline in front of the desired user.

What other interesting properties does this scheme have? For one, each user can have their own space, and arrange its "distortion field" in whatever way suits them. Also, a baseline drawn with a single finger specifies the field for the whole space, a very powerful yet simple interaction. The arbitrary shape of the baseline gives complete freedom in customizing a space, and spaces are no longer specific to a circular display.

Here's a concrete example. Say each user draws a straight baseline at their "bottom" of the display, approximately the width of their body. Then each space would have the following characteristics:

1. All elements in front of the user face her properly.
2. Elements to the sides are arranged in quarter-circles centred at the baseline's endpoints, facing towards the user at an angle.
3. Elements further away (up) from the user or to the sides become smaller with distance.

Control issues

There is a non-obvious problem common to both approaches: since an element is usually larger than a pixel, which point do you use when determining the applicable distortion? I can think of three options, none very inviting:

1. Use the centre point of the element for both determining and applying the distortion. This is nice, since the same point will always be used for each element. However, if the user doesn't "grab" the centre of the element when starting to drag it, the element could shrink/rotate out from underneath the user's finger, since the transformation will be about the centre. (For example, I grab a large element near a corner, and move it so its center is in a "small size" area. The element becomes smaller, and my finger is no longer within the element's boundaries.)
2. Use the centre point to determine the distortion, but apply it at the grab point. This is a terrible idea, since applying the transform will shift the element's centre, thus changing the distortion factors, necessitating another distortion, etc., ad infinitum if you're not lucky. The converse (determine at grab point, apply at centre) is equally bad.
3. Use the grab point to determine and apply the distortion. This leaves the element under the user's finger at all times, and gives a nice "physical" feeling to the movement. However, if the user drags the element using one corner, drops it, then grabs it by another corner, the applicable distortion changes suddenly and maybe unpredictably. You can also get into situations where two elements are visually located "in the same area", but have very different distortions applied since they were grabbed at different points.

I tend towards the last alternative, but I'm not sure how intuitive it would be. I'd be curious to find out what algorithm is used for the circular table.

A last remaining question is how does the table get partitioned into spaces? How are the borders determined, and what happens when a new space gets created? I don't have a good answer to this yet. I think users should be able to control the "strength" of each space, and this would provide the input to some kind of balancing algorithm that would settle the partition. When a new space is created, other spaces must adjust their borders, but it's unclear how to shift elements around. However, this would allow for some cool effects: imagine that you can drop elements into a trashcan. If you want to recover an element, you open the trashcan through some icon, and it "warps in" a space right over top of the icon with all the deleted elements in it, temporarily pushing away the elements in your space. When you close the trashcan, your space recovers its original shape and things return to normal. I think this could be modeled by using a 2D mesh for each space, kind of like "warp" deformations in typical paint tools.

Looking forward

As a final crazy idea, what if this scheme was applied to windows on a typical desktop? Granted, it might only make sense with a larger screen, and the orientation changes might be undesirable, but it's an interesting thought experiment. Compare and contrast this with the Mac OS X Exposé feature.

So, do you think there's still some interesting work to be done in this area? Are my ideas feasible and worth going for, or am I out to lunch? Or have I missed more literature and "it's all been done" already?