OSCulator 3.0 for whiteboard: grouped calibration?

[Antioche]

Hi,

We have so far been using WiimoteWhiteboard (http://uweschmidt.org/projects/wiimote-whiteboard) in the past to track IR points for our art performance Ooo-Ya-Tsu (https://2hd.uk/oyt).
This has not worked so well, with both a very limited set of different wiimote models that could connect to that software and rather unstable/messy connection process.

When it did work, though, the output was very simple to process in Processing (simulating mouse drag when a IR point appeared on the canvas).

As we were already using OSCulator to route the OSC from the Processing program and the musician's MIDI instruments, I looked yesterday into the possibility of using it also for the Wiimote and was delighted to discover that the new version supported whiteboard application with wiimotes! So first of all, thank you Camille! Of course, I immediately bought a second license for Osculator...

Our set up uses 2 wiimotes (or ideally more), overlooking from two different angles a video projection of a large paper canvas (roughly 1.5 x 4 m) laying on the floor, on which we are using real paintbrushes and pencils equipped with small IR LEDS. The two Wiimotes are useful to avoid self shading of the IR points while we work on the canvas. So this is basically a standard whiteboard setup.

I have been playing with the whiteboard functionality this morning to review how to calibrate the system.
It seems that, when the TUIO option is activated on a wiimote, it requires through a calibration process (4 points to "click" with the IR LEDS).

However, when using several wiimotes to "watch" the same canvas, the calibration needs to be run on each wiimote, rather that all calibrating at the same time.
Is this correct?

This, however, results in rather jumpy cursor position when assigning a Mouse (absolute X & Y) event to /tuio/2D/cur/0, as any slight misalignment between the different calibrations causes the cursor to wiggle.
WiimoteWhiteboard had an simple way to calibrate all wiimotes at the same time, so that the 4 calibration IR "clicks" would be seen at the same time by multiple wiimotes, resulting in very stable cursor position in use.

So my question: is there a way to calibrate all wiimotes at once?
Or am I misunderstanding how that works in OSCulator?

Note that I have tried to select multiple wiimotes (by selecting several units in the Wiimote device list with the Command key down) and clicking on the TUIO "Calibration" button, hoping that it would calibrate all the selected wiimotes at the same time.
But that didn't work: same wiggling cursor effect...
Looking forward to your feedback.

Thibaut
2hD Architecture Workshop

[camille]

Hi Thibaut!

This practical report is really useful.
Until now, the "Whiteboard" feature has been used in a live show, but only with one Wiimote I believe.

However, when using several wiimotes to "watch" the same canvas, the calibration needs to be run on each wiimote, rather that all calibrating at the same time.
Is this correct?

Yes at the moment.

[QUOTE]This, however, results in rather jumpy cursor position when assigning a Mouse (absolute X & Y) event to /tuio/2D/cur/0, as any slight misalignment between the different calibrations causes the cursor to wiggle.
WiimoteWhiteboard had an simple way to calibrate all wiimotes at the same time, so that the 4 calibration IR "clicks" would be seen at the same time by multiple wiimotes, resulting in very stable cursor position in use.[/QUOTE

Got it. I am not sure if this will improve cursor stability, but I understand the point of simultaneous calibration.
I am not sure how easy it would be to implement, though, but I can look into it.

So my question: is there a way to calibrate all wiimotes at once?
Or am I misunderstanding how that works in OSCulator?

No at the moment, as you guessed, calibration is watching events from a single Wiimote, but that can be improved in the future.

[...] But that didn't work: same wiggling cursor effect...

OK, have you tried with a single Wiimote?
Do you also have a wiggling cursor effect?
My point is that single Wiimote data should work well for a start in OSCulator, then we can look into adding more Wiimotes.

Another question: by using two Wiimotes, are you looking for more reliable, single TUIO stream, or a you looking for two independant TUIO streams?

Best,
Cam

[Antioche]

Hi Camille,
Many thanks for the swift reply.

The reason for using multiple wiimotes (2, ideally 3) simultaneously to track our IR LEDs is that as we move around the canvas where we draw and paint, we often obstruct the view from a single wiimote (either with our bodies, or by changing the direction of the LED IR beam by moving the tools holding the LEDs themselves). We have been constrained so far by ceiling heights to position our wiimotes away and at an angle (about 45 deg) to make sure that each remote could "see" the whole canvas on the floor. Our ceiling mounted projector (projecting onto this canvas) has a fairly wide angle, wider than the wiimotes, so we cannot position the wiimotes straight above to reduce self-shading.,,

OK, have you tried with a single Wiimote?
Do you also have a wiggling cursor effect?
My point is that single Wiimote data should work well for a start in OSCulator, then we can look into adding more Wiimotes.

The tracking with a single wiimote is very smooth and stable.
The wiggling cursor effect is clearly the result of the slightly different calibration of the two wiimotes in OSCulator. As I move the cursor with the IR point, the cursor oscillate between two positions, but this oscillation stops as soon as I obstruct the view from one wiimote, at which point the cursor stabilise (very smoothly) to the other wiimote's tracking. Although I was very careful to align the two calibration, the parallax between the two wiimotes probably offsets the calibration points slightly, resulting in different calculated coordinates.
Doing a very inaccurate of the two wiimotes on purpose results in even larger oscillation. This oscillation is also clearly visible when previewing the /tui/2D/cur/x position for instance.

Another question: by using two Wiimotes, are you looking for more reliable, single TUIO stream, or a you looking for two independant TUIO streams?

I am looking for a single unified stream that can cope with self-shading and obstructions of the sight line between one of the wiimotes and the IR LEDs.

I am not sure how easy it would be to implement, though, but I can look into it.

Not sure how easy it would be, but here's how I would imagine it:
At the moment it is possible to select multiple wiimote devices in the parameter window (CMD+Click) and to then check/uncheck options of the multiple devices at ones.
Maybe selecting the several wiimotes at once and clicking the TUIO calibration button would share the calibration of all these device at once also, so that the all use the same calibration?
When "clicking" on a calibration point, the /wii/#/ir/ coordinate of all selected wiimotes could be collected and the calibration of the /tui/2D/cur/ done for all selected wiimotes?
This would not require any UI change and would follow the existing behaviour (configuring multiple devices at once). And this way, several wiimotes could be used for a single TUIO stream and others left as independent TUIO stream.
Aside from being more accurate, this would also speed up setup and allow a lot of flexibility for white boarding applications.

Does this sound like a realistic/feasible approach?

Cheers,

Thibaut

PS: In fact, we have an artist residency for this project starting on Monday, before a live performance as part of a festival on Thursday 25/11. So if we could sort this out before then it would be awesome!

[camille]

Hi Thibaut,

At the moment it is possible to select multiple wiimote devices in the parameter window (CMD+Click) and to then check/uncheck options of the multiple devices at ones.
Maybe selecting the several wiimotes at once and clicking the TUIO calibration button would share the calibration of all these device at once also, so that the all use the same calibration?

Unless I miss something useful, I would rather use the list of currently connected Wiimotes. This would therefore require no user interaction: just connect your Wiimotes, click "calibrate", and you're done.

When "clicking" on a calibration point, the /wii/#/ir/ coordinate of all selected wiimotes could be collected and the calibration of the /tui/2D/cur/ done for all selected wiimotes?

Yes, but the calibration UI must be updated in order to see what Wiimote acquired the calibration point, and move to the next calibration point only when all Wiimotes acquired the current calibration point.

This would not require any UI change and would follow the existing behaviour (configuring multiple devices at once). And this way, several wiimotes could be used for a single TUIO stream and others left as independent TUIO stream.

So, I understand that you would like to "group" Wiimote streams into a single TUIO stream?
That is an interesting idea, but rather complex to implement properly; calibration is only a portion of the problem.

If I allow calibration for multiple Wiimotes at once, is it OK for your to deal with several TUIO streams?
This way I can quickly implement the feature, and leave time to think about the fusion of multiple TUIO streams.

Cam

[Antioche]

Hi,

Unless I miss something useful, I would rather use the list of currently connected Wiimotes. This would therefore require no user interaction: just connect your Wiimotes, click "calibrate", and you're done.

Yes that would work for me, but maybe too constraining for people who would like to manage several TUIO streams (for instance several wiimotes, looking at different surfaces and tracking independent objects?)

Yes, but the calibration UI must be updated in order to see what Wiimote acquired the calibration point, and move to the next calibration point only when all Wiimotes acquired the current calibration point.

WiimoteWhiteboard was using the following system:
When calibrating (always with all wiimotes connected), a click on one of the calibration point would add a green tick to it only if all wiimotes have seen that click. Otherwise, a red cross appears. In that case, the calibration must be started again (most likely because one of the calibration points was out of sight from one of the wiimotes and that wiimote's orientation needs to be adjusted to cover the whole calibrated area). Having the red cross is useful to inform the user that the click was seen by at least some wiimotes (rather than a broken IR LED!)
This is simple and fairly intuitive.

So, I understand that you would like to "group" Wiimote streams into a single TUIO stream?
That is an interesting idea, but rather complex to implement properly; calibration is only a portion of the problem.

I have no need for grouping them into different streams.
Some people might find that useful (for instance groups of two wiimotes looking different surfaces for IR points), but my use is simpler and fairly similar to what happens now:
I calibrate each wiimote to look at the same surface, calibrating them one after the other. The result is a single /TUIO/2D/cur that is teh synthesis of all these calibrated wiimotes.
This is perfectly fine for my application. The only missing bit is the global calibration for all wiimotes to get more stable coordinates when several wiimotes look at the same point.
Or some sort of averaging between the different cursor values of the different wiimotes?

Does this sound feasible?

Cheers,

T.

[camille]

When calibrating (always with all wiimotes connected), a click on one of the calibration point would add a green tick to it only if all wiimotes have seen that click. Otherwise, a red cross appears. In that case, the calibration must be started again (most likely because one of the calibration points was out of sight from one of the wiimotes and that wiimote's orientation needs to be adjusted to cover the whole calibrated area). Having the red cross is useful to inform the user that the click was seen by at least some wiimotes (rather than a broken IR LED!)

This is basically what I has in mind, thank you for the description.

I have no need for grouping them into different streams.
Some people might find that useful (for instance groups of two wiimotes looking different surfaces for IR points), but my use is simpler and fairly similar to what happens now:

I am not sure I was properly understood:

With two Wiimotes, you will get one TUIO stream with two cursors.
But in your use case, what you'd really want is one TUIO stream with one cursor, being the results of "grouping" or "consolidating" the results from both Wiimotes. This would be typically done by using a Kalmann filter, but requires careful signal analysis.

I calibrate each wiimote to look at the same surface, calibrating them one after the other. The result is a single /TUIO/2D/cur that is teh synthesis of all these calibrated wiimotes.

Yes, there is a single TUIO stream, but bear in mind that there are two cursors.
I understand you are looking for one consolidated cursor.

This is perfectly fine for my application. The only missing bit is the global calibration for all wiimotes to get more stable coordinates when several wiimotes look at the same point.
Or some sort of averaging between the different cursor values of the different wiimotes?

We will start with a single calibration for all connected Wiimotes, then we will see from there how it goes.

Thank you, this is very useful to share those practical details!

[Antioche]

Hi,

With two Wiimotes, you will get one TUIO stream with two cursors.
But in your use case, what you'd really want is one TUIO stream with one cursor, being the results of "grouping" or "consolidating" the results from both Wiimotes. This would be typically done by using a Kalmann filter, but requires careful signal analysis.

Hmm... In the current OSCulator, if I calibrate each wiimote independently, I still get only one cursor, don't I?
Attached is my current setup, converting the /tui/2D7cur/0 into Absolute mouse coordinates.
Once I have calibrated the two wiimotes (independently), I can move the mouse cursor with an IR point on the canvas. The cursor is "wiggling" (I assumed oscillating between the two sets of coordinates calculated by each wiimote), but if I Obstruct one of the wiimotes, the wiggling stops, presumably aligning with single coordinated of the other wiimote. So I guess that the /tui/2D/cur/0 was already a composite of the two wiimotes, forming a single cursor, as I was not converting any other TUIO cursor (such as /tui/2D/cur/1) to mouse coordinates. I thought I would only be able to get multiple cursor if several different IR points were seen by the Wiimote.

Or I am getting this wrong?
Is that the proper approach to white boarding with OSCulator?

We will start with a single calibration for all connected Wiimotes, then we will see from there how it goes.

That sounds like a great start!

Thank you, this is very useful to share those practical details!

Thank you for the great support!

Looking forward to testing this more.

Cheers,

Thibaut

[camille]

Hi Antioche,

Hmm... In the current OSCulator, if I calibrate each wiimote independently, I still get only one cursor, don't I?
Attached is my current setup, converting the /tui/2D7cur/0 into Absolute mouse coordinates.

Yes, sorry I was confusing Wiimote identifiers and TUIO cursors.
Nonetheless, I found a bug in Wiimote TUIO synthesis, where the source property would have the wrong identifier

Once I have calibrated the two wiimotes (independently), I can move the mouse cursor with an IR point on the canvas. The cursor is "wiggling" (I assumed oscillating between the two sets of coordinates calculated by each wiimote),
but if I Obstruct one of the wiimotes, the wiggling stops, presumably aligning with single coordinated of the other wiimote. So I guess that the /tui/2D/cur/0 was already a composite of the two wiimotes, forming a single cursor, as I was not converting any other TUIO cursor (such as /tui/2D/cur/1) to mouse coordinates. I thought I would only be able to get multiple cursor if several different IR points were seen by the Wiimote.

You are right, the wiggling was caused by incorrect calibration.
I will send you a test version soon.


Best,
Cam

[Antioche]

Hi Camille,
Just a quick word to thank Camille for his fantastic support of this issue and providing us with the fixes we needed to get this to work.
We have used the new TUIO/Whiteboarding functionality for our latest public performance of our art project Ooo-Ya-tsu (https://2hd.uk/oyt) and the system worked flawlessly through for several days in continuous use with two wiimotes monitoring a large floor area. The grouped calibration works well and the IR point tracking was very smooth.

The system we previously used for this tracking, the free and open-source WiimoteWhiteboard (http://uweschmidt.org/projects/wiimote-whiteboard) worked well too, but we (and most other users of the software on MacOS according to the forum) had tons of stability problems, both with stability in use and with the connection of wiimotes, which required complicated routines to by-pass MacOS' bluetooth stack and only worked with very few wiimotes models (basically first generation Nintendo wiimotes). I am very grateful to the developer of WiimoteWhiteboard, but there is no looking back after having tried OSCulator 3 for the same task: easy pairing, simple connections, works with basically any wiimotes I could throw at it (we had 6 different models/brands).

Now exploring the other capabilities of OSCulator!
Cheers,

Thibaut
2hD Architecture Workshop