Multimodal Affect Recognition for Unseen Visitors in a Free-Roaming VR Museum

Can wearable sensors tell a museum guide how a visitor feels, second by second, for a visitor it has never met?

Akshaj Somanpura Rajendran
MSc Interaction Technology · Final colloquium
Supervisors: Shenghui Wang and Funda Yildirim
University of Twente · 23 October 2026

The setup

Reading affect from a visitor the system has not encountered

A virtual reconstruction of a museum room holding five paintings. Visitors move through it without constraint, and a conversational agent responds to the painting a visitor’s gaze settles on.
Five sensor streams are recorded throughout: pupil and gaze, electrodermal activity, cardiac activity, speech acoustics and speech content.
Visitors rate their own affect once per second, on two continuous scales: valence, from unpleasant to pleasant, and arousal, from calm to excited.
The ratings were obtained by two methods, live during the visit and retrospectively from a replay. The two do not converge on a single ground truth, so they are analysed separately throughout.

The guide has never encountered this visitor. Every score presented is measured on a visitor held out of training.

VISITORS

19,534

RATED SECONDS

SENSOR STREAMS

The setup2

Agenda

Structure of this presentation

	Section	What it covers
1	The detection margin	How large a difference has to be before this evidence can call it a difference
2	Data characterisation	What the recordings actually contain, measured before any modelling
3	Five experiments	Sensors alone; sensors combined; an ordinal target; the attended painting and where the variance sits; personalisation
4	Conclusion	What it means, what it contributes, where it is limited, and what I would do next

Agenda3

The detection margin · method

The margin is fixed before any comparison is made

Any score carries noise. Without knowing how much, a real difference cannot be told from a fluctuation.
One reference cell was fixed in advance, the cardiac features against live arousal, and rerun twenty times.
Each run redraws which visitors go into training, so the spread measures how far a score moves when the sample of people changes.
The cell is not one the later comparisons go on to select, so the margin cannot be tuned to flatter a result.

This is the question that matters for an unseen visitor: would a difference survive a different sample of people?

The detection margin4

The detection margin · result

The margin is 0.08 on the continuous score, 0.21 on the ordinal

The twenty runs span -0.031 to +0.074, four of them negative. The margin is 0.08 CCC, and 0.21 on the ordinal metric. A difference smaller than the margin is reported as undetected, never as evidence that two candidates are equal.

The detection margin5

Data characterisation · method

Three passes check the recordings before any modelling

Coverage, on the raw recordings before any cleaning. A second counts as covered if the raw sensor produced at least one sample within it.
Signal quality. Sampling rates, timestamps, value ranges and completeness verified against what each instrument should have produced, so that no later result rests on a channel that was mislabelled, misaligned in time, or silently empty.
Visiting behaviour, read from the engine’s own log: first painting, visiting order, dwell, revisits, and engagement with the in-VR guide.

The third pass is descriptive and independent of every feature choice in the thesis.

Data characterisation6

Data characterisation · result

Twenty of 47 electrodermal recordings are unusable

Channel	What the audit found
Electrodermal	20 of 47 recordings carry no usable signal. 18 are negative for at least half their length, which a skin conductance cannot be; 2 more are almost flat. The failures cluster in time: 15 of 22 in the second collection wave against 5 of 25 in the first, consistent with electrode wear.
Eye tracker	One sample per rendered frame, so a median 23 Hz against a nominal 120, and slowest at the paintings themselves (18 samples per second there against 47 elsewhere). The two eyes agree at a median r = 0.92.
Blood-volume pulse	Exactly 64 Hz in every recording, so no samples are lost. Three sessions have no recording inside the session window.
Speech	Visitors spoke little: a median 6.9% of analysed seconds (1.2 to 23.0%). Eight transcripts are near-empty.
Annotations	In the live stream, 20.0% of valence seconds rest exactly on the scale midpoint, where an untouched joystick sits. In the retrospective stream, none do.

Data characterisation7

Experiment 1 · RQ1 · method

Five settings searched, eight model families compared

Step 1. For each sensor, search the preprocessing and feature choices the literature leaves open, five parameters, each decided inside the training folds.
Step 2. With those settings fixed, compare eight model families, from ridge regression to four gradient-boosting libraries, so that the choice of library cannot decide the result.

Every score is measured on a visitor held out of training, with imputation and tuning fitted inside the training fold only.

Experiment 18

Experiment 1 · result

No sensor separates from the chance-corrected floor

Mean per-visitor CCC in each sensor-by-slice cell. Zero is the score of a model carrying no information about the held-out visitor. Every value lies between -0.072 and +0.084, narrower than the single 0.08 margin.

Experiment 19

Experiment 1 · result

Neither the model family nor any single feature accounts for it

Eight families span 0.0124 CCC, approximately one seventh of the margin.
And before any model, every feature was tested directly against the ratings: 436 tests, none survives correction.

The result does not depend on the model chosen. At this cohort size no feature relates to the ratings beyond chance.

Mean CCC over all cells, by model family.

Experiment 110

Experiment 1 · findings

Experiment 1 settles the pipeline and the 0.08 margin

No sensor recovers the valence or arousal of an unseen visitor in this cohort. All twenty sensor-by-slice scores fall between -0.072 and +0.084, against 0 for a model carrying no information about that visitor.
Neither the settings searched nor the model families compared account for this. Five parameters were searched and eight families compared, spanning 0.0124, well inside the margin.
No feature relates to the ratings beyond chance at this cohort size: of 436 direct tests, none survives correction.
Carried into Experiment 2: these settings, extremely randomised trees, and the margin of 0.08.

Experiment 111

Experiment 2 · RQ1 · method

Two dimensions vary: sensor subset and combination mechanism

Which sensors. Five sensors give 31 non-empty subsets, from one sensor to all five.
Two further steps. Four ways of aligning the sensors’ timelines were compared first. And a block of engineered columns, measuring how two sensors move relative to one another, was offered to early fusion as an addition.

Mechanism	How the sensors are joined
Early fusion	Concatenate every feature into one table; a single model decides how to weigh the sensors.
Simple averaging	One model per sensor; average their predictions with equal weight.
Weighted averaging	One model per sensor; weight each by its own score on the training visitors, with a negative weight floored to zero so that a poor sensor cannot invert the prediction.

Experiment 212

Experiment 2 · result

The weighting rule can reduce a subset to a single sensor

rcea valence, top five rows

Combination	Mechanism	CCC
EDA	simple	+0.03888
EDA	early	+0.03888
pupil + EDA	simple	+0.02940
EDA + cardiac	weighted	+0.02878
EDA + cardiac	simple	+0.02680

retro valence, top five rows

Combination	Mechanism	CCC
cardiac	simple	+0.08224
cardiac	early	+0.08224
pupil + cardiac	weighted	+0.08224
EDA + cardiac	weighted	+0.08224
cardiac + speech acoustics	weighted	+0.08224

rcea arousal, top five rows

Combination	Mechanism	CCC
pupil + EDA + cardiac + speech	early	+0.05989
pupil + EDA + cardiac	early	+0.05392
cardiac + speech content	early	+0.05028
pupil + cardiac + speech	early	+0.04952
pupil + cardiac	early	+0.04789

retro arousal, top five rows

Combination	Mechanism	CCC
pupil + speech acoustics	early	+0.04933
pupil + speech ac. + content	early	+0.04481
pupil + speech content	early	+0.03330
pupil + speech acoustics	simple	+0.03242
pupil + speech acoustics	weighted	+0.03242

On retro valence the five best rows carry an identical score, because every sensor but cardiac was floored to zero weight. Across all 316 combination rows, 15 are identical to one of their own sensors; every one is weighted averaging, on that one slice.

Experiment 213

Experiment 2 · result

No combination exceeds the best single sensor

Per slice, the best single sensor against the best combination of two or more sensors. Two slices favour combining, by +0.016 and +0.021; two favour the single sensor, by 0.010 and 0.012. All four lie inside the margin. Overall +0.0707 against +0.0822.

Experiment 214

Experiment 2 · findings

Experiment 2 settles the fusion arm and a second margin

No combination of sensors scores above the best single sensor. The best combination of two or more sensors reaches +0.0707, against +0.0822.
No way of reconciling the sensors’ timelines separates from another. The four alignment methods span 0.006, against the margin of 0.08.
The cross-sensor columns do not score above their own rotated surrogate: they exceed it in 1 of 12 cells, where chance alone produces 0.6.
Carried into Experiment 3: early fusion of all five sensors with the cross-sensor block, at +0.0271, and a second margin of 0.21 on the ordinal score.

Experiment 215

Experiment 3 · RQ2 · method

The target is reduced to three to seven ordered levels

Two arms. The trained-on-levels arm bins the ratings first and trains a classifier to predict the level directly. The post-hoc arm keeps the continuous prediction and assigns it to a level afterwards, by the same fixed boundaries.
Level counts swept from three upward. Only three to seven are selectable: above seven, some level holds under one per cent of the rated seconds.
Metric: pooled quadratic weighted kappa, where zero denotes chance, with balanced accuracy as a cross-check.

The two annotation streams are binned separately, because only the live stream has a resting position at the scale midpoint.

Experiment 316

Experiment 3 · result

An ordinal target does not raise agreement above chance

Carried forward: three levels, trained-on-levels. The two arms differ by 0.0123; all five counts tie in every slice, with per-slice bests of 5, 5, 7 and 4. Nothing separates, so the convention takes over and the simplest count is carried.

Each point is one level count on one slice, for both arms. Forty values, almost all at or below chance, the whole result inside one margin. The two arms differ by 0.0123; across eighteen counts the trained-on-levels mean moves 0.054.

Experiment 317

Experiment 3 · findings

Experiment 3 settles three levels as the target

No count of ordered levels separates agreement from chance at this cohort size. Pooled agreement is negative on every slice but retro valence.
No granularity separates from another. Across eighteen counts the trained-on-levels mean moves 0.054, against the margin of 0.21.
Only five counts are selectable. Above seven levels some level holds under one per cent of the rated seconds, falling to 0.03% at twenty.
Carried into Experiment 4: three levels and the trained-on-levels arm.

Experiment 318

Experiment 4 · RQ3 and RQ4 · method

Localisation is tested, then the ratings are decomposed

Half one, localisation. Restrict prediction to the seconds a visitor is attending a painting, resolved from the engine’s gaze log, and compare against the whole-session model on identical rows, so that only the scored rows change. 48 paired tests.
Half two, variance. Stop asking what the sensors predict and ask where the variance in the ratings sits. A crossed mixed-effects model splits each rating into the visitor viewing, the painting viewed, and the remainder.

The second half uses no sensor and no model. It describes the ratings themselves.

Experiment 419

Experiment 4 · result

Scoping to the attended painting yields no detectable gain

48 paired tests on identical rows. After correction, none survives as a gain.
Two differences do survive, and both are losses, on painting B2: live arousal falls from +0.0264 to -0.0686, retro valence from +0.0398 to -0.0429.
Neither is credited, because the ordinal metric does not confirm them and the rule requires both.

The absence of a gain therefore does not rest on which seconds were scored.

PAIRED TESTS ON IDENTICAL ROWS

SURVIVE CORRECTION AS A GAIN

Experiment 420

Experiment 4 · result

The rating is attributable to the visitor, not the painting

The painting takes 0.1% to 1.5%; the visitor takes 36.4% to 64.5%. Two visitors viewing the same painting agree at 0.000 to 0.002. Dropping each painting in turn and refitting keeps the share small, but the estimate is imprecise: it runs as high as 17.3%, a consequence of having only five paintings.

Experiment 421

Experiment 4 · findings

The painting explains almost none of a rating

Scoping prediction to the attended painting produces no gain this cohort can detect. None of the 48 paired tests on identical rows survives correction.
Two losses on painting B2 survive on the continuous score alone. Neither is credited, because the ordinal metric does not confirm them.
The attended painting accounts for little of a rating: 0.1% to 1.5%, against 36.4% to 64.5% for the visitor.
Carried into Experiment 5: the whole-session pipeline, since localisation established no gain to carry.

Experiment 422

Experiment 5 · RQ5 · method

Three levers, and a constant-estimate baseline

Lever 1, baseline-window calibration. Uses a quiet opening window and requires no ratings from the visitor.
Lever 2, a labelled warm-up. The model sees the held-out visitor’s own first 30, 60, 120 or 180 seconds of labelled rating, then predicts the remainder.
Lever 3, a person-relative target. Predict each rating as a departure from that visitor’s own mean.
A baseline for comparison. Estimate the visitor’s opening mean and hold it constant, disregarding every sensor, to test whether a lever recovers anything beyond a visitor’s typical rating.

Every arm is scored on the same rows, so the arms are comparable to one another and not only to their own references.

Experiment 523

Experiment 5 · result

Consistent direction, but a constant estimate outperforms the warm-up

Pooled score rises in 21 of 24 cells, yet no paired test survives correction (q = 0.138 at best).

The control wins. Estimating the visitor’s own opening mean and then ignoring every sensor reaches 0.29 to 0.54 pooled kappa, against 0.08 to 0.27 for the trained warm-up, in all four slices. And more warm-up does not help: out to 600 s the per-visitor CCC and kappa are exactly zero at every length, and past 360 s only 25 visitors or fewer can supply it.

Experiment 524

Experiment 5 · findings

Personalisation is not established at this cohort size

Personalisation is not established at this cohort size. The pooled figures move in the expected direction in 21 of 24 cells, yet none of the 48 paired tests survives correction.
The labelled warm-up is the one lever that rises with length on three of four slices, reaching +0.3019 above its reference at 180 s. None of those differences survives correction.
That lever requires the visitor’s own moment-to-moment ratings. The only lever requiring no ratings lowers the continuous score on three of four slices, by up to 0.0393, while raising the ordinal one.

Experiment 525

Conclusion

The variation belongs to the visitor, not to the moment

No approach produced a gain this cohort can detect. No single sensor, no combination, no ordinal formulation of the target, no scoping to the attended painting, no personalisation lever.
The variation sits with the visitor. 36.4% to 64.5% of a rating is attributable to who is viewing; 0.1% to 1.5% to the painting viewed. A cross-person model has almost nothing shared to learn.
Personalisation recovers a visitor’s level, never their movement. A constant estimate of a visitor’s own opening mean outperforms a model trained on that same warm-up. Two properties usually treated as one are separated here.
The decomposition is an observation about the ratings, not a verdict on the sensors. Two visitors may respond in opposite directions to the same painting while each one’s signals continue to track their own rating.

On this evidence, a guide cannot rely on second-by-second affect decoding for an unseen visitor.

Conclusion26

Contributions

Three contributions, and only the method transfers

A measured null across five sensors, 31 subsets and four mechanisms, under a protocol with no leakage, in a literature that predominantly reports positive findings.
The detection margin as a methodological device. Measure how far a score moves when the sample of people changes, fix that margin before comparing, and report differences inside it as undetected rather than equal.
The variance decomposition as a pre-check that any researcher can run on their own collection to establish whether the target carries a shared component at all.

The method transfers to other collections. The figures do not; they are specific to this one.

Contributions27

Limitations

Four limitations, and cohort size is the binding one

38 visitors. The 0.08 margin is the direct price of this cohort size, and is wide enough to conceal a real effect.
Five paintings. This is why the painting share carries a wide interval, and why the claim rests on the gap rather than the value.
Measured data-quality problems. 20 of 47 electrodermal recordings unusable; the eye tracker at a median 23 Hz against a nominal 120.
The target itself. A self-report at 1 Hz sets a ceiling below which finer variation is not recoverable even in principle.

Limitations28

Future work

Check the target before modelling it

Examine the target before modelling it. Run the variance split first and confirm the stimulus component is not near zero.
Design for the person rather than the moment. A short labelled calibration is inexpensive and reliably recovers level, even where it recovered no timing here.
Design the elicitation to produce a shared response rather than assuming one exists. A free-roaming visit may not be a setting in which moments are shared between visitors.

The open question of greatest interest: would this same pipeline recover a signal on a collection where one is known to be present?

Future work29

Thank you

Questions are welcome: on the protocol, the null results, or the variance decomposition.

Akshaj Somanpura Rajendran
MSc Interaction Technology · University of Twente