Joint Alignment and Modeling of Correlated Behavior Streams

Liliana Lo Presti, Liliana Lo Presti, Stan Sclaroff, Agata Rozga

Research output: Contribution to conferenceOtherpeer-review

17 Citations (Scopus)


The Variable Time-Shift Hidden Markov Model (VTS- HMM) is proposed for learning and modeling pairs of cor- related streams. Unlike previous coupled models for time series, the VTS-HMM accounts for varying time shifts be- tween correlated events in pairs of streams having different properties. The VTS-HMM is learned on a set of pairs of unaligned streams and, thus, learning entails simultaneous estimation of the varying time shifts and of the parameters of the model. The formulation is demonstrated in the analysis of videos of dyadic social interactions between children and adults in the Multimodal Dyadic Behavior Dataset (MMDB). In dyadic social interactions, an agent starts an interaction with one or more “initiating behaviors” that elicit one or more “responding behaviors” from the partner within a temporal window. The proposed VTS-HMM explicitly accounts for varying time shifts between initiating and responding behaviors in these behavior streams. The experiments confirm that modeling of these varying time shifts in the VTS-HMM can yield improved estimation of the level of engagement of the child and adult and more accurate dis- crimination among complex activities.
Original languageEnglish
Number of pages8
Publication statusPublished - 2013

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Vision and Pattern Recognition


Dive into the research topics of 'Joint Alignment and Modeling of Correlated Behavior Streams'. Together they form a unique fingerprint.

Cite this