HUMAN POSTURE TRACKING AND CLASSIFICATION THROUGH

STEREO VISION

Stefano Pellegrini and Luca Iocchi

Dipartimento di Informatica e Sistemistica

Univesit

`

a di Roma “La Sapienza”

Via Salaria 113, Roma 00198, Italy

Keywords:

Human Posture Classification, Human Activity Recognition, 3D Modeling, Stereo Vision.

In this paper we present a system for posture tracking and classification that uses a stereo vision sensor, which

provides both for a robust way to segment and track people in the scene and 3D information about tracked

people. The presented method uses a 3D model of human body, performs model matching through a variant of

the ICP algorithm and then uses a Hidden Markov Model to model posture transitions. Experimental results

show the effectiveness of the system in determining human postures in presence of partial occlusions and from

different view points.

1 INTRODUCTION

Computer vision techniques for human posture recog-

nition have been developed in the last years by us-

ing different techniques aiming at recognizing human

in real environments. Consequently, all the works

presented in this area have limitations with respect

to a general applicability of the systems. In this pa-

per we present an approach to human posture track-

ing and classification that aims at overcoming some

of these limitations, thus enlarging the applicability

of this technology.

A major distinction among works in human posture

recognition is given by the presence of a model of the

human body. Methods that do not use human mod-

tion of person silhouette and thus are quite sensitive to

noise, occlusions and to the view point from which the

person is seen, since low-level features tend to ignore

some relevant information in the images, e.g., the po-

sition of recognizable body parts such as the head or

the hands. The work in (Cucchiara et al., 2005b) uses

a multi-camera setting to overcome problems related

to partial occlusions in posture classification, provid-

ing a solution to determine standing postures in pres-

ence of occlusions.

the fact that a predefined point of view or some con-

straints on the movements of the person being ana-

lyzed must be specified for the procedure to grant ef-

fective results.

use a 3D model, both in the case of 2D and 3D input

data. In (Sminchisescu and Triggs, 2003) the para-

meters of a complete 3D model are estimated from

monocular images using particle filtering, but the time

used for the analysis of each image is prohibitive to

tures are searched using a 3D complete model. After

image segmentation, the silhouette of the detected im-

age is compared to the virtual silhouette generated by

the model in some predefined postures. The posture

of the model which has the best match, according to

a projection histogram procedure, is chosen to be the

right one. (Bregler and Malik, 1998) use a complete

ferent view points, but have problems with occlusions

due to unpredictable variations of the person figures.

The above works deal with posture tracking in dif-

ferent ways. Some of them (Demirdjian et al., 2003;

Bregler and Malik, 1998; Sminchisescu and Triggs,

2003) use different tracking techniques for comput-

ing and updating the parameters of the model. These

works are indeed not focused on posture classifica-

tion, i.e., determining specific postures. Other works

instead propose a two-steps approach: first, model

when the goal is to recognize predefined postures,

temporal filtering allows for improving performance

with respect to frame by frame classification.

Finally, multi-camera setting has been used for

tracking human body movements: (Demirdjian et al.,

2003) use stereo vision and a 3D model of the upper

human body for real-time 3D tracking of head, torso

and arms, while in (Grammalidis et al., 2001) the pa-

rameters of an MPEG4 3D model are estimated using

the depth image coming from the person being ana-

Bahadori et al., 2005; Iocchi and Bolles, 2005))

has been successfully used for segmenting scenes in

which people move in the environment and are able to

provide not only information about the appearance of

a person (e.g. colors) but also 3D information of each

pixel belonging to the person.

In practice a stereo vision based people tracker pro-

vides, for each frame, a set of data in the form XYZ-

RGB containing a 2 1/2D model and color informa-

tion of the person being tracked. Moreover, corre-

implementing an effective posture classification tech-

nique as described here. Moreover, the use of stereo-

based tracking guarantees a high degree of robustness

also to illumination changes, shadows and reflections,

thus making the system applicable in a wider range of

situations.

method for posture tracking and classification given

a set of data in the form XYZ-RGB, corresponding to

the output of a stereo vision based people tracker. The

δ, σ, and the height h shown in Figure 1.

The parameters of the model that are measured dur-

ing model matching and then used for pusture classi-

fication are z =< α,β,γ,δ,h

norm

norm

is

the ratio between the height measured at the current

frame and the height of the person measured during

the training phase, while σ is not included, since it

does not contribute to posture detection.

of the pelvis joint and of the feet point of contact for

each image in the sequence, it also analyzes the se-

quence of observations over multiple frames, so gath-

ering progressively information about the symmetry

plane and the direction of the left hand.

Since the human model contains data that must be

adapted to the person being analyzed, a training phase

must be executed for the first frames in the sequence

(ten frames are normally sufficient), to measure the

person’s height and to estimate the bimodal color dis-

tion of the head ξ is also computed. Considering that a

progressively correct estimation of the height (and, as

a consequence, of the other body dimensions) is avail-

able, the points in the image which height is greater

than the neck level can be considered as head points.

Since the input data provide also color of each point

in the image, we can estimate a bimodal color distrib-

ution by applying the k-mean algorithm on head color

points, with k =2.

After the training phase the Algorithm 1 reported

is a threshold. Θ denotes

the internal state of the tracker and it is defined by

Θ={Π,τ,φ}, with Π={p

, p

, p

set of the three principal points, τ is the symmetry

plane of the model, φ is the direction of the model

within the symmetry plan (it can assume two values:

left or right). The algorithm, given the current input

data ω, computes the model parameters z and update



CONST: M, ξ, λ, λ

TH

M

i

i

i

i

) ∈ ω};

M

p

˜

p

H

F

))

˜

F

= project on floor(

˜

P

= filter plane(τ,Π



);

ˆ

Π



);

= filter(ρ, φ);

Π

),h

slightly preferring the leftmost posture), while the one

in Figure 2 b) is ρ =0since the leftmost pose is very

unnatural. The likelihood function can be instantiated

with respect to the environment in which the applica-

tion runs. For example, in an office-like environment

likelihood of situation in Figure 2 a) may be increased

(thus preferring more the leftmost posture). By fil-

tering these values uniformly through time, we get a

reliable estimate of the orientation of the person φ

angles of the model and hence z, that is used in the

subsequent classification phase. Moreover, Π

denote the internal state of the procedure that will be

used in the next cycle.

Our approach to posture classification is mainly char-

acterized by the fact that it is not made upon low-level

data, but on higher level ones that are retrieved from

in the previous section are uni-modal. Moreover, the

distributions for the different postures are well sepa-

rated each other thus making this space very effective

for classification. However, temporal integration of

these information allows for a more robust classifier

since it allows for modeling also transition between

postures.

procedures (that are compared in Section 6). They

use an observation vector z

each γ ∈ Γ. In addition, due to the uni-modal nature

of such distributions, they have been approximated as

normal distributions.

The first classification procedure just considers the

maximum likelihood of the current observation, i.e.

ML

= argmax

P (z

|γ)

The second classification procedure makes use of a

Hidden Markov Model (HMM) defined by a discrete

status variable assuming values in Γ. Probability dis-

tribution for the postures is thus given by

P (γ

|z

)

P (γ

|γ

)P (γ

|z

)

|γ) P (γ)

The transition probabilities P (γ

|γ



model transitions between the postures, while P (γ)

is the a priori probability of each posture. A discus-

sion about the choice of these distributions is reported

30 degrees from the horizon. A stereo vision based

tracker has been used to provide XYZ-RGB data of

tracked people in the scene. The tracker processes

640x480 images at about 10 frame per seconds, thus

giving us high resolution and high rate data. The sys-

tem described in this paper has an average computa-

tion cycle of about 180 ms on a 1.7 GHz CPU. There-

fore in combination with the tracker the overall sys-

tem can process about 3.5 frames per second. More-

over, code optimization and more powerful CPUs will

The experiments have been performed by consid-

ering a set of video sequences, chosen in order to

cover all the postures we are interested in, with dif-

ferent people, in different environmental conditions,

different orientations and also considering partial oc-

clusions. These videos have been grouped in differ-

ent ways in order to highlight different characteris-

tics of the system as explained later. For each video

we built a ground truth by manually labeling frames

with the postures assumed by the person. Moreover,

of frames that have been used to compute classifica-

tion rates in all the experiments is 2085.

sen a priori probability of 0.8 for the standing posi-

tion and 0.2/(|Γ|−1) for the others. This models

situations in which a person enters the scene in an

initial standing position. The transition probabilities

T

= i|γ

0.147 0.827 0.019 0.004 0.004

0.133 0.033 0.767 0.033 0.033

0.142 0.015 0.027 0.769 0.046

0.148 0.037 0.037 0.037 0.741

that have been manually computed by considering

tion requirements.

by analyzing postures with people in different orienta-

tions with respect to the camera. Here we present the

results of tracking bending postures in five different

orientations with respect to the camera. For each of

the five orientations we took three videos of about 200

frames in which the person entered the scene, bent to

grab an object on the ground and then raised up ex-

iting the scene. Table 1 shows classification rates for

acceptable for an application, it is necessary to tune

the tracker and/or to add additional processing in or-

der to provide for better segmentation.

Errors that are more related to our approach are

mostly determined by incorrect matching of the ICP

algorithm, specially in situations where movements

are too quick. This is a general problem for many

systems based on tracking. A minor problem arises

when the person do not pass through non-ambiguous

postures. In fact, until disambiguation is not achieved

The input to our system is a set of XYZ-RGB data

extracted by the tracker and the system is able to clas-

sify several main postures with high efficiency, good

accuracy and high degree of robustness to various sit-

uations. The approach is based on the computation of

significant parameters for posture classification, that

is performed by using an ICP algorithm for 3D model

matching; 3D tracking of these points over time is

then performed by using a Kalman Filter in order to

increase robustness to perception noise; and finally a

tracker system works very well when people are in

standing position, while quality of data worsen when

people sit, lay down, or bend. While the quality

of segmentation does not affect classification of the

LAID posture (that is mainly determined by the pixels

height from the ground), segmentation errors are the

main causes of classification errors for the other pos-

tures. Classification errors may be reduced by pro-

viding feedback from the posture classification to the

tracker. In fact, given these information the tracker

model parameters can be useful to determine for ex-

ample different ways in which people fall down.

REFERENCES

Bahadori, S., Grisetti, G., Iocchi, L., Leone, G. R., and

Nardi, D. (2005). Real-time tracking of multiple peo-

ple through stereo vision. In Proc. of IEE Interna-

tional Workshop on Intelligent Environments.

Beymer, D. and Konolige, K. (1999). Real-time tracking of

Bregler, C. and Malik, J. (1998). Tracking people with

twists and exponential maps. In IEEE Conf. on Com-

puter Vision and Pattern Recognition (CVPR’98).

Cucchiara, R., Grana, C., and Prati, A. (2005a). Probabilis-

tic posture classification for human-behavior analysis.

IEEE Transactions on Systems, Man, and Cybernet-

ics, 35(1):42–54.

Cucchiara, R., Prati, A., and Vezzani, R. (2005b). Posture

classification in a multi-camera indoor environment.

In Proc. of IEEE International Conference on Image

Demirdjian, D., Ko, T., and T.Darrel. (2003). Constraining

Goldmann, L., Karaman, M., and Sikora, T. (2004). Hu-

man body posture recognition using mpeg-7 descrip-

tors. Visual Communications and Image Processing.

Grammalidis, N., Goussis, G., Troufakos, G., and Strintzis,

M. G. (2001). 3-d human body tracking from depth

images using analysis by synthesis. In Proc. of

Iocchi, L. and Bolles, R. C. (2005). Integrating plan-view

tracking and color-based person models for multiple

Rusinkiewicz, S. and Levoy, M. (2001). Efficient variants

of the icp algorithm. Proc. of 3rd International Con-

Sminchisescu, C. and Triggs, B. (2003). Kinematic jump

processes for monocular 3d human tracking. Proc.