DEVELOPMENT OF ALGORITHMS FOR TRACKING OF MOVEMENT AND CONTROL POINTS ON THE FACE FOR REMOTE DETECTION OF PSYCHO-EMOTIONAL STATES
I.A. Znamenskaya1, E.Yu. Koroteeva1, V.V. Shishakov1, A.V. Khakhalin1, E.A. Kuzmicheva1, S.A. Isaichev2, A.M. Chernorizov2
1Lomonosov Moscow State University, Faculty of Physics, Moscow, Russia;
2Lomonosov Moscow State University, Faculty of Psychology, Moscow, Russia
znamen@phys.msu.ru, koroteeva@physics.msu.ru, shift@physics.msu.ru
Contents
2. Termografiya in psychophysiology
4. Analysis algorithm of thermograms
Abstract
This work is focused on estimating the relationship between the dynamics of heat fluxes from a human face captured by a thermal camera, and the physiological response of the autonomic nervous system registered by contact methods. The emotional arousal is identified based on the activity of facial muscles, cutaneous blood vessels and breathing using optical and infrared imaging of human faces. The algorithms are applied for motion compensation and juxtaposition of video and thermal images, together with the software implementing these algorithms. The machine-learning based method is proposed to allocate the key points on human faces.
Keywords: infrared thermography, image processing, pattern recognition.
1. Introduction
The problem of objective diagnostics of the functional statuses (FS) of the person in the types of activity connected to high risk of origin of technogenic catastrophes becomes not only a relevant, but also socially important problem. The reliability role of "a human factor" considerably increases if the periods of the vigorous activity requiring fast decision-making are replaced by the long periods of the monotonic operation. Such type of activities is characteristic of the majority of operator professions, drivers of vehicles, the military personnel, employees of security and law enforcement agencies i.e. where the probability of origin of extremal and emergency situations is high.
The scientific applied researches connected to development of noninvasive methods of objective identification of FS in real time are of particular interest. Development of methods, an optical-electronic equipment and software of identification of psycho-emotional statuses on specific features of behavior and appearance of the person is the relevant task for scientists and practicians in Russia and abroad.
The purpose of this work is development of technology of distantny renting of physiological information on psychoemotional conditions of the person (quiet wakefulness, emotions, a stress) on the basis of researches of activity of mimic muscles, integuments, superficially lying blood vessels of the person and also respiratory (gas) streams. The task of identification of emotional exaltation of the activity of mimic muscles recorded in optical and infrared (IR) ranges is set. The task is complicated by need of registration in the conditions of continuous shift of a surface of a face concerning scanners – the video camera and the thermal imager. In work this problem is solved by a combination of methods of registration in visible and Infrared ranges and due to development and adaptation of the program of the analysis of dynamically changing visual scenes.
2. Termografiya in psychophysiology
The thermography represents a way of measurement and visualization of the thermal, infrared radiation emitted by all heated bodies. The main condition for formation of IR of the image is existence of temperature contrast between an object and a background. Features of radiation of a skin of the person defined wide use of a thermography in physiological and biomedical diagnostics [1]. The skin has the high coefficient of radiation close to coefficient of radiation of absolutely black body therefore change of its temperature leads to appreciable change of power of IR of radiation. Besides, the low coefficient of reflection of a skin minimizes influence of the environment on determination of its temperature.
The most available to thermovision measurements is the area of the person. Under invariable external conditions the surface of a face of physically healthy person has the non-uniform distribution of temperature determined by physiological features of an organism. Local change of temperature fields in a face can be bound:
1) with activity of mimic muscles;
2) with expansion of blood vessels of the person;
3) with dynamics of respiratory streams (as showed experiments [2]);
4) with intensifying of a diaphoresis in certain face zones.
Certain areas on a human face have the greatest thermoesthesia to various internal and external stimuli [3]. Such areas on a thermogram are called areas of interest (Fig. 1).
3. Technique of experiments
In work experiments on simultaneous non-contact registration of dynamics of a caloradiance on behalf of the person and contact registration of activity of the autonomic nervous system were made.
Fig. 1. The thermogram of a human face with the allocated areas of interest
For carrying out infrared shooting in experiments two thermal imaging cameras were used (separately). The first camera of the FLIR SC7700 brand works in the range of lengths of waves of 3.7-4.8 microns (MWIR) and allows to receive thermovision images with a frequency up to 115 Hz with the spatial resolution of 640õ512 pixels and to 400 Hz with limited permission. Control of the camera and processing of heats pattern are carried out by means of the software of Altair and FLIR ResearchIRMax. The second camera of the COX CX640 model works in the range of lengths of waves of 8-14 microns (LWIR) and allows to receive thermovision images with a frequency up to 50 Hz with the spatial resolution of 640õ480 pixels. For receiving and the analysis of heats pattern the software of Thermal Imaging Analyzer is used. Shooting in the visible range was at the same time carried out by the Olympus OM-D E5 digital camera.
About 25 healthy examinees (men and women) aged from 18 up to 55 years participated in experiments. All measurements were taken in position of examinees sitting, after their adaptation to laboratory conditions. The thermal imager and the video camera settled down at distance about 1 m from the examinee. Time of shooting varied from 20 with up to 15 min. Frequency of shooting was 5-25 Hz. The fixed indoor temperature (20-22 °C) was indoors maintained.
For the analysis of correlation of thermal measurements with assessment of a psychoemotional status of the person synchronously with thermovision and video filming registration of physiological responses of the person by means of the multi-channel Entsefalan encephalograph analyzer was carried out. The course of carrying out an experiment is provided in Fig. 2.
4. Analysis algorithm of thermograms
Incorrect comparison of thermovision and video filming requires coercion of a thermal field on a human face to standard representation. During the decision of this task algorithms of compensating of movements of the person and comparison of images from the video camera and the thermal imaging camera were applied and also the software realizing the offered algorithms is used. Besides, the method of separation of key points on a human face on the basis of machine learning is offered.
In the used range of lengths of waves the circuit of the human head or person has very clear boundary because of a high difference of temperature of a background and a body of the person, however distinguish separate features can be difficult because of low contrast of such image within the person. The image of the person in the visible range has more expressed detailing allowing to define precisely layout on the image of areas of interest (Fig. 1), compensating possible distortions, such as turn of the head, mimic distortions, etc. and also to precisely evaluate parameters of these distortions. The solution of two tasks is for this purpose required: 1) search of the person on the image on basic circuits and points (eyes, a nose); 2) the detailed description of parameters of the person (circuits of eyes, a century, eyebrows, lips, nose wings).
Both functionality are realized in a wide range of the biometric software and also in program tools of the developer (SDK), such as OpenCV [4]. Detection of the person on the image is executed by means of the cascade filters of Haar. More detail assessment of turn of the person, detection of biometric (mimic) break points on a face is carried out by means of the active models of appearance (ActiveAppearanceModel, AAM) and method of the Active Models of Figures (ActiveShapeModels, ASM).
Combination of these AAM/ASM models with the thermovision image allows to define more precisely layout of the necessary points of temperature monitoring on the thermovision image. These methods allow to catch also a mimicry of the person and to use this information for assessment of his emotional status. Data application of methods on a video series allows to trace paths of break points in space and to analyze response characteristics of a mimicry. Use of a stereocamera allows to evaluate in more detail in a short-range zone a relief of a surface of a face, and in a distant zone – to evaluate distance to the researched object. Algorithms of restoration of a card of depth of a scene are realized in a set of tools for the developer (including OpenCV).
For the analysis of thermal images in this operation the software in the Python language with use of OpenCV, Dlib and OpenFace libraries was used [5, 6, 4]. Data of library, using Viola-Johnes [7, 8] method, allow to select the person on the image and also to find characteristic key points of the person (eyes, lips, a nose, etc.). Further, the CLNF [9] method on the image specifies layout, the form and spatial orientation of all features and also position of the head concerning the camera. It allows to compensate movement of the head concerning the camera, to select a specific point on the image of the person, to track change of temperature in this specific point and also to evaluate whether these sections on the image are visible, or not. These algorithms were applied both to thermovision video filming, and to visual (Fig. 3). In case of simultaneous shooting with the thermal imaging and normal digital camera for synchronization of a video series blinking of eyes was used.
Fig. 2. Part of experimental installation (left) and thermovision card of the respondent (right)
5. Preliminary results
A series of experiments on in advance prepared scenarios containing intervals in which examinees were in a quiet status, and intervals in which stressful conditions were modelled was conducted. In Fig. 4 the example of comparison of dynamics of a heat flux from area of interest (a nose tip) on a human face with results of record of the skin and galvanic response (SGR) is provided. SGR – bioelectric response which is widely applied in psychophysiological researches as a highly sensitive index of level of activity of sympathetic nervous system and also to assessment of neuromental tension of the person. In Fig. 4 it is visible that stressful response to a sharp sound leads to local increase, and intensive mental loading - to local lowering of temperature in the field of interest (most possibly, connected to its cooling due to gain of activity of sweat glands).
The software module is developed for comparison of the image from the video camera and the thermal imaging camera. It was initially planned that use of an additional synchronous video series in the visual range will help to evaluate more precisely provision of break points of the person on the image and it is better to compensate movement and turns of the person concerning the camera. However further, after accumulation of a certain number of the synchronous records with different examinees, it was succeeded to make learning selection of IR-images and information on the recognized persons from a visual video series that allowed "to train" OpenFace library to work with IR-images separately, to look for position of features and to estimate orientation of the head without hint from a visual channel.
Fig. 3. The scheme of work of an analysis algorithm of the image of the person, allocation of lines and assessment of turn of the head in common on IR and a visual video series
Fig. 4. Comparison of dynamics of an average temperature signal in the field of interest (it is allocated on the thermogram) and SGR signal during passing of the test: (a) deep breath; (b) sharp sound; (c) mental loading
Software module for coercion of a thermal field on a human face to standard representation. Diagrams of change of temperatures of the exhaled air (lower) and the surfaces of skin over the right nostril (upper), the compensation of movements of a human face received from the heat pattern with use of the used algorithms – taking into account offset of position of the person in the course of shooting are provided on fig. 5. It is visible that temperature of the exhaled air is higher than temperature of a surface of skin. The natures of oscillations of temperature of the exhaled air and the surface of skin differ from each other: temperature vibration amplitude on the surface of skin is less, than a vibration amplitude of the exhaled air.
Fig. 5. Diagrams of change of temperatures of the exhaled air (lower) and the surfaces of skin over the right nostril (upper) from time
6. Conclusion
As a result of operation the software module for comparison of the image from the video camera and the thermal imaging camera is created. The activity of a surface of face skin of the person in the optical and infrared ranges is probed. The received results allow to say that dynamics of thermal fields in a human face can serve one of indicators of change of its psycho-emotional status.
The analysis of the revealed psychophysical response in the form of local change of temperature fields in a face for different tests is carried out. Conclusions are drawn on age and gender distribution of the revealed responses.
Acknowledgment
Work is performed with use of the equipment acquired at the expense of means of the Program of development of the Moscow University and with financial support of RNF (a grant 16-18-00080).
References
1. Skripal A.V., Sagaydachny A.A., Usanov D.A. Thermovision biomedical diagnostics: Studies. grant. Saratov, 2009.
2. Znamenskaya I.A., Koroteyeva E.Yu., Hakhalin A.V., Shishakov V.V. Thermographic visualization and remote analysis of dynamic processes in a face. Scientific visualization. 8(5):1–8, 2016.
3. Ioannou S., Merla A. Thermal infrared imaging in psychophysiology: Potentialities and limits. Psychophysiology. 51, 2014, pp. 951–963.
4. OpenFace: http://www.cl.cam.ac.uk/research/rainbow/projects/openface/wacv2016.pdf
5. Dlib: http://dlib.net/
6. OpenCV: http://opencv.org/
7. Viola P. Jones M.J. Rapid Object Detection using a Boosted Cascade of Simple Features. proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR 2001), 2001.
8. Viola P. Jones M.J. Robust real-time face detection. International Journal of Computer Vision. vol. 57, no. 2, 2004., pp.137–154.
9. Baltrušaitis T., Robinson P., Morency L-P. OpenFace: an open source facial behavior analysis toolkit. IEEE Winter Conference on Applications of Computer Vision, Lace Placid, NY, 2016.