Virtual
environment systems have become widespread in many areas of human activity from
computer games industry to the tasks of training qualified specialists [1-4].
In these systems, a person is immersed in virtual environment by means of VR
headset, where virtual models replace the surrounding real objects. In modern
life, an important role is played by such objects as the means of recording and
displaying video signal on events occurring in some place, which are combined
into video surveillance complexes. For example, they are present in cars to
control blind spots by the driver, in unmanned aerial vehicles for manual
control of flight trajectory and territory survey, in spacecraft to ensure the
processes of docking with the International Space Station and landing on
planetary surfaces. Therefore, the development of effective methods and approaches
for simulation of video surveillance complexes in virtual environment systems
is an important and actual task.
Active
research is being carried out in the field of creation and development of
technologies for virtual modeling of video surveillance systems and their
elements. The main directions of such a research are implementation of virtual
observation devices taking into account parameters of their real prototypes,
and processing images from virtual cameras based on computer vision
technologies. For example, articles [5-7] describe methods and approaches to
modeling video cameras in virtual environment in the part of rendering images,
which are obtained by means of virtual devices, with imitation of real device
characteristics. The authors of paper [8] consider the task of implementing a
network of virtual cameras with automatic control in three-dimensional scenes,
which provides the ability to track moving objects within field of view of
these cameras. Despite good elaboration level of many important aspects in the
area under consideration, insufficient attention is paid to the task of complex
modeling of video surveillance systems in virtual environment, which includes
implementation of virtual devices for observation and displaying, as well as their
control by an operator immersed in virtual space.
This
paper proposes methods and approaches for video surveillance complex simulation
in virtual environment systems, including creation of basic element models of
such complexes in modeling system of 3D virtual scenes. The novelty of
developed solutions is implementation of new texture map type needed to create
models of display devices in virtual environment. Such textures make it
possible to use any virtual object or its part as a screen that shows images
from one of the scene's virtual cameras. In addition, original approach is used
to control elements of video surveillance system models, based on functional
schemes and three-dimensional virtual control panels.
To
ensure the possibility of designing video surveillance complexes in
three-dimensional virtual environment, this paper proposes a technology based
on integration of own software module (plugin) into the Autodesk 3ds Max modeling
system. This module allows implementing the most important elements of such
complexes, namely virtual models of observation devices and display devices
with their characteristic physical and optical properties. The plugin creation
is carried out by means of built-in scripting language for 3ds Max.
Virtual
model of observation device (video camera) in three-dimensional scene includes
geometric model and virtual camera object. Geometric model is a set of textured
triangle polygons that creates a visual representation of the device. It
usually copies some real prototype and can be implemented using standard tools
of 3ds Max. Virtual camera is used for rendering scene images seen by simulated
device, and is hierarchically linked with geometric model, being its child
node. Standard camera object in 3ds Max has a limited number of parameters and
does not cover the necessary set of physical and optical properties required to
simulate real device. Developed plugin adds new object "Physical camera"
to the modeling system. Along with the basic properties, this virtual camera
allows to set such parameters as physical size of device's matrix, the focal
length of a lens, diaphragm, and servo autofocus. Rollout of settings for the
Physical camera object is illustrated on the left side of Figure 1.
Fig.
1. Rollouts of settings for new virtual camera with physical parameters (left) and
new "Virtual screen" texture map (right) in 3ds Max.
Visual
representation of display device model (screen, monitor, etc.) is also based on
polygonal geometry with using materials and textures. To superimpose images of
the scene obtained by virtual camera on the required area of such model, this
paper proposes new type of texture map which we called "Virtual screen".
The right side of Figure 1 shows settings rollout for such texture, which
includes two blocks, namely parameters of simulated screen and parameters of
connection to observation device. The first block allows selecting resolution
of displayed images in pixels, adjusting backlight intensity for R, G, B color
channels of virtual screen's pixels, and setting display mode to grayscale or
full color. The main parameter of the second block is
Signal source,
which defines one of the scene's virtual cameras as the image source for
considered texture map. In fact, it establishes a link between observation and
display devices. In addition, it is possible to simulate signal distortions
that may occur in electronic systems of real devices and in information
transfer channels between them. This is provided by usingspecial coefficients
of synchronization failure (horizontal and vertical desync) and the amount of
so-called white noise. Proposed approach to the implementation of display
device models based on the Virtual screen texture makes it possible for them to
use an image output area with an arbitrary geometric shape.
One of
the main tasks solved in the proposed approach to simulation of video
surveillance systems is implementation of new texture map type "Virtual
screen" in the visualization system of three-dimensional scenes, taking
into account the parameters of virtual camera linked to this map. To solve it,
every time before rendering the main frame, it is necessary to perform a
pre-processing stage, which consists in synthesizing an image for each texture
of the Virtual screen type used in the scene. Figure 2 shows a diagram of such
process.
Fig.
2. Image synthesis diagram for the Virtual screen texture.
At first,
we determine the view frustum based on the focal length and matrix size of
virtual camera of the Physical camera type, which is linked to considered
texture. Width and height values (in pixels) that are specified in parameters
of the texture are set as viewport size. Next, graphics processor (GPU)
performs 3D scene rendering, during which visualization system writes result
color and depth of each pixel not to the frame buffer and z-buffer, but to
intermediate textures using off-screen buffer technology (FBO). These textures
are passed to an input of special shader for depth of field simulation (DOF) on
GPU. To implement the DOF effect, the shader uses special algorithm [9] of
blurring the incoming color texture by means of Gaussian filter in accordance
with current parameters of virtual camera (focal length, diaphragm and focus
distance). Resulting image is also saved in intermediate texture used at the
stage of signal distortion simulation.
Distortion
simulation shader is only executed if desync coefficients or white noise value
for considered Virtual screen texture are non-zero. White noise imitation is
performed by means of grayscale noise map whose texture coordinates are
computed using pseudo-random numbers. Synchronization failure is simulated by
rendering a quad strip (a chain of connected quads) that consists of identical
horizontal quads covering the entire area of output image buffer. Input texture
with undistorted image is mapped on this quad strip. Texture coordinates of
each pair of the strip vertices lying on the same horizontal line are shifted
to the left or right by an offset value depended on pseudo-random number and
horizontal desync coefficient. In addition, texture coordinates for all
vertices of the strip are simultaneously shifted up or down by the same offset
value depended on pseudo-random number and vertical desync coefficient. The
stage result of signal distortion simulation is also an intermediate texture.
At the
final stage, we perform GPU-accelerated simulation of such settings of the
Virtual screen texture as grayscale mode and screen backlight color. In this
work, two above-mentioned settings are considered mutually exclusive.
Therefore, RGB color correction is implemented only for full color mode
according to the following formula:
|
,
|
(1)
|
where
Ñsrc,
Ccorr
– source and corrected pixel colors of image for
virtual screen,
k
– screen backlight color specified in Backlight Color
field of the Virtual screen map’s parameters.
Mapping
synthesized textures to objects during the main rendering of virtual scene is
performed similarly to self-illumination textures. Fragment shader adds to
intensity
I, computed for each processed object fragment, a color
Cvscr
of the corresponding pixel from the texture map:
|
.
|
(2)
|
In
addition to implementing models of video surveillance complexes in virtual
environment, this paper considers the task of such model control by hands of an
operator immersed in virtual space. Proposed solution is based on two key
elements: virtual observer and 3D virtual control panel.
Virtual
observer is a combination of two virtual cameras and models of human hands. The
cameras are the operator's eyes in virtual environment. Images of
three-dimensional scene rendered with them are displayed in real eyes of a
personusing Oculus Rift VR headset. Positions and orientations of the
cameras are com
puted based on position and
orientation of the operator's head, which come from sensors of the headset. To
implement an impact of the observer on scene objects, realistic virtual models
of hands were created (Figure 3), which are controlled by a pair of Oculus
Touch controllers. The models copy movements of the operator's hands and
provide the possibility of bending and unbending virtual fingers. Apart from
polygonal geometry, virtual hands include invisible volumes (boxes, spheres,
cylinders, etc.) bounding it. Impacts on objects of the scene is modeled by
detection and response of collisions between bounding volumes of hand models
and bounding volumes of these objects.
Fig.
3. An example of 3D virtual models of hands and control panel.
3D virtual control panel (Figure 3) consists of polygonal
model, bounding volumes, models of element actuation sensors, as well as
functional scheme. The scheme has inputs, logic elements, function blocks,
communication lines and outputs. The inputs are connected to the sensors, and
the outputs are connected to controlled scene objects. When fingers of virtual
hand impact on elements of the control panel (for example, pressing buttons),
the sensors generate non-zero signals at the corresponding inputs of the
scheme. Before rendering each frame, control signals at the outputs of the
scheme are computed in accordance with signals at its inputs. Formed signals
provide the ability to control parameters of virtual observation devices (focal
length, diaphragm, focus mode, view direction, etc.) and virtual display
devices (signal source, screen backlight color, etc.).
Methods
and approaches for simulation of video surveillance complexes proposed in this
paper were implemented in our virtual environment system VirSim [10] developed
at the SRISA RAS. Plugin for the Autodesk 3ds Max modeling system was also
created, which allows using virtual cameras with physical parameters, and
textures of the Virtual screen type when creating three-dimensional scenes.
Testing
of developed solutions was carried out using a scene of the virtual polygon,
which includes about six hundred thousand textured triangles. For this, virtual
models of observation devices (three copies), display device, and control panel
were placed in the scene (Figure 4). CCTV camera is used as observation device
prototype. Its model consists of polygonal geometry, virtual camera object of
the Physical camera type, and two joints with electric motors (Figure 5). The
latter provide the ability to change the yaw and pitch angles of the device in
order to select needed direction
V
of camera view. Table 1
presents parameters of simulated observation devices.
Fig.
4. Model of video surveillance complex in the scene of virtual polygon, including
three cameras, display and control panel.
Fig.
5. 3D virtual model and structure scheme of virtual observation device.
Virtual
display device is a model of ordinary monitor. Its screen plane is textured
using a map of the Virtual screen type, initially linked (in modeling system)
with the first observation device. Signal source selection, as well as changing
of other texture parameters, can be performed directly during the operation of
virtual environment system.
Table 1. Parameters
of virtual observation devices.
|
Device number
|
Matrix size
|
Focal length
|
Diaphragm
|
Yaw angle
|
Pitch angle
|
|
1
|
36 x 24
mm
|
20-200
mm
|
1/5.6
|
-70° …
70°
|
-45° …
45°
|
|
2
|
1/8
|
|
3
|
1/2.8
|
Developed
three-dimensional virtual control panel for video surveillance complex contains
11 buttons. Three of them allow selecting one of the available virtual
observation devices as the signal source. The remaining buttons control
selected device and perform the following functions: increase and decrease the
yaw and pitch angles, increase and decrease the focal length of a lens, perform
autofocus, turn on or off servo autofocus mode. To implement the operation of
this control panel, functional scheme illustrated in Figure 6 was created.
Button
coder
block included in the scheme computes and stores the code (from 0 to
2) of last pressed signal source button. This code is transmitted to control
inputs of demultiplexers, which switch signals of control panel buttons to
elements associated with selected observation device. The task of
Camera
control
blocks is to recompute values of the focal length and orientation
angles of the device when corresponding buttons on the control panel are
pressed. Computing voltages supplied to electric motors
ENGINE_YAW
and
ENGINE_PITCH
(placed in the joints of observation device models) is carried out using PD
controllers.
Fig.
6. Functional scheme for control of video surveillance complex.
Figure 7
shows how the operator, immersed in the scene of virtual polygon using Oculus
Rift headset, presses buttons of the control panel by means of virtual hand
models. Virtual hands copy the operator's real hand motion due to Oculus Touch
controllers.
Fig.
7. Virtual surveillance complex control using Oculus Touch.
Approbation
results of proposed methods and approaches in the VirSim software complex
showed that they make it possible to effectively implement virtual models of
video surveillance complexes in virtual environment systems, training complexes
and other applications.
This
paper presents original methods for implementing models of video surveillance complexes
in a virtual environment. The advantage of proposed solutions is an integrated
approach to simulation of such complexes, including their creation in a
three-dimensional scene, synthesis of video streams from observation device
models and visualization on display device models, as well as control of system
elements using virtual reality technologies. Results obtained in the paper can
be used in software development for training complexes and virtual environment
systems.
The publication is made within the state task of Federal
State Institution “Scientific Research Institute for System Analysis of the
Russian Academy of Sciences” on “Carrying out basic scientific researches (47
GP)” on topic No. FNEF-2022-0012 “Virtual environment systems: technologies,
methods and algorithms of mathematical modeling and visualization.
0580-2022-0012”.
1.
Garcia A.D., Schlueter J., Paddock E. Training astronauts
using hardware-in-the-loop simulations and virtual reality // AIAA SciTech
Forum, Orlando, FL, 2020.
2.
Bruguera
M.B., Ilk V., Ruber S., Ewald R. Use of virtual reality for astronaut training
in future space missions – spacecraft piloting for the Lunar Orbital Platform –
Gateway (LOP-G) // 70th International Astronautics Congress, Washington D.C.,
2019.
3.
VR
trenazher. Virtualnaia realnost v obuchenii. Neftianaia promyshlennost [VR
simulator. Virtual reality in learning. Oil industry].
https://www.youtube.com/watch?v=keXlfMKyxsI.
Accessed 25 April 2022. [in Russian]
4.
VR-trenazher.
Deistviia pri pozhare [VR Fire Safety Training].
https://www.youtube.com/watch?v=cm9vFeB18X8.
Accessed 25 April 2022. [in Russian]
5.
Bisagno N., Conci N. Virtual camera modeling for
multi-view simulation of surveillance scenes // 26th European Signal Processing
Conference (EUSIPCO), 2018, pp. 2170-2174.
6.
Kučiš
M., Zemčík P. Simulation of Camera Features // Proceedings of the
16th Central European Seminar on Computer Graphics, 2012, pp. 117-123.
7.
Barsky B.A., Horn D.R., Klein S.A., Pang J.A., Yu M.
Camera models and optical systems used in computer graphics: part I,
object-based techniques // Proceedings of the 2003 international conference on
Computational science and its applications, 2003, pp. 246-255.
8.
Qureshi
F.Z., Terzopoulos D. Surveillance in virtual reality: System design and
multi-camera control // Conference on Computer Vision and Pattern Recognition,
2007, pp. 1-8.
9.
Maltsev
A.V., Torgashev M.A. Raspredelennoe modelirovanie glubiny rezkosti pri
vizualizatsii trekhmernykh stsen na GPU [Distributed simulation of DOF when
rendering virtual scenes on GPU] // Trudy NIISI RAN, Vol. 9, No. 5, 2019, pp.
100-104 [in Russian].
10.
Mihaylyuk
M.V., Maltsev A.V., Timokhin P.Ju., Strashnov E.V., Krjuchkov B.I., Usov V.M.
Sistema virtual'nogo okruzhenija VirSim dlja imitacionno-trenazhernyh
kompleksov podgotovki kosmonavtov [The VirSim virtual environment system for
the simulation complexes of cosmonaut training] // Pilotiruemye polety v
kosmos, Vol. 4, No. 37, 2020, pp. 72-95 [in Russian].
