Within the framework
of the approach developed by the authors to systematization of design processes
of information visualization tools, the task of building a model of visual communication
which is adaptable to various situations of practical use is of high importance.
One of the known obstacles in this direction is the significant uncertainty in the
visualization tool user’s perception and actions, i.e. their subjectivity influenced
by many external factors.
Considering visualization
as one of the existing tools for organizing human-machine interaction, a large number
of issues can be pointed out that need to be resolved when supplementing context-sensitive
interfaces with the capability to adapt to the user’s cognitive characteristics.
Transition to the use of visual analytics tools adaptable not only to the features
of the analysis problem being solved but also to the conditions of their use (including
users’ individual capabilities and state) will make it possible to approach important
goals:
•
Wide
involvement in the visualized data analysis of perception characteristics of a potential
visualization tool user, his preliminary awareness as well as personal cognitive
models [1]. An obstacle here is the variability of the specified resource, its continuous
transformation depending on a variety of external causes.
•
Transition
to the meaningful use of modern technical developments in the field of computer
visualization [2]. The problems in this area are largely related to the formation
of “digital reality”, in which interpretation of visualized information is not verified
by accumulated personal experience.
•
Obtaining
balanced visualization technologies aimed at a rational combination of visualization
resource intensity levels and analysis task complexity [3].
•
Increase
in the speed of visual communication as one of its most significant characteristics
due to the efficient coordination of cognitive and visual models.
•
Expanding
the capabilities of scientific visualization tools by attracting passive resources:
the user’s emotional states, visual perception aesthetics, movement interpretation,
etc.
•
Directed
use of visualization teaching potential, the purpose of which may be formation of
the user’s new conceptual apparatus and a system of reliability criteria that are
of decisive importance, for example, in the development of decision support systems
based on providing the decision maker with visualized information.
The basis for the
transition to the adaptable model of visual communication can be the similarity
between visual and speech communications investigated at the level of their functional
features and their corresponding tools. From this point of view, the dynamic characteristics
of visual communication are of fundamental importance, when visualization interpretation
subjectivity can be considered not as an accidental result but as a process controlled
by the inherent characteristics of the visualization tools. Consequently, in Frege’s
classical semantic triangle, which combines three concepts: sign, denotation and
sense, the informative image acquires additional meaning as a control element responsible
for the representation of external data, control over the user’s state and response,
interpretation of data and change in the system of concepts belonging to a particular
user [4].
Development of the
analogy between visual and verbal (linguistic) communications necessitates to determine
the meaning of some concepts, including “sign”, “word”, “phrase”, in information
visualization as objects with different purposes, properties and applicability substantiation.
Introduction of such definitions will make it possible to systematize visualization
tool developers’ efforts as a result of the transition to a reasonable use of the
semiotic approach in visualization [5].
The most obvious
difference between these concepts can be associated with the information content
of the corresponding objects, i.e. with the amount of new information obtained by
the recipient in the course of communication. For example,
•
a
visual sign refers to a concept included in the user's knowledge system;
•
a
visual word is a sign of greater capacity, i.e. an image that possesses self-sufficiency
properties, which includes both a content part and a comparison with a new visual
sign;
•
a
visual phrase is an informatively rich image interpretation of which is accompanied
by a purposeful cognitive effort associated with the interpretation of phrase structure,
meanings of its individual elements, interrelationships between them, etc. based
on additional perceived features.
Besides, in accordance
with the systemic language properties, visual phrase interpretation results are
also determined by the subjective influence of the user himself, including his awareness,
ability to recognize hidden meanings and implicit indications as well as the psycho-emotional
state and propensity for in-depth analysis of the reasons that led to the appearance
of an interpreted visual phrase.
Information content
assessment of a visual image or its individual components is an extremely ambiguous
process [6]. Therefore, an alternative (or additional) basis for the introduction
of definitions of visualization structural elements can be the time spent by the
user on the interpretation of a visual image. From this point of view, belonging
of a visual image to one of the named categories becomes dependent on visual communication
participants. For example, a rich informative image, if repeatedly represented to
the user, is interpreted in a way different from the initial observation. The signs
of similarity with the already known image identified in the resulting one provoke
a switch to retrieving the results of an earlier interpretation from memory. The
purpose of switching is usually to save the user's physical or cognitive effort
and associated time.
A common situation
in visual design is when a visual phrase, which may have a voluminous informational
meaning, becomes a visual sign, uses different interpretation mechanisms and most
likely does not justify some of the visualization developer’s efforts. Controlling
this process and supplementing visualization tools with options for promptly making
the necessary changes to their operation provide opportunities for reducing resource
consumption, including increase in speed of visual analytics tools.
In other situations,
visualization based on representation to the user of individual images (a visual
word), sufficient to establish a correspondence between an information event (fact)
and a visual image, may depend on the current characteristics of the user's perception.
For example, in case of the user’s insufficient preliminary information awareness,
it becomes necessary to accompany the visualization with additional explanatory
elements, and the visual word becomes more capacious, acquires the capabilities
of a learning tool and goes into the category of visual statements.
In the opposite case,
the user becomes confident that the presented visual image is familiar to him and,
even in the absence of a verified interpretation, the user stops detailed study
of the image and simply establishes a correspondence between his knowledge and the
new image. Thus, the visual phrase becomes a sign again, which can lead to many
interpretation errors. The indicated difficulties accompanying visual communication
make it mandatory to adapt visualization tools to the user’s current state and general
capabilities.
There are many examples
of using visualization tools to solve highly specialized applied problems (medicine,
geology, engineering, education, etc.), in which the discrepancy between the purpose
and the means used discredits the value of visualization. For example, in the design
of many decision support systems, excessive information arises, the influence of
which on the result of visual communication is rather difficult to control. In this
case, there is an unreasonable attraction of resources from the point of view of
the goal of visual communication. The manifestations of over-informing include visual
elements with re-informing (visual tautology), unjustified display variations that
require additional cognitive efforts from the user, etc.
A characteristic
feature of visualization used in educational processes is the accompaniment of the
main content with elements that repeat previously stated information to form the
necessary sequence of inferences with the student. Adapting visual communication
to the conditions of its implementation can solve two common problems: distraction
of users’ attention [7] who already have a sufficient amount of preliminary knowledge,
or building alternative interpretations, verification of which requires additional
resources.
As an example, there
are two alternative situations encountered in education [8]. In the first case,
a student who does not have the necessary motivation to actively participate in
the educational process is focused exclusively on memorizing the incoming visual
information. Lack of cognitive effort in the communication process leads to passive
fixation of new information, which is easily replaced by new data in the near future.
In the opposite situation, the process of interpreting the visualized data can go
in a completely unexpected direction or cause the emergence of many hypotheses that
are insignificant for the problem being solved and corresponding to the randomly
arisen interest of the communication participant; this will become a significant
obstacle to achieving the initial learning goal.
A positive solution
to these problems may be to change the goal of visual communication. It is about
replacing the process of informing the user with the process of cognitive research
[9]. In this case, the lack of personal knowledge, which prevents the interpretation
of a data image as a previously known sign, initiates generation of assumptions
about its meaning which have signs of novelty for the user. The source of the generated
hypotheses about the meaning of the perceived image are visual elements, the context
of communication, subjective experience and the purpose of interpretation. Achieving
this goal involves testing the validity of hypotheses, establishing new relationships
between known facts, accumulating both positive experience and rejected erroneous
judgments. In this case, the cyclical nature of the approach to the correct interpretation
allows considering visual communication as a process corresponding to the results
of the directed training of the user.
Transition from one
type of visual communication to another occurs as a result of selection or change
of the way for representing analyzed data. This enables considering visual communication
as a tool with adjustable functionality; the opportunity to control it becomes an
independent task for visualization tool developers. The purpose of such control
is not only the choice of the required type of interaction with the user but also
the suppression of unwanted choice or switching made by the user under the influence
of uncontrollable factors.
Within the framework
of the semiotic approach to the development and use of visual analytics, it is necessary
to consider the interdependence of all its components: interpretation problems,
communication goal, visualization tool user’s characteristics, visual representation
method, means of influencing the user.
Based on this, control
of visual communication properties can be carried out in several ways:
•
Interface,
software for selecting the properties of data visual representation (sigmatic control).
•
Means
of achieving proportionality, compatibility, conflict-free visual metaphors used
for different data and problems (visualization semantics)
•
Means
of attention control organizing and changing the sequence of communication (visualization
syntactics)
•
Tools
for adapting visual representation to the needs of a particular user; or means of
influencing the observer, forming necessary, from the point of view of the problem
posed, psycho-emotional state (visualization pragmatics).
Expanding the range
of visual communication capabilities becomes the applied meaning of its controllability.
Consequently, inclusion of visual analytics tools by developers into their functional
set of adaptation tool becomes an additional but necessary action. The indicated
ways of managing visualization are inter-complementary in their capabilities and
implementation options, and the decision to use them depends on practical expediency.
Therefore, it is important to develop a theoretical model of adaptable visual communication
and the rules for its error-free adjustment.
The goal of the communication
model is deriving and studying internal visualization processes and their features.
The directed organization of such processes in the ways proposed by the developers
will lead to systematization and predictability of the results obtained in the practical
use of visualization tools. Within the framework of the semiotic model, the following
processes can be distinguished (Fig. 1) associated
with the application of visualization: definitions of acceptable notations (compliance
with technical capabilities and the existing tradition), choice of a representation
metaphor (compliance with data characteristics), organization of a discussion (cyclical
hypothesis formation and verification), making and preservation of the achieved
decision (compliance with the original problem and the prospective application of
the results obtained).
For the visualization
tool user, the same processes can be correlated with his own actions (Fig. 1) aimed
at solving the problem at hand. These actions usually represent a search for answers
to local questions, some of which, being quick patterns of perception, are not even
formulated explicitly. The processes of perception, interpretation, analysis and
decision making are compared to the corresponding elements of the semiotic model
and therefore can be realized by comprehensible linguistic means.
Fig.
1. Correspondence
between visual communication stages, visualization tool developer’s goals and
user actions
Assessment of uncertainty
(errors) in actions or reactions of communication participants is proposed as one
of the ways to test the proposed model of user interaction with visualized data,
i.e. communication between data, the user of visual analytics tools and their developer.
To obtain such an assessment, an observation of the user's actions in simulated
practical situations of interaction with unfamiliar objects has been carried out.
The subjects were
asked to determine the purpose and capabilities of an industrial design object relying
on assessment of its appearance (Fig. 2). It
was assumed that this was a model version of the problem of interpreting visual
data by a visual analysis tool user. The point of modeling such a task ensuring
the necessary cognitive problem for the subject was the observation of a conceptual
design object that had a well-defined set of functions, but the appearance, consistent
with the functional content of the device, had significant novelty. In other words,
the visual image acted as an unfamiliar semiotic object but with a given meaning.
Fig. 2. Interpretation of the appearance
of an industrial design object by levels of contextual information
When interviewing
the participants in a group of subjects (40 people), the following goals were pursued:
determining visual image elements that were significant for the observer, assessing
their information content, identifying the reasons for misinterpretation, selecting
visual communication features that contributed to its efficiency increase. Within
the framework of the experiment, the subject's interaction with the unknown object
was limited to visual observation in three-dimensional space. Kinesthetic, auditory
and any other perception were excluded from the communication process. A limitation
artificially introduced when testing the proposed interaction model was the need
for the subject to make a final decision (interpretation of visual data) with a
minimum number of erroneous hypotheses.
In one variant of
the experiment, the user was asked to evaluate a new device and determine the purpose
of its controls. The obstacle for the subjects was the fact that the device appearance
was completely unusual from the point of view of everyday experience. The conditions
of such an experiment corresponded to the situation of using visualization tools
in which the user would see for the first time a data image formed within a representation
metaphor unknown to him. As expected, the correct interpretation of visual information
occurred only in a small number of cases (<10%).
“What is it?”
(Fig. 2, À)
If the
initial data for the user are information about the field of application of the
device and therefore the associated subjective experience then the situation, according
to the correspondence scheme (Fig. 1), will be characterized by the choice of a
specialized sign system (sigmatics). The significant error level (
∼
65%) at
the sigmatic level is due to the lack of connections between experience, purpose
and representation of information.
“How does it work?”
(Fig. 2, B)
The transition
to the next level (semantics) occurs after adding contextual information to the
information available to the user. In this case, the user selects simple information
structures in the analyzed image that make it possible to establish the necessary
connections between the purpose and individual experience (through representation).
The number of erroneous judgments formulated by the user is reduced to 40-45% according
to the obtained rough estimates.
“How to use it?”
(Fig. 2, C)
The error
level reduces significantly after changes are made to the observation conditions,
that is providing interpretation with feedback. As a result of the procedure of
repeated visual communication the number of formed and tested hypotheses increases;
this corresponds to the accumulation of subjective experience by the user.
The reduction in
the number of erroneous decisions at this stage (syntactics) is the result of coordination
of the communication goal and the interpretation results. Detailing the initial
data (analogs, operation principle, device characteristics) changes the user's understanding
of the problem being solved, reduces the error level (almost to 20%) and leads to
the emergence of new knowledge for the user corresponding to the study of a new
symbolic object.
“What will it give
me?”
Finally, the most significant results in reducing the interpretation error level
made by the user (∼5-10%)
are obtained when three semiotic components
(experience, purpose, representation) are consistent due to the interaction between
the user and the object of study (Fig. 3).
It should be noted
that, as a result of the research, the thesis about the possibility of manipulating
the process of user interpretation by the visualization tool developer was confirmed.
This means that the results of practical application of visualization tools that
use metaphors of visual representation unfamiliar to the user can form both new
knowledge and persistent false associations that can influence the effectiveness
of the use of visualization tools.
The
need to
comply with the listed correspondences is fulfilled at the design stage of visual
analytics tools and leads to changes in methods used both when creating visualization
tools and when applying them. The main goal of the proposed changes is to shift
the efforts towards actively attracting user competencies at all stages of solving
a research problem – from preliminary analysis of conditions to final decision making.
This corresponds to the reasonable refuse to use decision support systems based
on machine learning capabilities in situations where the development resource intensity
of such systems does not correspond to their practical application.
Fig.
3. Coordinated
functioning of the semiotic model elements
and its impact on visual communication efficiency
Thus, the problem
arises of identifying a set of tasks of visual data interpretation for which the
user's own knowledge is sufficient to obtain the correct solution. An example of
a possible modification of traditional visual data interpretation method from the
user point of view is appearance in the visual communication process of operations
of initial problem statement modification (reinterpretation). The ultimate purpose
of manipulating the conditions of the problem and the process of its solution is
to reduce the volume of optional (redundant) operations with the data under study.
An analogy can be
drawn using a technique called “reframing” [10], which is used at the stage of setting
the goal of visual research. In this case, reframing is a technique for changing
the interpretation context based on the selected representation metaphor in order
for the user to achieve new or operational understanding of the visualized data
meaning. The meaning of the actions performed by the user within the framework of
the semiotic approach to visualization is to achieve the closest possible consistency
between the components: data representation (sigmatics), user's own knowledge (semantics),
research problem (pragmatics), decision hypothesis connecting data, i.e. creating
a model of the event under study (syntactics). For the visualization tool user,
such consistency looks like an opportunity, supported at the level of the involved
tools, to look at the task from another (various) perspective(s), evaluate its goal
from a different angle, identify features in the initial data that are not available
when using the traditional or familiar representation metaphor.
Reframing methodology
implementation poses a task for the visualization tool developers to provide the
user with a technical opportunity to create a variable data representation in the
course of visual communication. Consequently, it becomes mandatory to add to visualization
tools the capability to control interactively both the represented data (filtering,
scaling, coordination, etc.) and visual display methods.
A useful consequence
of the proposed scheme for coordinating semiotic model elements (Fig. 3) is the
opportunity to increase the efficiency of visual analytics functional definition
of the adaptable visualization concept proposed in the paper, which considers visualization
as a process efficiency of which depends on the consistency of all its elements.
First, adaptation of visual communication implies coordination of expressive means
used by visual analytics tools both with the properties of initial data and with
the goal of communication being performed. Second, communication control aimed at
achieving consistency between user perception and visualization properties must
ensure that the visualization tools perform their role without errors. Otherwise,
the use of decision support tools in an unusual role of learning tools and, therefore,
the unattainability of the set goal may result from an incorrectly chosen visualization
metaphor.
Based on the considered
features of visual communication, it can be argued that their reasonable organization
is capable of expanding the applicability of visualization tools including their
use as standalone tools with a number of unique advantages.
For example, the
results of experimental assessments of consequences of using interactive visualization
tools [8] allow for asserting the existence of two-way consistency within which
not only the user's knowledge is used to interpret and analyze the initial data,
but also his experience is actively manipulated.
The cognitive effort
made by the user while forming intermediate hypotheses and verifying them corresponds
to the iterative learning effect consistent with the considered semiotic approach.
A useful result of the continuous accumulation of experience by the user is reduction
in the time it takes to solve similar problems because memorized visual sequences
of visualization tool states or their fragments, due to the capabilities of perception,
also become new signs (or their combinations) and take part in interactive visual
communication at the next step.
However, as follows
from the correspondence scheme (Fig. 1), transitions between the levels of the semantic
model can occur under the influence of many external factors which are not always
controlled by the user. For example, transition to the pragmatic level can be performed
without going through the syntactic level in the presence of strict time constraints
for visual communication. In this case, absence of consistency between subjective
experience and communication goal can become a source of erroneous decisions.
Understanding the
reasons why visual analytics tools can perform a function different from the one
for which they are designed or involved in the task at hand will allow for making
timely adjustments and avoid misconceptions. Analysis of the visualization semantic
model and the results of experimental assessments allows for the following conclusions
(Fig. 4):
•
Adaptation
of visualization tools to visual communication goals can be realized both when using
the capabilities of interactive control and as a result of changes in the external
conditions of communication.
•
The
task of informing (sigmatics), using a certain sign system, in the case of expanding
this system or changing (clarifying) the area of its application, goes into the
class of learning tasks, which, in turn, leads to different organization of the
communication process.
•
Acquaintance
with a new visual metaphor defining the meanings of unfamiliar images leads to a
change in the user’s interpretation of visual data, complementing the previous experience
and knowledge. This initiates the formation of subjective interest and new hypotheses
explaining contradictions present in the image. To solve the problem of analysis
(syntactics) becomes the communication goal. It implies directed cyclical communication.
•
Ambiguity
of hypotheses formulated by the user based on the capabilities of the selected sign
system and the goal of visual communication can be supplemented by the capability
to save and compare verification results of these hypotheses. The level of selection
or decision making about optimal compliance with the desired result (pragmatics)
should also differ in its organization from the other levels in order to preserve
the goal of visual communication and prevent transitions to simpler levels.
The findings are
in good agreement with the experience of designing visualization tools [11] for
interpretation and analysis of heterogeneous data [12-13]. For example, the same
data, being the basis of visual images using different representation metaphors
(Fig. 5), correspond to the tools of informing (A), research (B) and selection (C).
In
the informing problem (A), the goal is to transmit the initial data to the user
with minimal distortions arising from the transition to the visual representation.
In the research problem (B), visualization options are proposed that focus the user's
attention on the features of the data that are relevant to the purpose of the analysis.
In the selection problem (C), the initial data visualization is supplemented with
visual images of the hypotheses being formed, for example about the internal dependencies
necessary to select one of the possible solutions.
The visualization
option offered for decision support systems assumes the capability to visualize
not the initial data but only the results of intermediate interpretation.
Fig.
4. Changes
in the levels of visual communication occurring as a result of coordination
of its goals, user capabilities and visualization tools used
|
|
|
A
|
B
|
C
|
Fig. 5. Visualization options with different
functional purposes
Thus, the paper proposes
a scheme for adapting visualization tools to the specifics of the tasks for which
they are developed.
Analysis and interpretation of significant volumes of
multidimensional data, including those obtained as a result of computational experiments,
should be considered as one of the promising areas of visual analytics development,
to which the results obtained in this work may be valuable
[14].
Application
of adaptable visualization techniques will reduce the level of interpretation errors
and increase the efficiency of existing and developed visualization tools in their
practical use.
1.
Zakharova,
A.A., Shklyar, A.V.: Informative Features of Data Visualization Tasks. Scientific
Visualization, 7 (2), 73-80 (2015).
2.
Bondarev,
A.E., Galaktionov, V.A.: Generalized Computational Experiment and Visual
Analysis of Multidimensional Data. Scientific Visualization 11 (4), 102-114
(2019). doi: 10.26583/sv.11.4.09
3.
Bondarev,
A.E.: On the Construction of the Generalized Numerical Experiment in Fluid
Dynamics. Mathematica Montisnigri XLII, 52-64 (2018).
4.
Solomonik,
A.: Ocherk obshhej semiotiki [Essay on general semiotics]. URSS, Moscow, Russia
(2018). [in Russian]
5.
Averbukh,
V.L. Approach to Semiotic Theory of Computer Visualization. The Advances in
Computer Science: an International Journal (ACSIJ), 4 (1), 44-54 (2015).
6.
Ware,
C.: Foundation for a Science of Data Visualization. In: Information
Visualization. Perception for Design (Third Edition). 1-30 (2013). doi: 10.1016/B978-0-12-381464-7.00001-6
7.
Green,
T.M., Ribarsky, W., Fisher, B.: Building and Applying a Human Cognition Model
for Visual Analytics. Information Visualization, 8 (1), 1-13 (2009). doi: 10.1057/ivs.2008.28
8.
Zakharova,
A.A., Vekhter, E.V., Shklyar, A.A.: The Applicability of Visualization Tools in
the Meta-Design of an Educational Environment. European Journal of Contemporary
Education 8(1), 43–51 (2019). doi: 10.13187/ejced.2019.1.43
9.
Sacha,
D., Sedlmair, M., Zhang, L., Lee, J.A., Weiskopf, D., North, S., Keim, D.:
Human-Centered Machine Learning Through Interactive Visualization: Review and
Open Challenges. In: ESANN 2016 proceedings, European Symposium on Artificial
Neural Networks, Computational Intelligence and Machine Learning, pp. 641–646.
i6doc.com publ, Bruges, Belgium (2016).
10.
Martin, B.,
Hanington B.: Universal Methods of Design: 100 Ways to Research Complex
Problems, Develop Innovative Ideas, and Design Effective Solutions. Rockport
Publishers (2012).
11.
Zakharova, A.A.,
Vekhter, E.V., Shklyar, A.V., Pak, A.J.: Visual Modeling in an Analysis of Multidimensional
Data. J. Phys.: Conf. Ser. 944 012127 (2018). doi: 10.1088/1742-6596/944/1/012127
12.
Zakharova, A.A.,
Shklyar, A.V.: Visual Presentation of Different Types of Data by Dynamic Sign Structures.
Scientific Visualization, 8 (4), 28-37 (2016).
13.
Zakharova, A.A.,
Podvesovskii, A.G., Shklyar, A.V.: Visual and Cognitive Interpretation of Heterogeneous
Data. Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci., XLII-2/W12, 243-247
(2019). doi: 10.5194/isprs-archives-XLII-2-W12-243-2019
14.
Bondarev, A.E.:
On Visualization Problems in a Generalized Computational experiment. Scientific
Visualization, 11 (2), 156-162 (2019). doi: 10.26583/sv.11.2.12
