1. Introduction

Due to mass computerization, human-computer interfaces have entered the lives of almost all of humanity over the past decades. People perceive computer systems in general particularly via user interfaces. Computerization has changed the way people work dramatically. In some cases, these changes are positive, in others, they cause problems, which in turn alter the results of work. When some computer-system design decisions (mostly those concerning human-computer interactions) are poorly thought-out, this causes growing stress in users, which has little to do with the task at hand. Presumably, these negative effects are caused by lack of attention not only to human factors, but also to the goals and tasks of work, its motives etc. A serious analysis of the aspects of future product users activity is due in the interface design. Activity approach is one of the most important factors to creating the theory of human-computer interaction.

This paper is dedicated to the analysis of specialized (both professional and mass) interfaces serving as instruments in goal-oriented and productive activity. It refers to the theses of activity theory and some topics of psychology and physiology, the implementation of which the authors deem useful for the field of practical interface design and development. The history of interface design clarifies some nuances of modern interactive systems. Then we discuss examples of prototype implementations of service interfaces, and the prospects of introducing the activity approach into the practical design of specialized interactive systems. Also addressed are the problems of users-researchers activity, which arise in designing specialized systems of computer visualization using virtual reality.

2. Related Works

Activity theory, developed in the middle of the 20^th century, is primarily connected with A.N. Leontiev and S.L. Rubinshtein (Leontiev 1978, Rubinshtein 2005). The articles proposing the implementation of the activity theory in the design and development of human-computer interfaces were appearing from the mid-to-late 80s to the early 90s of the 20^th century. Among the pioneers of this approach one should mention V.P. Zinchenko (Zinchenko 1992). The publications of V. Kaptelinin (e.g., Kaptelinin 1992-a, Kaptelinin 1992-b, Kaptelinin 1996) made an important contribution to the development of the ‘activity’ approach to human-computer interaction. In the mid-90s, the book was published (Nardi 1996 – Ed.), which included research and review articles concerning the opportunities of the activity theory in regards to creating the theory of HCI (Kuutti 1996, Kaptelinin 1996, Nardi 1996). In this book, the experimental research of human-computer users’ activity was published. The main results of the research from the 90s and early 2000s are reviewed in an in-depth article (Rogers 2004). Later, the research on applying the activity approach in HCI and software engineering appeared, too. In 2002, a special issue of the Computer Supported Cooperative Work magazine was published, which was dedicated to the activity theory and software systems design in practice (CSCW 2002). A collection of academic papers discussing approaches to creating HCI systems was published in 2003 (Carroll J.M. (Ed.) 2003).

The papers published in the beginning of 2000s are discussing the implementation of the activity theory both in the design of HCI systems (Gould E., Verenikina I. 2003) and in preliminary pre-design analysis of tasks (Crystal A. and Ellington B. 2004). The articles of the 2^nd half of 2000s and the beginning of 2010s contain a variety of examples of implementing the activity theory and activity analysis in interactive systems design (e.g., Kaptelinin 2011). The paper (Canino-Rodríguez et al. 2015) addresses approaches to the development of smart air traffic systems. These approaches are close to those of the activity theory, in regards to interface design. In (Sjolie 2011) activity theory is considered in relation to brain-computer interaction.

A comparative analysis of the activity theory and several other lines of research into human cognitive abilities in regards to developing the theory of HCI was carried out in a number of works (e.g., Aboulafia A., Gould E. and Spyrou T. 1995, Carroll, J.M. (Ed.) 2003, Rogers, Y. 2004).

Presently, a wide range of papers and books has been published on the development of the activity approach to HCI design. Among them are monographs of V. Kaptelinin and B. Nardi (Kaptelinin V. & Nardi B. 2006, Kaptelinin V. & Nardi B. 2012), G. Gay and H. Hembrooke (G. Gay & H. Hembrooke 2004), D. Mwanza (Mwanza 2011). V. Kaptelinin did a full survey of the activity theory for HCI in 2012, in an electronic encyclopedia of human-computer interaction (Kaptelinin 2012). The paper (Bakke S. 2014) deals with the specifics of implementing the activity approach into interface design. The paper (Clemmensen T., Nardi B., Kaptelinin V. 2016) analyzes the activity theory in HCI since it emerged about 25 years ago. Our paper is dedicated to the analysis of specialized (both professional and mass) interfaces serving as instruments in goal-oriented and productive activity. Additionally, we consider user activity for specialized systems of Computer Visualization.

3. The Activity Approach to Interface Design

In the beginning of this section, we are briefly considering a number of aspects of the activity theory and several theories of physiological cycles, relevant to the design of human-computer interfaces. We will also address the works of S.L. Rubinshtein (Rubinshtein 2005), A.A. Ukhtomsky (Ukhtomsky 2002), N.A. Bernshtein (Bernshtein 1947), P.K. Anokhin (Anokhin 1968, Anokhin 1978).

Activity is always conscious and purposeful. In the course of an activity, an action is conscious when a partial result gained by it becomes a direct goal of the subject. It ceases to be conscious when the goal is moved further, and the previous action becomes only a means of accomplishing another action, aimed at a more general goal. That way, the action aimed at smaller goals is removed from consciousness and moves into the unconscious. This is how a hierarchical structure is established: activity – conscious actions – operations. That is, an activity breaks down into a number of conscious and motivated actions, which are implemented via a set of operations. Personal or group activity can also be broken down by finding out its goal, motivation, conditions, and personal characteristics.

Activity as a whole can be determined by its result. Decision-making is always oriented towards the result corresponding with the dominating motivation in the moment.

Activity can be implicitly represented by the following questions:

1.       What is the desirable result?

2.       When exactly is the result to be achieved?

3.       By what means is the result to be achieved?

4.       How can we determine if the result is sufficient?

In designing the user activity for service interfaces, it is important to provide consistency of attention and concentration, on the one hand, and ability to switch between different kinds of work, on the other hand. Therefore, the notion of a dominant is important. According to A.A. Ukhtomsky, a dominant is ‘a more or less stable focus of increased excitability [of the centers] … new excitations coming into the centers serve to intensify (confirm) the excitation in the focus, while inhibitory phenomena are widely developed in the rest of the central system’. The inertia of the prevailing excitation, i.e. the dominant of a currently experienced moment, may be viewed as a source of perception inertia and, subsequently, of false perception (as well as of biases, obsessive images or even hallucinations). The possibility of wrong perception is to be accounted for in designing interfaces. At the same time, this inertia facilitates stability of attention and structuring of perceived information. Activity has to keep being adjusted and remain free from pre-set schemes to be effective and adequate. Excessive monotony of actions leads to quick fatigue, attention loss and incorrect work execution as a result.

An activity consists of a set of conscious, goal-motivated actions, which, in turn, can be broken down into sets of operations. At every level of this hierarchy, one has to find and strictly define the goals relevant to the activity. In designing operations of an interface, consistency should be maintained. In other words, execution of the same (or similar) operations should produce the same or similar results. Taking into account psychomotor factors is essential in designing separate operations and their combinations. We shall consider several aspects of the theory of movement behavior.

An action is based on primary automatisms that form during earlier development of an individual. However, executing an action also generates new, more complex automatisms called skills. An action becomes a skill when an individual through exercise has gained the ability to perform an operation without making its execution their conscious goal. At the same time, an individual is not restricted from exercising conscious control over it, although this control may disrupt the fluidity of an automated movement. A skill is a component and a way of executing an action, and it depends on the semantic content of the latter. The automatic activation of a skill draws upon the semantic content of conditions in which it is executed. A skill may be a complex operation determined by complex semantic content. Skills develop through exercise. Conscious goal-oriented exercise is tuition; it is not only reinforcement, but improvement as well. A certain transfer takes place: the positive effect of exercising one skill spreads to exercises of other skills. To make such a transfer happen, certain conformity of elements (elementary movements), as well as components, aspects of a skill, is necessary. This conformity is not abstract, it should be perceived by the individual. Any skill, whether it relates to movement or knowledge, forms at different levels of brainwork. The goal of tuition is to achieve the automaticity of a skill, thus transferring it to a more basic level of movement behavior.

To analyze an activity (before starting to design instrumental interfaces), one first has to find its purposes and means of achieving a goal, to assess how well future users understand the goal and to determine their motives.

The task of an interface designer is to minimize the difficulty of an activity when using an interface and to assure the consistency of the interface. The latter means that similar tasks should be resolved by similar actions via similar (or analogous) operations. The resolution of every task inside an interface should not be achieved via complex actions, thus becoming a separate activity in itself.

Activity approach to designing human-computer interaction for a certain problem presupposes in-depth research of the work of future users in a ‘pre-computer’ phase, analysis of all encountered tasks, and description of activities necessary to solve them. Of great importance is defining the main goals and motives of the activity in question, describing separate stages of the activity and finding all entities encountered by the workers. Lastly, an activity analysis of the situation arising after the work has been computerized is due. It is crucial for actions and operations in popular interfaces to remain simple. In designing interfaces, one has to consider the level of difficulty users will experience in mastering them. Mastering an interface involves acquiring the relevant knowledge, proficiencies and skills. Simplicity of an interface is largely connected with automatic execution of the operations within it.

4. Activity Analysis of Programming and Interfaces

Modern computing has been developing since the 1940s. Along with computing hardware, computer programming has developed as a separate kind of activity. Software development generally includes analyzing the task at hand, choosing or creating methods and algorithms of task-solving, breaking down the whole task into a set of separate actions (subroutines, functions, procedures), and accomplishing the necessary actions to create a set of operations for a real or virtual machine. (Here, a virtual machine refers to the description of rules of a programming language. For instance, the notions of Fortran and Algol machines were mentioned.) The resulting computer program is verified by means of testing and debugging. Software may be developed by a single programmer or by a team, where separate stages of software creation or separate parts of code are distributed among the members. Note the similarity between the description of an activity per se and the process of programming. However, the human factor was almost entirely discarded from the notion of programming, as the latter was not considered an activity, because the main task was to attain the maximum efficiency of computing. At first, the complexity of programming was not regarded at all. Originally, computers were used mainly in scientific and engineering computing. Programmers then were specialists in applied mathematics, physics and engineering disciplines. Every interaction with computers was inseparable from programming. Program users had to interact with computers to inspect the progress of modeling and modify the work of their programs by establishing new parameters of their execution. The immediate human interaction with computers was determined by the existing level of electrical engineering and electronics. In stock-produced computers for data input and output paper media were used (for example, perforated cards and perforated tapes for input, and paper tape for output via printing devices). This case of exchanging information can be described as an exchange in monologues (Voiskounskiy A.E. 1990). Later a set of hardware for human-computer interaction was developed, which included character and graphic displays and means of input: a character and function keyboard on a display, a rotary dial, a light pen, and a mouse. Later joysticks, trackballs and touchscreens were introduced. But still, interaction with a computer (or rather, with a program) included elements of programming and remained within the scope of programming activity.

Computerization in the 70s and 80s encouraged the launch of office automation. It was necessary to support the customary office activity connected with text input and working with different spreadsheets. This task was solved via word processors and form-based languages, which later on appeared in Microsoft Office and its counterparts. An idea was suggested for developing visual (iconic) languages, which would make solving some of office automation tasks easier. Later, operating system interfaces were created on the basis of desktop metaphor. Interfaces in the framework of desktop metaphor are based on tacit programming of computer systems and some not explicitly given virtual devices. In many cases, users of these interfaces had used some forms of programming. Note that interface designers did not strictly define (or possibly, they were not aware at all of) the ‘programming language’ or a ‘virtual device’ that was to be ‘programmed’.

The next stage of human-computer interface development is connected with the Internet. Recreational websites and sites of electronic commerce and service provisioning became an important source of ideas for interface development. Interface quality evaluation became closely connected with evaluation of effectiveness of electronic commerce and advertising placed on websites. The effectiveness of the latter can be evaluated primarily via the number of ‘clicks’ on an advertising banner (click-through rate). It appears that this is one of the most important reasons for using the (behavioristic) ‘stimulus-reaction’ model in interface evaluation.

Often without noticing it, the users of professional and mass interfaces become involved in the process of programming some virtual devices, which are defined by relevant program systems. In some cases, this tacit programming can be grasped intuitively, yet, in others, it may annoy the users, even causing stress.

 

5. Analysis of Instrumental Interfaces

This section deals with instrumental interfaces (see also Beaudouin-Lafon, 2000). We shall assume that service interfaces are to be understood as interfaces for specialists in a certain area, which the they use as a means of performing their professional activity, as well as instrumental interfaces for general use. Two classes of instrumental interfaces are under examination here: professional and mass interfaces. One may define professional interfaces as interfaces for specialists in those areas where a major part of activity constitutes professional work with people and/or documents, whereas interfaces are used as a tool for professional activity. Mass interfaces are those intended for general public use including utilization of various socially significant services. In case of mass interfaces, a designer, while defining the demands to an interface, takes part in forming the future activity. A user cannot abandon the use of a certain system because it provides access to vital services, resources, information etc. Mass interfaces should be aimed at the ‘weakest link’, i.e. a person with minimal ability in input, perception and analysis of information should be able to use it successfully. In case of professional interfaces, the goal of a user’s activity is pre-defined. Description of a task in general formulates the demands to an interface. A professional cannot abandon the use of an interface either because their activity is strictly regulated. An interface designer should examine the goals and peculiarities of a certain activity, so as not to corrupt it and to prevent any additional difficulties. We suppose that programming should not be present in professional interfaces as a separate activity, which would add to the core responsibilities of a professional. Analysis of professional and mass interfaces should be conducted from the viewpoint of specialists (for instance, office workers and health workers) who have personal experience in their professional activity. Accordingly, mass interfaces should be designed upon the experience of common users (Averbukh et al. 2014-a).

In the course of their activity professionals are dealing with a number of entities. For example, they may process personal documents, fill in the forms of internal documents, interact with customers, sometimes accept money and give checks. Computerization adds a new kind of activity and creates a new entity – interaction with a program. There are examples of interfaces that continuously switch the attention of operators, overloading them with additional tasks and interfering with their customer interactions. Thus, the analysis of activity caused by computer interfaces should be carried out, to evaluate both possible ‘redundancy’ and ‘insufficiency’ of computerization. Generally, the number of entities that a professional has to deal with should be reduced and not increased. That is why a designed interface should be able to do all the working with an entity itself. This way, an interface will not become a new, additional and complicating entity in an activity of a professional.

Quality and usability criteria used for evaluation of recreational websites and social networks, such as page visibility time, click counts for a certain picture, or personal opinions of a small number of respondents, are not applicable for evaluating instrumental interfaces. The criteria in question must be based on performance evaluation for individual users and a computerized organization as a whole. The quality of ‘instrumental’ mass interfaces can be measured by the time spent to achieve the desired result and the level of stress while trying to do that. For that end, concise interfaces with minimal demands for user’s memory and attention are required. Hence the requirement for saving and restoring the current state and context of the interface. Menu-based interfaces or any kinds of programming techniques are hardly applicable here. The quality of a professional instrumental interface can be measured by the quantity of people satisfied by the work of an enterprise within a set period of time. Presumably, the stress level of a professional using a certain interface would affect the stress level of a customer due to possible delays, failures and general annoyance. Designing instrumental interfaces is inseparable from solving more general problems, such as proper organization of an institution’s workflow (in which the interface will be used), documentation maintenance, securing access to data etc. Yet, all these decisions are generally outside of a designer’s competence. An interface designer has to examine the goals and peculiarities of a certain activity, so as not to distort it or bring in additional difficulties. The naturalness of professional interfaces should be tied to the activity experience of a professional for whom the interactive environment is being developed. For instance, today typical surgical tools include not only (and not so much) scalpels but also various devices controlling complex hardware for medical operations, along with specialized human-computer interfaces. In this case, professional interfaces are those for controlling hardware, setting up devices and their conditions. Gesture interfaces may be implemented to control hardware and devices as well. Therefore, it would not be necessary to train medical professionals to perform additional procedures with their hands/fingers, and a ‘shift of context’ would not happen in the middle of a critical activity.

In (Averbukh et al. 2014-a), the description of a prototype of a computer system for managing a clinic also included descriptions of implementing interactive systems supporting the interface for patients, interface for general medical professionals seeing patients, and interface for procedure reports on cardiac interventions. The interfaces were designed from the perspective of the activity theory. In the process of design, several assumptions were made concerning the necessary infrastructure. After the analysis of activity of patients and medical professionals, the functionality of developed services and their interfaces were chosen. Certainly, in making this choice, a number of simplifications were allowed.

A patient’s interface is an example of a mass instrumental interface. In its design, possible goals and tasks of both the patient and the medical institution were accounted for. We considered the main goal to be referring the patient to a medical professional in a quick and easy way, and assuring that the latter gets all the necessary information about the patient. The interface was made as simple as possible, to ensure its usability for a diseased person, who at the time is not inclined to [quasi]programming activity. The patient, upon entering their personal data, is referred to a medical professional. As the analysis of symptoms done by a patient on their own or via a phone operator may be inaccurate and may lead to unwanted consequences, the patient on their first visit is referred to a medical professional for an initial diagnosis. During further revisits the choice of a medical professional is based on the patient’s clinical record, the latter being also able to choose the time of visit.

In designing the interface for medical professionals we decided to computerize such elements of their activity during a visit as output of objective data of a patient’s health and subjective patient’s complaints, updating the clinical record, assigning therapy, and writing prescriptions. In assigning therapy and writing prescriptions for drugs, contraindications for a certain patient were to be examined. The present state of a patient’s clinical record was shown. For a full-scale diagnosis of an illness the possibility to check previous treatment by medical professionals of other specialties was implemented. To support the diagnosing, an output of medical tests and visualization results data (for example cardiograms, X-ray pictures), added to the clinical record of a patient, was made possible. By design, test results were meant to be added to the patient’s record automatically from the relevant hardware. Links to images allowed full-scale examination to see minute details. Also a list (managed by a medical professional) of additional attachments to the medical record was designed, which may contain any data. The interface included a form to enter the initial diagnosis for a patient, and a form for therapy assignment and writing prescriptions. The prototype of the system also has a simple knowledge base on contraindications in therapy and drugs for every patient, formed according to the medical record and current prescriptions. (See Figure 1.)

A project of an interface for a cardiac pacemaker implantations surgery logbook was created as a practical application of the activity approach to interface design. Design and implementation in this case are based on a cardiac surgeon professional knowledge of activity. Documenting a course of treatment is an important task, both in professional and legal sense. Development of a specialized service interface was based on the method of direct manipulation. This method implies creation of an interface in which users, in solving their tasks, directly manipulate visual representations of the objects upon which actions are performed. The concept of direct manipulation was developed in the beginning of the 1980s by a renowned computer scientist B. Shneiderman (see one of the first accounts of this approach in (Shneiderman 1983, Jacob 1986)). Every user of iconic interface has encountered this popular concept in one way or another. However, it is often forgotten in service interfaces, where almost literal transfer of ‘paper’ documentation into electronic version take place instead. There is a description of a surgery logbook interface in (Averbukh et al. 2014-a). This interface implies step-by-step work of a surgeon user, who manipulates pictograms referring to cardiac pacemakers and electrodes. The resulting visual representation may be converted to text in a required format.

The paper (Averbukh et al. 2014-b) contains an account of developing natural interfaces, which can be eventually used by surgeons in an operating room, for instance, to facilitate sterility. At the same time, a sign language for equipment management was evaluated. Medical professionals noted the disadvantages of sign languages based on the demonstration of finger figures. Using such gestures requires actions and operations that are additional to activity of a medical professional already busy with diagnostics or a surgical operation. It distracts and interferes with a surgeon’s work.

 

Figure 1. Examples of a ‘medical professional’ interface (prototype).

 

6. User Activity for Systems of Computer Visualization

A cycle of computer modeling includes creating qualitative (physical) and mathematical models, choosing or developing algorithms and computing methods, programming, computing, analysis and interpretation of results. Within the cycle of computer modeling, visualization facilitates analysis and interpretation of computing results. There are several subdomains of computer visualization, such as scientific visualization, software visualization and information visualization. Visualization as creating visual images of mental models had existed long before the emergence of modern computers. Moreover, translation of data into graphic images may be viewed as part of our daily life. However, computer visualization (in contrast to ‘pre-computer’ one) is most often aimed at research and discovery of new knowledge. One account of computer visualization as a separate discipline in 1987 mentions that ‘visualization offers a method of seeing the unseen’. It also notes that the goal of visualization is insight, not pictures (Visualization in Scientific Computing, 1987). Visualization should form (or facilitate the formation of) holistic mental models and, consequently, create insight. The occurrence of insight is considered one of the main criteria in evaluating the visualization quality (North, Ch. 2006). The insight in visualization is connected with user's mental models. The insight forms a new mental model of entities under analysis and visualization. And, on the other hand, the insight is based on pre-existing user's mental models.

Scientific visualization is one of the subdomains of computer visualization, and it deals with illustrating objects, processes and phenomena modeled in scientific calculations. Currently scientific computing is mainly conducted on supercomputers, and still may take many hours or even days. The results are huge volumes of data describing objects with complex structures. Generally, the modeled objects lack any similarities in nature. In many cases there are also no conventional, habitual (for this scientific discipline) methods of depicting such objects.

The process of visualization means building a visual image upon abstract ideas of an object. These abstract ideas constitute a model of an object, a phenomenon or a process researched, which relates to a user’s cognitive structures describing this entity. Visual images representing a modeled entity serve to create or restore the cognitive structures upon it. The task of visualization is to obtain a visual image, by means of which a mental image (idea) of the object in question can be correctly restored.

Drawing upon the long-term experience, we can conclude that activity of a specialist working with visualized representations for different purposes is mostly analogous in goals and means to activity of a researcher described in the works of Brushlinsky. Search for methods of visual representation of the main entities of computing models that would allow full interpretation of modeling results is a major task of researches in the field of computer visualization.

A system designer should consider tasks, goals and motives of a future user’s activity. Besides knowledge of the subject field under consideration, a designer should understand the way researcher users conduct their work. It is often assumed that a user imagines with sufficient accuracy what is sought in an observable, yet large array of data. In scientific and software visualization, however, these conditions are often inapplicable. In many cases, mathematical and computer modeling creates fundamentally new objects. The activity of a researcher studying visualized representations draws upon many factors, including their own personality traits. In addition, the activity of a researcher does largely depend on the hardware used for visualizing and interfaces. Users of scientific visualization systems are researchers busy with computer modeling of complex phenomena and processes. Very often, only an extremely small number of specialists can analyze and interpret the results of such modeling. That is why the quality of visualization and the degree to which the perceptual peculiarities of a certain specialist are taken into consideration are vital for the results of any important work. Observing the users of visualization systems allows us to distinguish several types of work behavior. Sometimes a quick glance at an image enables the researcher to produce some conclusions about the problem in question. In other cases, visual inspection takes a long time. Here, different modes are possible; for example, monitoring changes in a dynamic image or a visual text reading with analysis of its details, all the while interacting with visual objects.

Such differences are also associated with the tasks of visualization, which determine its type (cognitive, proving or illustrative), features of a model and stages of working on it; they also stem from professional and individual differences in users’ work methods. All of this affects the choice of imaging and methods of visualizations. Different types of visualization result in the differences in the activity of specialists using it.

Ideas of A.V. Brushlinskiy (Brushlinskiy A.V. 2003) are of importance for analyzing user experience in computer visualization systems. An emerging insight is considered to be the criterion of a successful visualization. In A.V. Brushlinskiy’s works, insight is regarded as an event in which an individual ‘immediately formulates the basic thought arisen’.

According to A.V. Brushlinskiy, the process of thinking begins with awareness of a problem. Its analysis leads to formulating a task, to the separation of the known from the unknown. In solving a scientific task, an individual cannot exactly predict the unknown, the result of a thinking process, early on. A.V. Brushlinskiy shows that starting a cognitive activity of solving a task, an individual does not and cannot know the answer yet unknown: which characteristics and relations of an object cognized are to be revealed; what kinds of action and thought operations are to be formulated and so on. In other words, while the start of thinking has been already established, it has no ‘end’ because the final situation of thinking does not exist. Yet, still being at the initial stage, an individual starts, uncertainly, to anticipate the future solution. The process of solving a task lies in revealing the relations between its elements, its conditions and demands. The individual solving a task is performing analysis via synthesis. New characteristics and relations of the tasks elements are laid out and synthesized with each other, until at last the solution is found. Cognitive visualization is aimed at helping the researcher see all the elements of the task at hand, to evaluate their relations with each other.

One may say that a search for a solution by a scientific visualization system user largely matches the activity of a researcher busy with a scientific problem, and may include both instant and non-instant insight. According to A.V. Brushlinskiy, in the latter a thought is formed within several seconds before one’s eyes (it is not originally available and is actually formed, not simply formulated). Studying the activity of a researcher analyzing and interpreting data with the help of visualization is a major task, which would allow to improve the efficiency of computer modeling as a whole.

There is no general description of a researcher user activity for systems of computer visualization yet. However, application of the activity theory helps in solving specific tasks of designing specialized systems of scientific visualization. A vital task is visualizing the process of solving problems in mathematical physics. An approximate solution is achieved by means of grid generation.

The accuracy of modeling results depends on the number of grid refinements. The more complex is the structure of the grid is, the more accurately it depicts the phenomenon examined. The number of grid cells has increased from 10⁴ to 10¹⁴ over the past two decades. Researchers working with such a grid must be able to examine separate fragments of the grid. Therefore, designers of the system visualizing this grid must enable navigation in three-dimensional space. Navigation in the system is implemented via a complex of camera control functions. A projection system of data manipulation is used here. Functions of data manipulation are divided into groups like movement, rotation, drawing sections etc. (Averbukh V.L. et al. 2010). Moreover, the researchers are interested not only in a dimensional image, but also in the distribution of values on the grid surface; this is commonly depicted by color gradation. When a large amount of grid cells is present, interpretation of an image obtained is obstructed because the user is finding it difficult to visually grasp and evaluate a large quantity of significant objects. (See Figure 2.)

 

Figure 2. Visualization of Computational Grids (Averbukh V. et al 2016)

 

VR provides us with a wider view space than monitor screen that allows user to change some visualization parameters such as the thickness of the tree, tree scaling parameters and the size of the label leaf. In order to see an individual leaf or branch, user just needs to go closer to it by passing through VR environment. That is much more convenient than zoom in /out in 2D format. In this way, VR visualization is more flexible than 2D presentation. It provides a new quality of visualization and enables user to view a general picture while the details are simply visible (Forghani, Vasev, Averbukh, 2017). Figure 3 demonstrates three-dimensional visualization of the physico-chemical changes in protein in the form of phylogenetic tree.

 

Figure 3. Virtual reality presentation of circular tree. (Forghani, Vasev, Averbukh, 2017)

 

Using virtual reality environments dramatically improves the ability to analyze and interpret the results. In implementing specialized systems of visualization of computational grids with virtual reality goggles, several problems of organizing users’ work arise. One of them is connected with the way the state of presence affects the user’s ability for intellectual activity to analyze and interpret data with complex structure. It seems important to evaluate the influence of virtual reality as a workplace of a fundamentally new kind, as well as the influence of experiencing the phenomenon of presence on human activity.

While working in such a system, the user has to move inside or around the grid to highlight and move its fragments etc. Therefore, an issue arises of the human factor in dimensional orientation and navigation in spaces with complex structures, which are three-dimensional grids. Some ways of interacting with the environment or moving in it may become inconvenient for the user, causing discomfort, while others may distract them from the task.

However, planning a research based on a real problem uncovered an issue mentioned in the work (Baker M.P., Wickens Ch.D. 1995): the difficulty of organizing a research in which specialists analyzing scientific data would play the role of subjects. Current specialized system is targeted at a small group of people, who are able to grasp the material offered and draw conclusions from it, important for them and their work. The number of such specialists was found too small to make a representative sample. Therefore, a problem arose of choosing a model task, solving which the subjects would execute mental operations similar to those executed by researchers in the area of mathematical physics. It was for the needs of the latter that three-dimensional grid was designed, and it brought about the issues of compatibility of intellectual activity and experiencing the phenomenon of presence.

After analyzing possible solutions, the Kohs Block Design Test (a part of Wechsler’s intellect test) was chosen as the basis for modeling a user’s work with the specialized system of scientific visualization.

Kohs Block Design Test allows to determine spatial abilities of an individual, including the ability to construct clear images that do not get destroyed after transformation (Hegarty, M., Kozhevnikov, M. 1999), and the ability for flexible redistribution of attention resources to optimize the process of solving the task (Milne, E., Szczerbinski, M. 2009). Kohs Block Design Test evaluates the ability to execute basic intellectual operations: comparison, analysis and synthesis. Therefore, Kohs Block Design Test can be used as a model of mental activity because it embraces a wide range of cognitive abilities.

The subjects of the experiment were in the virtual environment which represented a room with a table.  The subject was required to take blocks that have all white sides, all red sides, and red and white sides, and arrange them according to a pattern. They were timed on this task and compared to a normative sample. One-third of the subjects pointed out experiencing a feeling of presence during an inquiry. These results correlated with those from the Presence Questionnaire (Witmer B.G., Singer M.J. 1998). The analysis did not show the timing differences while solving the test or the amount of correct answers among those who pointed out the feeling of presence and those who did not. The obtained experience of continued development of grid visualization specialized systems using virtual reality generally confirms the results of the experiment. (See Figure 4.) (Averbukh N. 2014), (Averbukh1 V.L., Averbukh N.V. 2017).

 

Figure 4. Example of the virtual environment realized in the Kohs Block Design Test

 

Conclusion

Computerization of any process brings serious changes to the activity of professionals and general population, and sometimes leads to emergence of new kinds of activity. By means of several examples we have reviewed the possibilities of the activity theory in developing popular and professional interfaces, and in creating the means of visualization. An interface is a ‘face’ of a computer system. An activity is formed via interface design. Therefore, the activity theory is applicable when designing new software complexes, and also for evaluating the existing interactive systems. In that regard, a need in activity analysis arises. It may be useful to include such specialists as an Activity Analyst and Activity Engineer/Designer into development teams. Their tasks would include general description of an activity, finding out all its participants, goals and motives, and determining actions and operations for executing the activity. Such analysis would serve as a basis for designing interfaces through which operations and actions are executed.

The main activity is the one cemented with a common goal, motive for its execution and a sense of unity of all actions included in it. Furthermore, any one of those actions does not possess a distinct goal, but serves only to accomplish a certain task necessary to execute activity as a whole. When dealing with an interface as part of a general activity, an individual switches to interact with a computer and back without stopping their main activity or perceiving their actions as something separate. When an interface demands some special activity, an individual has to stop carrying out other tasks and switch to interacting with a computer. Emergence of additional activity through computerization should not be allowed; however, it may happen when ‘paperwork’ techniques are directly translated to computer ones. Of necessity are those kinds of activity which really make the work easier and greatly increase its intensity. Note the occasional forming of activity via tacit programming of an undescribed quasi-computer device. In this regard, interfaces based on direct manipulations are more successful.

It should be taken into account that in the evaluation of interfaces not only the final outcome of a single usage is important, but also work conditions and general condition of a user during an extended period of time. Analysis and design should also be conducted with regard to general psychological and ergonomic criteria, including the presence of stress, both in users of interactive systems and in consumers of computerized institutions. Visualization is one of the final stages of computer modeling. Activity of specialist users in analysis and interpretation of visual representations has not been described generally. However, the emergence of insight is a criterion of visualization quality because its occurrence may lead to correct data interpretation.

Specialized systems are often targeted at a small circle of researchers, who are able to understand the proposed material and make the important conclusions based on it. Therefore, the development of specialized visualization systems involves the study and analysis of the general and local goals and tasks of users, the set of their actions and the operations required to achieve the result of their activities. As was already mentioned, in order to evaluate the suitability of visualizations and interfaces, it is important to consider not only the result of one-time using, but the working conditions and the overall state of the user over a long period of time. This analysis is especially important in the course of designing the visualization systems using virtual reality. These designs should take into account common psychological and ergonomic criteria, as well as psychological and physiological characteristics of potential users. The activity of researchers (users of visualization systems) in the analysis and interpretation of visualizations is not described in general terms. Nevertheless, it is necessary to evaluate the user activity generated by specialized visualization systems.

The evaluation of activity created by new computer means and its comparison with ‘pre-computer’ variants should be carried out. One may need to create models of certain kinds of activity. A model is possible and necessary under the condition of insufficiency of knowledge and methods of obtaining it (research methods). A good (accurate) model enables structuring future knowledge, providing information not only about the structure, but also about the interaction of elements and substructures of an object. Further formalization of the notion of activity would allow comparing interactive systems on a more complex basis, which would include, among others, work results, intellectual complexity of separate operations and actions, conditions of users, and customer satisfaction. In addition, one may consider other approaches to theoretical description of HCI. For example, in (Nicastro F., Pereira R., Alberton B., Morellato L.P.C., Baranauskas C., Torres R.S. 2015) a semiotics-based approach to mobile user interface design was proposed.

Finally, we pay attention to the wide distribution of gestural interfaces associated with smartphones, game consoles and similar devices. These interfaces are already being successfully adopted by children from the earliest age. There are examples of young children trying to apply gadget-based skills in everyday life, trying to zoom in on real objects (for example, observed through the window of a room) with characteristic ‘window expansion’ gestures. One can argue the ergonomics of these interfaces, but they are familiar to hundreds of millions of users. Consideration of this factor is necessary in the design of instrumental (both mass and professional) interfaces for specialized (non-entertaining) systems.

References

1.Aboulafia A., Gould E. and Spyrou T Activity Theory vs Cognitive Science in the Study Of Human Computer Interaction, Proceeding of 18th Conference for Information Systems Research in Scandinavia, Goeteborg, Sweden. pp. 29-38, (1995)

2.Anokhin P. Biology and Neurophysiology of Conditioned Reflex. M., (1968)

3.Anokhin P. Selectas. Philosophical Aspects of the Theory of the Functional System. M., (1978)

4.Averbukh N., Subjective-Situational Study of Presence. Lecture Notes in Computer Science. Volume 8525. pp. 131-138, (2014)

5.Averbukh1 V.L., Averbukh N.V. (2017) Evaluation of user's psychological states in virtual-reality environments in GraphiCon 2017. 27th International Conference on Computer Graphics and Vision. Conference Proceeding. Perm State University. Perm. pp. 87-91, (2017)

6.Averbukh V.L., Baydalin A.Yu., Bakhterev M.O., Vasev P.A., Kazantsev A.Yu., Manakov D.V. (2010) Experience in Developing of Specialized Scientific Visualization Systems. Scientific Visualization. 4. Vol. 2. N. 4. pp. 27- 39, (2010)

7.Averbukh V., Annenkova O., Artyomova N., Bakhterev M., Vasev P., Pestova M.,  Starodubtsev I., Ushakova O. The Development of Specialized System for Visualization of Computational Grids (2016) In XVI International Conference SUPERCOPUTING and MATHEMATICAL MODELING. Abstracts. Sarov. October 3-7, pp.15-16, (2016)

8.Averbukh V.L. Averbukh, N.V., Naimushina A.V., Semenischev D.V., Tobolin D. Yu.  Activity approach in the design of human-computer interaction: the example of medical interfaces. Moscow. URSS, (2014-a)

9.Averbukh V.L. Averbukh, N.V., Starodubtsev I.S., Tobolin D. Yu. The Use of Gestural Interfaces in the Interaction with Objects // Scientific perspective. No. 10 (56)/2014. pp. 57-66, (2014-b)

10.     Baker M.P., Wickens Ch.D. (1995) Human Factors in Virtual Environments for The Visual Analysis of Scientific Data. NCSA-TR032, (1995)

11.     Bakke S. (2014) Immediacy in User Interfaces: An Activity Theoretical Approach. In: Human-Computer Interaction. Theories, Methods, and Tools. Volume 8510 of the series Lecture Notes in Computer Science, pp. 14-22, (2014)

12.     Bernshtein N. Motion Synthesis M. Medgiz, (1947)

13.     Beaudouin-Lafon M. (2000) Instrumental Interaction: An Interaction Model for Designing Post-WIMP User Interfaces. Proceedings of the ACM CHI 2000 Human Factors in Computing Systems Conference, pp. 446-453, (2000)

14.     Brushlinskiy A.V., Actor, Thinking, Learning, Imagination. M, (2003)

15.     Canino-Rodríguez et al Canino-Rodríguez, J.M.; García-Herrero, J.; Besada-Portas, J.; Ravelo-García, A.G.; Travieso-González, C.; Alonso-Hernández, J.B. Human computer interactions in next-generation of aircraft smart navigation management systems: Task analysis and architecture under an agent-oriented methodological approach. Sensors 2015, 15. pp. 5228–5250, (2015)

16.     Carroll, J.M. (Ed.) HCI Models, Theories and Frameworks: Toward a Multidisciplinary Science. San Francisco. Morgan Kaufmann publishers, (2003)

17.     CSCW  A Special Issue of Computer Supported Cooperative Work (CSCW): Activity Theory and the Practice of Design. Volume 11, (2002)

18.     Clemmensen T., Nardi B., Kaptelinin V. (2016) Making HCI theory work: an analysis of the use of activity theory in HCI research. Behaviour and Information Technology 35(8):1-20, (2016)

19.     Crystal A. and Ellington B. Task analysis and human-computer interaction: approaches, techniques, and levels of analysis. AMCIS 2004 Proceedings. Paper 391, (2004)

20.     Forghani M., Vasev P., Averbukh V., Three-Dimensional Visualization for Phylogenetic Tree // Scientific Visualization. 2017. Quarter. 4, Vol. 9, N 4. Pp. 59 – 66.

21.     Hegarty, M., Kozhevnikov, M. Types of visual-spatial representations and mathematical problem solving. Journal of Educational Psychology. 91(4). – pp. 684-689, (1999)

22.     Gay G., Hembrooke H. Activity-Centered Design: An Ecological Approach to Designing Smart Tools and Usable Systems. The MIT Press. Cambridge, Massachusetts, London, England, (2004)

23.     Gould E., Verenikina I. (2003) An activity theory framework for computer interface design. Proceedings of the 25th International Conference on Information Technology Interfaces (ITI 2003), 301-307, (2003)

24.     Jacob R.J.K. Direct Manipulation. Proc. IEEE Int. Conf., Syst. Man And Cybern. Atlanta, Ga. Oct. 14-17. Proc. N.Y. 1986. V.1. pp.384-389, (1986)

25.     Kaptelinin V. Integration of computer tools into the structure of human activity: Implications for cognitive ergonomics. Proceedings Sixth European Conference on Cognitive Ergonomics. Balatonfured, Hungary, 1992. September 6-11. pp. 285- 294, (1992-a)

26.     Kaptelinin V. Human computer interaction in context: The activity theory perspective. Proceedings “EAST-WEST” International Conference on Human-Computer Interaction: EWHCI'92. 4-8 Aug., 1992, St.-Petersburg, Russia. M. ICSTI. pp. 13-28, (1992-b)

27.     Kaptelinin, V. Activity Theory: Implications for human-computer interaction. In B. Nardi, (ed), Context and Consciousness: Activity theory and human-computer interaction. Cambridge, MA: MIT Press. pp. 107 – 110, (1996)

28.     Kaptelinin V. Designing Technological Support for Meaning Making in Museum Learning: An Activity-Theoretical Framework. 44th Hawaii International Conference on System Sciences. pp. 1-10, (2011)

29.     Kaptelinin V. Activity Theory. Encyclopedia of Human-Computer Interaction. Chapter 16, (2012) http://www.interaction-design.org/encyclopedia/activity_theory.html

30.     Kaptelinin V., Nardi B. Acting with Technology: Activity Theory and Interaction Design. Cambridge: MIT Press, (2006)

31.     Kaptelinin V., Nardi B. Activity Theory in HCI: Fundamentals and Reflections. Synthesis Lectures on Human-Centered Informatics. Morgan & Claypool, (2012)

32.     Kuutti K. Activity Theory as a potential framework for Human-Computer Interaction research // Context and Consciousness: Activity theory and human-computer interaction. Cambridge, MA: MIT Press. pp. 17-44, (1996)

33.     Leont'ev, A. Activity, Consciousness, and Personality. Englewood Cliffs, NJ: Prentice-Hall, (1978)

34.     Milne, E., Szczerbinski, M. Global and local perceptual style, field-independence and central coherence: An attempt at concept validation. Advances in Cognitive Psychology. ¹ 5. pp. 1- 26, (2009)

35.     Mwanza D. Activity-Orientated Design Method (AODM): Towards an Activity-Oriented Design Method for HCI Research and Practice. Germany: LAP LAMBERT Academic Publishing GmbH & Co. KG, (2011)

36.     Nardi B. – Ed. Context and Consciousness: Activity theory and human-computer interaction. Editor - B. Nardi, Cambridge, MA: MIT Press, (1996)

37.     Nardi, B. Activity Theory and Human-Computer Interaction In B. Nardi, (ed), Context and Consciousness: Activity theory and human-computer interaction. Cambridge, MA: MIT Press. pp. 7-16, (1996)

38.     Nicastro F., Pereira R., Alberton B., Morellato L.P.C., Baranauskas C., Torres R..S. Guidelines for Evaluating Mobile Applications: A Semiotic-Informed Approach. In: Hammoudi S., Maciaszek L., Teniente E., Camp O., Cordeiro J. (eds) Enterprise Information Systems. Lecture Notes in Business Information Processing, vol 241. Springer, Cham. pp 529-554, (2015)

39.     North, Ch., Toward Measuring Visualization Insight. In IEEE Computer Graphics and Applications, Volume: 26, Issue: 3, pp. 20-23, 2006

40.     Rogers, Y. New Theoretical approaches for Human-Computer Interaction. Annual Review of Information, Science and Technology, 38. pp. 87-143, (2004)

41.     Rubinshtein S. Principles of General Psychology. Moscow. Piter, (2005) 

42.     Shneiderman B. Direct manipulation: a step beyond programming languages. IEEE Computer 16(8). pp. 57-69, (August 1983)

43.     Sjolie D. Reality-based brain-computer interaction. Licentiate thesis, June 2011. Department of Computing Science. Umea University. Umea, Sweden, (2011)

44.     Ukhtomsky A. Dominance. St. Petersburg, Piter, (2002)

45.     Visualization in Scientific Computing Special Issue, ACM SIGRAPH Computer Graphics, V. 21, N 6, November, (1987)

46.     Voiskounskiy A.E. I say, we say... Essays about human communication. – M, (1990)

47.     Witmer B.G., Singer M.J. Measuring Presence in Virtual Environments: A Presence Questionnaire. Presence: Teleoperators and Virtual Environments. Vol. 7 Issue 3, June 1998. pp. 225-240 , (1998)

48.     Zinchenko V.P. Activity theory: Retrospect and prospect. Proceedings “EAST-WEST” International Conference on Human-Computer Interaction: EWHCI'92. 4-8 Aug., 1992, St.-Petersburg, Russia. M. ICSTI. pp. 1-5, (1992)