Tachi_Lab - Virtual Reality

What's Virtual Reality

According to the American Heritage Dictionary (3rd edition), “virtual” is defined as existing in essence or effect though not in actual fact or form. Thus, virtual reality (VR) is an entity which contains the essence of reality and is effectively real. It can provide a basis for technology which enables humans to experience events and act in a computer-generated environment just as if they were in the real environment.

The more one thinks about it, the more one realizes that the term "virtual" is a very profound and important concept. Virtual is an adjective of virtue, which means goodness, merit, or efficacy; and in its original sense, it means “the inherent power that makes a thing what it is.” In other words, each thing has a superficial part and an essential part, and the essential part is virtue; and its adjective, virtual, means "not superficially so, but essentially so.

The superficial part is the appearance, externals, pretext, name, norm, etc., while the essential part can be thought of as the effect, action, function, or substance, so it means "not in appearance or name, but in effect or substance," and since it is an adjective, it modifies the noun XX to become virtual XX. Therefore, it is easy to understand if the noun "XX" is modified to become "virtual XX," which means "not apparently or nominally XX, but in effect or in substance XX.

For example, virtual money, meaning credit card or electronic payment, is "not money in appearance, but money in effect. The term "virtual president" also refers to "not nominally president, but effectively president." It is somewhat an old story, but Hillary Clinton, the wife of former President Bill Clinton, was often referred to as a virtual president and used as a joke favored by Americans. What this virtual president meant was that she was not actually holding the office of president, but what she was doing was just like the president himself. This is why Hillary was called the virtual president. This is a joke, of course, but the use of the word virtual is formal and accurate.

If we apply reality to the "XX" in "virtual XX," virtual reality (VR) is "something that does not appear to be reality, but is, in effect, reality. To put it simply, virtual reality is "the essence of reality" and "extracted reality.

Thus, VR has the essence of reality, but moreover, essence is never absolute. The essence differs depending on what the purpose is. If we could approximate the real world to the fullest extent and have all of its elements, it would be reality itself, but in general it does not have all of them, nor is it necessary to have all of them. Of all the elements, virtual reality is the one from which we extract only the essence, that is, the elements that are important for our purposes. The next question is what are the important elements that vary depending on the purpose.

For example, if the purpose of a "virtual helicopter" is to train pilots, then a flight simulator for airplanes would be the answer. If the purpose is to enjoy the sensation of riding in a helicopter, it might be a game machine. If the purpose is transportation, something like an airship could be considered a virtual helicopter. In this way, virtual reality is the essence of reality that matches the purpose.

The next question is, then, what is the antonym of virtual?

The antonym of “virtual” is "nominal," and virtual is never a counterpart of real. Interestingly, the opposite of nominal is real. This means that real and virtual are almost synonymous. This is no wonder that some dictionaries define "virtual" as "almost real. The opposite word that should be paired with the real is imaginary. Imaginary is the antonym of “real”, and does not mean “virtual.”

If we take the view that the world as we perceive it is a map of the real world projected onto our brain through our sense organs, then we can even argue that the world as we perceive it is also a virtual world created by our sense organs. Human vision detects only a very limited region of electromagnetic waves called light (0.40 to 0.75 μm), and hearing perceives only a limited portion of air vibrations (20 Hz to 20 kHz). The senses of touch, taste, and smell are even lower resolution sensory organs that perceive the world. Humans have expanded this virtual world through the advancement of science and technology. The images of the universe captured by the Hubble Space Telescope and the world of atoms observed through the STM (Scanning Tunnel Microscope) convey the essence of this universe to humans. What people consider virtual is also an important factor. In other words, what is shown in the virtual world changes depending on what people consider to be the essence of a thing.

Thus, virtual reality is essentially a tool for augmenting human capabilities, conveying the essence of the real world to humans beyond the limitations of time and space, and in this sense, it is inextricably linked to the technology of telexistence.

Relation between Telexistence and VR

The term "telexistence" is a coined word combining "tel" or "tele" and "existence". It is a concept of the expansion of human existence, in which a person can exist in a place different from his or her own existing place and act freely in that place, as well as a technological system to make this possible.

The place that differs from one's own existing place can be a real environment or a computer-generated virtual environment, and the latter case, i.e., telexisting in a virtual world, is called virtual reality (VR). When a virtual space is mapped to the real world, it is possible to access the real world through the virtual space, or to work with the virtual world superimposed on the real world. This is called augmented telexistence. In augmented reality, the distance between human and avatar is reduced to zero (Fig. 1).

Fig.1
Fig.1 Telexistence in the real world, telexistence in a virtual world, and telexistence in the real world through the corresponding virtual world

Figure 2 shows another schematic of the relation between VR and telexistence described above. There are two types of virtual reality worlds: virtual worlds created by human imagination and a virtual world that corresponds to the real world.

The former type of VR, which is a virtual world created by human imagination, is used for creation, games, and entertainment. On the other hand, VR that uses a virtual space modeled on the real world is used for education, training, design, and scientific elucidation.

In a narrow sense, telexistence means telexsitence to a real environment. If a virtual space corresponding to the real environment is superimposed on it, it becomes augmented telexistence. In this augmented telexistence, if the distance between the user and the robot is reduced to zero, this is the so-called augmented reality (AR).

Fig.2
Fig. 2 Relation between VR and telexistence

Recently, in addition to AR, the term mixed reality (MR) has been used. In the next section, we would like to clarify the relation between VR, AR, and MR.

Relation between VR, AR, and MR

The relation between VR (Virtual Reality), AR (Augmented Realty), and MR (Mixed Reality) was first defined in the following paper by Milgram, Takemura, Utsumi, and Kishino.

Paul Milgram, Haruo Takemura, Akira Utsumi and Fumio Kishino: Augmented Reality: A class of displays on the reality-virtuality continuum, Proceedings of Telemanipulator and Telepresence Technologies. pp. 2351–34, 1994.

The paper considers a continuum from real to virtual, as shown in Fig. 3, and positions AR and AV as intermediate between Real Environment and Virtual Environment, defining AR and AV together as MR.

Fig.3
Fig. 3 Reality-Virtuality Continuum

Here, as another attempt, we will discuss the relation between VR, AR, and MR not as a continuum, but as sets.

Figure 4 shows the relation between them using sets. There are two worlds, the real world and the virtual world created by the computer, and they overlap each other. The set of the real world (R) is shown as an ellipse surrounded by a red frame, and the set of the virtual world (VR world) is shown as an ellipse surrounded by a blue frame.

Fig.4
Fig. 4 Relation between VR, AR, and MR

The intersection of the set of the real world and the set of the VR world (purple region) is the composite region that is both the real world and the VR world. On the other hand, the red area on the far left is the pure real world, which does not contain any VR world, and the blue area on the far right is the pure virtual world, which does not contain any real world and is created entirely by computer graphics.

In the area shown in purple, the real world and the VR world are mixed, and there are two ways to mix them. One is to add the virtual world to the real world, which is called augmented reality (AR) because it augments and extends the real world with the virtual world. This is the left half of the purple area.

The other is augmented virtual reality (AV), which adds the real world to the VR world, enhancing the virtual reality by adding real scenes and real people. This is the right half of the purple area.

This AR and augmented VR (AV) together are called mixed reality (MR). In the figure, MR = AR + AV, where + means union of sets.

Therefore, MR (the purple region) includes AR. However, it does not include all of VR, since it does not include the pure VR world (blue region). On the contrary, as mentioned earlier, the entire ellipse surrounded by the blue frame in the figure is VR, so VR includes both AR and MR. For this reason, the Virtual Reality Society of Japan (VRSJ) refers to all of these, including MR and AR, collectively as VR.

Since VR has a wide range of meanings, when we want to emphasize that the real world is somehow included in VR, we call it MR, and when we want to emphasize that the real world is the main focus of MR, we call it AR.

Three Elements of VR

The three elements of virtual reality (VR) are: (1) the environment constitutes a natural three-dimensional space for human beings, (2) human beings can act freely in the environment while interacting with it in real time, and (3) the environment and the human being using it are seamless providing the user with a realistic projection of himself/herself as a virtual human. The above environment can be a computer-generated virtual environment or a real remote environment.

These are called "three-dimensional spatiality," "real-time interactivity," and "self-projection," respectively, and an ideal virtual reality system is one that combines all three of these elements (Fig. 5).

Fig. 5
Fig. 5 Three elements of virtual reality.

"Three-dimensional spatiality" refers to the fulfillment of conditions under which computer-generated three-dimensional visual space and three-dimensional auditory space extend around humans.　Three-dimensional movies, such as those we see at IMAX thearter, have this element. In this case, however, trying to see from a different angle, or trying to get behind an object, is not possible. Nor can you touch the object you are looking at or move it to another location. In other words, "real-time interactivity" is lacking. On the other hand, home computer games offer “real-time interaction” with objects, but they do not provide a three-dimensional space in which you are enveloped, just looking at the display in front of you. Furthermore, there is no "self-projection," in which you are seamlessly connected to the computer-generated environment without contradiction in terms of both depth and direction, as if you have entered the environment.

Let me briefly explain this “self-projection” more precisely. In fact, humans know where their hands and feet are and what posture they are in even with their eyes closed. This is due to a sense of proprioception, such as a somatic sense and a sense of equilibrium. In the real space we usually experience, this sense of proprioception coincides with the audiovisual information of the space we observe with our eyes and ears. For example, if we open our eyes and look at the position where we think our hand is with our eyes closed, we will see our hand there. In fact, the state in which there is no contradiction between different human sensory modalities is a characteristic of real space. It is "self-projection" that makes it possible to realize such a state without contradiction even in a computer-generated virtual environment. In other words, where we feel we can see our own hands in proprioception, we can see the avatar's hands we are using so that we feel that the avatar is our new body.

Thus, VR is a technology that aims to configure a system with these three elements and use a computer-generated artificial environment in a state virtually equivalent to that of a human being using an actual environment.

These three elements are also the system requirements of telexistence. In other words, these three elements allow a remote environment with the presence of an alter ego robot to be used without a human being present on the spot, using the avatar robot as his or her new body, in a state virtually equivalent to the existence of a human being on the spot.

These three elements express the requirements of a system, but if we read them from the standpoint of the person using the system, they become the three requirements of self-location, agency, and ownership.

The sense of self-location is the recognition that a person clearly exists in the new place where he or she is now located. In self-location, first-person vision is dominant in location recognition. In VR, the image from the viewpoint of the virtual human is linked to the head movement of the user to present the first-person viewpoint without any discrepancy to the user (life-size three-dimensional space).　

The sense of agency is the feeling that a certain bodily movement and the action performed by that movement are being performed by oneself. In other words, in order to obtain this sense, the "prediction of the outcome of one's own actions" and the "actual outcome of one's own actions" must be consistent. In VR, the virtual human is made to follow the user with a reduced time delay and positional shift to the extent that the user does not feel it (real time interaction).

The sense of ownership is the feeling that one's body is one's own, and Rubber Hand Illusion is an illusory phenomenon related to this feeling. In order to create a sense of body ownership, it is necessary to match the body schema between the user's physical body and the body of the virtual human, the avatar. In VR, the system is designed so that there is no contradiction between visual and proprioception sensory information (self-projection).

The localization of the self, the sense of agency, and the sense of self-ownership are important requirements that must be secured in VR. By guaranteeing these three requirements, we can feel the virtual avatar as our new body, and act on that body as our new body in VR space.

The same is true for telexistence. In telexistence, in addition to these requirements, when you act as if the physical avatar robot is your own body and see yourself in that state, you will experience so-called heautoscopy or out-of-body sensation. This is a phenomenon unique to telexistence that does not exist in VR.