Haptic Primary Colors

| Summary | Haptic sensation as an information medium | Principle of Haptic Primary Colors |

| Integrated haptic transmission module based on the haptic primary color principle |

| How to add haptic primary color components |

 

Summary

Haptic means “of or relating to the sense of active touch, in particular relating to the perception and manipulation of objects using both cutaneous sensation and proprioception.”  From S. Tachi. "Telexistence 2nd Edition," World Scientific, 2015, p.204.

Humans do not perceive the world as it is. Different physical stimuli give rise to the same sensation in humans and are perceived as identical. A typical example of this fact is human color perception. Humans perceive the light of different spectra as having the same color if the light has the same proportion of the different spectral components. This is because the human retina typically contains three types of color receptors called cone cells or cones, each of which responds to a different range of the color spectrum. Humans respond to light stimuli via 3D sensations, which generally can be modeled as a mixture of the three primary colors (red, green, and blue).

This many-to-one correspondence of elements in mapping from physical properties to psychophysical perception is the key to virtual reality. VR produces the same effect as a real object for a human subject by presenting its virtual entities with this many-to-one correspondence. We have proposed the hypothesis that a cutaneous sensation (that is, one relating to or affecting the skin) also has the same many-to-one correspondence from physical properties to psychophysical perception owing to the physiological constraints of humans. We call this haptic primary colors. As is show in Figure A, we define three spaces: physical, physiological, and psychophysical or perceptual.

In physical space, human skin physically contacts an object, and the interaction continues over time. Physical objects have several surface physical properties such as surface roughness, surface friction, thermal characteristics, and surface elasticity. We hypothesize that at each contact point of the skin, the cutaneous phenomena can be broken down into three components: force f(t), vibration v(t), and temperature e(t). Objects with the same f(t), v(t), and e(t) are perceived as the same, even if their physical properties are different. We measure f(t), v(t), and e(t) at each contact point with sensors that are mounted on the avatar robot's hand and transmit these pieces of information to the human user who controls the avatar robot as his surrogate. We reproduce these pieces of information at the user's hand via haptic displays of force, vibration, and temperature, so that the human user has the sensation that he is touching the object as he moves his hand controlling the avatar robot's hand. We can also synthesize virtual cutaneous sensation by displaying the computer­ synthesized f(t), v(t), and e(t) to the human users through the haptic display.

This breakdown into force, vibration, and temperature in physical space is based on the human restriction of sensation in physiological space. Much like the human retina, human skin has limited receptors. In physiological space, cutaneous perception is created through a combination of nerve signals from several types of tactile receptors located below the surface of the skin. If we consider each activated haptic receptor as a sensory base, we should be able to express any given pattern of cutaneous sensation through synthesis by using these bases.

Merkel cells, Ruffini endings, Meissner's corpuscles, and Pacinian corpuscles are activated mainly by pressure, tangential force, low-frequency vibrations, and high-frequency vibrations, respectively. On adding cold receptor s (free nerve endings), warmth receptors (free nerve endings), and pain receptors (free nerve endings) to these four vibro-tactile haptic sensory bases, we have seven sensory bases in the physiological space.

Because all seven receptors are related only to force, vibration, and temperature applied on the skin surface, these three components in the physical space are enough to stimulate each of the seven receptors. Thus, in physical space, we have three haptic "primary colors”: force, vibration, and temperature. Theoretically, by combining these three components we can produce any type of cutaneous sensation without the need for any “real” touching of an object.

 

Figure A: Haptic primary color model.

Different physical stimuli give rise to the same sensation in humans and are perceived as identical.

 

The aforementioned discussion can be summarized as follows: A target object has physical properties, such as the concavo-convex shape, surface texture, friction, thermal conductivity, viscosity, elasticity, inertia, and humidity. However, generally, human cutaneous receptors only sense force (skin deformation), vibration, and temperature change, which occur at the contact to the physical body when the person touches the target object. Therefore, the optimal sensors are the force, vibration, and temperature sensors that directly sense these quantities. These quantities are transmitted and presented to the person through the force display, vibration display, and temperature display. Figure B shows the diagram of the measurement, transmission, and presentation systems based on the principle of haptic primary colors.

 

Figure B: Diagram of the measurement, transmission, and presentation systems based on the principle of haptic primary colors.

 

| Summary | Haptic sensation as an information medium | Principle of Haptic Primary Colors |

| Integrated haptic transmission module based on the haptic primary color principle |

| How to add haptic primary color components |

 

Haptic sensation as an information medium

Current audiovisual media are essentially passive media that do not involve the viewer's physical movements. Currently, the development of passive media that only convey audiovisual information is saturated, and its limits are beginning to be seen. Against this backdrop, new media are being pursued around the world that will revolutionize and advance current audiovisual media. In addition to developing conventional audiovisual media into interactive and active audiovisual media, there is a need to create innovative information media based on "physicality" that involves active physical movement using virtual reality, robots, etc., in a more fundamental way.

Transmission of "haptic" sensation is essential to convey this "physicality. However, although ad-hoc haptic transmission technologies that can be used in specific situations have been developed, there is still no general-purpose haptic transmission technology available. That is, haptic sensations cannot be recorded, transmitted, and replayed in a versatile manner comparable to visual or auditory sensations. Furthermore, there is no technology that actively integrates visual, auditory, and haptic senses to record, transmit, and replay physical experiences.

The most essential difference between technologies for recording, transmitting, and replaying bodily experiences and conventional technologies for transmitting passive audiovisual experiences is the existence of a sense of self-attribution of bodily experiences through haptic information resulting from the interaction between a person and the external environment through body movements. In other words, enabling the transmission of haptic information is the most important key to the realization of the transmission of bodily experience.

In current research on haptic transmission technology, many haptic transmission technologies have been proposed, but they either reproduce special situations or are large and complex devices combining various conventional elements. In current industrial applications of haptic technology, vibration presentation modules using piezoelectric actuators, etc., are widely used in smartphones and entertainment devices. However, the sensory information transmitted by these modules is limited to simple vibrations, and does not transmit the fine haptic sensations that can be obtained from real objects or the full-body kinesthesia that is generated by body movement. Unlike vision and hearing, which are perceived only by specific body parts such as the eyes and ears, haptic sensation is a complex sensation that is perceived by integrating the responses of a wide variety of haptic receptors distributed throughout the body in physical movement, and thus cannot be reproduced by a simple combination of single haptic sensory elements such as vibration and force. Therefore, in order to realize full-scale industrial application of haptic technology, it is essential to have a module that can record and reproduce information equivalent to the sensory information that people feel in a versatile manner, and an essential and versatile presentation method that is not a mere combination, which is equivalent to an imaging module (e.g., CCD) and presentation module (e.g., LCD) in the field of visual sensation. The presentation method is not a simple combination, but an essential and versatile one.

In vision, the reason why there is a versatile module for recording and a presentation module for playback in a general-purpose rather than ad hoc manner is actually that this recording, transmission, and display are based on human visual characteristics.

Human vision detects only a very limited region of electromagnetic waves, 0.40 to 0.75 μm, called light. Natural colors are a mixture of light of various wavelengths. A system that records, transmits, and presents all of this information would not be feasible in practice, even if it were possible in principle. What makes the recording, transmission, and presentation of visual information possible is a method based on the three-primary color principle that utilizes the wavelength-selective characteristics of human cone cells. In other words, even if light with a different spectrum from that of natural colors is used for each cone cell responsible for RGB (red-green-blue), without using a spectral composition exactly the same as that of natural colors, the same firing pattern is excited, or by presenting the human eye with the same RGB light, it takes advantage of the fact that the colors will appear exactly the same to humans.

Therefore, visual information can be easily and reasonably transmitted and reproduced by breaking down natural colors into three primary colors, recording and transmitting the values, and then synthesizing them with the three primary colors. It is also possible to record and reproduce them. It is also possible to take recorded images and process or edit them. It is even possible to create completely new images. Printed matter, television, and all other forms of visual information utilize this very principle (Fig. 1). It can be said that visual information has become a medium. The three-primary color principle is a method based on human sensory characteristics, and for this reason, it has become a general-purpose method that we use today.

Is it possible to do the same with the sense of touch? If so, what principle should be used to measure, record, transmit, and present haptic information to people (Fig. 2)?

The "Haptic Primary Colors" principle is a proposal to solve this problem. The "Haptic Primary Colors" principle is a technological concept that attempts to break down, transmit, and present haptic sensations based on physiological knowledge of touch in the same way that the three primary colors of vision are based on the wavelength characteristics of the cone cells of the eye (Fig. 3). If this becomes possible, it will be possible to measure, record, transmit, reproduce, and present haptic sensations in a general-purpose manner. In conjunction with this, it will become possible to fabricate sensing and presentation modules and elements comparable to CCDs and LCDs, and industrial applications will also advance. In other words, haptic sensation will become an information medium comparable to visual and auditory sensation.

 

Figure 1: Visual sensation a as an information medium based on the principle of three primary colors.

 

Figure 2: How haptic sensation can be an information medium.

 

Figure 3: Haptic Sensation as an information medium based on the principle of haptic primary colors.

 

| Summary | Haptic sensation as an information medium | Principle of Haptic Primary Colors |

| Integrated haptic transmission module based on the haptic primary color principle |

| How to add haptic primary color components |

 

Principle of Haptic Primary Colors

The human senses are divided into "specific sensation" and "somatic sensation. Specific sensation refers to sensations that have a corresponding special sense organ, such as the eye for sight or the ear for hearing. Acceleration is not one of the five senses, but it is classified as a specific sensation in the sense that it has a corresponding sensory organ, the ear (including the vestibule: the semicircular canals and the utricle and the saccule).

On the other hand, somatic sensation means sensation of body (soma), and can be roughly divided into cutaneous sensation, which originates in the skin, and proprioception, which is the sensation of posture and movement originating in the internal muscles and tendons. Proprioception is also called deep sensation.

In a narrower sense, the term "tactile sense" refers to the sensation of touch (contact) and pressure within the cutaneous sensation, which also includes the sensations of warmth, coldness, and pain. This sense of vibration and force corresponds to the sensory organs in the skin, such as Merkel cells, Meissner corpuscles, Pacinian corpuscles, and Ruffini endings. When the entire skin is indented or pulled, the deformation or vibration is transmitted to the sensory organs, causing sensation.

There is also active tactile sensation, which is a total sensation that occurs when the body is moved to touch itself, stimulating not only the skin but also sensory receptors such as muscle spindles in muscles and Golgi receptors in tendons, which are called proprioception. In this way, we can see that the sense of touch is not simply the sensation of whether or not we are touching a single sensory organ, but is a synthesis of a wide range of sensations, including endogenous sensations.

Now, let us consider the process by which a human being recognizes an object, say an iron ball, by tactile perception in the broad sense. First of all, we know the shape by touching it, but even if we do not touch it directly with our fingers but put thick gloves on our fingers and touch the iron ball through them, we can still guess that it is a "ball" based on how the joints of our arms, hands, and fingers move and what shape it has. This clearly shows that the perception of rough shape is not based on the skin, but on proprioceptive senses such as muscle spindles and Golgi receptors. In addition to rough shape, sensations such as hardness, spring-like repulsion, and resistance when moving the arm in water are also derived from the proprioceptive senses.

The cutaneous senses are much finer, recognizing fine surface shape patterns called "textures.” This sensation does not occur when wearing thick gloves, but requires direct touch. This sensation is further enhanced by the active tactile sensation described above. This sense of touch with motion, which combines proprioceptive and cutaneous sensation, is called haptics (haptic perception, haptics). Haptic means “of or relating to the sense of active touch, in particular relating to the perception and manipulation of objects using both cutaneous sensation and proprioception.” From Susumu Tachi. "Telexistence 2nd Edition," World Scientific, 2015, p.204. Although we cannot identify a metal object as iron just by holding it and touching it, we can recognize that it is metal because of the texture and the cutaneous sensation of warmth and cold. Henceforth, this cutaneous sensation will be referred to and discussed as the sense of touch in a narrow sense.

If we have to present an object exactly the same as the actual object on the surface of the human finger in order to convey the sensation of directly touching an object with the surface of the finger, then we cannot treat tactile perception as an information medium. In the case of vision, for example, the color of an object may differ from the actual spectrum, but if the cone cells responsible for each of the RGBs in humans fire identically, they appear to be the same color, and this principle is what makes TV, color photography, and color printing possible. This is what we call the three primary colors. Can we not say the same thing about the sense of touch? In other words, do the primary colors of touch exist?

The fact that there are distinctly different types of sensory organs in the human sense of touch supports the existence of such a haptic color system. These are Merkel cells, Meissner corpuscles, Pacinian corpuscles and Ruffini endings. There are also free nerve endings that respond to warmth and cold pain. Neurophysiological studies have shown that Merkel cells and Ruffini endings detect pressure and shear forces, Meissner corpuscles detect low-frequency vibrations, and Pacinian corpuscles detect high-frequency vibrations. When in contact with an object, Merkel and Ruffini are said to detect displacement, Meissner velocity, and Pacini acceleration (Figure 4).

 

Figure 4: Modified from Å.B. Vallbo and R.S. Johansson: Properties of cutaneous mechanoreceptors in the human hand related touch sensation, Human Neurobiology, vol.3, pp.3-14, 1984.

 

A real object has many physical properties such as unevenness, friction, heat, and elasticity. If it were necessary to reproduce all of these, we would have to bring the actual object, which would make it impossible to present it as an information medium. However, if we consider the situation in which skin sensation occurs, we can see that when we touch an object, force, vibration, and temperature changes are generated on the skin surface, which are transmitted to the sensory organs, causing them to react, resulting in the sense of touch. If this is the case, then no matter what the physical characteristics of the object are, such as its uneven shape, friction, heat, or elasticity, if the sensory organs respond in the same way as when the object is touched, the same tactile sensation will be produced in humans. This is exactly the same as in the case of vision.

There are two main ways to induce the same firing in the sensory organs.

The first method is to seek a base in physiological space. This method uses electrical stimulation to selectively fire the underlying sensory organs themselves. Since the invasive method of stimulating all sensory organs by implanting electrodes is not practical, transcutaneous electrical stimulation from the skin surface is used to selectively stimulate the sensory organs. For this method, it is possible to selectively stimulate only Meissner corpuscles by anodal stimulation, and to stimulate Merkel cells via skin electrodes by cathodal electrical stimulation to convey a sensation similar to pressure perception. However, it is not possible to selectively stimulate Pacinian corpuscles, nor is it possible to selectively stimulate warmth or cold. Conversely, even stimuli that do not normally cause pain may be accompanied by painful sensations. Therefore, the method of direct selective stimulation of the base of physiological space has not yet reached the point where it can provide general-purpose stimulation.

The second method is equivalent to the method that uses the RGB of vision as the basis and selects the basis in physical space. If the "force", "vibration", and "temperature" are presented in the same way as when we touch an object, even when we do not actually touch it, we will feel the same sensations since Merkel cells and Ruffini endings respond to pressure and shear force, Meissner corpuscles to low-frequency vibration, Pacinian corpuscles to high-frequency vibration, and free nerve endings to warmth, coldness, and pain in the same way as when touching a real object. As this sensation changes with the active movement of the human being, it is necessary to faithfully reproduce it by following the human movement. This method is similar to the method used in vision, which does not reproduce the entire spectrum of light, but uses only the spectrum corresponding to RGB as a basis, and reproduces almost all visual information by synthesis based on that basis. In other words, instead of reproducing all the physical characteristics of an object, such as unevenness, friction, heat, and elasticity, the system records and transmits the changes over time in the three physical quantities of "force," "vibration," and "temperature" that are generated on the skin surface by contact with the object, and synthesizes various haptic sensations based on these basal values.

This second method, which uses the three physical quantities of "force," "vibration," and "temperature" as bases, is the basic form of the haptic primary color principle.

 

Figure 5: Three physical bases of Haptic Primary colors.

 

However, this method based on "force," "vibration," and "temperature" has a problem for miniaturization. The force presentation method requires a fixed point (ground) for applying force at a different location from the point where the force is applied, due to the characteristics of the physical quantity called force. This means that when the presentation device itself is fabricated, the device cannot be contained solely in the vicinity of its stimulation site. This inevitably requires a relatively large space, which is a major limitation for miniaturizing the device. In addition, the general method of presenting force on the skin surface can present pressure in the pushing direction, but not in the pulling direction. This makes it difficult to present a sticky sensation (Figure 6).

 

Figure 6: Onomatopoeia representations of haptic feeling plotted on a plane with three axes of the psychological space of haptic sensation. Modified from Tomohiko Hayakawa, Shigeru Matsui and Junji Watanabe: Classification Method of Tactile Textures Using Onomatopoeias, Transaction of the Virtual Reality of Japan, vol.15. no.3. pp.487-490, 2010.

 

On the other hand, it has been shown that force sensation can be produced by stimulating Merkel cells with selective electrical stimulation. In addition, force stimulation using electrical stimulation can produce the sensation of force even in the direction of pulling, making it possible to present a “NECHO-NECHO” sensation, for example. In light of the above, a third method that includes "electricity" as a base in addition to "force," "vibration," and "temperature" is promising. In other words, the first method is added to the second method. Even if "force" is replaced entirely by "electricity," it is sufficient to assume that the involvement of "force" in the four bases is zero, and if "electricity" is set to zero, it becomes the second method.

Electricity does not occur in actual contact with ordinary objects except in special cases such as feeling static electricity, but since it is a physical quantity, there is no problem with it as a basis for physical space. Since this electrical stimulus also produces the sensation of pressure, vibration, and even pain, it has an effect on the five bases in the psychological space depending on how the electrical stimulus is given. In particular, the sensations of hardness, softness, dryness, and wetness are given to broaden the range of sensations that can be presented only by force, vibration, and temperature. Figure 7 shows the tactile primary color schema with electricity added to the bases.

 

Figure 7: Four physical bases of Haptic Primary Colors.

 

| Summary | Haptic sensation as an information medium | Principle of Haptic Primary Colors |

| Integrated haptic transmission module based on the haptic primary color principle |

| How to add haptic primary color components |

 

Integrated haptic transmission module based on the haptic primary color principle

The haptic primary color principle considers the sense of touch as a synthesis of the spatiotemporal relationships among pressure and shear force (perceived by Merkel cells and Ruffini endings, respectively), low frequency vibration (perceived by Meissner corpuscles), high frequency vibration (perceived by Pacinian corpuscles), cold, warm, and pain (perceived by free nerve endings, respectively). Based on this haptic primary color principle, we have developed a module that can acquire and present force, vibration, and temperature in an integrated manner, thereby realizing the transmission of highly realistic haptic sensations by integrating force, vibration, and temperature in a spatiotemporal manner. The new system enables the transmission of tactile sensations with a high degree of realism.

However, there are several technical challenges in realizing a compact haptic sensation transmission module. First, pressure and low-frequency vibration sensations are generally presented by using motors, air pressure, or other actuators to generate physical force on the skin, but this method cannot be miniaturized beyond a certain level. The key to miniaturization is to generate pressure and low-frequency vibration sensations freely without physical stimulation by using selective stimulation of tactile receptors by transcutaneous electrical stimulation to electrically stimulate the nerves that connect to Merkel cells, which perceive pressure, and Meissner corpuscles, which perceive low-frequency vibration sensations. So far, it has been shown that it is possible to selectively stimulate both the pressure and low-frequency vibration senses by choosing the stimulation pattern and polarity of the electrical stimulation. For the stimulation electrode, a thin, high-density electro-tactile display using a flexible printed circuit board can be used, and it has been shown that a pitch of 2 mm is sufficient for the presentation of pressure sensation, even at the finest fingertip, due to human tactile perception characteristics. For the acquisition of pressure sensation, a thin-type pressure sensor that enables multi-point measurement corresponding to the above-mentioned electro-tactile display can be used with a flexible substrate, and a thin-type sensor and display that enables acquisition and presentation of high-density, multi-point pressure sensation can be configured by using these sensors. Figure 8 shows the actual values of the modules researched and developed in JST CREST " Construction and Utilization of Human-harmonized "Tangible" Information Environment" (Principal Investigator: Susumu Tachi) conducted from October 2009 to March 2015. The following table shows the target values of the modules to be developed under the ACCEL project " Embodied Media Technology based on Haptic Primary Colors" (Principal Investigator: Susumu Tachi), which was carried out from December 2014 to November 2019. In the actual module developed (Figure 9), dimensions 27 x 14 x 6 mm; weight 6.3 g; pressure sensation presentation of 30 points at 2 mm pitch; 1 to 1000 Hz HiFi vibration presentation; and stepless control of temperature have already been achieved as of 2017.

 

Figure 8: The actual values of the modules researched and developed in JST CREST (right side of the table), and the target values of the modules to be developed under the ACCEL project (left side of the table).

 

Figure 9: Three-haptic-color module developed in 2017 under ACCEL project.

 

Many vibration actuators have been developed for the presentation of high-frequency vibration sensation, but most of them are designed with resonance frequencies set in the frequency range most easily perceived by humans, around 200 Hz. While this is effective for presenting strong stimuli with low power consumption, on the other hand, it is not suitable for the presentation of delicate tactile sensations, including the diverse vibration frequencies that occur in natural tactile movements. Therefore, in order to realize a vibration actuator that is optimal for the presentation of vibration sensations, it is necessary to use a small vibration actuator that can present vibration sensations with flat characteristics over a wide vibration frequency range of 1Hz to 1000Hz. It is known from physiological knowledge of tactile sensation that Pacinian corpuscles, which mainly perceive high-frequency vibration, are distributed only about one per 1-2 cm2, and therefore, it is sufficient to design a vibration actuator with a size of 2 cm2.

For the presentation of cold/warm sensation, it is known that in the haptic primary color principle, it is necessary to reproduce the heat transfer between the human skin and the object in contact, rather than reproducing absolute temperature. In order to achieve this, it is necessary to have bidirectionality in the acquisition and presentation of cold/warm sensation, i.e., the sensor that acquires the cold/warm sensation must function as a display that reproduces body temperature equivalent to that of a human being. A high-speed cold/warm sensation presentation method that enables temperature changes of about 1 Hz by arranging and controlling multiple temperature presentation elements in a matrix, utilizing the cold/warm perception characteristics of humans, makes it possible to transmit warm and cold sensations that change spatiotemporally and temporally.

The haptic sensation that a person senses sensitively is defined as the psychological space of haptic sensation, the physiological elements such as tactile receptors that generate the nerve pulses that constitute the haptic sensation are defined as the physiological space of haptic sensation, and the physical stimuli that activate the tactile receptors are further defined as the physical space of haptic sensation. According to the definition, the tactile transmission module developed can be considered a hybrid configuration of stimulation in the physiological space of Merkel cells and Meissner corpuscles by electrical stimulation, and stimulation in the physical space of vibration and temperature.

Therefore, an algorithm that creates arbitrary haptic information by editing and processing information recorded from the real world and converts it into appropriate haptic control signals along with body movements during the experience, and an algorithm that converts haptic information recorded from the real world into new haptic information corresponding to different body movements, will be constructed as a generally available "haptic primary color encoder" and provided together with haptic transmission modules.

In addition, when presenting actual haptic sensations, for example, when trying to present a "soft" haptic sensation to a hand, there is no haptic signal that can present "softness" by itself in the haptic presentation module, and the haptic signal to be presented in each haptic element of the haptic primary colors must be changed spatio-temporally. Furthermore, the user's body motion does not always coincide perfectly between the scene in which the physical experience is recorded and the scene in which it is experienced, and even if the same haptic control signal is presented under different motion conditions, it will be perceived as a different haptic sensation. Therefore, it is necessary to perform appropriate conversion according to the differences in physical motion, rather than simply presenting the recorded haptic information as it is.

 

| Summary | Haptic sensation as an information medium | Principle of Haptic Primary Colors |

| Integrated haptic transmission module based on the haptic primary color principle |

| How to add haptic primary color components |

 

How to add haptic primary color components

When a person actually touches a real-world object with his or her finger, the action points of the three stimuli (force, vibration, and temperature) applied to the skin surface of the finger are identical. However, it is generally difficult to match the action points of force, vibration, temperature, and electricity in a human display device.

On the other hand, when we examine the human senses, it is possible to produce the same sensation even if the points of action are different. It is also possible to generate sensations elsewhere than at the point of action. A good example of the latter is phantom sensation. This is a phenomenon of cutaneous sensation similar to the binaural localization of sound identified by Georg von Békésy. Under certain conditions, by applying stimuli to two locations on the skin, the stimulus image can be given to a different location than the original stimulus points between the two stimulus locations. Moreover, the position of the image can be controlled by the difference in stimulus intensity between the two stimulus points. The number of stimulus points can be increased to three to control the position of the perceived image in the plane. This is known to occur for vibration stimulation as well as for electocutaneous stimulation (Figure 10)1,2).

 

Figure 10: Phantom image created by the electrocutaneous stimuli. The phantom image can be controlled by the relative strength among the stimuli.

 

The existence of this phantom sensation phenomenon means that vibration stimuli can be felt at a non-stimulated point. For example, this occurred when pressure and vibration information measured by the hand of a telexistence robot called TELESAR V, which was constructed by the authors, was presented by a display device installed in a glove. In the display device, pressure and shear forces were applied to the finger belly, and when vibration was presented from both lateral sides of the finger, the finger belly, which felt the pressure sensation, also felt the vibration stimulus (Figure 11). It is thought that the phantom sensation caused the vibration stimulus to be felt in the center. Furthermore, it was seen that there was also an effect of the stimulation being more easily felt at the location where the pressure sensation was given.

 

Figure 11: Force and vibration display device (left) and the glove with three display devices.

 

Even if the force presentation part is replaced by electrical stimulation, vibration stimulation can be presented to the electrical stimulation part by phantom sensation in the same way. The phantom sensation image is more clearly localized by the electric stimulus. In other words, the phantom sensation described above localizes the vibration stimulus to the location of force and electrical stimulation.

The same is true for temperature presentation. It is known that when a temperature stimulus is applied from the side, it does not localize to the fingertip on only one side, but when a vibration stimulus is applied to the fingertip, it localizes to that area. In this case, the phantom sensation, which is localized at the center when applied from the left or right side, is also present, and is clarified by vibration, force, and electrical stimulation. In the case of temperature, the characteristics of the element and the nature of heat cause a delay in presentation. To compensate for this delay, it may be necessary to incorporate a non-contact temperature sensor on the robot hand side in addition to a contact-type temperature sensor.

These allow for the following:

· To localize the sensation of the stimulus by phantom sensation by placing the force presentation at the point of action of the stimulus, and by placing the stimulator, either vibration, temperature, or electrical stimulus, or any combination of these, at a physically distant location. The localization image by phantom sensation is also clarified by the presentation of the force.

· To localize the electric stimulus presentation to the point of action of the stimulus, and to localize either vibration or temperature, or both, to a physically distant location, to localize to the point of force action by phantom sensation, and to match the presentation site of the electric, vibration, and temperature tactile primary colors by making the localized image by phantom sensation clearer by the presentation of the electric stimulus.

· To localize the vibration presentation to the point of stimulus action and the temperature to a physically distant location, and to match the vibration and temperature tactile primary color presentation sites by localizing them to the point of vibration action by phantom sensation and by sharpening the localized image by phantom sensation with the vibration presentation.

Based on the above, a large number of elements can be placed horizontally to provide coincident vibration, electrical, and temperature stimulation at the points where they are placed. In other words, it is possible to increase the number of matched stimulation points more than the number of elements. Figure 12 shows an example of a large number of elements arranged like a color mask for television. In the figure, E is an electrode for electrical stimulation, V is a vibrator such as a piezoelectric element, and H/C is a heater and Peltier element for warm and cold stimulation, respectively. The electrodes other than those used for electrical stimulation can be used as grounding (earth). The element arrangement structure can be made of a flexible material such as silicone rubber or a flexible sheet material to easily conform to the shape of the body, and is suitable for stimulating large surfaces such as the palms, arms, back, and abdomen.

In the example of the horizontal arrangement pattern in Fig. 12, the unit of repetition pattern is the electrode E next to the vibrator V, and the heater H, and the Pertier element C. However, this example is not limited to this case, and the composition unit of the repetition pattern can be changed as needed, for example, according to the sensory resolution of the body.

Electrical stimulation has better sensory resolution than cold/warm or vibration stimulation. Therefore, by arranging the electrodes E densely and the vibrator V, heater H, and Peltier element C sparsely, it is possible to output high resolution for tactile sensations that can be discriminated more finely (such as electricity and force sensations), and for tactile sensations that can be discriminated relatively coarsely (such as vibration and temperature sensations), it is possible to reduce the number of elements and reduce cost.

For details, please refer to Reference 3.

 

Figure 12: An Example of the element arrangement structure.

 

1) Kazuo Tanie, Susumu Tachi, Kiyoshi Komoriya and Minoru Abe: Basic Study on Discriminability of Mental Location of Electrocutaneous Phantom Sensation, Transactions of the Society of Instrument and Control Engineers, Vol.15, No.4, pp.505-512 (1979.8) [in Japanese] [PDF]

2) Susumu Tachi, Kazuo Tanie, Kiyoshi Komoriya and Minoru Abe: Electrocutaneous Communication in a Guide Dog Robot (MELDOG), IEEE Transactions on Biomedical Engineering, Vol.BME-32, No.7, pp.461-469 (1985) [PDF]

3) Susumu Tachi, Masashi Nakatani, Katsunari Sato, Kouta Minamizawa and Hiroyuki Kajimoto, Tactile Information Conversion Device, Tactile Information Conversion Method, Tactile Information Conversion Program, and Element Arrangement Structure, United States Patents US 10,621,837 B2, Apr. 14, 2020, Filed Oct. 5, 2018. [PDF]