The Airborne Ultrasound tactile Display (AUTD) produces tactile sensation remotely on a human skin surface. The AUTD does not require users to wear any devices on them.
Ultrasound transducers arrayed on the AUTD surface generate a ultrasound focus at an arbitrary position in the air. At the focal point with high energy density, static pressure called “acoustic radiation pressure” is produced, and it can push the skin surface. The focal position can be electrically controlled based on the phase array technique. In addition to creating a focus, you can produce a more general complex pressure pattern by solving an inverse problem in real-time.
The current limitations of the technology are that the maximum pressure is about 50 mN per square centimeter and that the spatial resolution is comparable to the sound wave length 8.5 mm for 40 kHz ultrasound. But under these constraints, radiation pressure pattern are controlled freely by a computer, which create various tactile sensations on the skin.
Haptics using such non-contact haptic feedback is called “midair haptics.” The initial studies of ultrasonic midair haptics were conducted by Hiroyuki Shinoda’s group from 2008 in the University of Tokyo, and the research group demonstrated the basic technologies and haptic effects in the first half of the 2010s. Currently, the field is growing and many universities and companies in the world are moving forward with development.
World-first demonstrations by Univ. of Tokyo
Airborne Ultrasound Tactile Display (AUTD) (youtube video SIGGRAPH 2008 E-tech)
The world first study of midair haptics (haptic feedback using ultrasound).
- Takayuki Iwamoto, Mari Tatezono, and Hiroyuki Shinoda: Non-contact Method for Producing Tactile Sensation Using Airborne Ultrasound, Haptics: Perception, Devices and Scenarios: 6th International Conference, Eurohaptics 2008 Proceedings (Lecture Notes in Computer Science), pp.504-513, 2008.
- Takayuki Hoshi, Masafumi Takahashi, Takayuki Iwamoto, and Hiroyuki Shinoda: Noncontact Tactile Display Based on Radiation Pressure of Airborne Ultrasound, IEEE Trans. on Haptics, Vol. 3, No. 3, pp.155-165, 2010.
Touchable Holography (youtube video SIGGRAPH 2009 E-tech)
The first demo of midair haptic feedback synchronized with visual floating images.
- Takayuki Hoshi, Masafumi Takahashi, Kei Nakatsuma, and Hiroyuki Shinoda: Touchable Holography, Full Conference DVD-ROM, Disc1, ACM SIGGRAPH 2009: Emerging Technologies, New Orleans, 3-7 Aug. 2009.
Tactile Projector (youtube video IEEE WHC 2013)
Projecting visual image on the skin with midair haptic stimulation. A large aperture AUTD synchronizing multiple AUTD-units enabled this system.
- Keisuke Hasegawa, Hiroyuki Shinoda: Aerial Display of Vibrotactile Sensation with High Spatial-Temporal Resolution using Large-Aperture Airborne Ultrasound Phased Array, Proc. IEEE World Haptics Conference 2013, Oral, pp.31-36, Daejeon, Korea, April 14-18, 2013.
*Finalist of Best paper award, IEEE World Haptics Cconference (WHC) 2013
*Best Student Paper Award, IEEE World Haptics Conference (WHC) 2013
- Keisuke Hasegawa, Hiroyuki Shinoda, “Aerial Vibrotactile Display Based on Multiunit Ultrasound Phased Array,” IEEE Transactions on Haptics, vol. 11, no. 3, pp. 367-377, 1 July-Sept. 2018.
Haptomime, Mid-Air Haptic Touch Panel (youtube video)
Floating touch-panel with midair haptic feedback.
- Yasuaki Monnai, Keisuke Hasegawa, Masahiro Fujiwara, Seki Inoue, and Hiroyuki Shinoda, “HaptoMime: Mid-Air Haptic Interactions with a Floating Virtual Screen,” Proc. 27th ACM User Interface Software and Technology Symposium (UIST2014), pp.663-667, Hawaii, USA Oct. 5-8, 2014. (DOI: 10.1145/2642918.2647407)
*ACM UIST 2014 People’s Choice Best Demo Award
*Special prize by SIGGRAPH, Industry Special prize, Innovative Technologies 2014
— Computational design of 3D acoustic energy-density field.
The world first demo of static 3D haptic images. They could touch and perceive a static 3D shape haptically by their fingers and hands.
- Seki Inoue, Yasutoshi Makino, Hiroyuki Shinoda: Active Touch Perception Produced by Airborne Ultrasonic Haptic Hologram, Proc. 2015 IEEE World Haptics Conference (WHC), pp.362-367, Northwestern University, Evanston, Il, USA, June 22–26, 2015.
*Best Demo Award Winner, IEEE WHC 2015
— Advanced scheme of acoustic hologram and application to levitation. Boundary hologram provides acoustic field that produces proper restoring force to any minute displacement and rotation of the object.
The algorithm enabled the world first ultrasonic levitation of a macroscopic non-spherical object significantly larger than the wavelength. It is applicable to arbitrary-shape objects considering the diffraction and reflection by the object.
- Seki Inoue, Shinichi Mogami, Tomohiro Ichiyama, Akihito Noda, Yasutoshi Makino, and Hiroyuki Shinoda, “Acoustical boundary hologram for macroscopic rigid-body levitation,” Journal of the Acoustical Society of America 1. 145, pp.328–337, 2019.
Symmetric mutual telexistence by 3D haptic & optical field reconstruction. A pair of people can touch the realistic 3D image of each other with haptic sensation.
- Yasutoshi Makino, Yoshikazu Furuyama, Seki Inoue, Hiroyuki Shinoda: HaptoClone (Haptic-Optical Clone) for Mutual Tele-Environment by Real-time 3D Image Transfer with Midair Force Feedback, Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pp. 1980-1990, San Jose Convention Center, San Jose, CA, USA, May 7–12, 2016.
Recent results by Univ. of Tokyo
Lateral Modulation: A Method to Intensify Haptic Sensation by Ultrasound
Reciprocating motion of a focus along the skin surface (Lateral modulation, LM) can generate stronger sensation than Amplitude Modulation (AM) where the radiation pressure changes vibrationally at a spatially fixed point. The threshold of the radiation force of LM is 10 dB less than that of AM in the displayed maximum force, and LM can create haptic sensation on a hairy skin as well as a glabrous skin.
- Ryoko Takahashi, Keisuke Hasegawa, and Hiroyuki Shinoda, “Lateral Modulation of Midair Ultrasound Focus for Intensified Vibrotactile Stimuli.” International Conference on Human Haptic Sensing and Touch Enabled Computer Applications (Euro Haptics), pp 276-288 ,Pisa, Italy, June 13-16, 2018.
- Ryoko Takahashi, Keisuke Hasegawa, and Hiroyuki Shinoda: “Tactile Stimulation by Repetitive Lateral Movement of Midair Ultrasound Focus,” IEEE Transactions on Haptics, (Open Access) Early access, Oct. 2019.
Reducing Unnecessary Sound
Quick phase change (as seen in LM) causes unexpected amplitude change. The amplitude change produces unexpected tactile perception and sound noises. The following shows a practical effective method to reduce it. (The sound generation and amplitude change by phase shift was first pointed out by Hoshi in 2016.)
- Shun Suzuki, Masahiro Fujiwara, Yasutoshi Makino, and Hiroyuki Shinoda, “Reducing Amplitude Fluctuation by Gradual Phase Shift in Midair Ultrasound Haptics,” IEEE transactions on haptics (2020). (Open Access) Early access
3D Object Manipulation by Fingers
We are clarifying what information is essential to make the user feel the substance of a virtual object. Connected AUTDs surrounding fingers can generate a 5 mm diameter force spot, and create fine pressure distribution on the fingers. The distribution is controlled in real-time reflecting the contact state between the fingers and the virtual object. We have examined experimentally if the fingers can handle an object blindly.
- Atsushi Matsubayashi, Yasutoshi Makino, Hiroyuki Shinoda, “Direct Finger Manipulation of 3D Object Image with Ultrasound Haptic Feedback,” Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, Scotland UK, May 04 – 09, 2019. (Open Access)
- Atsushi Matsubayashi, Hiroki Oikawa, Saya Mizutani, Yasutoshi Makino and Hiroyuki Shinoda, “Display of Haptic Shape Using Ultrasound Pressure Distribution Forming Cross-Sectional Shape,” Proc. 2019 IEEE World Haptics Conference, pp.419-424, Tokyo, Japan, July 9-12, 2019.
In human vision, smooth pursuit eye movement is the basic ability to visually follow a moving object by keeping it at the sight center. In this study, we validated that a human hand has a similar ability to track a midair haptic stimulus, i.e., a human palm exposed to a point vibration by AUTD can follow the continuous movement of the stimulation point. The ability of motion tracking by hand can be applied to haptic guidance.
- Azuma Yoshimoto, Keisuke Hasegawa, Yasutoshi Makino, and Hiroyuki Shinoda: “Midair Haptic Pursuit,” IEEE Transactions on Haptics, Vol. 12, No.4, pp. 652-657, Oct.-Dec. 2019. (Open Access)
There is another strategy where the user find and follow the designated path actively. Consider a situation the path, from start to finish (the ‘goal’), is presented by scanning ultrasound focal points along the path. The scanned focal point produces vibrotactile stimuli on the hand, and users perceive the local path-direction easily. Then user can follow the path without strong concentration.
— Guidance along a palm
- Shun Suzuki, Masahiro Fujiwara, Yasutoshi Makino, and Hiroyuki Shinoda, “Midair Hand Guidance by an Ultrasound Virtual Handrail,” Proc. 2019 IEEE World Haptics Conference, pp.271-276, Tokyo, Japan, July 9-12, 2019.
— Guidance perpendicular to a palm
- Shun Suzuki, Keisuke Hasegawa, Yasutoshi Makino and Hiroyuki Shinoda, “Haptic Tracing of Midair Linear Trajectories Presented by Ultrasound Bessel Beams,” International Conference on Human Haptic Sensing and Touch Enabled Computer Applications(Eurohaptics), pp 209-220, Pisa, Italy, June 13-16, 2018.