Adding tooltips to a 3D model in Augmented Reality can enhance the learning experience. Tooltip functionality is part of Microsoft’s Mixed Reality Toolkit and can be implemented using Unreal Engine.
These development notes are from AnatoMe, an experimental project to teach students about human anatomy through an interactive hologram. Further details can be found on the AnatoMe development page or other project related pages below:
- AnatoMe development
- Building a UI for a HoloLens 2 app
- Using the Variant Manager to control an AR scene
- Adding tooltips in Augmented Reality (you are here!)
Creating annotations
Tooltips is an experimental feature of the UX Tools plugin, containing a set of UI controls that are developed and optimised for use with the HoloLens 2. Tooltips are panels of text in 3D space that rotate to stay in view as the camera moves and are anchored to a particular part of a 3D model. The process of creating Tooltips is to build a widget for the text, add anchors to the target model where you want to annotate, and then add the Tooltip to your level and position it in 3D Space.
To test these components worked as intended, I added an anchor to the top and bottom of a sphere and then created two Tooltips to point at each anchor. As I moved around the scene, the Tooltips rotated so the text was always readable. I then added eight placeholder Tooltips to the arm model, with the next step to only display the relevant tooltips for the currently selected layer.
Configuring tooltip views
This is where the earlier decision in the project to use the Variant Manager paid off. Rather than use a complicated scripting method to show and hide tooltips based on the current layer, I was able to configure the visibility of the Tooltips with full control directly within the Variant Manager. This system is great as it’s visual and easy for someone else coming into the project to view and understand how it works.
A second Variant Set called vs_tooltips was set up with tooltip visibility adjusted dependant on what layer was active. The last step was to add the relevant tooltips variant as a dependency on the main layer variant so tooltip switching was conducted automatically based on the active layer.
Testing in the Unreal Engine editor worked as intended. For each menu button press, the arm layer switched, activated its dependency variant, and changed the visibility of the tooltips to only show the ones related to the current layer.
Summary
The UI, tooltips, and variant manager configurations have been set up. The proof-of-concept project has been tested with the anatomy lecturers with feedback implemented and a new arm model has been acquired and incorporated into the project. Review the AnatoMe project page for a full overview of the project or view the video below.
Additional development notes for this project can be found via the links below:
- AnatoMe development
- Building a UI for a HoloLens 2 app
- Using the Variant Manager to control an AR scene
- Adding tooltips in Augmented Reality (you are here!)