Development and application
Augmented reality (AR) is the seamless merging of virtual objects with the real world. For more than 10 years, the HIT Lab NZ has been developing and researching AR technology. We have developed solutions for tracking, visualisation, and interaction in a wide variety of fields, including telecommunications, entertainment, and education.
Development tools for tracking and visualisation
Tracking and visualisation is a fundamental component of AR technology, and the HIT Lab NZ is dedicated to putting these tools in the hands of designers.
- ARToolkit: This open-source AR library is renowned in the field of AR research. It is the world’s first and most widely-used library of its kind.
- OPIRA: This is a natural feature-tracking and AR visualisation library. The library is robust under perspective distortion.
- Outdoor AR: To enable designers to easily create simple outdoor AR applications, the Lab has also been developing the Outdoor AR software framework for mobile and wearable devices.
Interacting using augmented reality
The Lab also investigates natural interaction between users and AR content. We make use of real-world metaphors and knowledge to help people interact with digital content. We started by using physical props as tangible user interfaces for AR applications, and over the years our research has expanded into investigating gestures, speech recognition, and spatial audio.
Applying augmented reality
We are currently focusing on using AR interfaces to enhance telecommunications. We are actively investigating how augmented visual communication cues (such as a mouse pointer) can be integrated with video conferencing. To further enhance remote collaboration experiences, we are exploring the possibilities of scaling up AR-enhanced social interaction to other platforms.
We are also working on industrial applications in geospatial survey and heavy vehicle operation. Our solutions will help surveyors and forklift operators work more efficiently and safely.
Our success with AR technology is evidenced not only by our partnerships within these areas, but also through a number of spin-off companies through the entrepreneurship of our graduates. For example, Quiver (formerly colAR) is a 3D colouring application for iOS and Android that uses AR technology to transform traditional colouring-book pages into interactive 3D characters. Students from the HIT Lab NZ started QuiverVision to commercialise this technology.
Academic staff member Gun Lee talks about working in AR
Gun Lee researches mobile applications and wearable technology. He spent five years at the HIT Lab, first as a postdoctoral fellow and then as a research scientist, and in late 2016 he moved to the University of South Australia in Adelaide.
What is augmented reality (AR)?
Augmented reality (AR) is about interfaces. These interfaces extend and add on to the experience of the real world. In AR, we take data — digital information — and put it in a real-world context.
What is an example of one of your projects?
One of my first projects at the HIT Lab NZ was called CityViewAR. As smartphones were becoming more popular among the general public, I created an app that allowed users to see Christchurch as it was before the 2010 and 2011 earthquakes. We tested it at the Re:START mall, and we found that with the app, people were more likely to actively explore the content and walk around the area than when they only had map-based information.
What are some applications of AR?
As in the CityViewAR app, AR can provide a virtual experience, but it can also increase the efficiency of many tasks. By removing barriers to the digital space, AR technology gives users greater access to information.
For example, we worked on developing AR solutions for geospatial survey and construction in collaboration with Trimble. To help survey people to navigate on the construction site without having to keep holding up a bulky industrial-grade GPS devices, we made a wearable AR survey guide that would allow people to move around the site more easily.
Another example is heavy vehicle operation. Currently, earth-moving vehicles might have a navigation device in the dashboard that tells operators how deep to dig and how high to pile dirt, but projecting this information on the scene using AR would be safer and more efficient. Likewise, with forklifts, drivers often lean out of their vehicles to see high platforms. We are working with Crown Lift Trucks to design AR-guided operation.
What are the challenges associated with this research?
The first challenge is that the field of AR is fast-paced. With such leading-edge technology, you have to keep up. You have to learn quickly and be constantly aware of new developments in the field.
Another challenge is that AR is multi-disciplinary. Not only are technical skills important in building AR interfaces, but also design and psychology. To be successful, you need to have a breadth of knowledge. Overall, you need to have a willingness to learn.
What advice would you give prospective students of the HIT Lab?
If you want to work in AR, open up your imagination. Don’t try to follow the crowd.
Some people are unsure whether they have enough technical skills, but you don’t need to be an expert in programming to work in AR. In the early days of AR, being able to write code was critical. Now, however, there are many tools that enable you to build AR interfaces, and it is your idea that is most important. Nothing is out of reach.
What does Snapchat have in common with Pokemon Go? Besides topping global charts, both apps use augmented reality (AR) to engage users in social experiences. For instance, Snapchat lets users annotate videos and send them to their friends, and the recently-released Pokemon Go is giving people a reason to explore their own backyards.
Alaeddin Nassani is a PhD student at the HIT Lab NZ who is researching this type of AR. He studies the use of wearable technology to share social experiences. “Maybe this application is not purely useful, but it is part of the field of AR that is currently less explored. There is potential for me to contribute.”
He predicts that the better AR gets, the less we will notice it. “When AR becomes mainstream, people won’t call it AR,” he says. “It will be part of normal life. You won’t notice the difference between actual reality and augmented reality. You will just see the world and share your experiences.”
For his thesis, Alaeddin experiments with wearable AR technology, like Google Glass. For the first stage of his research, he experimented with using head-mounted displays (HMDs) to share panoramic videos of the great outdoors . “You could take a nice hike in the mountains and let your friends and family see it through your eyes,” he says. “A static image is not a conversation, so users can add annotations and comments to the panorama.”
Now, he is using wearable technology to allow people to place and interact with virtual tags . The tags become annotations, augmenting the user’s real-world surroundings. He uses a Google Glass headset to display visual information to the user, and he uses Tango, a hi-definition depth sensor worn on the chest, to track the user’s motion. Just by speaking, the user can place the tags in the environment.
During his PhD programme, Alaeddin has as at times struggled with knowing whether he is taking his work in the right direction. “I always ask myself, ‘Is this topic worthy? Am I asking the right questions?’ Doing the PhD is so unlike other degrees, where you just go to class, study, take exams, and pass. In the PhD programme, you have to develop your own ideas. You can work with others, but eventually you have to take ownership of your own work.”
Alaeddin came to the HIT Lab NZ in 2013, driven by a fascination with augmented reality and its potential. At the time, the lab was under the leadership of Prof. Mark Billinghurst, who has published extensive research on worker-helper collaboration using AR.
At the HIT Lab NZ, Alaeddin is unique in that he is a part-time student. During the day, he works as a programmer, and at night, he comes to the lab. “Although I wish I had more time for both my job and the PhD programme, I appreciate the flexibility,” he says.
He decided to be part-time in order to balance the commercial and academic approaches to the field of AR. “I have continued exposure to the industry while growing my knowledge through research,” he says.
Alaeddin is inspired by his thesis supervisors at the HIT Lab NZ, Mark and Gun Lee. “They have a lot of students, but they still make time to help me refine my ideas.” Alaeddin also enjoys the feeling that the lab is a family. “We care for each other and give each other constructive feedback.”
After he completes his PhD, he plans to continue in the field of AR. He wants to do research and development, either commercially or in academia. “I want to make significant contributions to this field,” he says.
 Reichherzer, C., Nassani, A., & Billinghurst, M. (2014, September). [Poster] Social panoramas using wearable computers. In Mixed and Augmented Reality (ISMAR), 2014 IEEE International Symposium (pp. 303-304). IEEE.
 Nassani, A., Bai, H., Lee, G., & Billinghurst, M. (2015, November). Tag it!: AR annotation using wearable sensors. In SIGGRAPH Asia 2015 Mobile Graphics and Interactive Applications (p. 12). ACM.