Translated by

Microsoft Hololens

Product
Developers: Microsoft
Last Release Date: February, 2019
Branches: Pharmaceutics, medicine, health care

Content

Microsoft HoloLens - points of virtual reality. They represent the hoop which is put on the head with the tinted lenses located before eyes with wavy prismatic structure which refract and send to eyes of the user of the image from the microdisplays located on each side.

Microsoft HoloLens - points of the mixed reality. Use the 32-bit Windows Holographic operating system (the version of Windows 10). The device is independent and does not require connection to the PC, the smartphone or video game console.

In addition to the central and graphic processors points have the holographic processor (engl. holographic processing unit) developed by Microsoft especially for HoloLens.

2019

Beginning of sales of HoloLens 2. The price — $3500

On November 7, 2019 Microsoft announced the beginning of sales of the headset of the mixed reality of HoloLens 2. Its cost was the same which the American corporation called during the announcement of a product — $3500 (more than 223 thousand rubles at the rate of for November 7).

The first residents of the USA, France, Germany, Ireland, New Zealand, Australia and Britain will be able to purchase HoloLens 2. Terms of emergence of a product in other countries, including in Russia, Microsoft are not called.

Microsoft announced the beginning of sales of the headset of the mixed reality of HoloLens 2
Microsoft announced the beginning of sales of the headset of the mixed reality of HoloLens 2

The producer addresses HoloLens 2 to the corporate sector, mainly the companies which are engaged in production or repair work. For the mezzanine monthly board business can purchase the software of Microsoft Remote Assist intended for troubleshooting in real time.

Journalists of The Verge who managed to test HoloLens 2, noted some differences of the device from the predecessor. First, developers increased a viewing angle from 34 to 52 degrees. Secondly, abilities to manage were significantly expanded by gestures: if in the first version they were limited to clicking virtual objects, then it is possible to scale and move objects to HoloLens 2 under review. Besides, in the new headset it is possible to open a pop-up menu by clicking the holographic button a wrist.

The director of communications of Microsoft Greg Sullivan says that the company retains support of HoloLens of the first, but some developers can begin to create the applications requiring the management of gestures implemented only in HoloLens 2.

According to Sullivan, among those who left requests for acquisition of HoloLens 2 there were as new clients, and those who decided to purchase the device instead of HoloLens of first generation.[1]

Release of the emulator for HoloLens 2

In the middle of April, 2019 Microsoft released the emulator for HoloLens 2 which will allow to test applications on the computer without use of the device of the mixed reality. HoloLens Emulator is very useful product, considering the high cost of the gadget — $3500. Read more here.

Announcement of HoloLens 2

On February 24, 2019 the presentation of a helmet of the mixed reality of the second generation of Microsoft HoloLens took place. The device is addressed to exclusively corporate clients and will cost expensive.

In comparison with the first version the new headset received the doubled field sight when preserving holographic density in 47 pixels by overview degree. The company explains that improvement is equivalent to transition from the TV with the permission 720p on 2K for each eye. 

HoloLens 2 allows to monitor the movements of hands without any additional controllers. During demonstration on the air one of the staff of Microsoft started in 3D - space different applications, including a virtual piano. 

Microsoft provided a helmet of the mixed reality of HoloLens 2. Price
Microsoft provided a helmet of the mixed reality of HoloLens 2. Price

HoloLens 2 contains a new display system which allows to show devices the increased performance at low power consumption.  Using new time - the flying sensor of depth (time-of-flight depth sensor) in a combination to the built-in artificial intelligence and semantic understanding, the wearable device allows to interact with holograms just as it happens in the real world to physical objects.

The device is supplied with sensors of tracking eyes that does interaction with holograms even more natural. Also a system of face recognition Windows Hello is supported. Developers claim that a new helmet "easier and more convenient" in comparison with the previous model.

Microsoft notes that HoloLens 2 is focused on use in the companies which will be able to create the programs and app stores for the gadget. Besides, a helmet will support installation of third-party web browsers, and Mozilla already began to create such product.

After the announcement of HoloLens 2 the Microsoft company began to take preorders on purchase of a helmet of the mixed reality. The price — $3500.[2]

2018

As the AR headset of Microsoft helps surgeons on transactions

By the end of November, 2018 surgeons began to use the headset of the mixed reality reasonably to plan reconstruction lower extremity operations – the new HoloLens AR headset of Microsoft company helps to understand what fabrics and vessels can be used in recovery transactions with the autotransplantation help.

To patients appoint standard KT-researches with contrast which provides visualization of passable vessels and allows to select the most viable fabrics. Then using 3D - reconstruction the extremity and all its vessels is modelled, and data are transferred to HoloLens. The surgeons using the headset see the projection of blood vessels, bones and muscles imposed on the patient's leg and also tags of radiologists thanks to what can precisely predict the volume of the forthcoming transaction.

The AR headset of Microsoft helps surgeons on transactions
The AR headset of Microsoft helps surgeons on transactions

Traditionally for assessment of arrangement of blood vessels before transaction were used doppler-UZI, however this technique concedes to HoloLens in reliability of data retrieveds. Besides, application of HoloLens allows to reduce transaction time, so, and anesthesia time that matters not only for work of department, but also for patients – is known that reduction of time of surgery and anesthesia improves an outcome and reduces risk of complications.

Having proved that the HoloLens system can effectively be applied in work of some departments, Microsoft developers began to consider other methods of the application of the AR headset - in particular, it is already used in operations on removal of a colorectal cancer.

Nevertheless there is a number of problems which researchers hope to solve before the HoloLens system begins to be used more widely. Developers try to reduce time of processing of contrast images which takes about an hour now, to reach higher resolution of images and to improve compatibility between HoloLens and the software of hospitals.[3]

Application in build process of the spaceship

On October 23, 2018 it became known that engineers of Lockheed Martin who by request of NASA assemble the Orion spaceship use Microsoft Hololens glasses in work. The headset helps to visualize instructions and the end result of assembly, saving from need to study thousands of pages of the paper guides. Orion is developed for delivery of astronauts and loads to the ISS, and further for commission of the piloted flights to the Moon and to Mars.

Thanks to Hololens engineers can see the holograms simulated in the special program from Scope AR which are imposed on elements on which they work. Each part of model is supplied with the comment on how it should be set, and information on that, properly to twist bolts, is output near the corresponding opening. Also for each engineer only those parts which correspond to its specialization are highlighted by a certain color.

Before Microsoft and NASA started the Sidekick joint project within which astronauts of the ISS used Hololens headsets under repair, when carrying out experiments and for communication with the engineers who are on Earth[4].

Exit of Dynamics 365 Layout and Dynamics 365 Remote Assistant for HoloLens

In September, 2018 the Microsoft company submitted the new applications using artificial intelligence and the mixed reality for Dynamics 365. With their help the organizations and the enterprises can fully use possibilities of a helmet of the mixed reality of HoloLens for business challenges. Read more here.

Application for navigation of blind people

In May, 2018 in California Institute of Technology showed the application which allows people to be guided with the lost sight in space, using a helmet of virtual reality Microsoft HoloLens.

This device is equipped with several cameras, including depth camera and also loudspeakers located on perimeter. It and capability of the headset to create the volume map of the surrounding territory the group of engineers under the leadership of Markus Meister from  California Institute of Technology used.

They developed the program allowing blind people to move in premises and "see" objects (steps, furniture , etc.) around by means of audionavigation.

The application can conduct the person along the set route, announcing him the coming turns and various obstacles. Points take the video image, take important data on a surrounding situation and transmit this information through acoustical augmented reality. So, the person in a Microsoft helmet hears such hints as "Up a ladder", "A handrail on both sides", "Idi to the right", etc.

The VR navigator was tested on seven volunteers, it was offered to them to overcome the crossed route on two floors. According to developers, all participants of an experiment passed a way from the first.

Markus Maister hopes that after carrying out a research it will be possible to create devices which people with the weakened sight in public places, including in shopping centers and hotels will be able to use.

According to World Health Organization, by May, 2018 about 253 million people are visually impaired worldwide or are completely deprived of sight. Development of California Institute of Technology has high potential, however it is necessary to do a lot more work. For example, the application cannot build routes which were not laid in advance by someone, and does not react to people who can be in the same premises.[5]

Application in difficult reconstructive transaction

In February, 2018 the Microsoft corporation announced that physicians from Imperial College London used the device of the mixed reality of HoloLens in reconstructive and plastic surgery. Within the made experiment 3D - the picture of the computer tomography (CT) was loaded into the headset and saw a doctor in the form of the hologram during transaction. The similar technology allows to set exact location of blood vessels, bones and patient's muscles that considerably simplifies performing transactions at changes and also allows to reduce potential risks, specified in Microsoft.

Surgeons used Microsoft HoloLens in difficult reconstructive transaction
Surgeons used Microsoft HoloLens in difficult reconstructive transaction

As you know, feature of transactions at a change and change of fabrics is need of existence of an exact picture of anatomic features for successful and most operational connection of vessels and the subsequent recovery of a blood-groove of the operated area. According to doctors, existence before eyes of the pictorial diagram of arrangement of vessels, bones and muscles can become a key factor of success of the transaction which is usually requiring thorough preliminary training.

When carrying out similar interventions surgeons traditionally use the hand ultrasonic scanner for detection of blood vessels under skin on the movement of a blood-groove. However this process very labor-consuming also does not give exact data on location of vessels under skin.

Demonstration of an experiment, February, 2017

Application in training of doctors to accept difficult childbirth

As it became known in January, 2018, the CAE company created the LucinaAR simulator designed to help doctors to fulfill actions at different options of patrimonial complications. The basis of the simulator is formed by detailed dummies of mother and baby. Points of the mixed reality of HoloLens using which it is possible to see the movement of a fruit are also included in the package and to estimate efficiency of the taken measures.

In particular, the simulator is intended for training of doctors in work at a shoulder distotion — a rare species of a complication, it can meet to once per thousand of cases. If the obstetrician is familiar with technicians who help to release a shoulder of the baby, it considerably increases chances of successful childbirth. Detailed simulation in this case is irreplaceable.

«
LucinaAR is this break in educational simulation thanks to which students will be able to see anatomy of the patient with own eyes — the president of CAE doctor Robert Amio told. — The simulator will be combined with our platform of the mixed reality because we are sure of immediate and powerful effect of preparation for childbirth with a shoulder distotion.
»

The Canadian company CAE which specializes in equipment manufacturing for training in avionics and medicine in 2017 showed the first product using HoloLens — the VimedixAR project directed to training of medical students in performing ultrasonographies.

2017

Application in operation on removal of a tumor

Being in different corners of the world, three outstanding surgeons from India and Britain using Microsoft Hololens augmented rReality glasses could be present virtually at one operating room and jointly execute operation on removal of a tumor. On October 23, 2017 in the blog BMI Healthcare, the group of private clinics, largest in Great Britain, reported about it.[6]

Microsoft AR glasses helped surgeons to perform operation on removal of a tumor from the different countries
Microsoft AR glasses helped surgeons to perform operation on removal of a tumor from the different countries

Operation for a colorectal cancer took place based on the London hospital The Royal London Hospital. The surgeon from Great Britain Shafi Ahmed was assisted by colleagues — professor Shailesh Shrikhande, onkokhirurg from Tata Memorial Hospital in Mumbai, the largest oncological clinic of India, and Hitesh Patel, the advising kolorektalny surgeon from private clinic BMI The London Independent Hospital. Ian Nott from the USA, the cofounder and the technical director of the company of Aetho which developed Thrive AR-application for Microsoft Hololens glasses became one more virtual consultant.

Specialists worked in AR points and could not only hear, but to see each other in the form of graphic figures avatars as though they really were in one operating room. Tomographic pictures of internals of the patient which were displayed in the form of three-dimensional holograms were available to surgeons, and doctors could do on them virtual marks, discussing the transaction course.

Thrive AR-application for Hololens glasses was developed by Aetho company
Thrive AR-application for Hololens glasses was developed by Aetho company
«
"We really integrated technologies with medicine. Having brought together together specialists from different parts of the world and time zones, we showed that we can perform operations more safely, to provide the best results of treatment of patients and to democratize surgical practice" — professor Shafi Ahmed commented.[7]
»

Use of HoloLens glasses in oncological transaction, on October 24, 2017

It should be noted that it is not the first experiment of Shafi Ahmed on use of innovations in the operating room. In 2014 he with the help VR- Google Glass glasses organized online broadcast of transaction which was seen by several thousands of medical students from 115 countries.

Microsoft HoloLens 2.0

According to representatives of Microsoft, the second generation of HoloLens glasses, in addition to the "holographic" processor module (Holographic Processing Unit/HPU 2.0), will also be equipped with the coprocessor of artificial intelligence (AI coprocessor) thanks to which support of technologies of deep neural networks (Deep Neural Networks, DNN) will become possible.

The executive vice president of division for artificial intelligence and researches of Microsoft (Artificial Intelligence and Research Group) Harry Shum in July published the first details about the second generation of HoloLens glasses which development continues.[8]

Photo: s9.tarhely.com
Photo: s9.tarhely.com

Among a set of additional functions and opportunities of HoloLens 2.0 the coprocessor of artificial intelligence (AI coprocessor) which will allow the device to get support of technologies of deep neural networks is of the greatest interest. It means that HoloLens will be able to process "on the fly" large volumes of data, without resorting to external cloud resources.

As Gary Shum specified, the AI coprocessor will be completely reprogrammable and will be able to work in real time that, in turn, will allow the second generation of HoloLens independently and without delays to analyze what is seen and heard by the user. Besides, the chip will be used by developers of a software platform HoloLens 2.0 to creating applications of the mixed reality with the activated AI function.

As well as in a case with HoloLens of first generation, the "holographic" processor module (Holographic Processing Unit/HPU 2.0) which is responsible for data processing from all sensors of a system in real time including the sensor of depth of real time of own development of Microsoft, cameras for tracking of provision of the head of the user, sensors of the block of measurement of inertia (IMU) and the infrared camera will also be feature of a hardware platform of HoloLens 2.0 glasses.

Hardware

HoloLens 2.0 is based on the processor with a performance not below Intel Mobile Core i5, the built-in graphic module not more weakly than Intel HD Graphics 620 (GT2) with support of API Direct3D 12 is responsible for processing of graphics. The device is equipped not less than 8 GB of two-channel memory, the HDMI 1.4 interface with support of mode 2880x1440 with clock rate of 60 Hz or HDMI 2.0 with the resolution of 2880x1440 and clock rate of 90 Hz, the hard drive with a capacity from 100 GB, USB 3.0 or USB 3.1 port and also Bluetooth 4.0.

Application as the guide on the museum of the Japanese temple

Microsoft HoloLens, (2016)
Microsoft HoloLens, (2016)


On August 18 the Microsoft corporation announced that points of the mixed reality of HoloLens are used as the guide on the museum of the Japanese temple Kennin-dzi now. Ancient exhibits will recover using visual effects, sounds and pop-up information. Thus, the management of the temple Kennin-dzi expects to attract new visitors and also to strengthen their impression of an exhibition.

One of the oldest a zen - Buddhist temples of Kyoto Kennin-dzi will use Microsoft HoloLens as the guide on the museum
One of the oldest a zen - Buddhist temples of Kyoto Kennin-dzi will use Microsoft HoloLens as the guide on the museum

One of the first artifacts involved in the project — a 400-year painted screen with images of the Japanese gods of wind and thunder, Fudzin and Raydzin. In the mixed reality it will be complemented with realistic sounds and 3D - the elements developed by the Hakuhodo group in partnership with Microsoft. Having put on HoloLens glasses, the visitor will see under cloud deities from which to the earth strike lightnings. The electronic "performance" which is followed by the voice certificate of an exhibit will last from three to five minutes.

«
We want that people looked in a new way at our national property. It is the first exhibition at which HoloLens glasses will be applied in this way — Kadzukhiro Suda, the creative director of Hakuhodo Inc advertizing agency emphasized. — I consider that each museum should try to use the mixed reality including to attach to technology of normal visitors.
»

Use of points in difficult backbone operation

In February, 2017 it became known of use of points of virtual reality of Microsoft HoloLens during difficult and risky backbone operation. It was carried out at Institute of orthopedics and traumatology of Instituto de Ortopedia e Traumatologia de Jaraguá do Sul in the State of Santa Catarina (Brazil).

Surgeons Henrique Lampert and Bruno Gobbato applied HoloLens in an artrodeza — transactions of creation of an artificial anchylosis of a joint at which the complete immovability of a joint is reached. The gadget of Microsoft helped doctors to install precisely screws for fixing of a backbone in those places where the movement hurt to avoid injuries of nerves or destruction of an implant.

Through HoloLens surgeons visualized the three-dimensional model of a backbone of the patient with optimal arrangement of screws created on the basis of the computer tomography (CT). Scientists note that such visualization is very difficult as it requires use during transaction of the expensive KT-equipment which not all hospitals have. HoloLens is the portable device and costs $3 thousand that it is much less in comparison with specialized medical solutions.

Enrique Lampert and Bruno Gobbato hoped that new versions of HoloLens will be able to output the volume picture with higher resolution and picture frequency. At the same time, by words Lampert and Gobbato, they liked a "limited" viewing field of the device as it allows to monitor other screens, records and elements which are not connected with the headset at the same time.

Doctors intend to develop the training course devoted to use by the implemented HoloLens of holographic technology wearing spectacles in surgeries.[9]

The version for builders

Microsoft together with the American company software makers TRMB and the Cambridge university developed the HoloLens augmented rReality glasses held for use in the construction industry, the announcement is expected in 2017.

The device is created based on points of the mixed reality of Microsoft HoloLens and resembles a VR helmet superficially. Points will work at the Windows 10 system.

Points have the built-in sensors which read out technical parameters of constructions and reveal as the mistakes made at design and construction, and the purchased construction damages. Besides using the gadget it is possible to create the 3D model of construction - the building or an infrastructure facility, for example, of the bridge - and to save it for the subsequent computer processing.

In 2016 to year in sale send the Hololens glasses allowing to work with an interior and models of buildings.

2016: The computer executed in a type of glasses

Microsoft HoloLens is the headset which is put on the head, a hoop with the tinted lenses and wavy prismatic structure located before eyes. Lenses refract and send to eyes of the user of the image from microdisplays, about spirit of the parties.

Points are equipped with means of adjustment, individual adjustment to physiological features of the head of the user: interpupillary distance and others. Loudspeakers are located at bottom edge of the device, they provide giving of sounds of virtual reality and external noise.

Microsoft HoloLens, (2016)
Microsoft HoloLens, (2016)

Product development began in 2012. The Microsoft company provided a prototype in January, 2015. In March, 2016 sales of the version for developers in the USA and Canada began. In October, 2016 the preorder of the device became available in Great Britain, Ireland, France, Germany, Australia and New Zealand.

Microsoft HoloLens, (2016)
Microsoft HoloLens, (2016)

In construction of HoloLens are used:

  • 64-bit 4-core processor Intel Atom x5-Z8100 with a frequency of 1.04 GHz.
  • graphic processor
  • the holographic processor developed by Microsoft especially for HoloLens.

The holographic processor in the body 12 × 12 mm uses 28 digital signal processors of production Tensilica for the processing and data integration arriving from sensors, space scanning of the premises, recognition of gestures, a voice and the speech. SoC (system on chip) and the holographic processor divide 8 MB of the built-in memory of SRAM and on 1 GB of LPDDR3.

The built-in data warehouse with an amount of 64 GB, from which ~ 10 GB - the operating system. The amount of OZU — 2 GB.

As the developer company stated, Microsoft HoloLens is the computer executed in a type of glasses. The picture output to their glass is imposed on the world around and built in it by means of the data obtained from device cameras. Management in interactive applications is executed by gestures or voice.

You look also Virtual reality



Notes