Augmented Reality – Healthcare Use Case

This blog demonstrates creating augmented reality application using Unity game engine and Vuforia.

Augmented reality (AR) can be defined as the overlaying of digital content (images, video, text, sound, etc.) onto physical objects or locations, and it is typically experienced by looking through the camera lens of an electronic device such as a smartphone, tablet, or specialized viewing devices like google glass or Microsoft hololens.

Objective

The purpose of this blog will be to help a patient taking prescription medications to view the pharmaceutical details of the medication under a mobile phone by just scanning the prescription label on the medication bottle.

Examples are shown below. They might not be medically accurate but for the purpose of this case study have just been picked up as images.

When the bottle with the label (We shall call this our Image Target) shown below is scanned using a laptop with a camera, the medication literature as shown below is displayed to the user for reading (We shall refer to this as our Image Overlay).

Download and Installation

• Navigate to : https://unity3d.com/get-unity/download/archive and download Unity 2019.4.0
• In order to install Vuforia Engine for Unity2019 please use the script available under below link: https://developer.vuforia.com/downloads/sdk
• Add Vuforia Engine to a Unity Project or upgrade to the latest version (add-vuforia-package-9-1-7.unitypackage (2.57 KB) )
• Download and install the Java Development Kit 8 for your operating system.

Setting up Unity

1. Navigate to the GameObject dropdown menu and select “Vuforia > AR Camera.” If a dialog box appears requesting that you import additional assets, select “Import.”
2. Select “Vuforia > Image” in the GameObject dropdown menu to add an Image Target to your scene.
3. Locate the directional widget in the top right corner of your scene panel. Click the green y-axis option of your directional widget. This will make the green arrow disappear and shift your perspective in the scene panel to a y-axis birds-eye view of your project.

Go to the Vuforia developer portal to create your free Vuforia account and generate a development license key for your application.

1. Select “Register” in the upper right corner.
2. Once your account is setup, select “Develop” in the menu panel and then “License Manager.”
3. Select “Get Development Key” on the License Manager page.
4. On the “Add a free Development License Key” page, give your application a name and agree to the terms.
5. To access your license key, return to the “License Manager” page in the Vuforia developer portal and click the application name you just created. Copy the alphanumeric string on the right.
6. Return to Unity and File > Build Setting and click the “Player Settings” button in the bottom of the pop up window.
7. Navigate back to the Inspector panel and click the “XR Settings” option located at the bottom of the accordion menu. Select “Vuforia Augmented Reality Support.” Accept the Vuforia license in Unity.
8. Select ARCamera in the Hierarchy pane. In the inspector panel, click the “Open Vuforia Configuration” button in the Vuforia Behavious (Script)” component. Copy and paste your license key in the App License Key text field.
9. Navigate to “File > Save Scene as…” and give your scene a name. By default, Unity will save your scenes to the “Assets” folder of your project.

Convert your Image Target to a Dataset

Upload the medication bottle image shown above to Vuforia inorder to code it with the tracking information.

1. Go to the Vuforia developer portal and click “Develop” in the top menu and then select “Target Manager.”
2. Click the “Add Database” button on the Target Manager page. In the dialog box that appears, give your database a name and select the device option. Click “Create.”
3. Once your database has been created, return to the Target Manager page and click your database. Select “Add Target,” choose “Single Image” as the type, add the width and height of your image, and upload by clicking on the “Browse” button.

To determine the width of your image, right-click the image file and choose Properties. Select the Details tab in the dialog box that appears and scroll down to the “Image” section to find the width of the image.

Vuforia assigns target images a feature tracking rating on a scale of 1 to 5 stars. Navigate to the Image Targets database in the Vuforia Developer Portal and select the image you just uploaded. click the Show Features link to reveal the image’s trackable features.

The yellow markers represent the areas in the image that your application will use to determine if it is looking at the proper image target. If you notice that Vuforia is tracking unstable elements in your image (e.g. shadow, person, etc.), then you will need to re-edit or choose a different image.

If your image is given a good augmentability rating (anything between 3-5 stars should work), access the Target Manager page, then select your image and click “Download Database.”

Clicking download database will create a package file like the one shown below. Doubleclick and Import it

Import the Image Target

Select the ImageTarget GameObject in the project Hierarchy panel. Navigate to the inspector panel and select the drop-down menu of the Database parameter in the Image Target Behaviour. Select the name of the database you created earlier. The ImageTarget game object should now be associated with your book cover.

Add an Image Overlay

You will need to add an overlay to test if your AR camera is tracking the image. The overlay is the multimedia content that your application will superimpose onto your trigger image. In Unity, you can add images, videos, audio files, and animated 3D models as overlays. In our case it is the Rx detailed information Fig 2 above.

Drag the image into the “Assets” folder in the project panel at the bottom of the editor. Once the image has finished importing, select it and change its Texture Type to “Sprite (2D and UI)” in the inspector panel. Click “Apply.”

Next, drag the sprite you just imported onto your “Image Target” game object in the project Hierarchy.

The sprite is now a “child” of the Image Target game object and will only appear if the ARCamera recognizes the Image Target.

Create a User Interface

Now we shall create a simple user interface that guides the patient to scan their pharma Rx label.

Import a .jpg or .png file of your target image into your Assets folder.

You can do this by simply clicking and dragging the image file from its current file location into the Unity Assets folder. Once the image is imported, convert it to “Sprite (2D and UI)” in the inspector panel like you did for the author image overlay.

Next, navigate to the Hierarchy panel and select “Create > UI > Image.” Select the “2D” option in your scene view menu and center the image you just added by selecting it and clicking “F” on your keyboard. Click and drag the blue corners to center the image in your Canvas frame. Then, drag your Rx label into the “Source Image” parameter in the inspector panel.

Next, add some instructions to your UI by

1. Selecting “Create > UI > Text.”
2. In the inspector panel, write some instructions in the text field, such as “Scan this Prescription Label.”
3. Position your text box directly below your book cover and modify the font and color accordingly.

Before you can add functionality to the UI, you will need to combine your Canvas elements into a single game object.

1. Select “Create > Create Empty.” This will add an empty GameObject to your project Hierarchy.
2. Rename this GameObject “Finder” and make it a child of the Canvas by dragging it into the Canvas game object.
3. Drag the Image and Text game objects and make them children of your new “Finder” game object.

For this next step of the UI design, you will need to modify Vuforia’s “Default Trackable Event Handler” so that your “Finder” game object disappears from the user’s view once the Image Target has been found by the application.

Locate the script number packages folder and rightclick on open c# project as shown below.

Clicking open c# project will open Visual Studio editor like below

For this simple case study. We might just be able to modify the code through Unity console.

Select ImageTarget in Hierarchy and find the DefaultTrackableEvent in Inspector. Initially there will not be entries under On Target Found () and On Target Lost () methods.

Click the “+” button.

Select values for the 3 fields under On Target Found and On Target Lost as below. Note the values for the check box under each.

These settings tell the application to disable the Finder game object if the target image is found and to enable the Finder game object if the target image is lost.

Test Your Scene

Save your scene and press the play icon in the menu above the scene view.

Hold your prescription label in front of your webcam. You should see your UI overlaid onto the camera view. Place your label into the camera view and the prescription information / medication literature details image should appear overlaid on top of it.