Final Result
We want our app to display information according to a scanned barcode. The information comes from a web server and should hover nicely next to the scanned barcode. The result is shown in the image below.
Actual screenshot from the mobile device
System Overview
We use the following Tools:
| Tool | Version | Tasks |
|---|---|---|
| Unity | 2018.2.0f2 | Development environment |
| Vuforia | 7.1.31 | Image target tracking and camera positioning |
| ZXing | 0.15.0.0 | Barcode reading and decoding |
| RestClient | 1.2.2.0 | Web server communication |
Step 1 - Image Target Tracking with Vuforia
Vuforia Engine is an out-of-the-box toolset for a lot of Augmented Reality use cases. For using Vuforia, we just start a new Unity project and enable the Vuforia Unity 2018.2.0f2 plugin.
Further, we need to create a Vuforia account on developer.vuforia.com. The necessary steps for creating an account are not described in this article.
We get access to the camera image by using the following code:
using Vuforia; public class CameraImageAccess : MonoBehaviour { private bool mAccessCameraImage = true; public RawImage rawImage; private void Start() { Vuforia.VuforiaARController.Instance.RegisterVuforiaStartedCallback(OnVuforiaStarted); Vuforia.VuforiaARController.Instance.RegisterOnPauseCallback(OnPause); } private void OnVuforiaStarted() { CameraDevice.Instance.SetFrameFormat(Vuforia.Image.PIXEL_FORMAT.RGB888, true) CameraDevice.Instance.SetFocusMode(CameraDevice.FocusMode.FOCUS_MODE_CONTINUOUSAUTO); } private void OnPause(bool paused) { if (paused) CameraDevice.Instance.SetFrameFormat(Vuforia.Image.PIXEL_FORMAT.RGB888, false); else CameraDevice.Instance.SetFrameFormat(Vuforia.Image.PIXEL_FORMAT.RGB888, true) } private void OnTrackablesUpdated() { if (mAccessCameraImage && rawImage != null) { Vuforia.Image image = CameraDevice.Instance.GetCameraImage(Vuforia.Image.PIXEL_FORMAT.RGB888); if (image != null && image.IsValid()) { byte[] pixels = image.Pixels; if (pixels != null && pixels.Length > 0) { Texture2D tex = new Texture2D(Screen.width, Screen.height, TextureFormat.RGB24, false); tex.LoadRawTextureData(pixels); tex.Apply(); rawImage.texture = tex; rawImage.material.mainTexture = tex; } } } } public Vuforia.Image GetCameraImage() { if (mAccessCameraImage) { Vuforia.Image image = CameraDevice.Instance.GetCameraImage(Vuforia.Image.PIXEL_FORMAT.RGB888); if (image != null && image.IsValid()) { byte[] pixels = image.Pixels; if (pixels != null && pixels.Length > 0) return image; } } return null; } } Step 2 - Barcode recognition with ZXing
ZXing is an extensive library for working with barcodes. ZXing can not only identify and decode barcodes, but also create barcodes in many different formats. So, we download ZXing, unpack, and place the zxing.unity.dll in our project assets.
Decoding the barcode takes only a few lines of code:
using Vuforia; using ZXing; using ZXing.Common; public class DecodeBarcode : MonoBehaviour { private CameraImageAccess cameraImage; private BarcodeReader barcodeReader = new(); public bool Detect(out string barcode) { Vuforia.Image cameraFeed = cameraImage.GetCameraImage(); if (cameraFeed == null) return false; Result data = barcodeReader.Decode(cameraFeed.Pixels, cameraFeed.BufferWidth, cameraFeed.BufferHeight, RGBLuminanceSource.BitmapFormat.RGB24); if (data == null) return false; barcode = data.Text; return true; } } Step 3 - Data exchange with REST API
As we were scanning a label containing data from a vessel in liquids production, we might want to get at least the name and the status of the vessel. We decide to also ask the server for some more information. The request for the vessel data has the following final form:
[$address] ? $filter=equipment_label eq 'MB710' & $format=JSON & $select=equipment_id, equipment_label, equipment_name, dirty_holding_date, cleaning_date, expiry_date, equipment_status
And we receive a nice response containing all information from the database:
As a last step, we need to display this information in Unity next to the Vuforia tracking target.
Conclusion
In the end we had not one hurdle. All tools we used were just out of the box. Vuforia worked just awesome and all libraries we used were easy to integrate. However, personal mobile phones and tablets are not designed for industrial use, so we received a lot of feedback that operators couldn't read the text because it was way too small.
Addendum July 2019: In the meantime, the operators received their own industrial tablets. Looking forward to working with them again.