You can access the full course here: Applied Computer Vision with Unity and Azure
Hey everyone, my name’s Daniel Buckley, and I’ll be instructor for this course. We’ll be making an app that will allow you to take a picture through your camera and then have that picture sent up to an API, where the text of that picture will be extracted, sent back down, and then converted from text to text to speech through another API.
The first thing we’re going to be learning about is Microsoft Azure’s cloud computing, and we’ll be setting up an account and creating two API’s on this. We’ll first of all be creating Computer Vision API, which will allow us to send in an image and return to us the extracted text from that image. Then, we’ll create a Speech API, and this will allow us to send text up to the API and return to us a text to speech audio file.
We’ll also be using JSON files. This is the file format that we’ll get in return when we send a request to the Computer Vision API. It will return to us a JSON file, including all the text that is displayed on the screen. We’ll learn how to go through it, how to understand it, what it is, and how to extract the text that we need from it.
Unity Web Requests are something else that we’ll be using. This is Unity’s form of sending and receiving server web requests over the network. C# has their own built-in system for this already, but Unity’s is much simpler, much easier to use, and in many ways more versatile, as we have to enter in less code. It’s much more concise and good for what we need to use.
To tie this all together, and display it on the screen, we’ll be using Unity’s UI System, allowing us to project the camera view onto an image on the screen, and have text displayed at the bottom. In order to project the camera view onto the image, we’ll be using Webcam Textures. These are textures that Unity creates that allows us to render whatever our camera sees, that being a webcam or a device camera, onto a texture that we can apply to anything really. We can apply it to images on the UI, like in our example. We can even apply it as a normal texture onto cubes, 3D models, etcetera.
ZENVA is an online learning academy with over 400,000 students. We feature a wide range of courses, for people who are just starting out, or for people who are just wanting to learn something new. The courses are also very versatile, and you can learn many different ways. If you want to follow along with the tutorial videos, we have included course project files that you can use. Or you can just watch the videos along at your own pace. So that all said, let’s get started on our project.
Alright, the first thing you wanna do is go to the Microsoft Azure website here. It’s just azure.microsoft.com, and it should take you to the homepage right here. Now, we then need to log in. So, we just click on the sign in button up here. And then you can choose if you want to log in with an existing account, or create a new Microsoft account. If you have a Microsoft account, just log in with that, as any sort of Microsoft account will work. So, I’ll just log in.
So, Microsoft Azure is a Cloud computing service provided to us by Microsoft. It has many different apps and functions that we can use inside of applications. For us specifically, though, we’re gonna be using the Computer Vision and the Text-to-Speech services. These are part of the Cognitive Services pack. So we can go here up to Products, and inside here we can then click on where we see AI + Machine Learning and Cognitive Services.
Now, Cognitive Services are basically machine learning sort of APIs and SDKs we can connect to. If we scroll down, we can see a bunch of the different ones here. We have Vision, which is the one we’re gonna be connecting to. We’ll send over an image to the API, and then it will analyze the text, and return that to us. We also have the Speech here. There’s also many other smaller functions inside each of these large categories.
So what we’re going to do now is actually sign up for Azure, and make it so that we can start creating some of these resources. Now, what we can do, is we can then click on the Portal button up here. This will take us to our Portal, which basically just has a list of all our different resources. All the ones we can make and allows us to manage those resources and our apps. So, we’ll click on the Portal button here.
Okay, when we’re at the Portal now, what we want to do is first of all, click on the Create a resource button here. Because what we need to do is actually set up our free account, that will allow us to use these resources. So, it doesn’t really matter which one you click on. We’ll just click on the Windows Server 2016. We’re not gonna get this, but it will require us to create a free account. This is what we’ll need to be able to actually use and create resources.
Now, with the free account you may see here that- you may see at the $200 price mark, but that just means we get $200 credit for 30 days. This is basically like a free trial. You will need to sign up with your credit card and phone number, but you won’t be charged unless you go past 30 days, or you choose one of the actual pricing tiers when we create a resource.
So, we can just click on the Start Free button here, and then it will take us to this page here. We can click on Start Free again. And then you just want to go through and fill out all of this information here. It will ask you to just enter in your name, your phone number. You don’t have to enter in your ABN. You’ll then have to verify by adding in your credit card, or bank card, or debit card. It won’t do any charges. I’m fairly certain, but it’s just there to verify you. And if you do want to then sign up further, you can then just do it much quicker then.
Alright, so when that’s complete, you should be taken back to this page now. It shouldn’t look too different, but what we can do now is start creating resources and that’s what we’re going to be doing in the next video. You might also see up here that it says, you got your free trial, and it has remaining credit. So, that you know that you are in the free trial. And yeah, then we were good to go.
So, I’ll see you next lesson where we’ll start creating an actual computer vision resource.
All right, welcome back. In the last lesson, we created our Microsoft Azure account and ended up here on the portal. In this lesson, we’re gonna be creating our Computer Vision Resource, which will allow us to connect to the Computer Vision Resource API, sending an image, and then the API will extract the text from the image and then return that to us as JSON file.
Now before we continue, let’s go over what a JSON file is. Here’s an example of it. It’s basically a text file that contains objects and properties. So here we have, basically a list of users here and then we have an object for each user. You can tell an object by it having these squiggly brackets and a list or an array by having square brackets here. So each object here in this example has a name and an age. The name is a string and the age is 25.
So this is basically a way of- so this is the format that we are going to receive our text as. It will have multiple objects as well, as string containing information that just comes with the cognate service, but we’ll be looking specifically for very specific objects, lists and going down the hierarchy until we find the text that we actually want. We’ll go over more of what the actual JSON file for our computer vision API looks like once we got to scripting it, as we need to know when we are doing that.
All right, so let’s return to the portal here, and what we wanna do first of all is click on the Create Resource button at the top left. This will take us to a list where we can select something or search. What we’re gonna do is we’re gonna be searching for computer vision, select this and we want to create a new Computer Vision Resource. So we click on the create button And then this will take us to here. What we wanna do here is fill in some information.
The first thing we need to do, actually, is create a resource group. Now a resource group- it’s not that important that we do as it’s not required for when we start scripting in unity. But it is just good to pair the services that you going to be using together basically in one group. So, create a new group here, and let’s just call this ImageReaderApp- and we click okay.
And now this about the image reader app. For the name, we can call this ImageReaderComputerVision. Like so, and the subscription we wanna set to free trial. Location, you can choose whatever location you are in- whatever location here you see is close to you. I am from Eastern Australia so I’ll have it on Australia East, and for the pricing tier what we wanna do is select the F0 pricing tier. I have already created a Computer Visionary Resource before to actually test out the app, but you should have an F0 option here.
And then what we can do is click on create, and I have already created one so I’m gonna go to mine right now. But just click the create button, and it should then take you back to the actual portal home here. And in the top right if we click on the bell, you should see there’ll be a notification here. It should say something about the resource is deploying. So it will have a loading bar here showing you that it’s deployed. And once that’s complete there should be a button that says go to resource and when you click on that, it will take you to this page here.
We don’t really need to be on this page here by default, so what we what we gonna do is we gonna click on the overview button up here and this will take us to the overview page. Now there’re a few things, there are actually two things that we need in order to connect to the API once we’re in Unity.
And that is Endpoint URL here. So we need the Endpoint URL here, so just copy that and paste it to a Notepad document- just have this page opened for when we get up to scripting point. And then we need to click on short access keys, because in order to connect to your specific resource of the API, we need a specific access key. So you can click on that and it will show us we have two access keys here. Now we only need the key one, you can use the key two. It doesn’t really matter which one you use, but we’ll just use key one here. So just copy that and again keep it in a Notepad document (or just have this page opened, ready for scripting).
In the next lesson we’ll be setting up a second resource which is going to be, text to speech using the speech services and there are a few different things we need to get from that one. So, stay tuned for that.