AI102: Create computer vision solutions with Azure AI Vision

Analyze images

Azure AI Vision

• Description and tag generation
• Object detection - location of specific objects within the image.
• People detection - detecting the presence, location, and features of people in the image.
• Image metadata, color, and type analysis
• Category identification -
• Background removal - เหมือนมือถือ
• Moderation rating - adult or violent content.
• Optical character recognition - reading text in the image.
• Smart thumbnail generation - identifying the main region of interest in the image to create a smaller "thumbnail" version.

Sample Code

using Azure.AI.Vision.ImageAnalysis;

ImageAnalysisClient client = new ImageAnalysisClient(
    Environment.GetEnvironmentVariable("ENDPOINT"),
    new AzureKeyCredential(Environment.GetEnvironmentVariable("KEY")));

ImageAnalysisResult result = client.Analyze(
    new Uri("<url>"),
    VisualFeatures.Caption | VisualFeatures.Read,
    new ImageAnalysisOptions { GenderNeutralCaption = true });

Possible Feature (Result)

Exercise - Analyze images with Azure AI Vision mslearn-ai-vision (microsoftlearning.github.io)

จาก Exercise บางตัวอย่าง Background Removal ไม่มีใน VisualFeatures

Knowledge check: Analyze images

Image classification with custom Azure AI Vision models

• custom models include Image classification, Object detection, and Product recognition (เน้นหา Brand เช่น Coke)

Create a custom vision project

1. Create your blob storage container and upload just the training images.
2. Create the dataset for your project, and connect it to your blob storage container. When creating your dataset, you define what type of project it is (image classification, object detection, or product recognition).
3. Label your data in your Azure Machine Learning Data Labeling Project, which creates the COCO file in your blob storage container.

COCO file

• images: Defines the image location in blob storage, name, width, height, and ID.
• annotations: Defines the classifications (or objects), including which category the image is classified as, the area, and the bounding box (if labeling for object detection).
• categories: Defines the ID for the named label class.

4. Connect your completed COCO file for the labeled images to your dataset.
5. Train your custom model on the dataset and labels created.

3-5 images per class to train a custom image classification

6. Verify performance and iterate if the trained performance isn't meeting expectations.

Ref: Image classification with custom Azure AI Vision models - Training | Microsoft Learn

Creating your dataset

curl -X PUT https://<endpoint>/computervision/datasets/<dataset-name>?api-version=<version>\
  -H "Content-Type: application/json" \
  -H "Ocp-Apim-Subscription-Key: <subscription-key>" \
  --data-ascii "
  {
    'annotationKind':'imageClassification',
    'annotationFileUris':['<URI>']
  }"

Exercise - Classify images with an Azure AI Vision custom model

Knowledge Check - Image classification with custom Azure AI Vision models

Detect, analyze, and recognize faces

Azure AI Vision (Face service) ทำอะไรได้

• Face detection
• facial feature analysis อารมณ์เหมือนดูโหงวเฮ้ง

💡 Head pose / Glasses ใส่แว่น / Blur / Exposure / Noise / Occlusion / Accessories ใส่ prop อะไร แว่นหูฟัง / QualityForRecognition

• Face comparison and verification.
• Facial landmark location

predict Location.

• Facial recognition
• Facial liveness : real or fake

การทำ Face Analysis ต้องทำตาม Responsible AI โดยต้องเน้น Data privacy and security / Transparency / Fairness and inclusiveness

Compare and match detected faces

• When a face is detected by the Face service, a unique ID (GUID) is assigned to it and retained in the service resource for 24 hours.

Implement facial recognition

1. Create a Person Group that defines the set of individuals you want to identify (for example, employees).
2. Add a Person to the Person Group for each individual you want to identify.
3. Add detected faces from multiple images to each person, preferably in various poses. The IDs of these faces will no longer expire after 24 hours (so they're now referred to as persisted faces).
4. Train the model.

Exercise: Detect, analyze, and identify faces

Knowledge Check: Detect, analyze, and identify faces

Q: You need to verify that the person in a photo taken at hospital reception is the same person in a photo taken at a ward entrance 10 minutes later. What should you do? 
A: Verify the face in the ward photo by comparing it to the detected face ID from the reception photo.

Read Text in images and documents with the Azure AI Vision Service

Azure AI Vision options for reading text

• Image Analysis Optical character recognition (OCR) - อ่านจากรูป / handwritten text. ****เช่น ป้าย

Results contain blocks, words and lines, as well as bounding boxes for each word and line.

• Document Intelligence - อ่านเอกสาร ภาพถ่ายเอกสาร PDF Examples include: receipts, articles, and invoices

Use the Read API

• C#

ImageAnalysisResult result = client.Analyze(
    <image-to-analyze>,
    VisualFeatures.Read);

• Rest

https://<endpoint>/computervision/imageanalysis:analyze?features=read&...

Exercise: Read text in images

Knowledge Check: Read text in images

Analyze video

Azure Video Indexer วิเคราะห์จาก VDO โดยหลักๆ จะคล้ายกับ Image

• Facial recognition - detecting the presence of individual people in the image. This requires Limited Access approval.
• Optical Character Recognition - reading text in the video.
• Speech transcription - ทำซับ
• Topics - identification of key topics discussed in the video.
• Sentiment - analysis of how positive or negative segments within the video are.
• Labels - label tags that identify key objects or themes throughout the video.
• Content moderation - detection of adult or violent themes in the video.
• Scene segmentation - a breakdown of the video into its constituent scenes.

นอกจากนี้ยังดู custom insights ได้แก่ people / brand / Language

ใช้งาน Azure Video Indexer

• ผ่าน Azure Video Indexer widgets - เอา VDO มารัน เดวมันทำ Azure Video Indexer ให้
• ผ่าน REST API

https://api.videoindexer.ai/Auth/<location>/Accounts/<accountId>/AccessToken

Ref: https://api-portal.videoindexer.ai/

Exercise - Analyze video

Knowledge check - Analyze Video

Q: You want Azure Video Indexer to analyze a video. What must you do first?
A: Upload the video to Azure Video Indexer and index it.

Q: You want Azure Video Indexer to recognize brands in videos recorded from conference calls. What should you do?
A: Edit the Brands model to show brands suggested by Bing, and add any new brands you want to detect.

Reference

• Create computer vision solutions with Azure AI Vision - Training | Microsoft Learn