AI-102: Develop solutions with Azure AI Document Intelligence

Plan an Azure AI Document Intelligence solution

Azure AI Document Intelligence (Form Recognizer เดิม) is an Azure service that you can use to analyze forms completed by your customers, partners, employers, or others and extract the data that they contain โดยใช้ผ่าน

• Azure AI Document Intelligence Studio
• SDK C# NET / Java / Python / JavaScript

Responsible use of AI: Microsoft AI Principles

Azure AI Document Intelligence outputs data in JSON format, which is widely compatible with many databases, other storage locations, and programming languages.

0.Document analysis models: which take an input of JPEG, PNG, PDF, and TIFF files and return a JSON file

1.prebuilt models

• Common
  - Read text + Lain handwriting
  - General document - entities extract
  - Layout - structure document / size and position of its bounding box. / header / table
• Specific
  - Invoice
  - Receipt
  - W-2 US tax declaration
  - ID Document
  - Business card
  - Health insurance card

2.custom model - extract more specific information

3.composed model. With a composed model, users can submit forms of different types to a single service,

ดูเพิ่มเติม Choose a model type - Training | Microsoft Learn

Azure AI Document Intelligence and Azure AI Vision

• Azure AI Vision OCR เพียวๆอ่าน Text จากรูป
• Azure AI Document Intelligence อ่าน doc ที่มีโครงสร้าง + รูป

Plan For Using มีหลายแบบตอนแรก อาจจะเริ่มจากแบบฟรีก่อนก็ได้

• Create Document Intelligence Resource มี Free (F0)
  - แต่ใช่ composed model - 5 custom models
  - ได้ที่ละ 1 concurrent)
  ถ้าใช้หลาย Service Azure AI Service resource: a multi-service subscription รู้ว่าใช้เพียงอันเดียว Azure Document Intelligence resource
• Connect via End Point / Access Key

using Azure;
using Azure.AI.DocumentIntelligence;

string endpoint = "<endpoint>";
string key = "<access-key>";
AzureKeyCredential cred = new AzureKeyCredential(key);
DocumentIntelligenceClient client = new DocumentIntelligenceClient (new Uri(endpoint), cred);

Uri fileUri = new Uri ("<url-of-document-to-analyze>");

AnalyzeDocumentOperation operation = await client.AnalyzeDocumentFromUriAsync(WaitUntil.Completed, "prebuilt-layout", fileUri);

AnalyzeResult result = operation.Value;

Knowledge check - Knowledge check - Training | Microsoft Learn

Use prebuilt Document intelligence models

Features of prebuilt models

• Text extraction. All the prebuilt models extract lines of text and words from hand-written and printed text.
• Key-value pairs. Spans of text within a document that identify a label or key and its response or value are extracted by many models as key-values pairs. For example, a typical key might be Weight and its value might be 31 kg.
• Entities. Text that includes common, more complex data structures can be extracted as entities. Entity types include people, locations, and dates.
• Selection marks. Spans of text that indicate a choice can be extracted by some models as selection marks. These marks include radio buttons and check boxes.
• Tables. Many models can extract tables in scanned forms included the data contained in cells, the numbers of columns and rows, and column and row headings. Tables with merged cells are supported.
• Fields. Models trained for a specific form type identify the values of a fixed set of fields. For example, the Invoice model includes CustomerName and InvoiceTotal fields.

Requirements

• The file must be in JPEG, PNG, BMP, TIFF, or PDF format. Additionally, the Read model can accept Microsoft Office files.
• The file must be smaller than 500 MB for the standard tier, and 4 MB for the free tier.
• Images must have dimensions between 50 x 50 pixels and 10,000 x 10,000 pixels.
• PDF documents must have dimensions less than 17 x 17 inches or A3 paper size.
• PDF documents must not be protected with a password.

Compare prebuilt models

More

• Use the General Document, Read, and Layout models - Training | Microsoft Learn
• Use financial, ID, and tax models - Training | Microsoft Learn

Exercise - Analyze a document using Azure AI Document Intelligence
Knowledge check - Knowledge check - Training | Microsoft Learn

Extract data from forms with Azure Document intelligence

พวก pre-built มันซ้ำกับของก่อนหน้า

Train custom models

To train a custom model:

1. Store sample forms in an Azure blob container, along with JSON files containing layout and label field information.

You can generate an ocr.json file for each sample form using the Azure Document Intelligence's Analyze document function. Additionally, you need a single fields.json file describing the fields you want to extract, and a labels.json file for each sample form mapping the fields to their location in that form.

2. Generate a shared access security (SAS) URL for the container.
3. Use the Build model REST API function (or equivalent SDK method).
4. Use the Get model REST API function (or equivalent SDK method) to get the trained model ID.

OR

Use the Azure Document Intelligence Studio to label and train. There are two types of underlying models for custom forms custom template models or custom neural models.

• Custom template models accurately extract labeled key-value pairs, selection marks, tables, regions, and signatures from documents. Training only takes a few minutes, and more than 100 languages are supported.
• Custom neural models are deep learned models that combine layout and language features to accurately extract labeled fields from documents.This model is best for semi-structured or unstructured documents

Ref: Use the Azure Document Intelligence Studio - Training | Microsoft Learn

Using the API

string endpoint = "<endpoint>";
string apiKey = "<apiKey>";
AzureKeyCredential credential = new AzureKeyCredential(apiKey);
DocumentAnalysisClient client = new DocumentAnalysisClient(new Uri(endpoint), credential);

string modelId = "<modelId>";
Uri fileUri = new Uri("<fileUri>");

AnalyzeDocumentOperation operation = await client.AnalyzeDocumentFromUriAsync(WaitUntil.Completed, modelId, fileUri);
AnalyzeResult result = operation.Value;

Sample Response ดูค่า confidence ก่อนใช้จริง ควรจะสัก 80-100%

{
	"status": "succeeded",
	"createdDateTime": "2023-10-18T23:39:50Z",
	"lastUpdatedDateTime": "2023-10-18T23:39:54Z",
	"analyzeResult": {
		"apiVersion": "2022-08-31",
		"modelId": "DocIntelModel",
		"stringIndexType": "utf16CodeUnit",
		"content": "Purchase Order\nHero Limited\nCompany Phone: 555-348-6512 Website: www.herolimited.com Email: accounts@herolimited.com\nPurchase Order\nDated As: 12/20/2020 Purchase Order #: 948284\nShipped To Vendor Name: Balozi Khamisi Company Name: Higgly Wiggly Books Address: 938 NE Burner Road Boulder City, CO 92848 Phone: 938-294-2949\nShipped From Name: Kidane Tsehaye Company Name: Jupiter Book Supply Address: 383 N Kinnick Road Seattle, WA 38383\nPhone: 932-299-0292\nDetails\nQuantity\nUnit Price\nTotal\nBindings\n20\n1.00\n20.00\nCovers Small\n20\n1.00\n20.00\nFeather Bookmark\n20\n5.00\n100.00\nCopper Swirl Marker\n20\n5.00\n100.00\nSUBTOTAL\n$140.00\nTAX\n$4.00\nTOTAL\n$144.00\nKidane Tsehaye\nManager\nKidane Tsehaye\nAdditional Notes: Do not Jostle Box. Unpack carefully. Enjoy. Jupiter Book Supply will refund you 50% per book if returned within 60 days of reading and offer you 25% off you next total purchase.",
		"pages": [
			{
				"pageNumber": 1,
				"angle": 0,
				"width": 1159,
				"height": 1486,
				"unit": "pixel",
				"words": [
					{
						"content": "Purchase",
						"polygon": [
							89,
							90,
							174,
							91,
							174,
							112,
							88,
							112
						],
						"confidence": 0.996,
						"span": {
							"offset": 0,
							"length": 8
						}
					},
					{
						"content": "Order",
						"polygon": [
							178,
							91,
							237,
							91,
							236,
							113,
							178,
							112
						],
						"confidence": 0.997,
						"span": {
							"offset": 9,
							"length": 5
						}
					},
                    ...

Exercise - Extract data from custom forms
Knowledge check - Knowledge check - Training | Microsoft Learn

Create a composed Document intelligence model

What are composed models?

You can create custom models of two types:

• Custom template models. Use custom template models when your forms have a consistent visual template. The formatting and layout should be consistent across all completed examples of the form.
• Custom neural models. Use custom neural models when your forms are less consistent, semi-structured or unstructured.

How to Create Compose Model

• Assemble composed models - Training | Microsoft Learn

Exercise: Create a composed model
Knowledge Check - Knowledge check - Training | Microsoft Learn

Q: You have a composed model that consists of three custom models. You're writing code that sends forms to the composed model and you need to check which of the custom models was used to analyze each form. Which property should you use from the returned JSON?

A: docType The docType property includes the model ID of the custom model that was used to analyze the document.

Q: You're trying to create a composed model but you're receiving an error. Which of the following should you check?

A: custom models were trained with labels. Only custom models that have been trained with labeled example forms can be added to a composed model.

Build a Document intelligence custom skill for Azure AI search

เอา Azure AI Search เอามา Integrate Document Intelligent

1. Document Cracking. In document cracking, the indexer opens the content files and extracts their content.
2. Field Mappings. Fields such as titles, names, dates, and more are extracted from the content. You can use field mappings to control how they're stored in the index.
3. Skillset Execution. In the optional skillset execution stage, custom AI processing is done on the content to enrich the final index.
4. Output field mappings. If you're using a custom skillset, its output is mapped to index fields in this stage.
5. Push to index. The results of the indexing process are stored in the index in Azure AI Search.

ทำ Skill เพิ่มไปเข้าได้ด้วยนะ จะเหมือนกับส่วนของ AI Search

• กำหนด End point + Token
• กำหนด json เพื่อรับส่ง

More

• Understand Azure AI Search enrichment pipelines - Training | Microsoft Learn
• Build an Azure AI Document Intelligence custom skill - Training | Microsoft Learn

Exercise: Build and deploy an Azure AI Document Intelligence custom skill
Knowledge check - Knowledge check - Training | Microsoft Learn