[AZ-204] Develop event-based solutions

Explore Azure Event Grid

- Azure Event Grid

• Azure Event Grid -
  • uses the publish-subscribe model (Not guarantee order for event delivery, so subscribers may receive them out of order.)
  • Event Grid allows you to easily build applications with event-based architectures
• Concepts in Azure Event Grid
  • Events - What happened. (size 64 KB )
  • Event sources - Where the event took place.
  • Topics - The endpoint where publishers send events.
    • System topics - Provide by Azure Service
    • Custom topics -  third-party topics
  • Event subscriptions - The endpoint or built-in mechanism to route events, sometimes to more than one handler. Subscriptions are also used by handlers to intelligently filter incoming events.
  • Event handlers - The app or service reacting to the event.
• Discover event schemas - ถ้าใหญ่ไปต้องแบ่งส่ง
• Event schema
  • Event properties เดี๋ยวต้องไปดูเพิ่ม

[
  {
    "topic": string,
    "subject": string,
    "id": string,
    "eventType": string,
    "eventTime": string,
    "data":{
      object-unique-to-each-publisher
    },
    "dataVersion": string,
    "metadataVersion": string
  }
]

• CloudEvents v1.0 schema
• Explore event delivery durability
  • Retry schedule - ตั้งส่งใหม่
  • Retry policy
    • Maximum number of attempts
    • Event time-to-live (TTL) - หน่วยนาที
  • Output batching (Improved HTTP performance in high-throughput scenarios.)
    • Max events per batch
    • Preferred batch size in kilobytes - ขนาดที่เหมาะสมของ Event แต่ถ้าขนาดมากกว่า
  • Preferred Size มันยังส่งนะ
  • Delayed delivery
  • Dead-letter events - เกิดตอน ส่ง Event ไม่ทัน
  • Custom delivery properties

- Control access to events

• Event Grid ส่วนใหญ่ใช้ Azure AD + RBAC จัดการได้ ยกเว้น publishing events ให้ Event Grid topics / domains ต้องใช้ SAS ช่วย
• Built-in roles
  • Event Grid Subscription Reader - read Event
  • Event Grid Subscription Contributor - manage event
  • Event Grid Contributor - create and manage Event Grid resources.
  • Event Grid Data Sender - send events
• ถ้า event handler ไม่ใช่ WebHooks ต้อง เช่น ใช่ Azure Event Hub / Azure Queue ต้องมาขอ write permission Microsoft.EventGrid/EventSubscriptions/Write ด้วย
• Permissions for event subscriptions
  • System topics - ยุ่งกับ resource
  • Custom topics - ยุ่งกับ event grid

- Receive events by using webhooks

• Service ของ Azure ที่ Support WebHooks
  • Azure Logic Apps with Event Grid Connector
  • Azure Automation via webhook
  • Azure Functions with Event Grid Trigger
• Endpoint validation with Event Grid events
  • Synchronous handshake - ถ้าทำเสร็จมันจะส่ง validationCode กลับมา
  • Asynchronous handshake -
• ไม่รองรับ Self Cert

- Filter events

• Event type filtering

"filter": {
  "includedEventTypes": [
    "Microsoft.Resources.ResourceWriteFailure",
    "Microsoft.Resources.ResourceWriteSuccess"
  ]
}

• Subject filtering

"filter": {
  "subjectBeginsWith": "/blobServices/default/containers/mycontainer/log",
  "subjectEndsWith": ".jpg"
}

• Advanced filtering - ดูจาก Field ใน Event (Key) และค่าของมัน (Value) โดยสามารถเขียน Operation ควบคุมได้

"filter": {
  "advancedFilters": [
    {
      "operatorType": "NumberGreaterThanOrEquals",
      "key": "Data.Key1",
      "value": 5
    },
    {
      "operatorType": "StringContains",
      "key": "Subject",
      "values": ["container1", "container2"]
    }
  ]
}

Knowledge check - Explore Azure Event Grid (Knowledge check)

Explore Azure Event Hubs

Azure Event Hubs (PaaS) is a big data streaming platform and event ingestion service.

• Real-time and batch processing
• Scalable
• Rich ecosystem (Compatible Apache Kafka)

- Key concepts

• Event Hub client
• Event Hub producer
• Event Hub consumer
• Partition is an ordered sequence of events that is held in an Event Hub.
• Consumer group - view of an entire Event Hub. เข้าถึงได้พร้อมกันมากที่สุด 5 Partition
• Event receivers: Any entity that reads event data from an event hub (ใช้ AMQP)
• Throughput units or processing units: Pre-purchased units

- Explore Event Hubs Capture

• Captured data/Event (รูปแบบ Apache Avro format) ใน Event Hub สามารถเอาไปเก็บที่ Azure Blob / Azure Data Lake Storage ได้
• Capture windowing - ขนาดของข้อมูล - naming convention

{Namespace}/{EventHub}/{PartitionId}/{Year}/{Month}/{Day}/{Hour}/{Minute}/{Second}

• Scaling to throughput units
  • 1 MB per second or 1000 events per second of ingress
  • twice that amount of egress. 
  • ถ้าอยากได้เพิ่มต้องซื้่อ Throughput units

- Scale your processing application

• Requirement: For designing the consumer in a distributed environment
  • Scale
  • Load balance
  • Seamless resume on failure
    • checkpoint or offset - จุดที่ consumer ก่อน fail
  • Consume events
• Partition ownership tracking
• Receive messages
• Checkpointing - process by which an event processor marks or commits the position of the last successfully processed event within a partition
• Thread safety and processor instances -
  • function ที่จัดการกับ Event ทำงานตามลำดับ (sequentially) ในแต่ละ partition
  • แต่ทำ Parallel ได้ ถ้ามี Partition หลายๆอัน

- Control access to events

• Azure built-in roles for authorizing access to Event Hubs
  • Azure Event Hubs Data Owner: give complete access to Event Hubs resources.
  • Azure Event Hubs Data Sender: give send access to Event Hubs resources.
  • Azure Event Hubs Data Receiver: give receiving access to Event Hubs resources.
• Authorize access
  • managed identities
  • Microsoft Identity Platform
  • Event Hubs publishers with Shared Access Signatures
  • Event Hubs consumers with shared access signatures

- Perform common operations with the Event Hubs client library

ใช้ The Azure Event Hubs SDKs มาเขียน Code

• Namespace : Azure.Messaging.EventHubs
• Class EventHubProducerClient
  • Inspect an Event Hub
  • Publish events to an Event Hub
• Class EventHubConsumerClient
  • Read / Read all events from an Event Hub
• Class EventProcessorClient
  • Process Event
  • NOTE ใช้ Read แทน EventHubConsumerClient เหมาะกับ Production มากกว่า

Reference

• AZ-204: Develop event-based solutions - Learn | Microsoft Docs