About the Data Connector
Keep these main points in mind about the Data Connector:
The Data Connector contains four serverless functions.
The Function app maintains two queues in the Azure storage queue: one is in the main queue and the other is a backlog queue. Notifications read from the AWS SQS are placed in the mainQueue until the max size queue count is reached. After that, events are placed in the backlog queue.
Consumption function app deployment: If events in the backlog queue are growing, it's an indication that additional function apps need to be deployed to handle the load.
Proceed with these steps:
Queue Trigger: Run it each time an item is added to the Azure storage queue.
Timer Trigger: Run it every 10 minutes. It is responsible for fetching the files from SQS/S3 and adding them to the Azure storage queue.
Timer Trigger: Run it every 10 minutes. It is responsible for managing the backlog and main queue. Each time there are items in the backlog queue, the Timer Trigger checks if there is space in the main queue and removes items from the queue.
Timed Api Trigger: Run it once every hour. It makes GET workloads requests to the PCE, summarizes the response, and stores it in log analytics.
If you are deploying additional function apps, use the same steps as above and follow these guidelines:
For every new function app, ensure that it is deployed in a new app service plan.
Select the same storage account for every new function app deployment.
For premium function app deployment, you can deploy a virtual network and configure the same for the storage account. This ensures that the storage account is not open to public access.