I am writing this blog to record my trouble shooting of an Azure Function Blob Trigger. I give an unacceptable solution for the error below.
Possible thread pool starvation detected.First, I will tell you about my Azure function.
My Azure Function Blob Trigger
Firstly, this starts with the Logic App that save the invoices to blob store.
Secondly trigger an Azure function that sets Blob store index tag on the blob. Searching the blob storage is easier with tags.
The C# function code is
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Threading.Tasks;
using Azure.Storage.Blobs;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Azure;
using Microsoft.Extensions.Logging;
namespace Bidone.EDI.Batch.FunctionApp
{
public class SetBlobIndex
{
[FunctionName("SetBlobIndex")]
public static async Task RunAsync([BlobTrigger("edi-batch-extract-archive/{name}",
Connection = "EDIBatchStorage")]Stream myBlob,
string name,
ILogger log)
{
log.LogInformation($"C# Blob trigger function Processed blob\n Name:{name} \n Size: {myBlob.Length} Bytes");
string[] words = Path.GetFileNameWithoutExtension(name).Split('_');
log.LogInformation($"connection :{Environment.GetEnvironmentVariable("EDIBatchStorage")}");
BlobClient blobClient = new BlobClient(Environment.GetEnvironmentVariable("EDIBatchStorage"),"edi-batch-extract-archive", name);
Dictionary<string, string> tags =
new Dictionary<string, string>
{
{ "invoice_credit_number", words.LastOrDefault()},
{ "buyer_order_ref", words.SkipLast(1).LastOrDefault()},
{ "site", words.SkipLast(2).LastOrDefault()}
};
/* string invoice_credit_number = words.LastOrDefault();
log.LogInformation($"invoice_credit_number :{invoice_credit_number}");
string buyer_order_ref = words.SkipLast(1).LastOrDefault();
log.LogInformation($"buyer_order_ref :{buyer_order_ref}");
string site = words.SkipLast(2).LastOrDefault();
log.LogInformation($"site :{site}"); */
await blobClient.SetTagsAsync(tags);
}
}
}The configuration values are
[
{
"name": "APPINSIGHTS_INSTRUMENTATIONKEY",
"value": "hidden",
"slotSetting": false
},
{
"name": "AzureWebJobsStorage",
"value": "DefaultEndpointsProtocol=https;AccountName=storagedev;AccountKey=hidden==",
"slotSetting": false
},
{
"name": "EDIBatchStorage",
"value": "DefaultEndpointsProtocol=https;AccountName=storagedev;AccountKey=hidden==",
"slotSetting": false
},
{
"name": "FUNCTIONS_EXTENSION_VERSION",
"value": "~4",
"slotSetting": false
},
{
"name": "FUNCTIONS_WORKER_RUNTIME",
"value": "dotnet",
"slotSetting": false
},
{
"name": "WEBSITE_CONTENTAZUREFILECONNECTIONSTRING",
"value": "DefaultEndpointsProtocol=https;AccountName=storagedev;AccountKey=hidden==",
"slotSetting": false
},
{
"name": "WEBSITE_CONTENTSHARE",
"value": "edi-batch-dev-app",
"slotSetting": false
},
{
"name": "WEBSITE_ENABLE_SYNC_UPDATE_SITE",
"value": "true",
"slotSetting": false
},
{
"name": "WEBSITE_RUN_FROM_PACKAGE",
"value": "1",
"slotSetting": false
}
]Finally, I am using a consumption App service plan.
The problem
Developing the Azure function and testing it locally was a breeze and complete in less than 15 minutes. Next deploying to our DEV environment using devops also went well. Furthermore, unit testing gave positive results. I thought sweet that was easy, but pride comes before a fall.
Deploying with devops to our UAT environment also went well. The only difference between DEV and UAT environments is that they are in different subscriptions.
Unfortunately completing the same unit tests in UAT failed miserably. The function seems to have problems starting and I see these warnings in the application insights.
Hostmonitor is a feature of the Azure Functions Runtime that monitors various performance counters. The goal is to temporarily stop the host from doing more work when thresholds for any of the counters are about to be exceeded, to avoid hitting hard limits that could cause a shutdown.
If I leave the function running long enough, I see this error.
Exception while executing function: SetBlobIndex One or more errors occurred. (Exception binding parameter 'myBlob') (Exception binding parameter 'name') Exception binding parameter 'myBlob' Retry failed after 6 tries. Retry settings can be adjusted in ClientOptions.Retry. (The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) ..... The read operation failed, see inner exception. Cannot access a disposed object.
Object name: 'SslStream'. The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout. A task was canceled. The number of inner exceptions was 20 which is larger than 10, the maximum number allowed during transmission. All but the first 10 have been dropped.Now, you might jump to the conclusion that it just Mark’s bad code not disposing of something. But I cannot reproduce this error in the development environment. Furthermore, troubleshooting with boiler plate code shown below exhibits the same error in the UAT but not in the DEV .
Troubleshooting
The first thing was to just deploy the boiler plate code for an Azure function with a blob trigger from the Microsoft example to the UAT app. The function code is
using System;
using System.IO;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Host;
using Microsoft.Extensions.Logging;
namespace Bidone.EDI.Function
{
public class SetBlobIndex
{
[FunctionName("SetBlobIndex")]
public void Run([BlobTrigger("edi-batch-extract-archive/{name}",
Connection = "AzureWebJobsStorage")]Stream myBlob, string name, ILogger log)
{
log.LogInformation($"C# Blob trigger function Processed blob\n Name:{name} \n Size: {myBlob.Length} Bytes");
}
}
}
}
Even with this code the Azure function does not run in UAT and I get the same warnings in Application insights.
The second try was to change from an in-process host to an isolated host. The function code is a bit different but once again I get the thread starvation warnings.
System.IO;
using System.Threading.Tasks;
using Azure.Storage.Blobs;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Extensions.Logging;
namespace Bidone.EDI.Batch.FunctionApps
{
public class SetBlobIndex
{
private readonly ILogger<SetBlobIndex> _logger;
public SetBlobIndex(ILogger<SetBlobIndex> logger)
{
_logger = logger;
}
[Function(nameof(SetBlobIndex))]
public async Task Run([BlobTrigger("edi-batch-extract-archive/{name}",
Connection = "AzureWebJobsStorage")]
Stream stream,
string name)
{
using var blobStreamReader = new StreamReader(stream);
var content = await blobStreamReader.ReadToEndAsync();
_logger.LogInformation($"C# Blob trigger function Processed blob\n Name: {name} \n Data: {content}");
}
}
}The third try is to create the blob trigger on an app service plan instead of a consumption plan as suggested by Azure Blob Storage trigger Function not firing – Stack Overflow. Deploying to a premium app service plan solves the issue 😂. It removes many poison messages in the beginning.
Summary so far
An Azure Blob Storage Trigger Function is firing in our Test but not in our UAT subscription. After a while we see a warning “Possible thread pool starvation detected. The same Azure Blob Storage Trigger Function is deployed to both environments.
App service plan: Consumption
Trigger: Blob Storage Trigger
Runtime: dotnet or dotnet-isolated
Function code: Boiler plate from template
Changing the deployment to the settings below solves the issue.
App service plan: Premium
Trigger: Blob Storage Trigger
Runtime: dotnet-isolated
Function code: My code
But why does a consumption plan not work?
