Best Way To Fix Issue : Azure Function App Returning 4040

Introduction

Hello Guys! We encountered an unusual error recently while working with our Development team. The Azure Function App was reporting a sporadic, 404 error. This means that 95% of Azure Function requests were successful. However, around 5% were failing with 404 errors without any log or notification.

This blog post will explain why it happened and what we have done to fix this Azure Function App Returning 4040 .

Also Read: error code: m7121-1331

A brief introduction to Function Apps

Before I explain the error and show you how to fix it let me briefly explain Azure Function App Returning 4040. Function apps are server-less technology that allows you to run code (Python or C#, JavaScript and other) at scale.

Function App can scale resources to process many requests. It doesn’t require you to worry about the operating system and hardware. You only have to pay 0.20$ for every million executions, and $0.000016/GBS for every million executions. There is also a free grant of 400000 GBs and a million executions. Prices are effective as of the 25th April 2021 in the West Europe region.

Background story

One of our customers required real-time request processing. The average request was 100k per hour. In peak times, there were 30 requests per second. There was no variation in the workload between days and weeks. Pick times were between 10-12 AM on Mondays and Fridays. The request processing was very simple and took only a few seconds.

We have concluded that Function App and Serverless are a great fit for this project.

Error 404, Function Apps

Our solution was implemented and worked flawlessly at DEV, Test and Production with workloads that were similar to Production. It was flawless for him. All million of requests were processed without issue. All requests (100%) were processed with status code 200.

We noticed that around 5% of requests were getting 404 errors when we deployed to production. We began our investigation in the App Insights to determine the source of the problem. We could not find any errors, which means that the 404 requests never reached our Function App.

This is an unusual issue that we have thought about and have researched possible root causes. This issue is most likely not ours, but is related to the Azure platform.

Deployment of Azure Function Apps and High-Level Architecture

Let’s briefly discuss the Azure Function architecture and the deployment process to understand the root cause. Although I may be simplifying it, this is just to briefly explain what is going on in the background.

Simply put, workers execute your code. The more you request your function, the more workers will be assigned to it. This is how you reach your scale.

What happened in our case was

Visual Studio code was used to create our code.
Azure DevOps Pipelines can be used to help you.
The code was compressed into a zip file, and then went to the deployment into Azure Functions Ecosystems.
It is important to remember that Function App servers may have other functions that are not ours. In our case, we used a consumption plan and there was no isolation.
Our zip package was deployed during the Internal Deployment process by Function Apps.
It successfully deployed our package with function to one of the servers in the Function app cluster
However, some nodes failed to be deployed.
A load balancer responsible for distributing our traffic was notified “where”, but it wasn’t aware of any deployment/caching problems (point 7).
When an External Request was submitted, it was sent to the load balancer, who tried to distribute it evenly.
It was handled correctly by a properly deployed Function App in most cases. We were able to achieve our desired status of 200 in these cases.
Sometimes, however, the load balancer sent our request to a node that did not have our function app. We received our 404 error in these cases. It was difficult to track down.

Investigate and find the root cause

We spent about an hour diagnosing the problem and then it was quite simple to fix it.

We redeployed the Function App once more, and it was deployed correctly to all Function App cluster nodes.

Conclusions

Function apps are very useful and don’t require you to worry about OS, provisioning, etc.
As with all IT technologies, things may go wrong sometimes.
Be cautious when you deploy your Function App. Double-check that it is properly deployed.
It would be useful to have a watchdog that could query your function from the outside to monitor it and detect unexpected Azure Function App Returning 4040.

This blog post was enjoyed by me. Please feel free to comment and give feedback. Perhaps you’ve been in this situation before.