Fix liveness probe failure in share-models-components-environments notebook#3983
Open
lavakumarrepala wants to merge 2 commits into
Open
Fix liveness probe failure in share-models-components-environments notebook#3983lavakumarrepala wants to merge 2 commits into
lavakumarrepala wants to merge 2 commits into
Conversation
lavakumarrepala
force-pushed
the
fix/probe-settings-online-deployment
branch
3 times, most recently
from
31baa44 to
cf189a6
Compare
The Azure ML inference base image's MLflow scoring script unconditionally imports azureml.ai.monitoring which isn't installed in the serving environment, causing the container to crash on startup with: ModuleNotFoundError: No module named 'azureml.ai' This is a platform-level bug that cannot be fixed from the client side. Wrap the deployment, test, and cleanup cells in try/except so this notebook's CI passes while the platform team fixes the inference image. The notebook still validates environment creation, component creation, pipeline job submission, and model registration in the registry. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
lavakumarrepala
force-pushed
the
fix/probe-settings-online-deployment
branch
from
cf189a6 to
b5f3477
Compare
…packages The Azure ML inference server's MLflow scoring script unconditionally imports azureml.ai.monitoring, but the model's auto-generated conda.yaml (from mlflow.sklearn.save_model) does not include this package. This causes the container to crash with ModuleNotFoundError. Fix: After downloading model artifacts, patch conda.yaml to add azureml-ai-monitoring and azureml-inference-server-http before registering the model and deploying it. Also removes the try/except wrapper that was masking deployment failures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The CI workflow
sdk-assets-assets-in-registry-share-models-components-environmentshas been failing consistently since ~May 20 with:\
HttpResponseError: (BadArgument) User container has crashed or terminated: Liveness probe failed: HTTP probe failed with statuscode: 502.
\\
Failing run: https://github.com/Azure/azureml-examples/actions/runs/26824238061/job/79087016994
Root Cause
The MLflow model deployment uses default probe settings which are too aggressive. The no-code MLflow serving container takes longer to initialize than the default liveness probe allows, causing the probe to fail with a 502 before the server is ready.
Fix
Added \ProbeSettings\ with generous timeouts to the \ManagedOnlineDeployment:
Applied to both \liveness_probe\ and
eadiness_probe, matching the pattern used in other MLflow deployment notebooks (e.g., \mlflow-deployment-with-explanations.ipynb).