Skip to content

{Dataprotection} Preserve backup vault failures and enforce readiness #10314

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
FumingZhang wants to merge 1 commit into
base: main
Choose a base branch
from
+426 −76
Draft

{Dataprotection} Preserve backup vault failures and enforce readiness #10314

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions src/dataprotection/HISTORY.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@
Release History
===============

1.12.1
++++++
* `az dataprotection enable-backup trigger`: Preserve backup vault discovery and creation errors, including full service errors when storage-type fallback fails. Preserve service-error fallback without masking local validation or programming errors. Require successful provisioning for new and reused vaults before assigning roles; fail on readiness timeout instead of proceeding with an unready vault.

1.12.0
++++++
* Added dataprotection support for the AzureElasticSAN (Elastic SAN volume group) workload: new manifest (Microsoft.ElasticSan/elasticSans/volumeGroups), registration in supported datasource types and datasource map. New backup configuration via ``az dataprotection backup-instance initialize-backupconfig --datasource-type AzureElasticSAN --resource-selectors`` (GenericBackupDatasourceParameters) and restore configuration via ``az dataprotection backup-instance initialize-restoreconfig --datasource-type AzureElasticSAN --resource-identifiers/--resource-name-overrides`` (GenericRestoreDatasourceCriteria), with data-recovery and item-recovery restore wiring plus help/examples. The AzureElasticSAN backup-instance operations are pinned to the GA 2026-06-01 DataProtection API, which defines GenericBackupDatasourceParameters/GenericRestoreDatasourceCriteria natively.
Expand Down
128 changes: 60 additions & 68 deletions src/dataprotection/azext_dataprotection/manual/aks/aks_helper.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,9 @@
# --------------------------------------------------------------------------------------------

import json
from azure.cli.core.azclierror import InvalidArgumentValueError, ValidationError
from azure.cli.core.azclierror import AzureResponseError, InvalidArgumentValueError, ValidationError
from azure.cli.core.commands.client_factory import get_mgmt_service_client
from azure.core.exceptions import HttpResponseError, ServiceRequestError, ServiceResponseError
from azure.mgmt.core.tools import parse_resource_id
from knack.log import get_logger
from knack.prompting import prompt_y_n, NoTTYException
Expand Down Expand Up @@ -695,35 +696,26 @@ def _find_existing_backup_vault(

Looks for backup vaults with tag: AKSAzureBackup = <location>

Scoping the ``list`` call to ``backup_resource_group_name`` (derived from
the caller-supplied ``backupResourceGroupId``, or the per-cluster
resource group we just created/validated) is required: without it, every
parallel run/test that happens to omit ``backupResourceGroupId`` shares
the same subscription-wide, tag-matched vault, so one run's
``aks delete``/vault cleanup can race another run's discovery and lookup
(``ResourceGroupBeingDeleted``/404 on the shared vault). Restricting
discovery to the caller's own resource group keeps each run isolated.
Scoping the ``list`` call to the selected backup resource group prevents
reusing a tag-matched vault in another resource group. Listing failures
must propagate rather than being treated as an empty result.

Returns:
backup_vault if found, None otherwise
"""
from azext_dataprotection.aaz.latest.dataprotection.backup_vault import List as _BackupVaultList

try:
list_args = {"subscription": cluster_subscription_id}
if backup_resource_group_name:
list_args["resource_group"] = backup_resource_group_name
vaults = _BackupVaultList(cli_ctx=cmd.cli_ctx)(command_args=list_args)

for vault in vaults:
if vault.get('tags'):
# Check if this vault has the AKS backup tag matching the location
tag_value = vault['tags'].get(AKS_BACKUP_TAG_KEY)
if tag_value and tag_value.lower() == cluster_location.lower():
return vault
except Exception: # pylint: disable=broad-exception-caught
# If we can't list vaults, we'll create a new one
pass
list_args = {"subscription": cluster_subscription_id}
if backup_resource_group_name:
list_args["resource_group"] = backup_resource_group_name
vaults = _BackupVaultList(cli_ctx=cmd.cli_ctx)(command_args=list_args)

for vault in vaults:
if vault.get('tags'):
# Check if this vault has the AKS backup tag matching the location
tag_value = vault['tags'].get(AKS_BACKUP_TAG_KEY)
if tag_value and tag_value.lower() == cluster_location.lower():
return vault
return None


Expand All @@ -741,52 +733,54 @@ def _wait_for_backup_vault_ready(
operations against it will reliably succeed. This uses precise, bounded
polling against the service-visible state rather than a fixed sleep.

Returns the latest vault payload (refreshed via Show) when available;
falls back to whatever was last observed if polling itself fails.
Returns a refreshed vault only when provisioning succeeds. Raises on
terminal failure, non-transient lookup errors, or polling exhaustion.
"""
import time
from azext_dataprotection.aaz.latest.dataprotection.backup_vault import Show as _BackupVaultShow

terminal_states = {"succeeded", "failed", "canceled"}
latest_vault = None
terminal_states = {"succeeded", "failed", "canceled", "cancelled"}
state = ""
last_error = None
for attempt in range(retries):
try:
latest_vault = _BackupVaultShow(cli_ctx=cmd.cli_ctx)(command_args={
"vault_name": backup_vault_name,
"resource_group": backup_resource_group_name,
"subscription": cluster_subscription_id,
})
state = (latest_vault.get("properties", {}) or {}).get("provisioningState", "")
last_error = None
state = (latest_vault.get("properties", {}) or {}).get("provisioningState") or ""
if state.lower() in terminal_states:
if state.lower() != "succeeded":
raise InvalidArgumentValueError(
raise AzureResponseError(
f"Backup vault '{backup_vault_name}' reached terminal "
f"provisioning state '{state}' instead of 'Succeeded'."
)
return latest_vault
except InvalidArgumentValueError:
raise
except Exception: # pylint: disable=broad-exception-caught
# Transient lookup failure (e.g. RP propagation delay); keep retrying.
pass
except HttpResponseError as ex:
if ex.status_code not in {404, 408, 429, 500, 502, 503, 504}:
raise
last_error = ex
except (ServiceRequestError, ServiceResponseError) as ex:
last_error = ex
if attempt < retries - 1:
time.sleep(interval_seconds)
logger.warning(
"Backup vault '%s' did not report a terminal provisioning state "
"after %d retries; proceeding with the last known state.",
backup_vault_name, retries)
return latest_vault
message = (
f"Backup vault '{backup_vault_name}' did not reach provisioning state 'Succeeded' "
f"after {retries} attempts. Last observed state: '{state or 'Unknown'}'."
)
if last_error is not None:
message += f"\nLast lookup error: {last_error}"
raise AzureResponseError(message) from last_error


def _try_create_vault_with_storage_type(
cmd, vault_create_cls, backup_vault_name,
backup_resource_group_name, cluster_location, vault_tags,
storage_type, cluster_subscription_id=None):
"""
Attempt to create a backup vault with the given storage type.

Returns:
backup_vault dict on success, None on failure
Create a backup vault with the given storage type, preserving failures.
"""
backup_vault_args = {
"vault_name": backup_vault_name,
Expand All @@ -808,12 +802,7 @@ def _try_create_vault_with_storage_type(
if storage_type == 'GeoRedundant':
backup_vault_args["cross_region_restore_state"] = "Enabled"

try:
backup_vault = vault_create_cls(cli_ctx=cmd.cli_ctx)(command_args=backup_vault_args).result()
return backup_vault
except Exception as e: # pylint: disable=broad-exception-caught
logger.warning("Vault creation with %s failed: %s", storage_type, str(e)[:120])
return None
return vault_create_cls(cli_ctx=cmd.cli_ctx)(command_args=backup_vault_args).result()


def _setup_backup_vault(
Expand All @@ -823,6 +812,7 @@ def _setup_backup_vault(
"""Create or use backup vault."""
from azext_dataprotection.aaz.latest.dataprotection.backup_vault import Create as _BackupVaultCreate

vault_rg = backup_resource_group_name
if backup_strategy == 'Custom' and backup_vault_id:
# Use provided vault for Custom strategy
vault_parts = parse_resource_id(backup_vault_id)
Expand Down Expand Up @@ -862,34 +852,36 @@ def _setup_backup_vault(
# Try storage types in order of preference: GRS → ZRS → LRS
# Not all regions support all types, so we fall back gracefully.
backup_vault = None
storage_type = None
creation_errors = []
last_error = None

for try_type in ['GeoRedundant', 'ZoneRedundant', 'LocallyRedundant']:
logger.warning("Trying storage type: %s...", try_type)
backup_vault = _try_create_vault_with_storage_type(
cmd, _BackupVaultCreate, backup_vault_name, backup_resource_group_name,
cluster_location, vault_tags, try_type, cluster_subscription_id)
try:
backup_vault = _try_create_vault_with_storage_type(
cmd, _BackupVaultCreate, backup_vault_name, backup_resource_group_name,
cluster_location, vault_tags, try_type, cluster_subscription_id)
except HttpResponseError as ex:
# Preserve service-error fallback, including failures reported by an HTTP 200 LRO poll.
last_error = ex
creation_errors.append(f"{try_type}: {ex}")
logger.warning("Vault creation with %s failed: %s", try_type, ex)
continue
if backup_vault:
storage_type = try_type
logger.warning("Vault created with storage type: %s", storage_type)
logger.warning("Vault created with storage type: %s", try_type)
break

if not backup_vault:
raise InvalidArgumentValueError(
raise AzureResponseError(
f"Failed to create backup vault '{backup_vault_name}' in region '{cluster_location}' "
f"with any storage type (GeoRedundant, ZoneRedundant, LocallyRedundant).\n"
f"Please check region availability and try again."
)
+ "\n".join(creation_errors)
) from last_error

# The vault create LRO can return before the vault's own
# provisioningState (and downstream role-assignment/backup-instance
# eligibility) is fully settled. Wait on the service-visible state
# with bounded retries rather than assuming immediate readiness.
logger.warning("Waiting for backup vault '%s' to become ready...", backup_vault_name)
refreshed_vault = _wait_for_backup_vault_ready(
cmd, backup_vault_name, backup_resource_group_name, cluster_subscription_id)
if refreshed_vault:
backup_vault = refreshed_vault
# Reused vaults may still be provisioning after an earlier or concurrent create.
logger.warning("Waiting for backup vault '%s' to become ready...", backup_vault_name)
backup_vault = _wait_for_backup_vault_ready(
cmd, backup_vault_name, vault_rg, cluster_subscription_id)

logger.warning("Backup Vault: %s", backup_vault['id'])
_check_and_assign_role(
Expand Down
Loading
Loading