feat(train): Auto-detect subscription recipe hyperparameters in SFTTr…#5844
Merged
mollyheamazon merged 1 commit intoMay 11, 2026
Merged
Conversation
mollyheamazon
requested changes
May 11, 2026
…ainer
When a model has subscription-gated recipes (IsSubscriptionModel: true
in RecipeCollection), automatically attempt to fetch the recipe's
override_params from the S3 access point and merge additional
hyperparameter keys into the trainer's _specs schema.
This allows subscribed users to natively set datamix hyperparameters
(e.g. customer_data_percent, nova_*_percent) via trainer.hyperparameters
without any explicit flag or workaround.
For non-subscribed users, the fetch fails silently (AccessDenied) and
only standard recipe hyperparameters are available. The extra latency
only occurs when subscription recipes exist in the hub metadata.
Changes:
- After loading standard override_params, check if any recipe has
IsSubscriptionModel: true
- If found: resolve {customer_id} placeholder with caller's account ID,
download override_params from access point, merge extra keys
- Handle S3 access point ARN URI format for GetObject
- Silent fallback on failure (non-subscribed users unaffected)
- Add unit tests for positive, negative, and fallback cases
bba6d97 to
5489954
Compare
mollyheamazon
approved these changes
May 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Auto-detect subscription recipe hyperparameters in SFTTrainer
Issue
When using
SFTTrainerwith models that have subscription-gated recipes (e.g., Nova Forge datamix), the trainer's hyperparameter schema (_specs) only includes keys from the standard recipe. Subscription recipe hyperparameters likecustomer_data_percentandnova_*_percentare rejected withAttributeError.This is because the subscription recipe's
SmtjOverrideParamsS3Uripoints to an S3 access point that requires the customer to be subscribed. The SDK currently only downloads override_params from the standard (publicly accessible) recipe.Solution
After loading the standard recipe's override_params, auto-detect if any recipe in
RecipeCollectionhasIsSubscriptionModel: true. If found, attempt to download its override_params from the S3 access point using the customer's credentials and merge the additional keys into_specswithdefault: None.Key behaviors:
_specs(settable) but withdefault: NoneNonedefault are NOT serialized into_dict()unless explicitly set by the userChanges
sagemaker-train/src/sagemaker/train/common_utils/finetune_utils.py:{customer_id}placeholder in the S3 URI with caller's account IDdefault: None(non-destructive, won't serialize unless set)sagemaker-train/tests/unit/train/common_utils/test_finetune_utils.py:Testing
Validated end-to-end in IAD gamma with both cases:
Datamix job (user sets datamix HPs):
trainer.hyperparameters.customer_data_percent = 70works nativelynova_lite_2_0_p5_gpu_sft_text_with_datamix)Non-datamix job (user only sets standard HPs):
nova_lite_2_0_p5_gpu_sft)All 58 unit tests passing.
Customer Experience
Subscribed user wanting datamix:
Subscribed user NOT wanting datamix:
Non-subscribed user: