Skip to content

fix(qa-mc2dp): add disease_type/primary_site to aggMDS GDC adapter mapping - #575

Merged
jawadqur merged 1 commit into
masterfrom
fix/qa-mc2dp-aggmds-gdc-mappings
Nov 18, 2026
Merged

jawadqur merged 1 commit into
masterfrom
fix/qa-mc2dp-aggmds-gdc-mappings

Conversation

@jawadqur

Copy link
Copy Markdown
Contributor

Follow-up to #573. The GDC adapter added there crashes the aggMDS job:

adapters.py:1426, in normalizeToGen3MDSFields
  ] = f"Genomic Data Commons study of {normalized_item['disease_type']} in {normalized_item['primary_site']}"
KeyError: 'disease_type'
Error occurred during mds population. Existing indexes are left in place.

GDCAdapter always reads disease_type and primary_site from the mapped fields, and the mapping didn't define them. So nothing was written, and the old indexes (including the VPODC data we're trying to remove from QA) stayed in place.

This adds the two mappings. I ran MDS master's GDCAdapter and parse_config locally against this config and the live GDC API:

  • before: KeyError: 'disease_type'
  • after: 2 records (VAREPOP-APOLLO, APOLLO-LUAD), so populate.py gets past its "0 records → keep existing indexes" guard and swaps the indexes.

.secrets.baseline was regenerated by the repo's detect-secrets hook (v1.5.0). It drops 3 stale entries for this file that the scanner no longer flags.

https://claude.ai/code/session_019WtPvAbe8JKDofTACKqa7D

…pping

GDCAdapter builds each study description from the mapped disease_type and
primary_site fields, so the mapping added in #573 crashed with
KeyError: 'disease_type' and MDS left the existing indexes (VPODC data) in
place. With the two fields mapped the adapter returns the 2 APOLLO projects,
which lets populate.py swap the indexes and drop the old data.

.secrets.baseline regenerated by the detect-secrets pre-commit hook.

Claude-Session: https://claude.ai/code/session_019WtPvAbe8JKDofTACKqa7D
@jawadqur
jawadqur merged commit a6a995c into master Nov 18, 2026
1 check passed
@jawadqur
jawadqur deleted the fix/qa-mc2dp-aggmds-gdc-mappings branch November 18, 2026 15:31
jawadqur added a commit that referenced this pull request Nov 18, 2026
- Resolve the conflict with #575: keep _unique_id=path:submitter_id and the
  disease_type/primary_site mappings GDCAdapter requires.
- subject_type_of_exposures: map exposures[*].tobacco_smoking_status (as prod
  mc2dp does). "path:exposures" yields a list of dicts, which the string
  schema type can't convert, crashing the job with
  "TypeError: sequence item 0: expected str instance, dict found".
- Append ",id" to the cases fields list. GDCAdapter appends
  "?expand=summary&from=..&size=.." to mds_url, which gets glued onto the
  last field, so project.project_id was never returned
  (data_source_project was null for every record).
- Restore the closing brace missing from .secrets.baseline on master, which
  made the detect-secrets hook fail.

Tested locally with MDS master's GDCAdapter against the live GDC API:
232 subjects (APOLLO-LUAD 87, APOLLO-OV 70, APOLLO-BRCA-1 68, VAREPOP-APOLLO 7).

Claude-Session: https://claude.ai/code/session_019WtPvAbe8JKDofTACKqa7D
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant