support custom TopCP images

[MattShirley]

Backend work for supporting custom TopCP images from whitelisted Docker registry repositories: ssl-hep/ServiceX_frontend#658

Allowed images can be defined using the codeGen.topcp.allowedImages value, where the value should be a list of strings. Each string in the list will be tested to see if it is a valid prefix of the custom image. If it is, it will be allowed. The default value will allow any tags for the sslhep/servicex_science_image_topcp repository.

codeGen:
  topcp:
    allowedImages:
      - "sslhep/servicex_science_image_topcp:"

Sample files that work with a custom TopCP image but not with the default/base image:

# reco.yaml
CommonServices:
  runSystematics: False

PileupReweighting: {}

EventCleaning: {}

Trigger:
    triggerChainsPerYear:
        '2015':
            - 'HLT_2mu6_bBmumuxv2'
            - 'HLT_2mu4_bBmumuxv2'
        '2016':
            - 'HLT_2mu6_bBmumux_BsmumuPhi_delayed'
            - 'HLT_2mu6_bBmumux_BsmumuPhi_delayed_L1BPH-2M9-2MU6_BPH-2DR15-2MU6'
            - 'HLT_2mu6_bBmumux_BsmumuPhi_L1BPH-2M9-2MU6_BPH-2DR15-2MU6'
            - 'HLT_mu10_mu6_bBmumux_BsmumuPhi_delayed'
            - 'HLT_2mu6_bBmumuxv2'
            - 'HLT_2mu6_bBmumuxv2_delayed'
        '2017':
            - 'HLT_mu6_mu4_bBmumux_BsmumuPhi_L1BPH-2M9-MU6MU4_BPH-0DR15-MU6MU4'
            - 'HLT_2mu6_bBmumux_BsmumuPhi_L1BPH-2M9-2MU6_BPH-2DR15-2MU6'
            - 'HLT_2mu6_bJpsimumu_L1BPH-2M9-2MU6_BPH-2DR15-2MU6'
            - 'HLT_2mu10_bBmumuxv2'
        '2018':
            - 'HLT_mu6_mu4_bBmumux_BsmumuPhi_L1BPH-2M9-MU6MU4_BPH-0DR15-MU6MU4'
            - 'HLT_2mu6_bBmumux_BsmumuPhi_L1BPH-2M9-2MU6_BPH-2DR15-2MU6'
            - 'HLT_2mu6_bJpsimumu_L1BPH-2M9-2MU6_BPH-2DR15-2MU6'
            - 'HLT_2mu10_bBmumuxv2'
    noFilter: True
    noGlobalTriggerEff: True

Muons:
  - containerName: AnaMuons
    WorkingPoint:
      - selectionName: loose
        quality: Loose
        isolation: NonIso
      - selectionName: medium
        quality: Medium
        isolation: NonIso

BPhyVertex:
  - vertexName: 'BPHY28BsKKMuMuCandidates'
    decorations: []


BPHY28:
  - vertexName: 'BPHY28BsKKMuMuCandidates'
    inputMuons: 'AnaMuons'
    decorateTrackParameters: True
    decoratePixelHits: True
    decorateDedx: True

# After configuring each container, many variables will be saved automatically.
Output:
  treeName: 'reco'
  vars: []
  metVars: []
  containers:
      # Format should follow: '<suffix>:<output container>'
#      vtx_: 'BPHY28BsKKMuMuCandidates'
      mu_: 'AnaMuons'
      '': 'EventInfo'
  commands: []

AddConfigBlocks:
  - modulePath: 'TopCPToolkit.BPHY28Config'
    functionName: 'BPHY28Config'
    algName: 'BPHY28'
    pos: 'Output'
  - modulePath: 'TopCPToolkit.BPhyVertexConfig'
    functionName: 'BPhyVertexConfig'
    algName: 'BPhyVertex'
    pos: 'Output'

# spec.yaml
General:
  OutFilesetName: "fileset_ntuple_v1_af"
  OutputDirectory: "."

Sample:
  ## data18
  - Name: data18_periodD
    Dataset: !Rucio data18_13TeV:DAOD_BPHY28.44180527._000001.pool.root.1
    Query: !TopCP |
      reco=reco.yaml,
      docker_image=sslhep/servicex_science_image_topcp:25.2.50-v2.18.0_v0.2

[codecov]

Codecov Report

❌ Patch coverage is 96.96970% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 86.16%. Comparing base (7095747) to head (19601a7).

Files with missing lines	Patch %	Lines
...PToolkit/servicex/TopCP_code_generator/__init__.py	87.50%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1163      +/-   ##
===========================================
+ Coverage    85.92%   86.16%   +0.23%     
===========================================
  Files           94       94              
  Lines         3233     3260      +27     
  Branches       371      378       +7     
===========================================
+ Hits          2778     2809      +31     
+ Misses         380      378       -2     
+ Partials        75       73       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[BenGalewsky]

Is there any way we can move the image selection retrieval and verification into the TopCP CodeGen? I really don't like having a hardcoded test for topcp in the submit transform handler.

I don't think we need to change the app at all to support this special case

[BenGalewsky]

Why iscustom_image still here?

[BenGalewsky]

Do we still need this in the App config?

[BenGalewsky]

Could this be moved to the codegen's __init__ so that if there is a formatting error in the ALLOWED_IMAGES it will fail at startup instead of the first time someone submits?

[MattShirley]

The function call can't go in __init__.py because there's nothing to compare it to. However, I added a check in create_app to validate the JSON parsing of of TOPCP_ALLOWED_IMAGES. And I removed type guarding from validate_custom_docker_image.