USD and Farm Rendering

[MustafaJafar]

Is there an existing issue for this?

• I have searched the existing issues and added correct labels.

Description

Current Behavior

Rendering locall work but rendering on farm doesn't.

I tried figuring out the problem so here are what I've found so far:

• Typically, the ayonUsdResolver plugin is added on run time. However, the hook responsible for that doesn't run on farm causing the resolver to not be present, see here
• once the hook works, the resolver logs

  2025-08-07 19:03:00:  0: STDOUT: [2025-08-07 19:03:00.610] [warning] not logged in 401 
  

  after some debugging, it turns out it fails to login becasue the expected value of AYON_API_KEY is not the AYON API, but rather it's a bearer token. we get one when we login to the server. this is mentioned in my example about testing the AYON CPP API in 360 fix outdated example of cpp api ayon-documentation#382 . yes, this is not related to the resolver itself, it's related to the AYON cpp api.
• about pinning
  • When this logic in extract pinning runs in Houdini, it always exports the pinning file to a temp location which is deleted shortly after submitting the DL jobs. which I assume is wrong as the DL jobs expects to access that file.
  • PINNING_FILE_PATH I don't see this environment variable set on the submitted DL Job.
  • The UsdPinningRoot hook has a bug here where it should be roots = anatomy.roots

I'll keep updating this list.

Steps To Reproduce:

you can reproduce it by publishing a render while loading a product that uses entity URI. e.g., in Houdini

Additional context:

Version

Version of the server, ayon-core and the addon that's affected

What platform were you running when you found the bug?

Windows
Linux - Rocky9
Linux - Ubuntu
MacOS

You can also provide any additional information relevant to the issue.

Relevant log output:

[MustafaJafar]

For reference, I was able to make the Houdini DL job run successfully by:

1. add farm to launch types in the resolver init hook
   

   ayon-usd/client/ayon_usd/hooks/pre_resolver_init.py

   Line 17 in 7b15373

   launch_types = {LaunchTypes.local}

   launch_types = {LaunchTypes.local, LaunchTypes.farm_publish}

2. remove the () from
   

   ayon-usd/client/ayon_usd/hooks/usd_pinning_root.py

   Line 24 in 7b15373

   roots = anatomy.roots()

   roots = anatomy.roots

3. Remove the first try except from here. this fixes the issue that pinning file is added to a temp location. this aligns with the current status of Houdini.
   

   ayon-usd/client/ayon_usd/plugins/publish/extract_skeleton_pinning_json.py

   Lines 59 to 67 in 7b15373

   	try:
   	staging_dir = Path(instance.data["stagingDir"])
   	except KeyError:
   	self.log.debug("No staging directory on instance found.")
   	try:
   	staging_dir = Path(instance.data["ifdFile"]).parent
   	except KeyError as e:
   	self.log.error("No staging directory found.")
   	raise KnownPublishError("Cannot determine staging directory.") from e

           try:
               staging_dir = Path(instance.data["ifdFile"]).parent
           except KeyError as e:
               self.log.error("No staging directory found.")
               raise KnownPublishError("Cannot determine staging directory.") from e

4. Add environment variables to my submitted DL Job. I did it via the DL monitor.
   

---

Also, in USD addon settings, I've pushed the level of the logging.

so, my log looks like this

2025-08-07 21:35:20:  0: STDOUT: Loading plugin 'ayonUsdResolver'.
2025-08-07 21:35:20:  0: STDOUT: [2025-08-07 21:35:20.000] [info] AyonApi::loadEnvVars()
2025-08-07 21:35:20:  0: STDOUT: [2025-08-07 21:35:20.000] [info] Loaded AYON_API_KEY and AYON_SERVER_URL
2025-08-07 21:35:20:  0: STDOUT: [2025-08-07 21:35:20.000] [info] Found SideId
2025-08-07 21:35:20:  0: STDOUT: [2025-08-07 21:35:20.000] [info] Loaded Environment Variables(funcEnd): AYON_API_KEY=fa936d1e36c27ab47d706f7d361a141db6f2b5ebdb7456df44f80130a19c6c99, AYON_SERVER_URL=http://192.168.1.12:5000, AYON_SITE_ID=military-mouse-of-jest
2025-08-07 21:35:20:  0: STDOUT: [2025-08-07 21:35:20.000] [info] Init AyonServer httplib::Client
2025-08-07 21:35:20:  0: STDOUT: [2025-08-07 21:35:20.000] [info] Constructor Getting Site Roots
2025-08-07 21:35:20:  0: STDOUT: [2025-08-07 21:35:20.000] [info] AyonApi::getSiteRoots()
2025-08-07 21:35:20:  0: STDOUT: [2025-08-07 21:35:20.000] [info] AyonApi::GET(/api/projects/Experiments/siteRoots)
2025-08-07 21:35:20:  0: STDOUT: [2025-08-07 21:35:20.019] [info] found site Roots: 
2025-08-07 21:35:20:  0: STDOUT: [2025-08-07 21:35:20.019] [info] configs, H:\AYON\configs
2025-08-07 21:35:20:  0: STDOUT: [2025-08-07 21:35:20.019] [info] work, H:\AYON\projects
2025-08-07 21:35:20:  0: STDOUT: Begin Path Mapping
2025-08-07 21:35:21:  0: STDOUT: b''
2025-08-07 21:35:22:  0: STDOUT: b''
2025-08-07 21:35:22:  0: STDOUT: End Path Mapping
2025-08-07 21:35:22:  0: STDOUT: ROP type: usdrender
2025-08-07 21:35:22:  0: STDOUT: Rendering frame 1001 to 1003
2025-08-07 21:35:22:  0: STDOUT: Loading plugin 'usd'.
2025-08-07 21:35:22:  0: STDOUT: Loading plugin 'usdNc'.
2025-08-07 21:35:22:  0: STDOUT: [2025-08-07 21:35:22.122] [info] AyonApi::resolvePath(ayon+entity://Experiments/assets/shapes?product=usdAsset&version=v004&representation=usd)
2025-08-07 21:35:22:  0: STDOUT: [2025-08-07 21:35:22.122] [info] AyonApi::SPOST endPoint: /api/resolve?pathOnly=true, jsonPayload: {"resolveRoots":false,"uris":["ayon+entity://Experiments/assets/shapes?product=usdAsset&version=v004&representation=usd"]}, sucsessStatus: 200
2025-08-07 21:35:22:  0: STDOUT: [2025-08-07 21:35:22.122] [info] AyonApi::serialCorePost() endPoint: /api/resolve?pathOnly=true, Payload: {"resolveRoots":false,"uris":["ayon+entity://Experiments/assets/shapes?product=usdAsset&version=v004&representation=usd"]}, sucsessStatus: 200
2025-08-07 21:35:22:  0: STDOUT: [2025-08-07 21:35:22.140] [info] AyonApi::getAssetIdent({"entities":[{"filePath":"{root[work]}\\Experiments\\assets\\shapes\\publish\\usd\\usdAsset\\v004\\Expr_shapes_usdAsset_v004.usd"}],"uri":"ayon+entity://Experiments/assets/shapes?product=usdAsset&version=v004&representation=usd"})
2025-08-07 21:35:22:  0: STDOUT: [2025-08-07 21:35:22.140] [info] AyonApi::rootReplace({root[work]}\Experiments\assets\shapes\publish\usd\usdAsset\v004\Expr_shapes_usdAsset_v004.usd)
2025-08-07 21:35:22:  0: STDOUT: [2025-08-07 21:35:22.140] [info] AyonApi::rootReplace(H:\AYON\projects\Experiments\assets\shapes\publish\usd\usdAsset\v004\Expr_shapes_usdAsset_v004.usd) rooted
2025-08-07 21:35:22:  0: STDOUT: Loading plugin 'arHFs'.
2025-08-07 21:35:22:  0: STDOUT: Loading plugin 'usdHGeo'.
2025-08-07 21:35:22:  0: STDOUT: Finished Rendering
2025-08-07 21:35:23:  0: STDOUT: [2025-08-07 21:35:23.116] [info] AyonApi::~AyonApi()

[MustafaJafar]

Tagging @philippe-ynput @antirotor @BigRoy for visibility.

[BigRoy]

@MustafaJafar can you make a PR with the required changes please?

[MustafaJafar]

@MustafaJafar can you make a PR with the required changes please?

Replied in DMs. but let me quote: Some of these change can be done quickly but some other require some time for fixing and cleaning it properly.

The fixes include:

• (fixed by Chore: Auto-detect service api key or user token ayon-cpp-api#38) Enhancing the login/auth logic in the AYON C++ api to support loggin is as service. AYON CPP API doesn't support using AYON serivce api key ayon-cpp-api#37
• (fixed by Clean up usd publish plugins #93) Fix launch types in ayon-usd/client/ayon_usd/hooks/pre_resolver_init.py
• (fixed by Clean up usd publish plugins #93) Fix typo in ayon-usd/client/ayon_usd/hooks/usd_pinning_root.py
• (fixed by Clean up usd publish plugins #93) Fix getting staging dir in ayon-usd/client/ayon_usd/plugins/publish/extract_skeleton_pinning_json.py
• Add something to set the PINNING_FILE_PATH on submitting DL jobs. its value should be the path to pinning file from extract_skeleton_pinning_json.py.
• (fixed by Clean up usd publish plugins #93 and Remove ayon-usd related plugin. ayon-deadline#197) [Not required but recommended] Checking if it works as expected and moving it to USD addon https://github.com/ynput/ayon-deadline/blob/develop/client/ayon_deadline/plugins/publish/global/collect_usd_pinning_env_vars.py

It can be fixed in multiple small PRs but I prefer to make a big PR as they refer to one topic/feature USD resolver and farm publishing.
And, we can also set some temporal solutions in case we post poned the cpp fix.

[MustafaJafar]

PR #92 partially solves this issue. we might need additional work for adding environment variables to submitted DL Job and the AYON_API_KEY issue.

[philippe-ynput]

@MustafaJafar can you please make a single PR for this issue ?

[MustafaJafar]

@MustafaJafar can you please make a single PR for this issue ?

I've some draft PRs that should partially fix this issue.

• Clean up usd publish plugins #93
• Remove ayon-usd related plugin. ayon-deadline#197

I'm going to give them another test and mark them as ready for PR reviewing.

For this issue to be completely fixed:

1. we need this PR 17 ayonapiget does not check for failed httplib connections ayon-cpp-api#35 merged.
2. Add some plugin in deadline addon to set the PINNING_FILE_PATH environment on DL jobs. Although, I'd prefer to disable pinning feature and skip this fix for now. and wait until we make a decision about what should happen to the pinning mode.

Please @philippe-ynput let me know if I need to do any further actions.

[MustafaJafar]

Update:
PR #93 Should Fix farm rendering with resolver enabled / pinning generation file enabled. (The current status of the PR should work although, the PR need some cleaning up.)
But, for this issue to be resolved, we would need a new release the AYON USD Resolver with the CPP API fix ynput/ayon-cpp-api#35