flatShaded mode will render black if viewport is not set to flatShaded prior to capture

[joshdicarlo]

Sporadic issue in maya2022, I run a series of 4 playblasts where the main modifications are to the viewport_options:

{
  'displayAppearance': 'flatShaded' if flat_shaded==True else 'smoothShaded',
  'displayLights': 'all' if lit==True else 'default',
  'displayTextures': 1 if tex==True else 0,
}

A simplified sample function would look like (sorry, haven't tested the exact function, but should have all the main bits):

def custom_capture(name, tex = True, flat_shaded = False, lit = True, start_frame=0, end_frame=10):
  cam_name = 'persp'
  viewport_ptions = {
      'displayAppearance': 'flatShaded' if flat_shaded==True else 'smoothShaded',
      'displayLights': 'all' if lit==True else 'default',
      'displayTextures': 1 if tex==True else 0}
  display_options = {
            "displayGradient": 0,
            "background": (0, 0, 0),
            "backgroundTop": (0.535, 0.617, 0.702),
            "backgroundBottom": (0.052, 0.052, 0.052)}
  viewport2_options = {
            "multiSampleEnable" : 1,
            "lineAAEnable" : 1,
            "ssaoEnable": 1,
            "enableTextureMaxRes": 1}
  capture.capture(camera=cam_name, start_frame=start, end_frame=end, filename=name, format="qt", compression="none", width=2048, height=2048, quality=100, maintain_aspect_ratio=False, overwrite=True, viewport_options=viewport_options, viewer=True, off_screen=True, display_options=display_options, viewport2_options=viewport2_options)

Then these are run in the following sequence:

# generate smooth shaded, lit, with textures
custom_capture(name = 'smooth_lit_tex',tex=True, flat_shaded = False, lit = True)
# generate flat shaded, lit, with no textures
custom_capture(name = 'flat_lit_notex',tex=False, flat_shaded = True, lit = True)
# generate flat shaded, lit, with textures
custom_capture(name = 'flat_lit_tex',tex=True, flat_shaded = True, lit = True)
# generate flat shaded, unlit, no textures
custom_capture(name = 'flat_unlit_notex',tex=False, flat_shaded = True, lit = 0)

It seems that fairly consistently, if the active viewport is not set to flatShaded - flatShaded playblasts turn out blank. Similarly true with the "displayTextures" setting. If the main viewport is not first set, the renders turn out blank.

A bit difficult to put your finger on, though, as in some sessions everything works. But upon opening a clean maya session and re-running, typically any except the "smooth_lit_tex" one above will turn out black.

The hacky workaround I found was to temporarily force all modelPanel's to use my target "viewport_settings" in code, run the capture command, and then reset. The suspicion being that the panel created within the capture command is actually inheriting something from the viewports and it's not being successfully overwritten due to some maya 2022 bug.

Platform: Windows 10
Maya Version: 2022.3.0-18.0.PFIX

[BigRoy]

Great report - thanks! However, it would still be even better if we can point to the specific line of code or snippet that "always" makes it happen so we have a reproducable. I've seen issues before with consecutive playblasts where some things were not getting inherited.

If there's any chance you could investigate slightly deeper that would be of great help.
Could you try hardcoding a cmds.refresh(force=True) call between your custom captures. Does that help?

[joshdicarlo]

Yeah, I hard coded that into capture.py right before the playblast command is called just to get as close to the source as possible and still had the issue.

The hack that I put in place just to force things was a line right before capture runs that looks like this:

panels = cmds.getPanel( type='modelPanel' ) or []
panel_configs = {}
for panel in panels:
  panel_configs[panel] = {k:eval(f"cmds.modelEditor('{panel}',q=True,{k}=True)") for k in viewport_options }
  cmds.modelEditor(panel, edit=True, **viewport_options)

I restore it after capture runs like this:

for panel in panels:
  panel_config = panel_configs.get(panel)
  if panel_config:
    cmds.modelEditor(panel, edit=True, **panel_config)

Not elegant, but just a clue as to what's going on. So far this has resolved my issue.

[joshdicarlo]

The most reproduceable case I found was:

1. Open Maya
2. Open Scene you want to playblast (should have some lights and a mesh)
3. Set maya viewport to smooth shade all, textured, lit (7 key).
4. In capture viewport options, change { 'displayAppearance': 'flatShaded' } when running

viewport_options['displayAppearance'] = 'flatShaded'

capture.capture(
  camera=cam_name, 
  start_frame=start, 
  end_frame=end, 
  filename='my_test_pb', 
  format="qt", 
  compression="none", 
  width=2048, 
  height=2048, 
  quality=100, 
  maintain_aspect_ratio=False, 
  overwrite=True, 
  viewport_options=viewport_options, 
  viewer=True, 
  off_screen=True, 
  display_options=display_options, 
  viewport2_options=viewport2_options)

This fairly consistently produced black renders for me. If it doesn't the first time, try again by first rendering with smoothShaded

My default settings for those params are:

viewport_options = {'cameras': 0,
                        'controlVertices': 0,
                        'deformers': 0,
                        'dimensions': 0,
                        'displayAppearance': 'smoothShaded',
                        'displayLights': 'all',
                        'displayTextures': 1,
                        'dynamicConstraints': 0,
                        'dynamics': 0,
                        'fluids': 0,
                        'fogColor': [0.5, 0.5, 0.5, 1.0],
                        'fogDensity': 0.10000000149011612,
                        'fogEnd': 100.0,
                        'fogMode': 'linear',
                        'fogStart': 0.0,
                        'fogging': 0,
                        'follicles': 0,
                        'grid': 0,
                        'hairSystems': 0,
                        'handles': 0,
                        'headsUpDisplay': 0,
                        'hulls': 0,
                        'ikHandles': 0,
                        'imagePlane': 0,
                        'joints': 0,
                        'lights': 0,
                        'locators': 0,
                        'manipulators': 0,
                        'motionTrails': 0,
                        'nCloths': 0,
                        'nParticles': 0,
                        'nRigids': 0,
                        'nurbsCurves': 0,
                        'nurbsSurfaces': 0,
                        'pivots': 0,
                        'planes': 0,
                        'polymeshes': 1,
                        'rendererName': 'vp2Renderer',
                        'selectionHiliteDisplay': 0,
                        'shadows': 1,
                        'strokes': 0,
                        'subdivSurfaces': 0,
                        'textures': 0,
                        'useDefaultMaterial': 0,
                        'wireframeOnShaded': 0}

display_options = {
            "displayGradient": 0,
            "background": (0, 0, 0),
            "backgroundTop": (0.535, 0.617, 0.702),
            "backgroundBottom": (0.052, 0.052, 0.052),
        }
viewport2_options = {
            "multiSampleEnable" : 1,
            "lineAAEnable" : 1,
            "ssaoEnable": 1,
            "enableTextureMaxRes": 1
        }

In most cases merely changing 'smoothShaded' to 'flatShaded' (while the current viewport is in 'Smooth Shade all') will reproduce the issue.

My viewport options are explicit rather than sparse here - made by querying the maya modelPanel settings for all of the ViewportOptions keys in capture.py when I had the viewport set precisely as desired in maya - just to rule out maya setting extra options I was unaware of when using flatShaded.