Monitoring path tracer frame accumulation

[jakubtomsu]

This adds a SpFunc to query the internal path tracer frame accumulation counter (previously not available from python), and updates the PT example frame loop to make use of this counter instead of blindly stepping.

The reason why this is even a problem in the first case is not 100% clear to me, but I have run into issues with the frame counting many times before. Even if we call instance.step(num_frames=num_pt_frames, single_step=True) with the correct frame count and try really hard to ensure the scene never gets invalidated, it still sometimes breaks. Just this morning I ran the path tracing example with 64 samples and got this (on the last commit and something similar on the one before that):

This is clearly completely wrong and nowhere close to the 64 samples, possibly because async loading. Of course, it would be really nice to properly fix the loading too, but this adaptive frame loop at least makes sure the render is correct even in non-perfect conditions.

[mikeroberts3000]

I disagree with this proposed approach. It is brittle (has a hardcoded index 0), will increasingly clutter the capture component implementation as it grows (when we want to return a different combination of rendering stats we will need to constantly add more clutter), doesn't compose well (the capture component is forced to become a wrapper for the lower-level FSceneViewStateInterface class in addition to all of its other responsibilities), and is not a good fit for what SpFunctions are useful for (passing NumPy arrays to and from Python).

It would be better to create a wrapper class in SpUnrealTypes in a separate file called SpSceneViewStateInterface.h like this:

#include <Kismet/BlueprintFunctionLibrary.h>
#include <Whatever/HeaderIsNeededForFSceneViewStateInterface.h>

#include "SpSceneViewStateInterface.generated.h"

UCLASS()
class USpSceneViewStateInterface : public UBlueprintFunctionLibrary
{
    GENERATED_BODY()
public: 
    UFUNCTION(BlueprintCallable, Category="SPEAR")
    static uint32 GetPathTracingSampleIndex(uint64 ViewState)
    {
        #if RHI_RAYTRACING
            FSceneViewStateInterface* view_state_ptr = reinterpret_cast<FSceneViewStateInterface*>(ViewState);
            SP_ASSERT(view_state_ptr);
            return view_state_ptr->GetPathTracingSampleIndex();
        #else
            return 0;
        #endif
    }

    UFUNCTION(BlueprintCallable, Category="SPEAR")
    static uint32 GetPathTracingSampleCount(uint64 ViewState)
    {
        #if RHI_RAYTRACING
            FSceneViewStateInterface* view_state_ptr = reinterpret_cast<FSceneViewStateInterface*>(ViewState);
            SP_ASSERT(view_state_ptr);
            return view_state_ptr->GetPathTracingSampleCount();
        #else
            return 0;
        #endif
    }
};

I like this approach better because it doesn't clutter the capture component and is completely out of the way (no other C++ source file even needs to reference it). It can grow organically as we want to expose more rendering stats because we can just add more functions to USpSceneViewStateInterface (not a big deal because it is so out of the way). Many of the classes in SpUnrealTypes are wrappers that follow this pattern.

To support my proposed approach, I've already created a GetViewStates UFUNCTION that returns an array of pointers cast to uint64 (necessary because FSceneViewStateInterface is not visible to the reflection system and is therefore not allowed in UFUNCTION signatures).

From Python, a user could do something like:

sp_scene_view_state_interface = game.get_unreal_object(uclass="USpSceneViewStateInterface")

# ...

view_states = component.GetViewStates()
if len(view_states) > 0: # 0 index assumption lives in user code
    view_state = view_states[0]
    sample_index = sp_scene_view_state_interface.GetPathTracingSampleIndex(ViewState=view_state)
    sample_count = sp_scene_view_state_interface.GetPathTracingSampleCount(ViewState=view_state)
else:
    sample_index = 0
    sample_count = 0

[jakubtomsu]

Right, the separate UFUNCTION wrapper does indeed look nice. Thank you for the suggestion, I used SpFunc as it was the easiest/quickest way to get it working.

[jakubtomsu]

I will update the PR with your code and run my tests, unless you want to do the changes yourself

[mikeroberts3000]

Please go ahead 😄

[mikeroberts3000]

SP_ASSERT(view_state_ptr);

[mikeroberts3000]

This is a bug. BP_CameraSensorPathTracer is needed because the example code isn't setting the show flags for path tracing any more. That only happens in BP_CameraSensorPathTracer. So we need to use BP_CameraSensorPathTracer.

[mikeroberts3000]

assert len(view_states) > 0
view_state = view_states[0]
sample_index = sp_scene_view_state_interface.GetPathTracingSampleIndex(ViewState=view_state)
sample_count = sp_scene_view_state_interface.GetPathTracingSampleCount(ViewState=view_state)

[mikeroberts3000]

Not needed.

[mikeroberts3000]

Adjusting show flags not needed because now it happens in the definition of the camera sensor.

[mikeroberts3000]

        # When path tracing begins, Unreal begins asynchronously building various Nanite data structures, and
        # as these data structures finish building, they internally reset the progress of the path tracer.
        # Unfortunately Unreal doesn't expose any way to check if these asynchronous build tasks are complete,
        # so we proceed with a heuristic approach that attempts to front-load the build tasks as much as
        # possible, and then we render a conservative number of warm-up frames to ensure the build tasks are
        # complete before we start rendering a final image.

        game.console_service.set(name="r.RayTracing.Nanite.ForceUpdateVisible", value=1)                     # force all Nanite builds to be scheduled on a single frame
        game.console_service.set(name="r.RayTracing.Nanite.MaxBuiltPrimitivesPerFrame", value=256*1024*1024) # set the max number of built Nanite primitives per frame to be very large (default is 8*1024*1024)
        game.console_service.set(name="r.RayTracing.Nanite.MaxStagingBufferSizeMB", value=2048)              # set the size of the Nanite staging buffer to be large (default is 1024)

[mikeroberts3000]

Setting bUseSceneViewExtension explicitly is not needed any more. I changed the camera sensor to auto-detect if a scene view extension is needed based on the show flags and various other things.

[mikeroberts3000]

    # inserting an extra frame or two can fix occasional render-to-texture initialization issues (advances a minimum of 3 frames)
    game.async_loading_service.wait_for_engine_idle()

    spear.log("Rendering warm-up frames...")

    # advance an extra few frames to give Nanite a chance to finish building its data structures
    instance.step(num_frames=args.num_warmup_frames)

    # force Nanite to stop building its data structures so it can't invalidate the path tracer any more
    with instance.begin_frame():
        game.console_service.set(name="r.RayTracing.Nanite.Update", value=0)
    with instance.end_frame():
        pass
   
    spear.log("Finished rendering warm-up frames.")

    # The path tracer accumulates one sample per pixel per rendered frame, exactly like the editor's
    # path-tracing viewport, and stops once it reaches r.PathTracing.SamplesPerPixel (set to args.num_frames
    # above). Moving the camera or anything in the scene invalidates the accumulated samples and restarts
    # from scratch. We poll the internal path tracer accumulation counter to ensure that the image has been
    # fully rendered. This way we avoid potential loading issues. The engine then applies the denoiser (if
    # one was requested) to the converged result.

    spear.log("Path-traced rendering beginning...")

    for i in range(args.num_frames):

        with instance.begin_frame():
            # Explicitly reset the path tracer's accumulated samples on the first frame. Nothing moves
            # in this example, but frames rendered above (e.g. during wait_for_engine_idle()) already accumulated
            # samples against this component's persistent view state, so without this reset, sample_index would
            # start ahead of 0 below.
            if i == 0:
                final_tone_curve_hdr_component.RequestPathTracerReset()

        with instance.end_frame(single_step=True):
            view_states = final_tone_curve_hdr_component.GetViewStates()
            assert len(view_states) == 1
            view_state = view_states[0]
            sample_index = sp_scene_view_state_interface.GetPathTracingSampleIndex(ViewState=view_state)
            assert sample_index == i + 1

    spear.log("Path-traced rendering finished.")

Set a reasonable default value for args.num_warmup_frames that works reliably for you.

[mikeroberts3000]

#include <SceneManagement.h> // FSceneViewStateInterface
#include <RenderingThread.h> // FlushRenderingCommands

[mikeroberts3000]

Great.