NvFBC host stops producing frames on modeset

[Xyene]

I'm filing this for posterity as it's technically a bug, but it's pretty low-impact. This occurs with B5, but I suspect it's been around forever.

Guest desktop running at 2560x1440.

Host desktop running at 2560x1440.

xyene@lynwood:~$ swaymsg -t get_outputs
Output HDMI-A-1 'Goldstar Company Ltd LG TV SSCR 0x00000101' (focused)
  Current mode: 2560x1440 @ 119.998 Hz

Game (specifically, osu!) set to modeset to 3840x2160.

osu! config

This goes in the .cfg file in %userprofile%\AppData\Local\osu!.

BeatmapDirectory = Songs
VolumeUniversal = 30
VolumeEffect = 80
VolumeMusic = 80
AllowPublicInvites = 1
AutoChatHide = 1
AutomaticDownload = 1
AutomaticDownloadNoVideo = 0
BlockNonFriendPM = 0
Shaders = 0
BloomSoftening = 0
BossKeyFirstActivation = 1
ChatAudibleHighlight = 1
ChatChannels = #osu
ChatFilter = 0
ChatHighlightName = 1
ChatMessageNotification = 1
ChatLastChannel = #osu
ChatRemoveForeign = 0
ChatSortMode = Rank
ComboBurst = 0
ComboFire = 0
ComboFireHeight = 3
ConfirmExit = 0
AutoSendNowPlaying = 1
CursorSize = 0.99
AutomaticCursorSizing = 0
DimLevel = 80
IHateHavingFun = 0
Display = 1
DisplayCityLocation = 0
DiscordRichPresence = 1
DistanceSpacingEnabled = 1
EditorTip = 0
VideoEditor = 0
EditorDim = 0
EditorGrid = 0
EditorDefaultSkin = 0
EditorSnakingSliders = 1
EditorHitAnimations = 0
EditorFollowPoints = 1
EditorStacking = 1
ForceSliderRendering = 0
FpsCounter = 1
GuideTips = 1
CursorRipple = 0
HighlightWords = 
HighResolution = 0
HitLighting = 0
IgnoreBarline = 0
IgnoreBeatmapSamples = 0
IgnoreBeatmapSkins = 0
IgnoreList = 
KeyOverlay = 0
Language = en
LastPlayMode = Osu
AllowNowPlayingHighlights = 0
LastVersion = b20211225
LastVersionPermissionsFailed = b20200831.2
LoadSubmittedThread = 1
LobbyPlayMode = -1
ShowInterface = 1
ShowInterfaceDuringRelax = 0
LobbyShowExistingOnly = 0
LobbyShowFriendsOnly = 0
LobbyShowFull = 0
LobbyShowInProgress = 1
LobbyShowPassworded = 1
LogPrivateMessages = 0
LowResolution = 0
ManiaSpeed = 13
UsePerBeatmapManiaSpeed = 0
ManiaSpeedBPMScale = 0
MenuTip = 12
MouseDisableButtons = 1
MouseDisableWheel = 0
MouseSpeed = 0.99
Offset = 0
ScoreMeterScale = 1
DistanceSpacing = 0.8
EditorBeatDivisor = 1
EditorGridSize = 32
EditorGridSizeDesign = 32
Height = 2160
Width = 3840
HeightFullscreen = 2160
CustomFrameLimit = 200
WidthFullscreen = 3840
MsnIntegration = 0
MyPcSucks = 0
NotifyFriends = 1
NotifySubmittedThread = 1
PopupDuringGameplay = 1
ProgressBarType = Pie
RankType = Top
RefreshRate = 60
OverrideRefreshRate = 0
ScaleMode = WidescreenConservative
ScoreboardVisible = 1
ScoreMeter = Error
ScreenshotId = 0
MenuSnow = 0
MenuTriangles = 1
SongSelectThumbnails = 1
ScreenshotFormat = Jpg
ShowReplayComments = 0
ShowSpectators = 1
ShowStoryboard = 0
Skin = Default
SkinSamples = 1
SkipTablet = 0
SnakingSliders = 0
Tablet = 0
UpdatePending = 0
UserFilter = All
UseSkinCursor = 0
SeasonalBackgrounds = Always
UseTaikoSkin = 0
Video = 0
Wiimote = 0
YahooIntegration = 0
ForceFrameFlush = 0
DetectPerformanceIssues = 1
Fullscreen = 1
MenuMusic = 0
MenuVoice = 0
MenuParallax = 0
RawInput = 1
AbsoluteToOsuWindow = 0
ConfineMouse = Fullscreen
ShowMenuTips = 0
HiddenShowFirstApproach = 1
ComboColourSliderBall = 1
AlternativeChatFont = 0
Username = 
DisplayStarsMaximum = 10
DisplayStarsMinimum = 0
AudioDevice = 
AudioCompatibility = 0
SavePassword = 0
SaveUsername = 0
Password = 
TreeSortMode = Difficulty
TreeSortMode2 = Difficulty
Letterboxing = 0
LetterboxPositionX = 0
LetterboxPositionY = 0
FrameSync = VSync
ShowUnicode = 0
PermanentSongInfo = 0
Ticker = 0
CompatibilityContext = 0
CanForceOptimusCompatibility = 0
keyOsuLeft = Z
keyOsuRight = X
keyOsuSmoke = C
keyFruitsDash = LeftShift
keyFruitsLeft = Left
keyFruitsRight = Right
keyTaikoInnerLeft = X
keyTaikoInnerRight = C
keyTaikoOuterLeft = Z
keyTaikoOuterRight = V
keyPause = Escape
keySkip = Space
keyToggleScoreboard = Tab
keyToggleChat = F8
keyToggleExtendedChat = F9
keyScreenshot = F12
keyIncreaseAudioOffset = OemPlus
keyDecreaseAudioOffset = OemMinus
keyQuickRetry = OemTilde
keyIncreaseSpeed = F4
keyDecreaseSpeed = F3
keyToggleFrameLimiter = F7
keyVolumeIncrease = Up
keyVolumeDecrease = Down
keyDisableMouseButtons = F10
keyBossKey = Insert
keySelectTool = D1
keyNormalTool = D2
keySliderTool = D3
keySpinnerTool = D4
keyNewComboToggle = Q
keyWhistleToggle = W
keyFinishToggle = E
keyClapToggle = R
keyGridSnapToggle = T
keyDistSnapToggle = Y
keyNoteLockToggle = L
keyNudgeLeft = J
keyNudgeRight = K
keyHelpToggle = H
keyJumpToBegin = Z
keyPlayFromBegin = X
keyAudioPause = C
keyJumpToEnd = V
keyGridChange = G
keyTimingSection = None
keyInheritingSection = None
keyRemoveSection = None
keyEasy = Q
keyNoFail = W
keyHalfTime = E
keyHardRock = A
keySuddenDeath = S
keyDoubleTime = D
keyHidden = F
keyFlashlight = G
keyRelax = Z
keyAutopilot = X
keySpunOut = C
keyAuto = V
keyScoreV2 = B
EditorVideo = 1
CredentialEndpoint = 

With DXGI this produces the below output when running the game. It is the wrong size, of course.

We can see the modeset from the client logs.

[I]    740314432              main.c:637  | main_frameThread               | Format: FRAME_TYPE_BGRA 3840x2160 stride:3840 pitch:15360 rotation:0
[I]    791598146              main.c:637  | main_frameThread               | Format: FRAME_TYPE_BGRA 2560x1440 stride:2560 pitch:10240 rotation:0
[I]    834073818              main.c:637  | main_frameThread               | Format: FRAME_TYPE_BGRA 3840x2160 stride:3840 pitch:15360 rotation:0
[I]    870712028              main.c:637  | main_frameThread               | Format: FRAME_TYPE_BGRA 2560x1440 stride:2560 pitch:10240 rotation:0

However, when trying this with NvFBC, the host gets put in a bad state.

Host logs

[E] 450339731493         wrapper.cpp:331  | NvFBCToSysCapture              | Unknown NVFBCRESULT failure -19
[E] 450339732077               app.c:784  | app_main                       | Capture interface reported a fatal error
[I] 450339732696         mousehook.c:141  | threadProc                     | Mouse hook thread received quit request
[I] 450339732697         mousehook.c:101  | updateThreadProc               | Mouse hook update thread received quit request
[I] 450339738939     force_compose.c:89   | threadProc                     | Force composition received quit request
[I] 450340191402               app.c:840  | app_main                       | Host application exited
[I] 450394345776              time.c:85   | windowsSetTimerResolution      | System timer resolution: 500.0 μs
[I] 450394346668               app.c:542  | app_main                       | Looking Glass Host (B4-566-g67d1073c0a+)
[I] 450394347598           cpuinfo.c:36   | lgDebugCPU                     | CPU Model: AMD Ryzen 9 5950X 16-Core Processor
[I] 450394348529           cpuinfo.c:37
   | lgDebugCPU                     | CPU: 4 cores, 8 threads
[I] 450394350503           ivshmem.c:131  | ivshmemInit                    | IVSHMEM 0* on bus 0x0, device 0x9, function 0x0
[I] 450394452353               app.c:559  | app_main                       | IVSHMEM Size     : 128 MiB
[I] 450394452809               app.c:560  | app_main                       | IVSHMEM Address  : 0x2CA0000
[I] 450394453257               app.c:561  | app_main                       | Max Pointer Size : 1024 KiB
[I] 450394453680               app.c:562  | app_main                       | KVMFR Version    : 14
[I] 450394454307               app.c:623  | app_main                       | Max Frame Size   : 62 MiB
[I] 450394454682               app.c:648  | app_main                       | Trying           : NVFBC (NVidia Frame Buffer Capture)
[I] 450394462775         wrapper.cpp:90   | NvFBCInit                      | NvFBC SDK Version: 112
[I] 450395141092             nvfbc.c:241  | nvfbc_init                     | DiffMap block    : 128x128
[I] 450395141462             nvfbc.c:242  | nvfbc_init                     | Cursor mode      : decoupled
[I] 450395166253     force_compose.c:81   | threadProc                     | Created window to force composition
[I] 450395242340               app.c:673  | app_main                       | Using            : NVFBC (NVidia Frame Buffer Capture)
[I] 450395242737               app.c:674  | app_main                       | Capture Method   : Synchronous
[I]      9662260              time.c:85   | windowsSetTimerResolution      | System timer resolution: 500.0 μs
[I]      9662839               app.c:542  | app_main                       | Looking Glass Host (B4-566-g67d1073c0a+)
[I]      9663227           cpuinfo.c:36   | lgDebugCPU                     | CPU Model: AMD Ryzen 9 5950X 16-Core Processor
[I]      9663570           cpuinfo.c:37   | lgDebugCPU                     | CPU: 4 cores, 8 threads
[I]      9664304           ivshmem.c:131  | ivshmemInit                    | IVSHMEM 0* on bus 0x0, device 0x9, function 0x0
[I]      9670035               app.c:559  | app_main                       | IVSHMEM Size     : 128 MiB
[I]      9670346               app.c:560  | app_main                       | IVSHMEM Address  : 0x2950000
[I]      9670634               app.c:561  | app_main                       | Max Pointer Size : 1024 KiB
[I]      9670939               app.c:562  | app_main                       | KVMFR Version    : 14
[I]      9672233               app.c:623  | app_main                       | Max Frame Size   : 62 MiB
[I]      9672536               app.c:648  | app_main                       | Trying           : NVFBC (NVidia Frame Buffer Capture)
[I]      9679545         wrapper.cpp:90   | NvFBCInit                      | NvFBC SDK Version: 112
[I]     10347289             nvfbc.c:241  | nvfbc_init                     | DiffMap block    : 128x128
[I]     10347621             nvfbc.c:242  | nvfbc_init                     | Cursor mode      : decoupled
[I]     10352868     force_compose.c:81   | threadProc                     | Created window to force composition
[I]     10448501               app.c:673  | app_main                       | Using            : NVFBC (NVidia Frame Buffer Capture)
[I]     10448872               app.c:674  | app_main                       | Capture Method   : Synchronous
[I]     10449164               app.c:348  | captureStart                   | ==== [ Capture Start ] ====
[E]    146236643         wrapper.cpp:319  | NvFBCToSysCapture              | Invalid parameter
[E]    146236925               app.c:784  | app_main                       | Capture interface reported a fatal error
[I]    146237282         mousehook.c:101  | updateThreadProc               | Mouse hook update thread received quit request
[I]    146237296         mousehook.c:141  | threadProc                     | Mouse hook thread received quit request
[I]    146238160     force_compose.c:89   | threadProc                     | Force composition received quit request
[I]    146249034               app.c:840  | app_main                       | Host application exited
[I]    147373017              time.c:85   | windowsSetTimerResolution      | System timer resolution: 500.0 μs
[I]    147373384               app.c:542  | app_main                       | Looking Glass Host (B4-566-g67d1073c0a+)
[I]    147373738           cpuinfo.c:36   | lgDebugCPU                     | CPU Model: AMD Ryzen 9 5950X 16-Core Processor
[I]    147374092           cpuinfo.c:37   | lgDebugCPU                     | CPU: 4 cores, 8 threads
[I]    147374763           ivshmem.c:131  | ivshmemInit                    | IVSHMEM 0* on bus 0x0, device 0x9, function 0x0
[I]    147380282               app.c:559  | app_main                       | IVSHMEM Size     : 128 MiB
[I]    147380589               app.c:560  | app_main                       | IVSHMEM Address  : 0x2D40000
[I]    147380877               app.c:561  | app_main                       | Max Pointer Size : 1024 KiB
[I]    147381176               app.c:562  | app_main                       | KVMFR Version    : 14
[I]    147381606               app.c:623  | app_main                       | Max Frame Size   : 62 MiB
[I]    147381891               app.c:648  | app_main                       | Trying           : NVFBC (NVidia Frame Buffer Capture)
[I]    147386156         wrapper.cpp:90   | NvFBCInit                      | NvFBC SDK Version: 112
[I]    147753632             nvfbc.c:241  | nvfbc_init                     | DiffMap block    : 128x128
[I]    147754307             nvfbc.c:242  | nvfbc_init                     | Cursor mode      : decoupled
[I]    147760952     force_compose.c:81   | threadProc                     | Created window to force composition
[I]    147855941               app.c:673  | app_main                       | Using            : NVFBC (NVidia Frame Buffer Capture)
[I]    147856525               app.c:674  | app_main                       | Capture Method   : Synchronous

Once in this state any one of the following is insufficient:

• restarting the host
• killing osu!

One must do both to get Looking Glass working again.

Some debug prints show that

LookingGlass/host/platform/Windows/capture/NVFBC/src/nvfbc.c

Lines 331 to 333 in b21d842

	static CaptureResult nvfbc_capture(void)
	{
	getDesktopSize(&this->width, &this->height);

correctly returns 3840x2160 after the modeswitch, and that the error occurs upon the third frame. A dumb hardcoding of width/height in getDesktopSize to 2560x1440 gets things working, but cropped weirdly.

Looking Glass is otherwise able to run on this machine when the desktop resolution is set to 3840x2160, it's just the modeset that breaks it.

[gnif]

Please update to B5 and test using DXGI. If this resolves the issue then it's a bug in the deprecated NvFBC API which we can do nothing about.