Hide native Windows 11 Start button when using a replacement

[YellowNest]

Summary

Hide the native Windows 11 Start button visuals while Open-Shell's replacement Start button is enabled.

Windows 11 renders the Start button through XAML, so the older HWND-based hiding logic cannot remove the native glyph. This caused custom Open-Shell buttons to be drawn on top of the Windows Start icon.

The change uses the Windows XAML diagnostics visual tree to:

• collapse the native Start glyph
• disable hit testing on the underlying native Start control
• preserve the native control's layout slot
• restore the original XAML properties when replacement is disabled or Open-Shell exits
• respect the All taskbars setting

The replacement button itself is not moved or resized by this code, so Windows remains responsible for centered taskbar layout.

Verification

Tested on Windows 11 25H2 with centered taskbar:

• Custom replacement button no longer shows the Windows Start glyph underneath.
• Replacement button remains in the correct centered Start position.
• Left-click and right-click behavior continue to work.
• Disabling Replace Start button restores the native Windows Start button.
• Re-enabling it hides the native button again.
• Exiting Open-Shell restores the native Start button without restarting Explorer.

GitHub Actions Build completed successfully for exact follow-up commit 9f572bb07419216a6b3fe61f6ff64d828b52dd33:
https://github.com/YellowNest/Open-Shell-Menu/actions/runs/37219583485

Fixes #1631

[ge0rdi]

This is super awesome.
I knew about the Explorer TAP interface from TranslucentTB project. But never managed to properly use it inside Open-Shell.

Glad to see it works nicely. And I guess it should be used for more customizations and improvements related to start button and taskbar itself.
For example there is this very ugly mouse hook HookDesktopThreadMouse that we are using to intercept mouse events that would be consumed by XAML framework otherwise. It would be great to get rid of this by changing XAML handler for start button.

Do you plan to work more in this area? I think it would be really appreciated.

[ge0rdi]

I have noticed that enabling custom Start button cause also that TaskView button dissapears. There is just empty place instead of the button.

[ge0rdi]

Another issue is that when I Exit Open-Shell (via right-click menu), it is not possible to start it again.

Starting Open-Shell Menu Settings shortcut doesn't do anything (no settings are shown, Start menu is not customized).

[YellowNest]

Thanks for testing this carefully. Both reports were valid and pointed to separate problems in the initial TAP implementation.

I pushed the follow-up in 9f572bb and ran a fresh Actions build against that exact commit; the build passes.

The Task View regression came from treating every ExperienceToggleButton#LaunchListButton as the Start control. Current Windows 11 uses that shape for multiple taskbar controls. The code now resolves the attached AutomationId and only classifies the actual StartButton for suppression, so Task View is left alone.

The restart problem was a separate TAP lifetime issue. Shutdown now restores the XAML properties synchronously, removes the visual-tree callback with UnadviseVisualTreeChange, tears down the dispatch window, and balances the additional module reference introduced by InitializeXamlDiagnosticsEx. There is also a shutdown guard preventing a late connection attempt from attaching again while Open-Shell is exiting.

The build for the follow-up is here:
https://github.com/YellowNest/Open-Shell-Menu/actions/runs/37219583485

The two paths that still deserve runtime confirmation are:

• custom Start button enabled while Task View remains visible and functional
• Exit Open-Shell, then start Open-Shell again without restarting Explorer

And yes, I plan to keep working in this area. Replacing HookDesktopThreadMouse with a narrower XAML-side solution looks like a good next target. I want to keep this PR focused and make sure the TAP lifecycle is solid first, then tackle that separately.

[ge0rdi]

I want to keep this PR focused and make sure the TAP lifecycle is solid first, then tackle that separately.

Definitely.
Each fix/feature deserves own PR.

I'm glad you plan to work on this more.

[YellowNest]

Thanks, I really appreciate that.

I use Open-Shell myself, so contributing here is useful to me too. I care about keeping it working well on current Windows builds, and if I can help take some of the load off, I'm happy to.

I also appreciate the trust you're putting in the work. Your reviews and testing are valuable, especially around these Windows 11 edge cases, so I'll keep changes focused and separate like you suggested.

I've already started looking at the XAML input side in a separate research branch. The first build is clean, but I'll keep it out of upstream until I've tested the behavior properly.

Glad to help.

[ge0rdi]

custom Start button enabled while Task View remains visible and functional

I can confirm this works properly now.

Exit Open-Shell, then start Open-Shell again without restarting Explorer

I'm now experiencing random Explorer crashes when Exiting Open-Shell and starting it again (using Open-Shell Menu Settings shortcut). This is on 26H2 (Windows Sandbox).
I was able to obtain one dump and the stack looked like:

  *** Stack trace for last set context - .thread/.cxr resets it
 # Child-SP          RetAddr               Call Site
00 00000000`0367d600 00007ffc`8e616071     combase!Ordinal130+0x358
01 00000000`0367d650 00007ffc`8e62eec4     combase!Ordinal95+0x2e11
02 00000000`0367d730 00007ffc`8e62eae1     combase!Ordinal164+0x35f4
03 00000000`0367d800 00007ffc`8e558579     combase!Ordinal164+0x3211
04 00000000`0367d850 00007ffc`8e5a6543     combase!NdrOleDllGetClassObject+0x4c99
05 00000000`0367d890 00007ffc`8e5a63f6     combase!CoLockObjectExternal+0x463
06 00000000`0367d8c0 00007ffc`8e65f463     combase!CoLockObjectExternal+0x316
07 00000000`0367d8f0 00007ffc`8e62e705     combase!Ordinal130+0x1993
08 00000000`0367d960 00007ffc`8e4e6072     combase!Ordinal164+0x2e35
09 00000000`0367da20 00007ffc`6eb329f0     combase!CoReleaseMarshalData+0x122
0a 00000000`0367daf0 00007ffc`6eb31f64     oleacc!ReleaseMarshallData+0xac
0b 00000000`0367db30 00007ffc`6eb31ef2     oleacc!AccInfo::PropInfo::~PropInfo+0x54
0c 00000000`0367db60 00007ffc`6eb320c5     oleacc!AccInfo::PropInfo::`scalar deleting destructor'+0xe
0d 00000000`0367db90 00007ffc`6eb3202b     oleacc!AccInfo::~AccInfo+0x55
0e 00000000`0367dbc0 00007ffc`6eb31091     oleacc!CPropMgrImpl::RemoveEntry+0x4b
0f 00000000`0367dbf0 00007ffc`6eb30cbe     oleacc!CPropMgrImpl::Clean+0xed
10 00000000`0367dc40 00007ffc`6eb30bc4     oleacc!CPropMgrImpl::SetPropValue+0x2a
11 00000000`0367dcc0 00007ffc`5a861615     oleacc!CPropMgr::SetHwndPropStr+0xe4
12 00000000`0367dd90 00007ffc`5a847687     StartMenuDLL!SetControlAccessibleName+0x75 [D:\a\Open-Shell-Menu\Open-Shell-Menu\Src\Lib\SettingsUIHelper.cpp @ 166] 
13 00000000`0367dde0 00007ffc`5a846769     StartMenuDLL!CSettingsDlg::OnInitDialog+0x727 [D:\a\Open-Shell-Menu\Open-Shell-Menu\Src\Lib\Settings.cpp @ 1316] 
14 00000000`0367df70 00007ffc`5a82ee28     StartMenuDLL!CSettingsDlg::ProcessWindowMessage+0x69 [D:\a\Open-Shell-Menu\Open-Shell-Menu\Src\Lib\Settings.cpp @ 1116] 
15 00000000`0367e010 00007ffc`6df61148     StartMenuDLL!ATL::CDialogImplBaseT<ATL::CWindow>::DialogProc+0x78 [C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\atlmfc\include\atlwin.h @ 3932] 
16 00000000`0367e0b0 00007ffc`8eeb426e     atlthunk!AtlThunk_0x06+0x18
17 00000000`0367e0f0 00007ffc`8eeb3604     user32!UserCallDlgProcCheckWow+0x1f2
18 00000000`0367e1e0 00007ffc`8eeb3526     user32!DefDlgProcWorker+0xc4
19 00000000`0367e290 00007ffc`8eeb52e6     user32!DefDlgProcW+0x36
1a 00000000`0367e2d0 00007ffc`8eeb4b6c     user32!UserCallWinProcCheckWow+0x356
1b 00000000`0367e430 00007ffc`8eedf593     user32!DispatchClientMessage+0x9c
1c 00000000`0367e490 00007ffc`906c5c04     user32!_fnDWORD+0x33
1d 00000000`0367e4f0 00007ffc`8e251384     ntdll!KiUserCallbackDispatcherContinue
1e 00000000`0367e578 00007ffc`8eeb73c6     win32u!NtUserMessageCall+0x14
1f 00000000`0367e580 00007ffc`8eec0192     user32!SendMessageWorker+0xa26
20 00000000`0367e620 00007ffc`8eebefc7     user32!InternalCreateDialog+0x1032
21 00000000`0367e800 00007ffc`8eec0318     user32!CreateDialogIndirectParamAorW+0x57
22 00000000`0367e850 00007ffc`5a84a2f6     user32!CreateDialogIndirectParamW+0x18
23 (Inline Function) --------`--------     StartMenuDLL!CResizeableDlg<CSettingsDlg>::Create+0xa6 [D:\a\Open-Shell-Menu\Open-Shell-Menu\Src\Lib\SettingsUIHelper.h @ 55] 
24 00000000`0367e890 00007ffc`5a82d00d     StartMenuDLL!EditSettings+0x1e6 [D:\a\Open-Shell-Menu\Open-Shell-Menu\Src\Lib\Settings.cpp @ 1934] 
25 00000000`0367e920 00007ffc`5a792e1d     StartMenuDLL!EditSettings+0x32d [D:\a\Open-Shell-Menu\Open-Shell-Menu\Src\StartMenu\StartMenuDLL\SettingsUI.cpp @ 5287] 
26 00000000`0367ea50 00007ffc`8eed3bed     StartMenuDLL!HookDesktopThread+0x48d [D:\a\Open-Shell-Menu\Open-Shell-Menu\Src\StartMenu\StartMenuDLL\StartMenuDLL.cpp @ 3694] 
27 00000000`0367f2a0 00007ffc`8eebb838     user32!DispatchHookW+0xad
28 00000000`0367f330 00007ffc`8eebb77c     user32!CallHookWithSEH+0x28
29 00000000`0367f380 00007ffc`906c5c04     user32!_fnHkINLPMSG+0x7c
2a 00000000`0367f3d0 00007ffc`8e251304     ntdll!KiUserCallbackDispatcherContinue
2b 00000000`0367f488 00007ffc`8eeb19df     win32u!NtUserPeekMessage+0x14
2c 00000000`0367f490 00007ffc`8eeb1968     user32!_PeekMessage+0x3f
2d 00000000`0367f500 00007ff7`4081a271     user32!PeekMessageW+0x168
2e 00000000`0367f570 00007ff7`40818aeb     explorer!CTray::_MessageLoop+0x2c1
2f 00000000`0367f6b0 00007ffc`8fab3a8c     explorer!CTray::MainThreadProc+0x7b
30 00000000`0367f710 00007ffc`8e8bcdf7     SHCore!_WrapperThreadProc+0x15c
31 00000000`0367f800 00007ffc`905dee4c     kernel32!BaseThreadInitThunk+0x17
32 00000000`0367f830 00000000`00000000     ntdll!RtlUserThreadStart+0x2c

I can share the dump eventually.

[YellowNest]

Thanks for the dump trace. I treated this as a teardown/lifetime problem rather than an accessibility-code failure and reworked the TAP shutdown path.

The latest head is 192b2e3.

The important changes are:

• keep the module reference acquired by InitializeXamlDiagnosticsEx until after the visual-tree callback and XAML/COM references are fully released
• run the XAML/COM teardown on the same thread that owns the TAP dispatch window
• only publish g_Tap after AdviseVisualTreeChange succeeds, with complete cleanup on failed setup paths
• hold a separate StartMenuDLL module reference for the asynchronous connection worker, released atomically with FreeLibraryAndExitThread, so Exit cannot unload the DLL while that worker is still executing

I also checked the current upstream master changes; they don't overlap this code.

The full upstream PR build for 192b2e3 is green:
https://github.com/Open-Shell/Open-Shell-Menu/actions/runs/37225846278

The critical runtime test now is repeated cycles on 26H2:

1. enable the custom Start button
2. Exit Open-Shell
3. start it again through Open-Shell Menu Settings
4. repeat several times without restarting Explorer

If Explorer still crashes, the full dump would be useful. I don't want to paper over this with another timing workaround; the remaining failure, if any, should be traced from the dump.

[YellowNest]

Follow-up on HookDesktopThreadMouse: I now have the XAML input-routing replacement isolated and building cleanly in a separate branch/validation PR (YellowNest/Open-Shell-Menu#30).

The approach keeps the existing WH_MOUSE path strictly as a fallback while the XAML route is probing, lets only WM_MOUSEMOVE through for verification, and removes the hook only after the Start control, XAML source and taskbar mapping have all been resolved and a routed pointer event has been observed.

Runtime route failures restore the fallback, and teardown drains queued bridge/state messages before StartMenuDLL unload.

I also reviewed the multi-taskbar and touch/pen cases and deliberately avoided making IsHitTestVisible=False independent of the replacement-button setting. That would remove the hook more aggressively, but it could change non-mouse input behavior on a visible native Start button.

The validation workflow is fully green:

• Build binaries: success
• Build installers: success
• Build setup and symbols: success
• Artifact uploads: success

Validation branch:
feature/win11-xaml-input-routing-reviewed

Validation PR:
YellowNest#30

I am keeping this out of #2547 itself so the input change stays a separate PR as requested. Once #2547 lands, I can rebase this onto current master and submit a focused PR containing only the HookDesktopThreadMouse replacement.