Played some more... In SETTINGS | GENERAL, under taskbar button options, I had it set to "Show only active window". That works fine for the taskbar itself, but ALT-TAB still showed all windows in the group.
Then I changed the option to "Show only one combined button". After it changed the taskbar setup, that combined button showed 5/6/7 windows from the group. It varied. I don't know how/why it picked the number to display when hovering over the combined button. And ALT-TAB still showed all windows in the group.
Then I changed back to the "Show only active window" option. The taskbar icon then showed (when hovering over it) ALL the windows in the group, just like it did before using Groupy 2 at all.
Then I switched between (rolled through, showed) all the windows in the group, going to each tab. And then not only did the taskbar button go back to showing just the active window/tab, ALT-TAB also showed just a single window!
I have no idea what happens behind the scenes in that scenario, but it worked.
Now if I close the tabs and open them again, I have to go through that whole process again. So that combination seems to be tricking something, whether it's Groupy 2 or Windows itself (Win11 in this case).