Looking at it the issue is being triggered by you using OpenShell. Which version of OpenShell are you using?
Also please don't use ShadowFX with Curtains as it is redundant given the shadow offerings in Curtains now.
It looks like some of my issues were a combination of all the other UI-tweaking apps I have installed.
To make debugging a little easier, I have uninstalled Aero Glass, Open Shell Menu, and ShadowFX. I created a "broken" style by copying the Cairo Dark style, and editing the Button.Background setting to use the "start-button.png" image. Now what happens when I click "Apply Changes" is this: the screen goes black for a second, and comes back but all the windows are back to the stock Windows 10 style. After ~25 seconds, the screen goes black again, with only the mouse pointer visible. Clicking doesn't seem to do anything, Alt-Tab shows the task list but doesn't seem to do anything else. Tapping the Windows key a couple of times brings the desktop back, but the Start menu, taskbar, and desktop are hung, and all the window frames have an old pre-windows-10 theme:
Checking process IDs before and after the first flicker and the long back screen, it looks like dwm.exe has crashed and restarted both times. If I go back to Curtains, select a working style, and kill dwm.exe again, everything goes back to normal. No crashdumps seem to be generated, though.
Hopefully this will be enough info to let you reproduce it at your end.