Why you cannot subclass Gfa_hWnd

In the course of interpreting the disassembly of both the GfaWin32.exe and the run-time I created quite some editor extension procedures. Most of them are rudimentary and should be developed further, but some of them are very useful (at least to me). I always hoped to put them together in one single GLL file and make that GLL available. However, I never came to do that. I had a problem I didn't understand, and thus didn't know how to solve. After unloading that GLL the IDE crashed.

To add new functionality a Windows developer will most likely start by subclassing the window. In fact, subclassing is the most elementary of all modification techniques. I started by subclassing both IDE windows, the main IDE window Gfa_hWnd and the editor window Gfa_hWndEd. The subclassing was performed in the Gfa_Init() procedure and undone in Gfa_Exit(), which seemed logical at the time. However, as said, this caused a (fatal) problem that at that time I couldn't resolve and I reserved it for a later moment to handle. I was convinced it was something I simply overlooked and at that stage it wasn't really an obstacle. The GLL was automatically loaded when the IDE was started and unloaded when it was closed. I never unloaded the GLL by hand using the Editor Extension Manager dialog box.

Sometimes I was annoyed by the problem and tried to find out the cause of it. It turned out that when the GLL was unloaded in the course of closing the IDE, the un-subclassing was performed after the main window was closed. So, the Gfa_hWnd window didn't exist anymore and re-installing the old window procedure didn't cause any problem. But, when I unloaded the GLL when the IDE was active, using the Extension dialog box, it always crashed the IDE. The undoing of the subclassing of Gfa_hWnd was the problem. In fact, you cannot subclass the Gfa_hWnd in a GLL.

My problem was that I didn't understand exactly what I was doing when I was subclassing a window. Let me illustrate by describing the path a WM_COMMAND (a posted) message follows to where it gets actually handled. Here is what happens when you select the GFA-BASIC 32's Editor Extension Manager dialog box, either from the menu-bar or through a shortcut First the OS puts a WM_COMMAND message in the application queue. The program obtains the message in an endless GetMessage loop and dispatches the message to the window procedure (note: the one I tried to subclass). The window procedure invokes the modal dialog box and a new message loop is started to handle the dialog box. Now, after subclassing, the OS invokes the new window procedure rather than the old one, and requires the new procedure to execute the old window procedure using the CallWindowProc() API. So, the old window procedure, which handled the WM_COMMAND, is called from the new window procedure located in the GLL. In fact, when the dialog box is put on the screen and a modal message loop is started, the program is still executing the new window procedure inside the GLL! After closing the dialog box the program will want to return to the location it was called from, the place where CallWindowProc() API was invoked. You see the problem now?

The old window procedure was invoked from inside the new window procedure (located in the GLL), which was just unloaded using the Editor Extension Manager dialog box. There is no place to return to! Ergo, the most elementary technique, subclassing a window to add new functionality, is not possible for Gfa_hWnd in an editor extension.