What determines the order in which icons appear in the Alt+Tab list?
The icons appear in the same order as the window Z-order. When you switch to a window, then it comes to the top of the Z-order. If you minimize a window, it goes to the bottom of the Z-order. The Alt+Esc hotkey (gosh, does anybody still use Alt+Esc?) takes the current top window and sends it to the bottom of the Z-order (and the window next in line comes to the top). The Alt+Shift+Esc hotkey (I bet you didn't know that hotkey even existed) takes the bottom-most window and brings it to the top, but does not open the window if it is minimized.
The presence of "always on top" windows makes this a little more complicated. The basic rule is that an "always on top" window always appears on top of a "not always on top" window. So if the above rules indicate that a "not always on top" window comes to the top, it really just goes as high as it can without getting on top of any "always on top" windows.
You may have run across the term "fast task switching". This was the term used to describe the precursor to the current Alt+Tab switching interface. The old way of switching via Alt+Tab (Windows 3.0 and earlier) was just like Alt+Esc, except that the window you switched to was automatically opened if it had been minimized. When the new Alt+Tab was added to Windows 3.1, we were concerned that people might prefer the old way, so there was a switch in the control panel to set it back to the slow way. (There is also a setting SPI_SETFASTTASKSWITCH that lets you change it programmatically.) It turns out nobody complained, so the old slow way of task switching was removed entirely and the setting now has no effect.
This does highlight the effort we take to try to allow people who don't like the new way of doing something to go back to the old way. It turns out that corporations with 10,000 employees don't like it when the user interface changes, because it forces them to spend millions of dollars retraining all their employees. If you open up the Group Policy Editor, you can see the zillions of deployment settings that IT administrators can use to disable a variety of new Windows UI features.