To the outside world, redline can mean to mark something for removal, or it could mean the maximum safe speed of an engine. But in the world of Microsoft design, the term redlines (pronounced as if it were written as the two words red lines, but the accent is on the red) refers to a diagram showing the exact position of visual elements. They typically take the form of a proposed screen shot, with arrows and lines superimposed to indicate the distances between items, which items align with each other, and so on. They also contain indications as to the exact colors to use for different elements.
Originally the lines and arrows were actually red, hence the name. Here's an example of something that gives you the idea:
These aren't real redlines because the diagram doesn't contain any indications about the colors to use, and more complete redlines would include diagrams showing the hover, pressed, and disabled states.