Refactor To Make Immediate Change Easier

Jeremy Miller gives his reasons for refactoring.  He gives two over-arching reasons.  The first is "To remedy a deficiency in code, design, or architectural quality" and the second, "To make a forthcoming change to the code easier."  I'm very much in agreement with the second, but the first makes me nervous.  Refactoring is not an end to itself.  Changing code to make it of higher quality without an immediate purpose in mind strikes me as wasteful.  Code doesn't need to be pretty until it is being maintained.  The compiler doesn't care how ugly the code is.  The reason to make code of higher quality is because someone will may have to change that code and it will be hard to change in its low-quality state.  However, if change is not imminent, cleaning up the code now is a non-optimal decision.  Suppose the code continues working and no change is necessary in that part of the code base.  Because the compiler doesn't care and no human looks at it, the effort to make it higher quality is wasted.  Instead, it is better to wait until the change is needed to make the updates.  This would seem to be in line with the Lean principle of waiting until the last responsible moment or just in time delivery.  It is better to wait than to refactor pre-emptively.  Refactoring can become a yak-shaving exercise if one is not careful.  To be fair, Jeremy does end with this advice:  "If a structural problem in the code isn't causing any immediate harm or friction, you probably leave it alone for now."