As I am myself exploring our new Microsoft SQL Server 2017, I encountered this error when installing a second instance of SQL Server.
TITLE: Microsoft SQL Server 2017 Setup ------------------------------ The following error has occurred: VS Shell installation has failed with exit code 1638.
What was more surprising is that the 1st instance installation went perfectly well. I suspect my interleave of SSMS installation between both might have been the cause.
Anyway, the important thing is that situation is covered by KB 4092997 and as indicated in the KB, a simple Repair of 'Microsoft Visual C++ 2015 Redistributable (x64)' followed by a server restart cleared the problem for me. We're working on making the interactive link of the pop-up message a bit more useful than it is today.