One-time Timer Jobs fail to start

I recently ran into an issue with a SharePoint 2013 farm in which one-time timer jobs were failing to run.  This was impacting the ability to deploy solutions, run Merge-SPLogFile, etc.  When running Merge-SPLogFile from an app server, it kept failing on the web front ends (WFEs) with a message of "WARNING:  Unable to start job CollectionLogFiles_<server>_####### on server <server> ".  We could see the one-time timer jobs get created on all servers.  The jobs on the app servers would execute and be removed, but the jobs on the WFEs would sit for a period of time and then they would get deleted, resulting in the command failing to pull logs from the WFEs.  If we ran the Merge-SPLogFile command from the WFEs, it worked across all servers.

First, we went after the usual culprits.  We confirmed that the timer service was running on each server.  Next, we cleared the configuration cache across all servers.  We disabled the AV solution.  We deleted all one-time timer jobs.  We ran psconfig and even tried a SharePoint software repair on one of the servers.  Additionally, we reviewed the ULS logs, event logs, etc., but found nothing out of the ordinary.  Nothing made a difference.  By chance, I noticed that the WFEs had a different time zone configured.  When I changed the WFE time zone to match the app server time zone (and rebooted), everything started working.  So the moral of the story is, always make sure the server time zones in a farm match.  One-time timer jobs leverage the local server time zone as part of determining whether there are jobs that need run on that particular server.