Fix several timing issues when using CLI as a config method