Graceful exit from hpc

[m-bossart]

If a training hits the hpc time limit it is terminated and data is not saved. Need to have some saving of parameters at checkpointing to be able to restart the training.