We recently had an issue where our production server's queue was cleared. Upon looking into this to see if someone cancelled the request, we found the entry in the audit log of where it was queued; but no entry of its cancellation. I then started to look at the server and discovered it was forcibly stopped on 6/23/2019 @ 6:24 PM. When inquiring with the windows server team we were informed that the VM Host had a catastrophic failure and therefore all servers were automatically evacuated to a new host. Can the way the queue is handled be modified to a more stable way where it will not be accidentally lost?

Our issue is we allow the customer to schedule the build (deployment) for the future. Due to this could you please store the future scheduled builds requested by a user (not the cron or scheduler builds) in the database. When the build starts, then remove or mark the database record complete and perform the normal build status/queue process in memory like you currently do.