Situation Analysis:
The client is one of the largest hedge funds in the world and operates a high-availability, 24-hour research and trading support service. The group produces critical strategy updates that influence trading around the globe.
Updates come from a combination of custom applications, batch analysis jobs and reports, data uploads, and spreadsheet models. The group was plagued by painful staff turnover, inadequate documentation, and frequent staff wake-ups due to errant system processes.
Goals:
Reduce staff wake-ups, shift the support load from senior staff to a reliable support layer to enable senior time for upstream fixes, and increase the predictability of incident resolution.
Approach and Results:
Phase 1 was a 3 month test with a single lead, two process runner team, embedded within a very over-worked support group. The team had trouble at first engaging senior resources in the documentation process. Management aligned incentives and we tweaked the process design in response. We started to make major progress against the group's pain points and pushed through to complete senior team backing.
Success Metrics:
80% reduction in overnight wakeups. 3X increase in runbook creation and incidents executed entirely by OntimeOps. Onboarding of additional systems outside of the initial scope and expansion of the steady state team by 100%.
