Call of Duty

If you are a part of a friendly data team, one day you inevitably find yourself overwhelmed with requests. Do you grumble about not being a service desk or do you capitalize on being in demand?

Probably, both! Data people like to grumble in general, because we like for things to make sense, have a structured reasoning, and a degree of predictability. Naturally, that doesn’t always match our reality. But, I’ve found that over-indexing on grumbling and running a strict JIRA backlog for new requests (or refusing to take them outside of the quarterly planning) can do more harm than good in the team’s journey towards becoming a trusted and valued thought partner.

So, what do you do? Do you just deal with constantly being interrupted and context-switching and forego meaningful progress on generative work? Not at all! On-call rotation to the rescue!

Now, it can definitely use a bit of a different branding. On-call in tech has a connotation of being woken up in the middle of the night by a PagerDuty alert. If your data team is responsible for data pipelines that feed into the user-facing product in some way (especially when it comes to critical points of user experience, like the ability to perform the key action in the product, right access levels for PII, payments, and so on) - that may still be your reality. 

But for teams that are focused on analysis, reporting, research - it is really not all that scary. So if you are introducing this concept to your team, call it something different so as not to get an immediate rejection (learn from my mistakes!). At Spring Health, the data engineering team called the ‘on-call’ person ‘the sprint guardian’ which admittedly sounds much more noble.

Here is how it could work…

I first started thinking about on-call at Peloton in 2021. At that point, my team was probably around 12 people, and we were supporting 30+ product pods across software and hardware development, plus regularly fielding questions and requests from our wonderful partner teams - Content, Brand Marketing, Strategy, Investor Relations, PR, Community. 

The analysts were starting to get overwhelmed. The product analyst life is full enough of context-switching even without ad hocs - between defining taxonomy, designing experiments, doing impact calculations, and launch analyses.

On the other hand, as the team became larger and the product team started to operate in a pod model, our knowledge started to get more siloed - if someone was working on the digital growth pod, they knew everything about the in-app onboarding funnel but nobody else did.

So I had an idea of picking a ‘point person’ who can field small requests on a given week while both allowing everyone else on the team to focus better, and for them to learn about subjects and stakeholders they normally don’t face. Together with my project manager (that role on my team was short-lived, it ended up not being as beneficial as I envisioned it) we interviewed some of our software engineering managers about their version of on-call, and I’d ‘cold’ reached out to the wonderful

Katie Bauer

after seeing a mention of her giving a talk on leveraging an on-call system at Reddit. 

As a result, after a couple of collaborative sessions with our team and an 8-week pilot, we came up with the following system:

• One person per week is on call with a time box of 10 hours (each person schedules as works for them - 2 hrs every day or two-three larger chunks), only during their regular work hours.

• All requests (except for ‘where do I find X?’) would go through a form that required the requestor to articulate the ‘why’ behind the request and explain the dependencies if the desired deadline was really tight

  • Check out this great article about designing a great intake form!

• The SLA for getting back on the request (not necessarily solving it fully) was 48 hours since we didn’t have any operational/production use cases in our scope

• The on-call analyst would triage the requests (with my help, as needed) and determine if some requests were too big to be done in a couple of hours (e.g. require a net new deep research project). In this case we would pick the person on the team the most versed in the subject and have them have a more deep conversation around business impact, scope, and prioritize it on their roadmap as it allows.

• The on-call analyst would also monitor our team slack channel for any ‘quick questions’ throughout the shift

• At the end of the weekly shift, the on-call analyst would write a quick recap in the shared doc to highlight if some projects were unfinished and had to be picked up, or if they’ve noticed a specific emerging pattern in requests

This structure alleviated concerns about on-call completely overtaking folks’ agendas and making it hard for their product prods to rely on their support and made it much easier for analysts to anticipate when they would be more distracted, so they didn’t plan any meaty projects for their on-call weeks resulting in logging off on Friday with a disappointed feeling. 

We’d schedule shifts in advance using a survey where folks could pick which weeks they coudn’t be on call (because of PTO or workload), so we had a primary + emergency backup on-call schedule set for a couple months out.

Each couple of cycles, I would survey the team - asking them if the on-call workload was manageable, if they’d learned anything new, and if they were feeling more focused when not on-call, to make sure the system was still working.

Side effects… or primary benefits?

One of the benefits of this system that combined a low-friction way to submit a request without having to worry about immediate pushback with the centralized request logging, plus the built-in weekly reflections was that it became easier for us to spot important trends in requests.

And yes, as you would expect, 60%+ of requests were something that a data scientist would call ‘unhelpful’ (either not actionable/just curious or the question asked was not the right question to solve the underlying ‘why’). But, first of all, it is great that a lot of people are ‘just curious’ and it should be encouraged. And understanding the patterns of what people are curious about can lead to identifying our own blind spots.

 If many people asked us about ‘what song was the most played in this month’s playlists?’ in the last few weeks - was there some strategic conversation about music happening that we weren’t a part of? Was there some hypothesis around changing patterns of impact of music on engagement? At the very least, we would make sure we build and socialize dashboards where stakeholders can self-serve insights about different touchpoints with music and playlists at Peloton, but even better if it was the birth of a new impactful deep-dive analysis.

Strength in Numbers

Of course, if you have a much smaller team, a lot of benefits of this system are less prominent. Being on call every three weeks is very different from ‘every 2 months’. But it still helps create more focus for the teammates and learn new datasets and sets of problems. With a three-person team at Bubble, we are doing a much less formal version of this system - without an intake form (for now) and very rigid time limits, but as the team grows, I will look to refine it more to relive the old glory days :) 

Are you leveraging a similar system on your data team? How do you go about ad hoc requests?