With the increasing demand for innovation in the tech industry, applied research teams have become integral in driving technological advancements and meeting customer needs. Effective management of research teams is crucial to ensure research objectives align with business goals and produce tangible outcomes. As an applied research team at Cloudinary, we’ve gained valuable experience balancing research, development, and business needs while navigating a dynamic and rapidly changing environment.
Agile methodologies are known to help development teams reach their goals on time while improving the communication between the different stakeholders. But when it comes to research teams, agile methodologies are often perceived as limiting creativity and depth of research. After years of trial and error, I believe that I finally have the secret sauce of doing this right!
In this article, I’ll explain how the Cloudinary research team has integrated tasks, like literature reviews, long research, benchmark creation, and baseline implementation, into an agile workflow.
Our first step in every research project is to perform a literature review. This process involves reviewing academic literature and technical information around a specific challenge we’re trying to tackle. Since this is an exploratory step, we don’t know in advance the amount of time it will take to finish the job or the outcomes. Different factors like the number of papers that need to be read, their relevance, and the quality of results are known only after we finish the task itself, making task planning challenging, especially in an agile workflow methodology.
Over the years we’ve learned that the literature review step is critical for efficiency. The conclusions we draw help us understand whether it’s feasible (or worthwhile) to solve a specific problem and whether we need to try to find a unique solution.
To fit the literature review task into an agile process, we’ll apply the following procedures:
New research topics are allotted 10 work days, while revisiting a known topic can take three to five days. In the case of a new topic, the first week is used to collect the different papers and perform a shallow review. This prepares us for the second week, in which we assess the most important papers that have a higher potential of providing valid solutions.
The literature review task ends with a one- to two-hour team meeting, in which the person who leads the task gives a summary of the literature review. To set up the necessary slides for the meeting, we reserve an additional work day. This formality is the most important part! Before we had this formal meeting, the outcome and quality of the review task varied drastically due to different reasons, among them, a researcher focused too much on a single paper, didn’t document their process, or took longer than expected.
This formal process ensures that everyone shares their knowledge before deciding how to tackle a new challenge. Team members, especially new ones, that need to handle an unfamiliar topic can have a better starting position, which will save them a lot of time further on.
Armed with more information we schedule a special planning session to elaborate on the next tasks and estimate the time they should take.
When developing a product or service, it’s important to establish specific, quantifiable goals to assess its performance and impact. Success criteria ensure that all stakeholders have a common understanding of what the project aims to achieve, what metrics will be used to determine success, and how progress will be evaluated. Having a shared understanding of the success criteria also aligns efforts towards the same goals and helps to prevent misunderstandings or disagreements later on
While the benefits of setting goals are clear, it is usually difficult to define these criteria and gather the necessary resources to measure them properly. We find it especially true when the future customer response is still unknown. Yet, we try to give the appropriate attention to this topic during our planning sessions, since we found that it helps us to direct the research better and to prioritize our tasks.
Having a long process about the success criteria might contradict the agile spirit, and since it might depend on different non-controllable factors, it might be very tricky to handle. To handle this correctly, we apply the following procedures:
This is a preliminary optional step that we use to define the benchmark ourselves and prepare a list of questions along the way. We try to answer these sets of questions during the next steps.
Usually centered around meetings with different people in our company, e.g., getting relevant information from our data team, contacting customers with the help of our Customer Success Managers, aligning with our product team regarding previous product constraints, etc. To estimate the time for all these meetings, we define a maximal time bucket for this task and subtract it from the overall working time. Usually, the time bucket is five days. The “definition” of done is a report with answers to a list of questions we prepared in advance.
If necessary, buying/collecting the necessary data that simulate an actual user scenario. This data might be different from our training data since it might simulate the end product and not exactly the algorithm under the hood.
Development tasks in which we implement the metrics that measure the success of the algorithm and the necessary scripts to evaluate any future solution against the collected test set. Usually, it is a mix of known metrics that are used in the literature and some specific metrics that target our business scenarios. This task is broken down into subtasks that take between 0.5-2 days each.
We evaluate a baseline solution using the created benchmark scripts and document the results. The result we get from this step will help us to understand whether we need to do further research and how far are we from having the minimal quality threshold. In a lot of cases, the baseline solution did a decent job and we could deliver a solution more quickly than expected.
Before we rush into an extended research project, we need to understand whether there’s some naive and simple solution that we can use, which might produce sufficient results. Besides having a baseline to compare to, we can use the baseline solution to create a POC for the product/feature. Engaging with potential customers early helps us to validate our assumptions, learn about usage patterns, and have more meaningful insights about our research challenges.
A few tips about this step:
- The outcomes of the literature review should point out the right baseline you should choose. Therefore, you can plan this step only after the literature review
- Definition of “Done” running the benchmark and documenting the results.
- Time estimations should take several days (three to five days) to implement, not including testing and documentation.
It’s difficult to measure the amount of work and time it would take to reach the success criteria. In the first stages of the process, we took the necessary measures to understand what should be our next steps, but we can’t know for sure whether these steps would work. More than that, like any agile workflow, our preliminary assumptions might change over time due to new information regarding the use cases, potential customers, and other restrictions we weren’t aware of. So it can be hard to communicate the need for a long research project to higher management and give them the necessary information for them to make good decisions.
On the other hand, researchers need time to think and explore, without taking any risks we’d end up with the Streetlight Effect. To handle these challenges, we apply the following procedure:
- Save time for exploratory research. It might change over time, but usually it is around 30% of our time. You shouldn’t skip the necessary steps for these research projects (i.e. literature review, baseline, etc.), but the decision process about continuing with the research project is less strict than others.
- Write down your steps. Writing down your next research steps and reviewing them during our daily meetings help others to review the process and give help if needed. The steps should have a clear “done” definition which involves comparing the result against the baseline solution. This step is important to avoid rabbit holes, in which a researcher wants to tackle an unimportant problem.
- Review your plans every two weeks. Planning sessions each month are long and thorough. However, after two weeks we review our plans to see that our assumptions, time estimations and other factors are aligned with our work.
- Plan two to three different possible research paths. Prioritize the different paths according to their estimated difficulty and their success potential. Start from the easiest solutions that still have potential to succeed.
- Get the stakeholders involved. Taking a decision about having a long research project is hard, especially if your resources are low. To help in this process, it is important to get the different stakeholders involved, showing them the current results and explaining the next steps. The current status might be good enough for several potential use cases, which might show we are on track and help them feel more comfortable.
- Challenge accepted!
- Source: Either a product manager request or an idea that came from some ideation session in the company.
- Understand the basics. Is it a common request? Who needs it and why? What are the known constraints? Deadlines?
- First decision with management: Do we prioritize this for the next quarter?
- Day 0: Understand the research challenge.
- Define the technical challenge. What are the technical issues our customers are facing?
- Does it need research? Why can’t it be solved using the current tools we have?
- Prepare for the literature review. What do we need to learn in order to gather knowledge for this challenge?
- Do we need to collect additional data to start the research project?
- Week 1-2: Literature review.
- Define who’s responsible for this task.
- Schedule the literature review meeting in advance (end of Week 3).
- Week 3: Quarterly planning session.
- Ensure you understand the success criteria.
- Ensure you have the necessary data. If not, we’ll define a data task to collect the necessary data that’s needed for training and evaluating the results on relevant use cases. We don’t proceed before we have the necessary data.
- Define the tasks that are needed to create the benchmark, baseline solution, and two to three other alternatives based on the literature review.
- Time estimations are given for each task.
- Communicate the plans with the different stakeholders. The time you need, when the next checkpoint is, what the challenges are.
- Week 3-7: Sprint work!
- Benchmark and the baseline solution are prioritized first.
- Drill down and define the tasks that are needed to implement the first steps. Try to break down the tasks into 1-3 days or less. Avoid having long tasks.
- Week 5: Mid-sprint short planning status.
- Has anything changed? Are the time estimations correct? Do we need to start over and go back to literature review?
- Week 7: Planning the next sprint.
- Review the results from the baseline solution on the benchmark. Does it reach the success criteria? If not, why?
- Drill down into our solution to understand what is needed to implement the first alternative solution.
- Save time for documenting the results and the implementation process.
- Communicate progress and results to the different stakeholders.
Applying agile methodologies to a research project requires a lot of attention. From my experience, after I could tweak it correctly, the team velocity changed drastically, even for the most vague and exploratory research projects.
Balance is the key here. Forcing the methodology “by the book” will kill creativity while a lack of structure and scheduling will lead to wasting time and make the team less impactful. If you want to use agile methodology in your research teams, invest time and effort to find this balance and adjust the process according to the present situation.
