Coinbase
Web3 Developer Screener
* This work is protected under an NDA.
ROLE
User Research Intern
TIMELINE
July 2022 - August 2022
GOAL
Create a set of screener questions that determine if someone is a Web3 developer to enable teams with effectively identifying candidates for user research or adjacent functions (e.g., job recruitment)
METHODOLOGIES
Questionnaire Development, Expert Interviews, Literature Review
Screening is an important aspect of data collection, because it helps ensure the target participants are found for a user research study. Not having clarity about what a Web3 developer should know or do makes it challenging to identify who is and who is not a Web3 developer. In order to support a growing developer ecosystem at Coinbase, it is necessary to create screener questions that determine if someone is a Web3 developer or not to effectively identify candidates for user research or adjacent functions.
Project Journey
As a user research intern on the Coinbase Cloud team, I was in charge of this project and performed the following tasks:
-
Creating research plan
-
Writing a moderator guide
-
Recruiting participants
-
Conducting interviews
-
Synthesizing insights to build understanding of Web3 developers
-
Creating a screener to effectively identify Web3 developers
​
Given the business impact of the screener, this project required cross-functional collaboration with internal Web3 developers and the HR team at Coinbase.
Business Objective:
-
Support a growing developer ecosystem at Coinbase with an effective way to identify candidates for user research or adjacent functions
​
Research Questions:
-
What are people's definition of Web3 developers?
-
What experience and knowledge should a Web3 developer have?
​
Research Goal:
-
Understand what qualities determine if someone is a Web3 developer or not
Foundational Research
In order to design an effective screener, I built my foundational knowledge of Web3 developers by conducting a literature review, 30-minute expert interviews with 5 internal Web3 developers, and conversations with 2 internal recruiters for Web3 developers.​
​
1. Understanding Web3 Developers
-
What is the definition of a Web3 developer?
-
What sets a Web3 developer apart from a non-Web3 developer?
-
What key concepts does a Web3 developer need to understand (e.g., blockchain and smart contracts)?
-
What essential tools does a Web3 developer use?
-
What experience should a Web3 developer have?
​
2. Foundational Research Artifacts
From my interviews with developers and recruiters, I collected the following artifacts as ways to categorize Web3 developers:
-
Categorization 1 - Specialty
​
​
​
​
​
​
​
​
​
​
​
​
​
-
Categorization 2 - Experience
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
-
Candidate-Role Map
Screener Development
From my foundational research, I determined that broad knowledge-based questions are the best approach to initially screen who is and isn't a Web3 developer. I iteratively refined a set of screening questions, collecting async feedback from internal Web3 developers each time. With each iteration, I narrowed down the set from 10, to 5, to 3 questions. The 3 questions focused on on smart contracts, nodes, and consensus mechanisms--representing the breadth of knowledge most Web3 developers should have. As a result, the screener would simplify Coinbase's recruiting process and make it easier to weed out non-Web3 developers.
Iterations of screener questions for Web3 developers
Challenge
There is no definitive method for knowing who is a Web3 developer. Based on my expert interviews, I gathered varied definitions of Web3 developers. I realized that we don’t have a simple and clear way of determining if someone is actually a Web3 developer for research recruitment. Thus, I needed to come up with screener questions that were broad enough to encompass many, if not all, categories of Web3 developers, but still specific enough to weed out non-Web3 developers.
Research Report
Besides documenting my methodology, research, and data collected from the project, I added two key insights (including the finalized screener) in my report. Afterwards, I distributed my report to the Research team on Slack and presented my findings to the company-wide Final Presentation at the end of my internship.
​
Key Insights:
-
There is no one type of Web3 developer. The web3 space is rapidly expanding and we’re identifying that Web3 developers are becoming highly specialized, gaining specific knowledge and experiences that can’t be easily rolled up into uniform criteria.
-
So what? Coinbase won’t be successful if we just build for “Web3 developers”. We must identify the attributes that meaningfully differentiate their needs, resources they have, and decision-making power which ultimately results in the products they use.​
-
-
There are 3 questions to identify Web3 contenders from the pretenders. We determined that broad knowledge-based questions are the best approach to initially screen who is and isn’t a Web3 developer.
-
So what? The screener questions will make recruiting developers more efficient and give the team greater confidence that we’re collecting data from the right audience.​
-
​
Finalized screener questions for Web3 developers
Outcome
At the end of this project, I submitted my final deliverables:
-
Research plan
-
Research report
-
Interview notes and recordings
-
Screener questions for Web3 developers
​
My work left a tangible impact for the Research and Marketing Insights teams:
-
Provided more data to reinforce Coinbase's needs to establish standardized Web3 developer segments to inform product, sales, and marketing decisions
-
Empowered teams with a tool to more efficiently and effectively screen Web3 developers
Reflection
Understanding a complex field. Conducting foundational research for both Web3 development and developers was harder than I expected. Despite having a software engineering background, I was confused by the terminology that I came across during my literature review. As someone new to the cryptocurrency space, it was difficult to grasp concepts, such as gas fees. However, the time and effort put into establishing foundational knowledge of Web3 helped me immensely with understanding my discussions with internal Web3 developers during expert interviews.