This article was originally posted on the Storj Labs blog.
In case you hadn’t heard, Storj Labs is building a decentralized cloud object storage service. Why would we do such a challenging thing? At a basic level, it’s because we believe the internet can be better than it currently is and we see how to improve it. We believe your data is worse off being stored in the centralized data centers of five multinational mega-companies.
Solving this grand problem requires us to solve many difficult sub-problems. Even though we are rigidly focusing on the simplest ways to solve these problems, the simplest solutions that can work in our space are still intensely complicated. To build our decentralized service, we need smart ideas and a capable team that isn’t afraid of solving challenges for the first time. If this sounds like fun to you, then you might be a good fit to join us on this adventure!
Today I want to introduce you to our engineering interview process.
Before we talk about the details, it’s important to list what we value, and what our goals are. We have two primary and co-equal objectives with recruiting:
We’ll talk about being welcoming first.
First off, pursuing a diverse team is the right thing to do, full stop. We recognize that the tech industry can be—and has been—an inaccessible place for people of underrepresented groups, so we value gate-opening (versus gate-keeping) wherever we can.
Fortunately, a welcoming, inclusive, diverse team has a number of benefits that make it the clear choice even if it wasn’t clearly the right thing to do! Monocultures breed all sorts of wacky pathologies, which is precisely the sort of thing we’re fighting against technically with our decentralized storage platform!
Did you know that workforces with more gender, ethnic, or racial diversity tend to perform better financially? According to McKinsey:
Companies in the top quartile for racial and ethnic diversity are 35 percent more likely to have financial returns above their respective national industry medians.
Companies with sustained high representation of women board directors (WBD), defined as those with three or more WBD in at least four of five years, significantly outperformed those with sustained low representation by 84 percent on return on sales, by 60 percent on return on invested capital, and by 46 percent on return on equity.
When you look at the hard data, it’s clear that more diverse teams are more flexible, efficient, and effective. We want those benefits for our team too!
As an aside, this is specifically a interviewing post, but diversity is not the end-goal of the journey. We said diversity and inclusion. Diversity is having many different backgrounds in your organization. Inclusion is having all of us feel welcome and wanting to be here. Literally centuries’ worth of company leadership advice talk about how important team morale and a sense of belonging is to all of us as human beings.
So, it’s not enough to just get employees through the door. To build a great team, you have to keep them! In a future blog, we’ll share more about what we’re doing to support inclusion through our new Diversity & Inclusion Council. We’ll also share more about what diversity means to us because it’s more than just hiring more underrepresented people, such as women or LGBTQ employees.
To solve the hardest problems in decentralized storage, we absolutely need a world-class team. Great developers love learning, and therefore enjoy and seek out opportunities where they can learn from and grow thanks to others on their team. This means a world-class team tends to attract more world-class developers. Why do so many engineers want to work at, say, AmaGooSoft? It’s not only about the money.
Unfortunately, identifying which new recruits are world-class is very hard, and a lot of companies take lazy shortcuts. One such shortcut is hiring based on time-in-the-industry. Don’t get me wrong, having more experience in distributed systems is a good thing for us, but requiring that someone has stuck around through the worst of our sometimes-polluted industry has too much of a filtering effect on our other top goal—a diverse and inclusive workplace. So, we need to find a more effective way to identify which candidates have top-shelf problem solving, communication, and programming skills, whether they have years of experience or are new to the industry. All backgrounds are welcome!
In my experience, the best predictor of how successful someone will be at solving complex problems after six months on the job is their learning rate. We’re specifically not looking for proficiency in a certain programming language, or knowledge of a specific skill, precisely because learning a language or skill should be the sort of thing our top candidates eat for breakfast. The best candidates are the sort of people who gravitate toward hard and confusing problems to tackle them and understand them, instead of flinching away from the discomfort of not knowing. We are building a decentralized cloud storage platform, after all.
So, while we value experience, we also want to find candidates that demonstrate they are eager and adept at throwing themselves at hard problems they don’t already know how to solve—and solving them. Great candidates are lifelong learners and this factors into our process.
To achieve our two major values, we’ve had to make calls on a number of trade-offs. There are definitely downsides to our recruiting and interviewing process, but we feel that on-balance, the trade-offs are currently worth it. We’re open to suggestions though! We’re constantly trying to learn and make improvements, and, considering interviewing ironically seems to have its own Full Employment Theorem, we will probably be on this journey indefinitely. (Again, we believe in lifelong learning.) Given our values, if you can think of a way to help improve our process, please leave a comment or shoot us an email!
The recruiting stage of our pipeline is simply finding people who may want to come work for us.
In an interview with Christiane Amanpour, Jon Stewart had this to say about diversity in comedy:
It started out as a male-dominated field. It’s not a particularly welcoming field—you sort of have to come out there and cut your teeth on it. I’ll tell you a story: So we had, on The Daily Show, there was an article about us that said it was a sexist environment, we didn’t have women writers. And I got very offended by that. I was very mad. I was like, “Are you saying I’m not a feminist?” I was raised by a single mother. She wore a T-shirt that said, “A woman needs a man like a fish needs a bicycle.” And me and my brother were like, “I think we might be men?” So I was mad—how can they say such a thing? And I went back to the writers room, and I was like, “You believe this, Steve? What do you think, Greg? Dave? Tom? Mike?” And then I was like, Oooohhh. And it was right.
But the reason it was right was not necessarily one that we had seen before. We had put in a system of getting writers where there were no names on it. We thought that’s color-blind, gender-blind, et cetera. But what you don’t realize is the system itself—the tributaries that feed us those submissions—is polluted itself…
But do you see what I mean? It’s a systemic issue, and I think what can mostly help change is when you open up new tributaries to bring in talent, and then they grow, and then they help grow their communities and tell their stories.
Does this sound like the tech industry to you? It’s not enough to simply say that you encourage candidates who might add diversity to your team to apply. You must find inputs to your recruiting funnel that represent greater diversity than the diversity in your field. It takes extra energy and effort.
Like The Daily Show, the main technical evaluation stage of our interview process is name-blind. We are interested in hiring the best candidates based as much as possible on relevant qualifications alone. But Jon Stewart’s observation is a key insight to our recruiting process: if we only have a certain type of candidate, then all hires will be that same type of candidate! So while we don’t have quotas, we must continually analyze our incoming funnel of candidates to determine whether we are finding candidates with diverse backgrounds and experience. We are of course interested in hiring the most qualified candidate regardless of any other trait, but we also believe it is worth doing significant extra work to help build a diverse and inclusive workforce, which we believe will help Storj Labs be the best company it can be. This means that if our incoming pool of candidates does not represent these values, then we need to cast a wider net in our search. We reject the idea that diversity efforts are about lowering the bar–increasing diversity requires that we widen the net and include more people so we can raise the bar.
We will continue to seek ways to improve our recruiting, and we welcome your thoughts on ways to make sure that everyone has an equal shot with Storj Labs. Again, if you have a suggestion on how we can improve, please let us know.
Each stage of our interview process takes work, so we want to quickly eliminate people from the process who aren’t going to be the right fit for that position.
In our screening stage, we want to ensure there won’t be some sort of logistical problem. Is the candidate requesting more compensation than we have budgeted for the position? Is the candidate applying for the right job listing? Is the candidate lacking relevant job experience entirely? Where does the candidate expect to work, and will it require a relocation? Does the candidate seem communicative and not a jerk? Can we answer any questions?
Hopefully most people sail on through this step.
This part of our interview process is the biggest, most time consuming, and potentially the most controversial, precisely because of how many trade-off decisions it makes. So, before we jump into what the downsides are, and why we do it anyway, let me just outline what we do.
First, we invite candidates to an interview-specific Slack channel with a randomly chosen pseudonym. Originally we chose random animal species as part of the pseudonym, but we then switched to names of stars, because you’re all stars! Interview candidates join the Slack channel to anonymously talk with the team.
Second, the hiring manager posts a link to a homework problem in the channel. We are trying to select a problem that:
We don’t set a deadline on the assignment because we want to be flexible with candidates’ schedules.
Third, we answer any questions the candidate has, discussing potential solutions and tradeoffs, and let the candidate get to work.
Finally, we run the homework submission against tests and evaluate their assignment’s code against a checklist. We pay $500 USD (in STORJ tokens) for problem solution submissions (whether or not they work completely).
So, why do we do it this way?
First, we want our interview to hinge on evaluating a candidate’s ability to do work in as real an environment as possible. We want them to use their IDE, their programming language, have access to documentation, relieve time pressure (if possible), and see how they communicate remotely (much of our own team is remote). We do our best to engage our candidates in a discussion about the problem; the Slack conversation is a big part of what we’re looking for. So, we want to evaluate the candidate on work that is as much like real work as possible.
Second, we use pseudonyms to let the candidate’s work stand for itself. At this point, we want this stage of our interview process to simply select the best possible candidates, independent of as many other factors as possible. This is how we use our focus on diversity to raise the bar—by including a wider pool of applicants, we can be even more selective at this stage.
Third, we use a hard problem that isn’t completely specified. Just like real tickets we unfortunately file sometimes, the problem statement has assumptions that aren’t clear and require some level of additional requirements gathering and clarification. Just like international math tests, we want a problem that most people won’t ace. The greater the fidelity of the test, the more an excellent candidate will stand out. This, combined with our lack of specific experience criteria, is intended to allow inexperienced but sharp candidates the ability to shine.
Fourth, we pay people. The major downside of our problem is it takes time, potentially time the candidate doesn’t have. This is something we’re pretty torn about. Unfortunately, a homework problem might eliminate people with busier home lives or who need to work elsewhere until they land the job with us, which is why we don’t set deadlines on the assignments. It’s challenging to compress our hard problem into the span of an hour, but we’re hopeful that compensating our candidates will help.
Fifth, we try to grade as evenly and as routinely as possible. We have a checklist and we have a test harness. Even though we’re already doing the interview at this stage name-blind, we still want to avoid as much bias as possible that might cause us to prefer anything besides what is explicitly stated as criteria in the homework problem description.
So, you’re probably thinking, “Some of the best candidates will most certainly balk at this huge hurdle you’re placing in their way.” To this we can only say, you’re right. Ultimately, we believe two things that make us pursue it anyway: we would rather have a process that occasionally rejects candidates that would have been a good fit than one that occasionally hires candidates that would have been a bad fit, and we believe great candidates are often just as interested in working on great teams (with stringent hiring criteria) as great teams are interested in hiring them. Many of our fantastic team members were more than willing to jump through this hurdle to join us!
However, we do make a concession. For candidates that simply do not have the time for the assignment and will be forced to pass on a sweet job possibility with Storj Labs otherwise, we are happy to consider some other sample of work. If the candidate is able to provide us with some code sample they have created with sufficient complexity to be graded using our homework grading evaluation checklist, we will anonymize it and pass it to our review team. We will ask the candidate to explain the project to us in the anonymous Slack channel, so that we get a feel for how the candidate communicates. Of course, we expect that any candidates using this option will make sure that the sample is something that they have permission to share with us–please do not send us some third party’s intellectual property. Remember, you may not have the right to share a work sample with us even though you created the sample. Your work for another company likely belongs to that Company. We will reject any candidate who sends us third party IP that they do not have permission to share, or another person’s work as if it was their own.
It’s a bit harder to grade these types of submissions evenly and fairly, so we are more picky with these types of submissions.
If the rest of this post has failed to convince you of the benefits of our system, that’s okay! We are also willing to do a whiteboard interview with difficult algorithmic challenges. Let’s just say these won’t be Fizz Buzz-style questions. These will be hard questions, and if you want to go this route (it is potentially the least time consuming), then we will be looking for you to dazzle us. We only leave this option available to be as flexible as possible, but be warned that we are the most picky for people who choose this option. We will certainly pass on some good candidates who choose this option.
Our last phase is a team interview. This is just as much an opportunity for us as it is for the interview candidate. We want you to get to know the team! We want bidirectional question asking. Interview candidates should grill us on anything and everything (if they want). Otherwise, we’ll be asking performance-based interviewing questions. Sidenote: the U.S. Department of Veterans Affairs of all places has the best collection of these we’ve found so far!
Assuming no one finds any red flags in the team interview, the formal interview process is complete. We may have a few more follow-up questions, and you might as well. Sometimes, we must be forced to make hard choices between well-qualified candidates. But ideally, for candidates who make it this far, it’s time for onboarding, which is worth a separate blog post.
We’ve spent a lot of time thinking about and tweaking this process. Aside from the time our homework assignment takes to complete and grade, we feel like this is one of the better interview processes we’ve seen. Please let us know how we can improve!