Interviewing Software Engineers

Summary: I recently went through the interview gauntlet in search of my next job. I was reminded of how badly most companies interview software engineers. Here are some anecdotes and suggestions for companies to improve their process.


Interviewing Software Engineers

I have done a lot of software engineer interviewing in my career. I've been on both sides of the table of course. Most of the interviewing advice you read is targeted at the candidate. "Study these principles," "memorize this model," "review these standard questions," "arrive early," "play their games," "come prepared with good questions," etc. There is very little advice for the interviewer available, and that is evident from the many, many bad interviews I've suffered through over the years.

I'll classify bad interviews into the following categories: Too Hard, Too Easy, Inconsiderate, Inconsistent, Unrepresentative.


Too Hard

This type of interview is the easiest to spot. It begins with a culture of elitism from the interview team. When I worked at HP one of the hiring managers in our group actually said, "If they didn't go to a top-ten school why are we even talking to them?" Young, hot-shot engineers often treat the interview as a pissing contest or a hazing ritual. Grizzled, grumpy veterans with a "what's wrong with kids today" attitude will be merciless.

Proponents will say, "We need to set the bar high." Fair enough, but interviews that are extremely hard are less effective at actually determining the quality of the candidate. Simply put, hard interviews assume the candidate has fundamental and mid-level skills and focuses on deep understanding in a small set of topics.

The result of the Too Hard interview is too many good candidates get passed over, and occasionally you hire an idiot that happens the know a lot about the one thing you grilled him on.

My advice: Start with easy questions and ramp up difficulty based on how well they are doing. Ideally spend only a short time on easy questions, about half of the time on medium, and a fair amount of time on hard. If there are doubts about the candidate have them back for a second interview focussing on suspect areas or specific skills.


Too Easy

I was once on an interview where, at the end, I wasn't sure that I had said a single thing that demonstrated I had the skills for the job. As it turned out the interviewer, my future boss, had no experience in this field or even as a manager. It was lucky for them I was qualified. But it was also stupid of me to take the job, which turned out to be the worst job I've ever had.

The interviewer is either incompetent, apathetic, or timid.

The result of the Too Easy interview is you'll hire anyone who walks in the door. Almost as bad, you put people in roles that are at the wrong level for their skill set (either too high or too low) because there is no way for the interviewer to distinguish good from great from okay. Finally, the candidate doesn't get a sense of what is actually required for the job and whether they would be a good fit for it.

My advice: Geez, just have good interviewers. How hard is that? Hard. Well, here are some one-off suggestions: Don't send a client engineer to interview a server engineer. Don't send a my-first-job engineer to interview a 15-year veteran. Reduce the likelihood of apathy by having the affected team interview the candidate, i.e. don't have a neutral hiring committee. Don't let timid or socially-awkward people interview. (Yes, this last one can be challenging.)


Inconsiderate

I once arrived for an interview and couldn't find the company in the building. I checked the recruiting email and the company's web site. I was in the right place. I called the recruiter. "Oh, you're at our old address. We recently moved."

A few weeks ago I interviewed at a startup. The HR rep told me to ask for her, but she wasn't there when I arrived. I was directed to a room by one of the engineers. I didn't know how long I was going to be there or who I was interviewing with. "Hi, my name's Mike. I'm the client lead. Well, let's get right into it. I want you to write code on the board to solve the following problem..." An hour later another engineer did practically the same thing. Then it happened again. After talking with the VP of Engineering and the CEO I finally met the HR rep. My performance was less than my potential, but I was pretty sure I didn't want to work there anyway.

When I was a hiring manager at EA the internal recruiter routinely would fail to greet the candidate in the lobby. The engineers would end up retrieving the candidate 10 minutes late. By the time the first session started they had lost 15 minutes.

When you are inconsiderate to candidates you make them not like you and not want to work for you. Pretty straightforward. What is less obvious is that you throw off their game. You agitate them and likely negatively affect their performance.

The result of the Inconsiderate interview is you miss out on good candidates because they didn't seem that good or YOU didn't seem that good.

My advice: Make sure it is clear who is responsible for recruiting, set clear expectations, and take action against those who aren't doing their job. (Note that there is no shortage of recruiters. If you have a crappy one fire him and get a better one.) Greet the candidate, give them a schedule, give them breaks, provide lunch if it's lunch time, thank them for taking the time to come in, make sure they know how to get to their hotel/airport/whatever, walk them out. Don't be a dick.


Inconsistent

I'll jump straight to advice here. Be consistent with who is interviewing and what types of questions they ask. Not only does this allow you to compare apples-to-apples with multiple candidates, but it allows you to inspect and refine your interview process. Questions that don't seem to predict ability of the candidate can be replaced in favor of ones that do. Interviewers that don't ever have much of value to add to the post-interview discussion can be replaced with ones that do.


Unrepresentative

This is my favorite category. And by favorite I mean least favorite. I'll go so far as to say I believe this is the most frequent bad interview type. Also, it's probably the worst for your company.

I once applied at a social gaming company for a Flash/ActionScript job. Even at the time I was a Flash expert with several years of experience and a few projects under my belt. After a successful phone screen they gave me a take-home programming test. I was to solve a problem... in JavaScript.

Many of the engineers I've worked with love to pose logic problems. "So you have this gold brick, and you can only cut it four times. Each day you need to pay blah blah blah." "You have two stables and all these horse. Each horse runs at a different speed. You want to get all the horses from the first stable to the second stable blah blah blah." Okay, these kinds of questions are bullshit. First, you are assessing if either they've heard the problem before or they're pretty smart. You don't know which. Second, while smart is good is doesn't directly translate to coding ability. Third, if you really need someone to solve these types of problems on the job there are known solutions to them that can be looked up. Forth, you don't need someone to solve these types of problems.

"Here's a problem. Now start coding the answer on the whiteboard. You explain what you’re doing with half your brain while the other half of your brain performs the entirely different activity of writing code. I'll sit behind you awkwardly judging your every move. Each step of the way I will interrupt you for things like missing parenthesis, off-by-one errors, and typos in variable names. You'll need to know the exact function signatures of any library call you make. Oh, and I don't know the language you want to code in, so I'll just have you code my favorite language that you haven't used in over five year." Yes, dreaded whiteboard coding happens pretty much every interview. Because on the job that's how you code. You don't do it quietly, by yourself, after careful consideration of the problem in context of the entire system. You don't have an IDE with syntax highlighting and code completion. You don't have reference material. You can't quickly write and test small increments of code. And you certainly aren't proficient in the language that you do your work in. Oh, wait. #sarcasm

I was at a conference on how to be successful at engineering outsourcing. In relation to hiring I heard excellent advice from a manager from BioWare Austin. They worked with a company in Russia to do a significant amount of work on a project. He said, "Interview like you work." The leads communicated via email, in English, with system specifications for the Russian developers. The developers did their work however they wanted, checked in code, and emailed back that it was done. For this type of relationship, he said, it didn't make sense to have a traditional conversational interview. Instead he handed them a piece of paper with a problem, provided them a laptop with a Java IDE, and left them undisturbed until they were finished.

For the Unrepresentative interview you waste time assessing skills that aren't actually needed for the job. You might end up with a puzzle-loving, unintimidated whiteboarder, but that doesn't mean you have a good engineer.

My advice: Include a take-home programming test BEFORE the candidate comes in. The test should take about four hours. Assuming it looks good bring them in and spend at least thirty minutes discussing it to make sure 1) they actually wrote it, and 2) that they can discuss design considerations and trade-offs meaningfully. NO WHITEBOARD CODING. Save the whiteboard for drawing architectural diagrams and other things that engineers actually do at work when discussing things with each other.


Conclusion

The bottom line is you want candidates to perform their best, you want to ask a broad range of questions to accurately assess their abilities, you want a consistent interview process with a qualified and professional interviewing team, and you want to interview in a manner and on the topics that are relevant to the position.