What we learned A/B testing job listing text and why you should do it too

As a hiring manager at Manifold, I want to make sure my job listings are as inclusive as possible, so we can find the best people who will be successful here and add to our culture.

And as a product manager with open positions and access to user testing tools, I naturally began testing. We want to test features and ideas as soon as possible — so why don’t we do this with something as important as who we hire, and how our companies present themselves to the world?

Striving for diversity in hiring is important because it’s the right thing, and it’s critical to our long-term success: we know the better we do at it, the smarter we’ll get, from the top down to individual teams seeing new opportunities and spotting more potential hazards as they work.

I’m lucky enough to have the autonomy to run with things like this and our tool allowed demographic splits, so I ran a series of groups:

• wide open
• identified as women
• identified as women and had children

I did this because:

1. I’m a white cis het man without kids, and I want to discover when I’m writing with blinders, or making assumptions, and I know that someone who is interviewing may not feel like they’re in a position of power to ask for things or point out problematic language, so I want to try and actively discover issues and address concerns up front.
2. in tech, especially in startups, we skew young and childless, and if you’re looking for “culture fit” you’re only going to hire people like you

(I did not look at non-binary gender because our user testing service forces testers into a binary gender choice. I have given them pointed feedback on this and would love to hear about more inclusive alternatives.)

I started small: I asked participants to read through a two versions of a problematic time zone bullet point and a benefits listing, answering questions (“what’s your opinion of x…”) after each. The first was very close to what we’d used in the past, and the second a new draft intended to be more expansive and inclusive.

I knew it wasn’t the most rigorous test, that there are many other intersections that need exploring, and that my variant text would need honing. But we build, we measure, we learn, right? So what’d we learn?

Time zones are the worst

Manifold is headquartered in Halifax, Nova Scotia, and about half the company is there. The rest is remote. With the vast majority of the company on the Atlantic and Eastern time zones, it’s very difficult to be far off that.

We’ve expressed that as:

We believe in having people work from where they are most productive. If you want to join us in our cozy office in Halifax, welcome! We’d love to have you. If you are remote, you are also welcome! Ideally, you live within 4 hours of the Eastern time zone.

That confused a surprising number of people. Some testers on the West Coast did not realize we’d hire them. A Jamaican tester who understood everything else was entirely baffled.

A cynical view might be “great, if they don’t understand relative time zones, you know they’re not worth hiring” and I would say “seriously? Have you ever worked with time zones? I want to hire people who don’t have a bunch of dendrites knotted up remembering that stuff.”

The variant tested was:

As a remote-centric company, you’re free to work anywhere as long as you’re online and available for our core hours of 12–4 Eastern, or 9–1 Pacific.

This eliminated entirely the question of where people lived. Everyone got it immediately!

It introduced a new problem, though: some testers then thought the job was only four hours a day. Two swooned at the thought. “This sounds like a dream job” one said. It is, but it’s not a half-time dream job.

I’m still working on wording. Time zones are the worst.

To the bennies

Here’s our first version:

We offer:

• health, vision, and dental insurance for you and your family
• 15 days paid vacation days to start
• competitive compensation
• equity opportunities
• available 401(k)

And the re-written one:

We offer:

• flexible working hours
• excellent health, vision, and dental insurance for you and your family that starts on your first day
• vacation you’ll be encouraged to use, plus an all-company end-of-year week off
• competitive salary
• equity opportunities
• 401(k) with no match (we’re brand new)
• yearly professional development allowance of $2,000 for conferences, training, or anything that’ll help
• health and wellness allowance
• home office allowance for remote workers
• we’ll issue you a sweet laptop, big monitor, and all that good stuff

All of that is true, by the way, and since I did these tests Manifold’s added 16 weeks of fully paid parental leave for anyone who becomes a parent (regardless of gender or if it’s by adoption or childbirth).

Things you think you know are likely right

The vacation you’re encouraged to use was lauded. Testers with kids noticed and appreciated the health care for them and their family that started immediately. Both of those were frequently cited in summary comments as indications that we’re a good place to work and care about our employees.

Yay!

“Competitive salary” is standard text that got a consistent “whatever, everyone says that” reaction. I know! I know. If you can’t offer meaningful ranges, what can you say here that is meaningful? “We conduct regular salary surveys to ensure our pay is in the top x%?” What if you’re a startup and you pay well enough, but much of the compensation is in startup equity and not cash?

You’ll discover hidden assumptions — the 401(k) problem

I tried two explanations of our 401k plan (for Americans), which does not have a company match. This is not unusual at your tiny new startups. I wanted to be very clear that there was not a match, though, and tried to slip in a why.

The original had “401(k)” as a bullet point. The variant said “401(k) no match (we’re new)”

This did not go well.

If you just say you have a 401k, there’s an assumption you have a match. This was called out explicitly in commentary (and I cringed every time) but really called out when we said we didn’t have it — it generated a very negative reaction: “Why no match?”

That set up another unexpected reaction — further on in second version, I listed out all of our fringe benefits. Wellness benefit to go towards gym membership! Professional development.

It ends up being a good chunk of bullets. What I didn’t expect was for people who had a negative reaction to the no match in the 401k, many felt even more negative about the additional benefits.

“Why are you offering all these other benefits but no 401k match?” one woman asked in the middle of the list, returning to it in later summary answers.

I had hoped there would be enough there to start off-setting the match. Who doesn’t think “Allowance towards home office? I can buy a V60 and make my own artisanal coffee?”

Everyone. If anything, reminding people we cared about the little things made not having a match more prominent.

You’ll introduce new problems that illuminate old issues

You probably already flagged one of the bullet points. We used to say

Whether you’re at Manifold HQ or remote, we’ll buy you your laptop, your choice of flavor, along with a monitor and the accessories you need to work and potentially travel well.

I thought that was too boring and formal, and also, it was the last bullet point and I was tired. I just waved my hand at it, and said:

We’ll supply you with a laptop, monitor, and all that cool stuff.

This… if you’re at a standing desk, please hold onto a solid surface — went over even worse than the 401(k) text. No one liked it. Zero comments that ‘this hip person gets that it’s table stakes.’ Almost everyone in an all-woman test group reading the line out loud delivered the second half with at best resignation. 100% of women with kids did. Some eyerolled so hard I could hear it. “I know what the writer was saying, but…” on woman said, trailing off.

Okay, so it’s lazy text. How big of a deal could it be?

Huge. It nearly set fire to the whole test. For many of the people who reacted, it destroyed the credibility of the entire company benefits description.

“Unprofessional” was a running theme. People who liked the benefits and seemed positive about us as a whole shifted into a “I don’t know…” tone for a bit, and then returned to mention it in summary comments. The more generous assumed we’d just screwed up writing that line.

I knew that line wasn’t great, but seeing how strongly people reacted and what it did to their overall impression reinforced the importance of consistent voice and paying attention that friendly didn’t wander into glib and unprofessional.

Moreover, it was a huge “your privilege is showing” moment. The original text I waved away did a good job of assuring people that they’d be supported even if they didn’t already have, or couldn’t afford to buy, a nice home computer setup so they could work remotely. This is huge for people who might be coming out of a career change, or otherwise don’t have cash in reserve for an expense like this.

Aaaaaand I just hand-waved the old, perfectly good text away without considering why it might be important because I thought it was boring.

Derek. Come on, man.

I know we all want to test incomplete things. And often, we need to remind ourselves it’s often okay to release something you’re a little embarrassed about, as long as customers can get value, because they can’t judge it against your Dreamstate v.3 release. But this was a welcome reminder that if I’m looking at something people will see, and my own reaction is “that’s just not good work” it’s worth holding. I’d have gotten way better results if I’d asked for help, or just slept on it.

What’s next

I loved doing this, and I recommend we all look for opportunities to use testing to learn where we have blindspots and expose hidden issues, so we can improve and iterate.

Manifold has a number of great positions open, particularly in development and marketing: check them out! And I’ll continue to test as I have the opportunity. I welcome feedback, thoughts, ideas, and particularly, anything in how in the world you express the time zone thing.

I’m at derek@manifold.co or @milhous on Twitter