How eHarmony Kills the Romance With A/B Testing

Editors note: Neil Patel from KISSmetrics will be running a half day workshop on “How to use A/B Testing, Analytics and Goal Measurement in your Web App” at the Future of Web Apps Miami – February 22-14 2010. Buy your tickets now.

eharmony-email

As a user experience designer, A/B testing is not only something I design for, but something I advocate that all my clients implement. It’s one of the best ways we can both provide users with the best and most effective experience and provide businesses with ongoing opportunities to optimize.

The reality is that like almost anything in design, some do it really well, while others fail in appalling and reasonably smile-inducing ways. Getting caught doing A/B testing by your most unassuming and uninformed user is like getting caught with your pants down—embarrassing.

One of the great examples of this occurred in 2000 when Amazon customers found themselves paying different prices for the same DVDs. Ouch!

This brings me to eHarmony, an online dating site that advertises some well-known commercials on television. If you’ve watched US TV in the past year chances are you know the Dr. Neil Clark Warren mantra I’m talking about;)

Throughout 2009 I found myself in a position where I experimented with online dating for two reasons:

  • the experience of how dating companies message and market to people
  • the dates (er, yes, I do mean the actual dates themselves)

Both taught me quite a bit.

Match, Chemistry and a handful of others never shocked me with their emails (some of the usernames that men choose did). However, eHarmony stood out of the crowd for a few reasons:

  • rapid-fire morning messages, usually between 4 and 8 in a row (like I was under fire from the “matchmaking tool”)
  • radically different branded email templates
  • really bad subject lines, all different, but with the same purpose

It’s the really bad subject lines that I’d like to focus on, and they relate to the rapid-firing in a row, because together they produced an experience that killed the romance.

Here’s a typical morning from 2009:

eharmony-2010

All of these messages, though different, say the same thing: “Get to know so-and-so.” They would have been more scanable and less distracting if they used the same subject line emphasizing the users’ names.

I never wanted to share these observations with anyone in my profession, because it’s inevitably tied to a much larger issue of my love life, and was ready to let this go on January 4 at the same time as my subscription ran out, except then eHarmony took thing a step further.

Beginning January 4, 2010, my inbox filled with an even more challenging set of differing subject lines. More challenging because the messaging was all over the place. Was I a user more interested in “activities,” “spark,” “unique,” “common,” or greatness? I felt like it was eHarmony having the problem ordering off a menu of men and not me. I wanted simple and given to me straight up.

Here’s a typical morning in 2010 (*pardon that they’re all in my trash now):

eharmony-2009

eharmony-2009

Distracting and disruptive, right?

This kind of A/B testing made me feel more like an experiment and less like a client. Moreover, my perception of eHarmony became characterized by the realization that this was not a thoughtful process that took pride in emotionally connecting, but that it was more of a massive warehouse churning out widgets.

A/B Testing Should Not Be:

  • Transparent
  • Radical
  • Isolated
  • Hurried
  • Fleeting

A/B Testing Should Be:

Subtle—User shouldn’t be aware that they are being tested

She should feel like she’s experiencing the very best your company has to offer from personalization to copy to look and feel. You’re learning from her actions, so make it impossibly easy for her to accomplish tasks that teach your team without drawing attention to your team.

Incremental—Don’t break things that are currently working well

Change the text on the button v. changing the button entirely simultaneously altering its shape, color, size, and text.

Aware—Web apps don’t exist alone in the world

News events, seasons, weather, traffic sources, search engine patterns, and even the economy shape the ways in which users discover, interpret, and engage with information.

As you A/B test you should make decisions based on data that’s measured and tracked over time and interpreted with a sensitivity to the world-at-large.

Paced—It’s not a sprint, and when developing apps there is no finish line online

A/B testing takes time and you need to understand your data in terms of mathematical relevance (e.g. how many unique visitors to sales conversions will it take based on traffic patterns to make a relevant sample size?)

Ongoing—Make user testing an ongoing part of your budget, design, and development

The best A/B testing occurs when you’re constantly learning from your users. As you grow your user base or expand your offerings new aspects of interaction will be introduced, and as web technologies change users will change predictable patterns.

Both large and small companies can A/B test successfully, but the same rule applies for making mistakes. Executing A/B testing isn’t rocket science, and it’s certainly not supposed to be harder than finding a great love in this life—e’hem eHarmony.

Free Workshops

Watch one of our expert, full-length teaching videos. Choose from HTML, CSS or WordPress.

Start Learning

Treehouse

Our mission is to bring affordable Technology education to people everywhere, in order to help them achieve their dreams and change the world.

Comments

0 comments on “How eHarmony Kills the Romance With A/B Testing

  1. I don’t believe this to be A/B testing though. From what I’ve read (eHarmony doesn’t let me use their site and I’m long since happily married), the messages have been this way for a long time. The purpose, I believe, is to catch people’s interests and cause them to open the e-mails and read them by specifically making them look like they aren’t form messages. Of course, that approach is pointless if they send them all at the exact same time.

    They probably DO test which headlines get the best response. But they are counting on the VARIETY to actually cause an increase in responsiveness all on its own, regardless of which headlines are better than others.

    • Drew,

      You make a great point. A/B testing can be delivered in any number of ways, divided between multiple users or based on a single user’s experience. In this case, I consider this A/B testing, because it’s clearly subject line testing. The content inside the messages is the same (exception the person’s name), but it’s eHarmony trying to learn what headlines compel users to open more frequently.

      What bugs me is what they do with that information? Do they fine tune based on individual users, so that I might wake up to a row of the same subject lines? Or, do they compile that data and use it to make community-wide decisions?

      In my experience with this series of emails, delivered over a 3 month subscription period, I never noted a change in receiving more of one kind v. the other. (and, I did measure this)

      A/B testing (even subject line testing) doesn’t stop at the user’s action – it goes on into how the company uses the data to create a better, more compelling, overly awesome experience that doesn’t result in a grumpy (me) user:)

      Cheers!
      CB

  2. I agree with Drew, this is the most basic form of ‘variety’ trying to inspire interest. The first thing I do as an eHarmony user is ditch all but one of their useless emails in my inbox, the remaining one being a reminder for me to go and check out the site at some point, probably not right away.

    If they sent one email with the same information (there is after all a body to an email, not just a subject) then the emails would be between 60% to 90% more effective, since they’d all stand a chance at getting read.

    • Paul and DB,

      So again great point building on Drew’s observation – but why isn’t this A/B testing?

      I didn’t explain that inside each of these messages (for the 3 month duration) the content and design of all of these emails is completely the same (exception the name of the person and his age). Nothing changes, not the color of the buttons, template, copy, etc.

      The only difference is the subject line.

      I’ve worked with multivariate testing for years (including subject line A/B), so I’m curious as to why you wouldn’t classify this as A/B testing – is it because they’re batch sent, which then just makes the campaign badly designed?

      Maybe I’m wrong:) Please help!

      :)CB

  3. Interesting post, I don’t see why they don’t just include all the different matches in one e-mail so your inbox isn’t flooded with so many messages each morning (that invariably end up in the trash). That would likely test better.

    • Kendle,

      Completely agree. Match.com sends a daily digest around noon with the photos of all of your latest matches. I’m far more likely to open a single email and click-through to a profile. I almost never clicked through on an eHarmony profiles for a few reasons:

      1.) no photos, which is strange and feels dishonest or like eHarmony is embarrassed of it’s users, Ok Cupid wrote a great article “Your Looks & Online Dating” http://ow.ly/Vbqd

      2.) had to open 4-8 per day (which if I wasn’t studying them, would have led to immediate batch deletes)

      3.) the core experience took me to one profile, and it wasn’t easy to navigate and match up the new matches with the emails in my inbox, so the experience was not succinct.

      Ciao!
      CB

  4. What a great topic! Whether and how this is an A/B/n test depends on what is being measured (if anything) on the back end, which I think is Drew’s point. Here are a few ways it could be going down:

    1. Each subject line is a separate variable in what is probably the world’s most poorly executed A/B/n test (this is your theory, I think, and a very reasonable one). In this case, the eHarmony folks would be looking to see which specific subject lines got more people to open/click through to a profile view. If this is the test, they are not taking into account the disruption and noise from sending different variables to the same person that you mentioned, and will have an epic amount of meaningless data on their hands.

    2. You are in an A/B test that is tracking multiple subject lines vs. a consistent subject line. Someone hypothesized that variety will inspire more interest (i.e., that the disruption you felt is actually a positive) and will result in increased engagement. In this world, there is another Chrissie out there who is getting the same emails all with the same (consistent) subject line, and you are being compared against here. This could actually result in good learnings for eHarmony (especially if everyone is reacting to the variety as negatively as you are).

    3. There is no test. You are just getting random subject lines because some guy thought that variety was a good idea, and the email program is being measured globally rather than by subject line. This, of course, would be really, really stupid, but it happens.

    Awesome post. Thanks!

    Jason

    • Wow, Jason, what great observations and kickass response that’s making me want to write more about the UX issues that come up in A/B testing. This just came out of my personal annoyance with a business that’s in a vertical known for excellent testing and measurement.

      1. My theory is that each subject line is a separate variable, based on the experience that the inside contents of the email are all the same (they were). I imagine eHarmony’s team is poised each morning to see which emails all the “Chrissies” out there open. The result is that it’s a terribly executed test.

      2. I measured these emails and this technique for 3 months, concluding on January 4, 2010, when I noticed that suddenly the subject lines got even nuttier, prompting this post. I would have thought that after 3 months (or another reasonable period of time to test) that the update to the strategy would have been to select one line over the other, not to launch a new campaign of even battier-sounding diverse subject lines. The strategy didn’t refine itself from last year’s execution.

      3. “There is no test.” Woah! In a world where even crazier things happen, this could be absolutely true. That eHarmony is simply outputting based on whimsy and not measurement, which is really, really stupid.

      Jason, you have me really thinking this through even more, and you articulated Drew’s points perfectly. Thank you! Checking out http://www.zaaz.com now:)

      Best from Brooklyn,
      CB

  5. Chrissie,

    Excellent post, really enjoyed reading it and the follow-up comments. I feel like I learned something!

    More please on UX testing :)

  6. While I didn’t bother me from a designer’s pov, because I’m not a designer. The entire user experience about eharmony was terrible. I hated exactly all the things you hated. What bothered me most, was that they pretend that they’re matching you, when in reality all they’re doing is spamming you. I think every single possible girl in my area was thrown at me, and when I tried to quit I got spammed like crazy to come back.

    The only thing I liked about it was that it did focus more on personality by making you fill the about you sections properly.

    But seriously, to me it seemed like a glossier version of email notifications from any classified service.

  7. Match used to focus on the “what,” matching people according to how they looked on paper.

    Which isn’t very interesting. Really, it’s not.

    I have heard eharmony does the same thing.

    okcupid does a lot better, in my opinion, by extracting a fair of the “who” you are. Far more interesting and relevant.

  8. This is an excellent post! Your points on what A/B testing should and shouldn’t be are dead on.

    I would like to comment though that I sincerely doubt that you experienced A/B testing. The subject line of eHarmony’s are the most important part of the emails that they send you: they are the grabber that either catches your attention (and gets you to their site) or not. In the first examples you showed, I agree that they all look very similar. I believe that the eHarmony admins realized the importance of those subject lines and decided to do an overhall of them to make them more attention grabbing. There would be no reason to change the contents of the email as again, it is the subject line that grabs your attention. From my perspective, there is no reason to believe that they are doing any specific testing of how you respond: I think they simply decided to make a slight update to their email system with the hopes that it will be more successful (ie with the hopes that the subject lines are more attention-grabbing).

  9. Hi Christine:

    What is more insulting is that eHarmony has done the same thing to you that they have done to multiple users; they suddenly find a huge crop of “matches” for you immediately before your free subscription runs out. It is a transparent marketing tactic, and an annoying one. They could have trickled those matches out with their batch e-mails rather than trying to “hook” you at the last minute with quantity.

    eHarmony also doesn’t let certain people sign up at all if you don’t answer the questions in the inital profile to their satisfaction; I was one of them. I believe my mistake was being overwhelmingly negative (I call it pragmatic) on the initial questionnaire. If I really cared though and I wasn’t just doing it as a social experiment, I would have answered the questions according to what they wanted to hear. Do you want dishonesty in your mate? Don’t choose eHarmony.