Mobile strategy is important for any Internet business, but it’s also a challenging market to operate in. At Myna we’re trying to make it a bit easier. Over the next few weeks we’ll be extending our real time A/B testing technology to mobile, giving Internet businesses unequaled ability to understand and optimise their offerings across the mobile and desktop Internet.
Compared to the web, mobile business is hard. Consider the following:
- Ranking is everything on both the iOS and Android app stores. Downloads quickly drop off as ranking falls.
- New apps have a short window when they are highly ranked, and if they can engage with users during this period they can maintain their ranking.
- Users hardly ever update their apps. Furthermore, if you’re targeting iOS, the lengthy App Store review process ensures that fixes are too late to have a significant effect.
- Limited data, only the total number of downloads and app store feedback is available.
This means that most businesses targeting mobile get only one chance to get it right — when they first launch — and they have to build their product before they receive any real feedback. Anyone who has run a business knows that nothing destroys assumptions faster than real users. In the last decade, online businesses have learned how to deliver steady improvements through controlled testing and rapid iteration. Can mobile businesses do the same?
We’re not the first to recognise the need for fast iteration in mobile apps. Trigger.io, Phonegap, and others provide HTML5 frameworks for building apps that are essentially websites. This allow updates to be pushed straight to mobile devices, a process known as hot deployment. Developers gain a fast feedback cycle, and they can use familiar analytics and testing tools, including Myna, to optimise their application. However, this approach cannot match the look-and-feel and performance of a true native application, which rules it out in many cases.
We’ve heard from a number of existing customers about their frustrations testing on mobile, and have decided to address it directly. We’re developing native Myna clients for iOS and Android, and also extending our API to allow you to push changes to your tests straight from Myna’s dashboard. This gives the best of both worlds: native application performance and hot deployment, bypassing the issues with app store updates.
We’re very excited about this new addition to Myna, which will benefit our existing web users as well. Myna will become the first testing platform that allows cross-channel testing, backed by super-fast bandit algorithms.
We will be launching Myna for mobile in closed beta. If you would like to be one of our beta testers you can sign up by emailing email@example.com.
“Growth Hacking and the Bandit Problem” is a recent talk by yours truly introducinga recent talk by the awesome Noel introducing multi armed bandits as a superior way of A/B testing. In case you missed it we decided to write it up as a blog post.
We begin by introducing our main character and hero of the story, the growth hacker. He is driven by one thing and one thing alone: pushing growth ever upwards and ever rightwards. Swoon.
To do this he follows 3 simple steps: Build, Measure, and Learn, a process handed down by the prophet of growth, Eric Ries. These 3 steps give our hero what he needs: a structured process to drive growth. Starting at the top, he gets something built, let’s say a new sign up page. He then takes some time to collect data and measure its effectiveness. Once this is over he sits down with his data and he learns, making a decision based on those results and informing the next iteration of the cycle. Round he goes again!
Faster is Better
Driven by a need for speed, our growth hacker takes a look at each step in his engine of growth to see where he can go faster. He starts off with build. Hmm there doesn’t seem to be much he can do here. Our growth hacker’s already pretty agile on the dev front.
Learning already seems to happen pretty fast, once he’s got all his data together. But measure? Now measure seems like a place where he might be able to speed up. At the moment he’s using A/B testing. Collecting all the data he needs to make a sound statistical decision takes a long time.
What if measuring and learning could happen together? What if we could turn our 3 step process into a 2 step process and speed it up dramatically? We could change our metrics chart to look like the green line instead of the orange, allowing our growth hacker to iterate and optimise as fast as he can! Well surprise surprise, you guessed it, our hero the growth hacker has just discovered the multi armed bandit, a way to drive growth faster than ever before!
The Multi Armed Bandit (Growth Hackers’ Secret Sauce)
“Woah there,” I hear you say. “Let’s just hold on a minute here and have a bit of background into this multi-armed bandit. Where is it from and what’s it all about?” Well, like all good secret powers the multi-armed bandit started off as a problem; The Bandit Problem.
Imagine walking into a casino. You head straight for a room full of slot machines or, as they’re called in the US, one armed bandits. You’re a clever egg, so no doubt you’re thinking that some of these machines are going to pay out more than others. You want to make sure you maximise your reward by finding and playing the bandit that pays out most. This is the bandit problem.
After a long hard think, and a lot of maths, you come up with a formula that helps you to find the machine that pays out most as soon as possible. Hurray! This means you don’t waste your money trying other machines that pay less often. The formula is called a bandit algorithm, and with this in hand our growth hacker receives the mighty powers of the multi-armed bandit!
Now Back to the Story
With his trusty multi armed bandit at his side our growth hacker can now set to the task in hand. Today he’s increasing conversions on a signup page. With 3 variants of the web page to choose from the MAB shows a different page to each visitor, and its reward comes when a visitor clicks on the orange button and converts. The scores are totted up and the process is merrilly repeated each time a new visitor comes to the site.
Two key Ingredients to the Secret Sauce
So far so good, but doesn’t this sound familiar? Up to this point our MAB has been purely exploring, trying variants at random and totting up their scores which is the main feature of A/B testing. But remember the goal of a MAB is to maximise the total reward, so to do this it’s going to bring another element into the mix, exploitation, which is showing the variants that have worked best in the past. This delicate balancing act between exploration and exploitation plays out for the duration of the test as the multi armed bandit happily goes about measuring and learning, at all times working to maximise total reward.
What are My Options?
We’ve just sketched an algorithm known as E-Greedy. When it comes to MAB there are lots to chose from, such as E-Greedy, Thompson Sampling, UCB-1 or Myna. Not all will perform in the same way or deliver the same results, as you can see in the chart below: (Ooh look, Myna’s the most successful. Who would have guessed?!)
Now I know what you’re thinking: fancy graphs based on simulated data are all well and good but I want to hear about some real life results. Look no further than one of our customers: Vizify, a startup working to create beautiful online portfolios. In order to improve their user engagement they decided to deploy Myna to optimise their email subject lines. Because Myna is so efficient with data, in just a few days Vizify had received a 500% increase in clickthroughs (pretty impressive for a startup with small amounts of traffic).
With Great Power comes Great Responsibility
When using MABs there are a few things to bear in mind:
- Your workflow will change dramatically. It’s going to become simpler as Myna is going to do all the work for you. It will be faster, because its so efficient with data you’ll get results at lightening speed. It’s also way more flexible. You will wave goodbye to setting parameters in advance (experiment length and p-value), and can add and remove variants at any time, testing almost anything you set your mind to!
- Defining rewards
- The ideal reward measure for any A/B test is most likely customer lifetime value but you probably can’t measure this very quickly. You need to have a fairly fast feedback cycle so the algorithm can adapt in a reasonable time period. Using simple measures like conversions are fine, but with any test you should check that this correlates with your true performance metrics.
- Stable Preferences
- The algorithms we’ve discussed only work when users have stable preferences. We don’t mean that all users act in the same way, but rather that their behaviour is similar in aggregate and stable over time. Broadly speaking, we assume what works today will continue to work tomorrow. For UI elements this is generally the case, but it is not, for example, true for news items where the value of the story is strongly time dependant.
And that’s it. Our growth hacker’s secret sauce, the multi armed bandit has been transformed into an almighty T-Rex. Jump on its back and ride off into the sunset and do what growth hackers do best: grow, fast! RAWWWRR!
We’re pleased to announce that we’re finally coming out of beta, which means the introduction of our least-requested new feature: pricing plans. These will come into effect on 15th December.
We’ve got plans for all sizes of business and website, including a permanently free plan for low-volume users or sample tests. Sign up now and start A/B testing for free - you can upgrade whenever you’re ready.
All our existing beta users (and any new sign-ups) will have the opportunity to choose a plan before December 15th. On that date, everyone else will be transferred onto the free plan. Don’t worry! All existing tests and data will remain intact.
We’ll make further announcements on billing closer to the time, but for now, all the details are available on the Myna pricing page.
Greetings from Myna HQ!
It’s been a veritable whirlwind of excitement since our last post.
Firstly, we’re pleased to announce that we’ve been accepted onto the Oxygen Accelerator program at BSP-A. This is a 13-week bootcamp for startups, providing access to mentors, introductions to investors, and (importantly) office space.
The information and advice we’re receiving is golden, and we’re having an absolute blast. We’ll admit to being a bit nervous about being locked in a room with a gang of hungry startups, but they’re a great bunch, with some fantastic ideas, and we really do wish them well on their way.
And if nothing else, we have a captive audience.
While we’re absorbing information like a pair of parched sponges, we’re also hard at work on Myna itself. We’re working on some refinements to the UI and reporting features, and rolling out changes to the support system – our help pages are getting a bit of a polish, and we’ve just given our internal metrics tracking a significant boost, integrating with Mixpanel.
We also seem to have adopted a small robot.
Isn’t he cute?
Follow us on Twitter for the most up-to-date haps, or subscribe to our RSS feed from the sidebar on the right.
Over the last few months we’ve been pouring effort into Myna, and we’ve been rewarded with some fantastic customers and equally fantastic feedback. We’re on the verge of leaving our beta and to celebrate we’re launching a competition to give free help to five businesses that rely on their websites to drive their business.
If you’re one of our winners, we’ll give you a free consultation to work out where you can best apply Myna and then guide you through integration with your site. As result you could see a substantial increase in sales for very little effort, and no cost to you.
It’s simple to apply. Just write to firstname.lastname@example.org telling us about your site and why you think you’d benefit from Myna. Make sure your email reaches us by Friday 17th August 2012 to be in the running!
Twelve thousand hits, over thirty emails, seven comments on the post, and over a dozen new beta testers. That’s what getting a blog post featured on Hacker Newsbrought us. We’ve been slowly developing Myna over the last few months, but this gave us the impetus to completely revamp the website. As you can see it’s still quite minimal, but it is certainly an improvement over the old site. Here are a few technical details that might be of interest if you’re trying to quickly build out a site:
The basic design of the site is Minima from Theme Forest. Well worth spending the $9 to get the general layout and some graphics.
We’ve heavily modified the Minima theme. It has a bunch of things we don’t need and didn’t support pages with lots of text. We used Less to get some abstraction over the CSS, which makes large changes a lot easier. Use it or use Sass. These tools are basically equivalent, so just pick one and move on.
We were lucky enough to find a public domain picture of a myna bird on Wikipedia. If we hadn’t, we’d simply have bought one from iStockphoto. We retouched the image a bit in Pixelmator, which does the bits of Photoshop we want at a price we can accept.
Now it’s time to get our new users live. Thanks, Hacker News!