Go to Pay4Bugs.com

The Pay4Bugs Blog

Pay4Bugs is the original pay per bug software testing marketplace. Our testers seek out and report problems with your software or website before they cause you to lose sales and customers.

Lessons from Apple's WatchOS 2 Delays & iOS 9 Bugs

Posted by Larry Salibra on · Comments

Apple’s latest and greatest mobile operating system, iOS 9, launched to much fanfare Wednesday. Missing was the newest software for the Apple Watch, leaving users hoping to take advantage of native apps on their wrist, waiting for a bit longer.

They took the unprecedented step (for Apple) of admitting that critical bugs had been found in WatchOS 2 and that it would not launch on schedule as they had publicly promised in the Apple special event a few weeks ago.

At the same time, as iOS 9 shipped to millions of excited customers, reports of bugs started rolling in on social media.

I experienced one myself when I tried to download the Twitter client Twitterrific only to for installation to freeze at “Installing” with no way for me to run or delete the app.

twitterific bug

In my case, this small annoyance was fixed by restarting the phone, but it was far from the only bug found within hours of launch.

What happened?

It’s not like Apple didn’t try to test iOS 9. Quite the contrary, in fact. For the first time, they opened up their public beta program to include iOS, so that members of the public had the opportunity to try out the latest iOS 9 builds before it was declared ready to ship. This was in addition to the tens of thousands of developers who have always had access to iOS beta builds.

Apple created their public beta program for the same reasons any company recruits beta testers. Even with the best QA practices, it’s impossible for one person or a small team to catch all problems in complex software.

There were, conservatively, at least a million eyeballs on its beta iOS 9 software for most of the summer and tens of thousands of developers using beta WatchOS 2 builds.

With all of this attention, how did these problems fall through the cracks? What caused perfectionist Apple to publicly admit a mistake and miss a ship date? Did the millions of eyeballs not notice the problems? What happened and we can we as builders of software products learn from it?

iOS 9 betas were missing a lot of code

First, we have to take a look at what people were actually testing for all these months. While both developers and testers were running iOS 9 and WatchOS 2 betas, all of the beta versions were missing important code and APIs that support features that were kept secret until the Hey Siri special event. Specifically, much of the code to support the new iPhone 6s including features such 3D Touch, Live Touch, always on “Hey Siri,” and its updated iPhone and iPad hardware wasn’t in any of the betas and didn’t get tested. It was hidden away in a secret branch of their source code version control system.

While code for newly announced was no doubt tested internally at Apple, one can imagine it be limited to those privy to the iPhone 6s wouldn’t get nearly the attention of the main code base. As any software developer can tell you, merging in new code also has the nasty habit of making bugs that you didn’t know about before suddenly appear…whackamole-style.

Show me the money

Second, let’s take a look at the incentive structure of Apple’s developer program and public beta program. People who access beta versions of software through the developer program are in the business of creating their own apps. They have paid somewhere from $99 to $299 for the privilege of distributing their apps and accessing other developer program resources so that they can build an app that customers love. One of these developer resources is early access to beta builds of upcoming operating systems. This gives developers the time they need to make sure their apps work properly on launch day when a new version of Apple’s operating ships so customers don’t experience broken apps.

Developers are busy. Like Apple they have businesses to run, customers to please and deadlines to meet. They’re using Apple’s
beta versions as necessary part of trying to create bug free software that delights their customers. Unless they run problems that directly affect their ability to deliver their product, spending 5 or 10 minutes filing a bug report with Apple just takes time away from making their own deadlines.

apple testing program

People who access Apple’s iOS and OS X betas through the public beta program are enthusiasts. They sign up because they want a chance to try out the latest and greatest features before everyone else. Sometimes this is for bragging rates, other times it’s because they’re already sold on the value proposition of the new features and want to enjoy the benefits now instead of waiting several months. These are often the same people that always buy the latest Apple gadgets.

Enthusiasts are using the public beta program for their own entertainment and gratification. When they run into problems they’ll often tell their friends who are also enthusiasts or share a screenshot on social media. This is fun. Systematically testing and retesting functionality and looking for edge case bugs isn’t most people’s definition of fun. Nor is writing up reproduction steps and filing bug reports with Apple.

beta program faq screenshot

Developers and enthusiasts have no incentive to tell Apple when they’ve found a problem. There’s nothing in it for them.
They don’t have some altruistic desire to subsidize the world’s largest company with their free time. There’s no reason they should.

Even if someone woke up feeling particularly generous wanting to send the $700 billion company some free bug reports, Apple doesn’t really make it a very nice experience. Their bug reporting tool has always been infamous for being stuck in the past and despite a recent update, little has changed.

bug reporter screenshot

The ghost of Scott Forstall still roams the halls of Cupertino years later. Pretty amazing for a company that agonizes over the design and attractiveness of their circuit board layouts. There’s not even an option to file a bug report for the Apple Watch hardware unless your Apple Watch looks like a Magic Mouse.

magic mouse

And it’s not like beta program members and developers don’t talk about the bugs that they find. Much to the contrary, they tell (ie. complain to) their developer colleagues and enthusiast friends. It’s something to talk about - a shared experience - and makes you feel better about the own bugs in your own software (or life) to talk about Apple’s. You’ll even see them shared on social media.

quote

Developers and enthusiasts like everyone else are rational actors. When asking themselves the question, “What’s in it for me?” when considering telling Apple about the problems they find, the answer is “Nothing.” In fact, it’s worse than nothing, it’s nothing in exchange for time spent documenting the problem, how to reproduce it, creating screenshots and screen recordings, and putting it all into a bureaucratic form for a company that is going to use the results to generate another year of recording breaking profits.

Lessons for your business

1. Many eyes make light work

Understand that there's a trade off between secrecy and quality of your software. There are often legitimate business reasons to minimize the number of people who know about your product before it launches. However, if no one sees your product before it launches, you customers are going to find lots of problems that you and your development team just didn't notice. Apple has so much brand capital that it can afford a short period of user unhappiness before they can roll out point fix updates to correct the problems. Can yours afford it?

There are both technical and legal ways to get more fresh eyeballs scouring your product for problems while maintaining reasonable confidentiality. At Pay4Bugs, we work with our Business and Enterprise customers to restrict their sensitive projects to hand selected testers and can ensure that these testers have signed the necessary confidentiality agreements before they get started.

2. Incentives drive behavior

It's simple, really. If users have no incentive to report bugs, they're not going to be interested in reporting bugs. If a payment or reward is offered, reporting bugs becomes much more attractive. When the reward is paid only for valuable bug reports, like we do at Pay4Bugs, users will be motivated to take the time to compile the screenshots and replication steps that make it easy for the problem to be quickly triaged by the project manager so that the development team can quickly and easily fix it.

If Apple’s beta program with millions of users clamoring to use the world’s most popular software fails to uncover so many common bugs, how many get missed in yours? At Pay4Bugs, we change the incentive structure. When testers can earn cash for useful & reproducible bug reports, you get (surprise!) lots of good bug reports. Get in touch with us to learn more.

Comments