Brad Hargreaves | Building Things

Brad Hargreaves on entrepreneurship, community and life

On Building General Assembly and My Next Steps

with 2 comments

Building General Assembly has been the most remarkable and inspiring professional experience of my life. We have helped more than 12,000 people pursue careers they love across 12 campuses worldwide and have been a part of some powerful stories along the way. When we set out to teach technology, business and design, we didn’t realize that we were going to create the first premium education brand to emerge in more than a century. That is truly humbling.


The GA team now comprises 400 of some of the smartest, most genuine people I know. We have seen incredible and talented individuals rise through our organization as first-time managers, while others have joined the senior team bringing decades of experience in education and leadership. I have immense faith in my co-founder Jake and the senior management team to guide GA through the next part of this journey.

Four and a half years after making the decision to start General Assembly with my co-founders Jake, Matt and Adam, I am excited to say that I am joining our lead investor Maveron as a venture partner and returning to my passion for early stage companies. I am not leaving entirely — I will be staying on part-time to help build our Credentialing Network as we develop a team around that critical initiative.

Maveron has been the ideal partner since making a bet on us three years ago, and I’m thrilled to join their team. As I’ve gotten to know Jason, Dan and all of Maveron, I’ve been impressed with their vision of building a truly consumer-only venture firm with deep expertise in product development, brand, and customer acquisition. And they’ve been able to execute on that vision with early bets on Zulily, Julep, and General Assembly, among others. As the home of great brands and consumer-savvy entrepreneurs, New York is a wonderful place to expand the Maveron family.

For most of my life, I viewed the education system as an institution immune to change. After building General Assembly, I see the power of communities of individuals challenging established systems and building better outcomes. GA’s growth is not a black swan event but rather a story that we will see with increasing frequency as entrenched industries feel the impacts of technology and design. This shift will be an incredible force for good, as sectors that have been traditionally slow to embrace technology — such as education, healthcare, and finance — are the largest in the world and touch every individual.

I have had the privilege of spending my life so far enabling that story by being an entrepreneur, and I hope to continue to do so. My co-founders and I are doubly fortunate that General Assembly is not just another company, but an organization that has had a transformational positive impact on so many lives. For that, I am incredibly thankful but remain hungry for more.

Written by Brad Hargreaves

November 24th, 2014 at 5:23 am

Posted in Uncategorized

Game Development’s Four Inflection Points

with 3 comments

Remember when gaming was one of the hottest verticals for venture investment? Four years ago, you couldn’t visit TechCrunch without reading about the latest social-mobile studio or game with dizzying stats.

No more. Zynga has been battered, traditional publishers never recovered, and new super-casual studios struggle to get traction in the public markets. Even companies riding the freemium wave like King are trading at 3-5X EBITDA multiples, numbers on par with industries like facilities management and for-profit education. Multiples that would’ve been unthinkable in 2010.

But there’s something going on in gaming right now, and it centers around four inflection points:

1. Casual game distribution is getting more expensive… way more expensive. CPIs (cost per installs) have risen by 56% on iOS in the year with no signs of slowing. The app marketplace is an oligopoly, dominated by a few big players that leverage owned channels — their successful games — to push new releases to the top.

Getting in the top 25 free apps — a critical channel to reaching a mainstream player base — has gone from expensive to borderline-impossible in the past two years. There are over 2,000 new apps released every day, the majority of them games. And most of them will generate little to no interest or revenue.

Given this, it’s not surprising that…

2. Casual game monetization is a race to the bottom. Mobile has always been a cesspool of questionable tactics and sleazy marketing. But the increased cost of distribution has meant that games need to resort to increasingly aggressive monetization strategies to keep LTV in line with CPI. Those who don’t are locked out of distribution channels. It’s a classic race to the bottom, with little incentive to favor user experience over monetization when targeting casual players.

It’s started to get regulatory attention and will only be so long before many of these tactics go the way of $9.99/month ringtone subscriptions.

3. AAA-quality game development is cheaper… way cheaper. Unity is absolutely dominating indie game development, and for good reason: it’s a flexible tool that allows small development teams to build high-fidelity games with orders of magnitude smaller budgets than were required just five years ago.

Other tools like Corona are playing a part too, but Unity is the clear leader here and is changing the way immersive gaming experiences are built.

4. Niche core and hardcore game distribution is easier than ever. Steam Greenlight has enabled games that never would have gotten mainstream coverage — let alone made it onto a Gamestop shelf — get in front of the right gamers at a fraction of the cost of past distribution channels.

Furthermore, core and hardcore games will get unique opportunities on new platforms (Oculus) and are finding new ways to reach savvier players who know are sick of the race to the bottom on mobile.

There may not be a venture-fundable, Zynga-scale model here — most of the success has been confined to hit-driven indies to date — but the landscape is changing.

Written by Brad Hargreaves

August 26th, 2014 at 12:05 pm

Posted in Uncategorized

Tagged with , ,

Experimenting on Human Beings

without comments

Since the Renaissance, scientific experiments were the domain of Science. And Science had its own way of doing things. Science derived questions from first principles, proposed hypotheses that may answer those questions, and designed the methodologies that would prove or disprove those hypotheses.

That’s how Science worked, and nothing else worked quite like it. Business was art; Business was Don Draper doodling his visions on a cocktail napkin. Politics was even further afield, an art buried in the fog of war. Science and Science alone ran real experiments.

But then some Bad Things were done in the name of Science. Things that shook the foundations of Science and threatened to bring down the whole operation. To prevent those Bad Things from happening again, Science began regulating itself, introducing concepts like Informed Consent and requiring experiments to be approved by Institutional Review Boards. Most people agree that those changes were for the better.

But then other people started figuring out what Science had been doing all this time. Finance was probably the first to find it, bringing experimental methodologies to trading in the late ’80s and early ’90s. Advertising soon followed suit and has become overwhelmingly sophisticated in the past ten years. Obama’s 2008 campaign was a watershed moment for Politics, with rigorous message testing and voter data methodologies at work for the first time.

One by one, domains of Business and Politics adopted the Scientific Method. Their subjects, naturally, were human beings. But since Science only regulated itself – not everyone who used its methods – these new domains don’t have the same procedures and oversights. Any organization with an audience could run experiments on them to see what message, user flow, landing page, ad campaign or button color worked the best. Some of these experiments could have a meaningful and lasting impact on their subjects – when an OkCupid test works, more people hook up, resulting in marriage, kids, STDs, emotional trauma and all kinds of chronic effects that would put any IRB into a tizzy.

And now Facebook is getting FTC attention for doing something every savvy web-based company has done in the past fifteen years. And OkCupid and others are defending them, claiming (correctly) that everyone does it.

Now everyone doing it doesn’t make it okay, per se. But it does raise questions of what “informed consent” means. One could argue that signing up for Facebook and accepting friend requests is consenting to seeing whatever those friends may post, even if Facebook is presenting them in a way that is likely to evoke certain emotions. And the informed consent argument is even stronger with OkCupid; they’re not trying to hide the purpose of their site from prospective users.

Of course, the specific methodologies of informed consent aren’t anywhere in play. There’s no paperwork or review, just a ToS that no one ever reads.

Perhaps the problem here is not that these companies are running experiments without informed consent, but that the implementation of informed consent needs to be totally re-thought when applied to Business and Politics. Signed forms and IRBs aren’t going to work – we need a methodology that fits companies that run thousands of experiments every day and test hundreds of variables, the vast majority of which have little to no lasting impact on their users.

Perhaps rather than going after Facebook, informed consent advocates would be better off tackling the issues with ToS and getting companies (and users) to care about the documentation that already exists rather than adapting new business practices to an old scheme that was created to solve different problems.

Written by Brad Hargreaves

July 28th, 2014 at 9:23 am

Posted in Uncategorized

Tagged with , ,

Price Gouging, Uber, and Alamo

with 4 comments

Last night I had to drive from Albany to New York.  My Virgin America flight from San Francisco got diverted, then had mechanical problems, and next thing I know I’m facing the prospect of spending Saturday night in the Schenectady Hampton Inn.

With the good fortune of sitting near the front of the plane, I was second in line at the Alamo counter at the Albany airport to rent a car to make the two-and-a-half-hour drive back to the city.  I was feeling pretty good about my prospects.  The man in front of me rented a mid-size for $80.  When I got to the counter and handed over my license, a troubled look came over the representative’s face.Screen Shot 2014-05-11 at 6.03.43 PM

“Oh wow — everything just changed.”

“What changed?”

“The prices.  The computers saw what happened with your flight.  It’s a lot different now.”

“How different?”

Alamo ended up charging me $600 for the pleasure of driving a rental car back to New York.  I was fortunate to find some fellow passengers to split it with me — company on the road is always nice too — but I still have a bad taste in my mouth.  It was an example of price gouging at its worst.

Uber has taken a lot of flak over practices that look similar at a high level.  Uber’s algorithms see surges in demand and raise prices accordingly.  Seems like old-school price gouging, the same kind that Alamo used against me in Albany.

But while I’m fuming at Alamo, Uber’s practices have never bothered me.  It really comes down to the elasticity of supply:  Regardless of how much Alamo charged, they weren’t getting more rental cars to Albany that night.  The first dozen people at the desk got cars and the rest were left with their hotel vouchers and the hope of a flight the next day.  The increase from $80 to $600 per car was simply a maximization of profit — an economic transfer from the customer to the company.  No additional economic value was created by the price increase.

Uber, on the other hand, has supply elasticity.  That is, supply changes quickly with the price offered.  If a customer is willing to pay $200 to get from the Upper East Side to the Meatpacking on a rainy Friday night, there’s going to be a driver willing to do it.  Increasing prices brings more drivers out, making it possible for more transactions to happen that “make sense” for both parties — even if those transactions are significantly more expensive than they would be on a sunny summer day.

Price gouging laws were built with the assumption of totally inelastic supply.  Price gouging laws exist to prevent Alamo from doing the kind of thing it did in Albany on Saturday night, or to prevent a grocery store from increasing the price of bread during a blizzard when more bread can’t be brought in.  When these laws were written, inelastic supply was the rule.  Logistic and supply systems didn’t exist on top of a digital layer enabling real-time changes in supply in response to demand.  But for some companies like Alamo, that digital layer still isn’t used to increase supply — just to raise prices.

I’m glad I got back to the city on Saturday night.  But it left me wondering why all the flak has landed on Uber for ensuring that supply can meet demand while Alamo is left to gouge and extract at will.



Written by Brad Hargreaves

May 11th, 2014 at 2:10 pm

A Golden Age of Management?

with one comment

It’s been exactly 60 years since Peter Drucker published The Practice of Management, bringing professional people management into the world of social science. And over the last twenty years, it’s gotten cheaper and easier to build a company, companies that are more often led by non-traditional managers with more leeway to try new things. Together, these forces are leading us into a golden age of people management.Picture from Valve's Office

In the history of organized work there has never been such a diversity of management tactics and structures. From holacracies to adhocracies to flat structures to four day weeks, entrepreneurs and managers are trying new ways to get things done and keep people happy at a faster rate than ever before. Just look at the headlines on Hacker News — you’re almost always able to find a couple front-page pieces on a new org chart or management hack.

I don’t agree with many of these tactics. Most won’t work. But some will, and aspects from others will get incorporated into more traditional management canon.

There’s been a lot of negativity floating around the tech world these days. This post isn’t about that. But it is about always seeking to do things better and iteratively improving the world around us, and in that spirit I’m optimistic.

Written by Brad Hargreaves

January 3rd, 2014 at 12:35 pm

Posted in Uncategorized

Brand Crossovers and Education

with one comment

In their December newsletter, University Ventures declared Bridgepoint’s licensing of Forbes’ brand to create the Forbes School of Business to be education’s “Mickey Mouse moment”. The letter compared the deal to Walt Disney’s creation of high-quality, branded theme park experiences that out-competed the questionable and unreliable carnies of decades past.

Brand crossovers in education have happened and will continue to happen. But this trend won’t result in high-quality experiences any more than blockbuster movie crossovers create high-quality video games.

From the letter:

For in a world where universities take centuries to establish themselves and their brands, there are myriad brands that currently have nothing to do with higher education but that could be very useful for higher education institutions interested in developing high quality programs that produce higher ROI for students and that clearly communicate those values to the marketplace. These brands could provide prospective students with shorthand information about the value proposition in a market where prospective students are crying out for better information about the likely return on their investment in a degree program.

Unfortunately, “quality” and “value proposition” are not unidimensional metrics that translate across industries and products. Brand crossovers in this industry have made a splash when education meets the self-help aisle but rarely beyond this. Think Tony Robbins and the Learning Annex; The Apprentice and Trump University. In those cases, students are looking not for a long-term skill set but for inspiration and a silver bullet, for which brands that communicate “immediate success” are highly valuable.

Unfortunately, licensed brands are often used as a stand-in for product quality rather than an indicator of it. The games industry offers a cautionary tale. For years, TV/movie/event IP was licensed to create new game titles, generating a bunch of famously terrible games.

Eventually, the tide turned. Games journalists looked for reasons to attack the next big-budget shovelware, framing the entire category as corporate garbage. Today, the entertainment-crossover-into-games genre is effectively dead, with interactive studios investing their biggest budgets in game-specific IP rather than licensed brands.

Education offers an even harsher environment for brand crossovers. Unlike games, education is typically viewed as an investment rather than consumption expense. Customers — especially those precious 10% whose tickets aren’t subsidized by the federal government — are suspicious and discerning. And unlike the late-90s gaming industry, the education space features hundreds of existing brands specialized in a variety of learning outcomes.

There’s value to the Forbes/Bridgepoint deal and think it will quite likely work out well for all parties involved. But brand crossovers are hardly a new thing, and it’s far too early to claim a watershed moment. The for-profit education theme park is still run by the carnies.

Written by Brad Hargreaves

December 28th, 2013 at 3:50 pm

Posted in Uncategorized

The First Thing That Breaks

with one comment

One distinguishing factor between inexperienced and senior software developers is their ability to answer the following question about an application they’ve built:

“As usage increases, what’s the first point of failure?”

Knowing which query or resource will need to be rewritten or replaced soonest is a meaningful — albeit simplistic — indicator that an engineer understands what he or she has built.

I like to apply the same heuristic to entrepreneurs. Experienced founders can be differentiated from novices by their ability to answer the following:

“As the business grows, what’s the first bottleneck you’ll encounter?”

Or stated another way, what problems will be need to be solved at each stage of the business’s growth? Let’s take a child’s lemonade stand. In this example, we have a particularly enterprising child who notices that the local supermarket can only supply 100 lemons per day, which equates to approximately 200 glasses of lemonade. At $1 per glass, that’s only $200 per day! The child thinks, “If I want to grow my business beyond that size, I need to buy wholesale” and begins spending time researching suppliers. But in this case, supply may not be the primary bottleneck. If only 100 cars come down the stand’s street every day — and only a percentage of those buy lemonade — s/he doesn’t need to find a wholesale supplier before finding a new location, or perhaps training an employee and opening a second stand.

It’s useful for an entrepreneur to think about their job less as growth and more as de-risking — identifying and working through the primary risks of the business. This is especially true if the entrepreneur wants to raise money. For the lemonade stand, the ability to buy wholesale isn’t a major risk — many small businesses do that with ease. Growing the customer base and scaling to multiple locations, on the other hand, is a problem worth tackling.

“Scaling prematurely”, “hiring ahead of the curve”, and “capital inefficient” are all phrases that VCs use to describe companies that tackle the wrong issues in their business’s growth. Entrepreneurs rarely err by implementing poor solutions to the right problems but rather by solving the wrong problems.

Written by Brad Hargreaves

June 23rd, 2013 at 11:51 am

Posted in Uncategorized

Spain Has a Tenured Faculty Problem

with one comment

By this point we all know things are bad in PIGS. But how bad, and why? Matthew O’Brien at The Atlantic included a simple graph in his argument that Spain is Beyond Doomed, charting the growth the long-term unemployed in Spain:

Spain Unemployment Chart

Anyone who has tried to do business in continental Europe — particularly Spain — knows that firing employees is effectively impossible. This can make businesses much more skittish when it comes to hiring full-time employees in boom times, favoring instead unprotected part-time contract labor.

And when I saw that chart, I was reminded of another:


Academic Faculty Chart

This one isn’t as dramatic, but the underlying problem is the same: a regime that protects a group of established individuals creates a “permanent underclass” of those on the outside looking in. In Spain, permanent workers are almost impossible to fire or downsize, just like tenured faculty in the US higher education system. The result in Spain is an underclass of indignados chronically under- and unemployed; the result in higher education is an underclass of poorly paid adjunct faculty with few prospects of advancement.

Neither system is sustainable. Both discourage effort and innovation, driving talent away. In academia, many of the best students flee research and end up in industry. Leaving a country isn’t as easy.


Written by Brad Hargreaves

April 28th, 2013 at 4:23 pm

Posted in Uncategorized

Tagged with , , ,

A Few Notes on Learning Rails

with 2 comments

In the spirit of eating your own dog food, I took General Assembly’s Intro to Rails this past fall. The whole process of learning Rails by trying out pretty much every online tool available led me to a few thoughts:

0) Prior knowledge is rarely clear-cut. I’ve done some combination of front-end development, game development and transactional SQL in the past. Most of which didn’t help and possibly hurt my efforts to learn Rails. Many people I meet have similar quasi-technical backgrounds and are never sure where to start learning. It’s important not to skip things just because you think you’ve seen it before.

1) You have to be passionate about what you’re building. I don’t think anyone can learn to code in the abstract, bouncing from one generic example to another. But…

2) You can be too passionate. Your first, second, third and twentieth attempt to build something will all suck. It may be easier to get started if you’re not building the end-all-be-all personal project to end all projects. Instead, build something to solve a really simple personal problem without any (obvious) commercial potential.

3) Embrace de-leveraging. You can’t walk in with a mindset that you’re just going to learn enough to know how to hire someone to do the hard work. While you may end up doing that, for the moment, you’re de-leveraging and writing the code yourself.

Written by Brad Hargreaves

January 28th, 2013 at 2:10 pm

Posted in Uncategorized

Tagged with , ,

Your Product Chooses Your Community, Marketing Scales It

with 5 comments

Great businesses are known and grown by the communities that use their products. Apple’s brand was reinforced by the hip designers and musicians that used it through the company’s worst days. Facebook’s initial band of affluent college students proved more sustainable than MySpace’s crowd. Entrepreneurs have noticed, and I’ve seen many try to carefully architect the initial community that uses their product and comes to speak for their brand. This is absolutely the right intuition, as great communities reinforce great products reinforce great businesses.

Unfortunately, I see many entrepreneurs go about this in the wrong way, attempting to curate a seed community by limiting user acquisition channels to only those with a similar brand or a “premium” audience. In addition to being unsustainable, this strategy hamstrings the business and opens the door for competitors.

Your brand’s community is chosen by your product. If you have crafted your product to speak to the community you want, you need not resort to user acquisition jiujitsu to keep the good crowd in and the bad crowd out. If you are relying on targeted marketing rather than a great product to build your community, you are resigning yourself to a giant game of whack-a-mole as you try to stay ahead of the wave of “bad” users which will inevitably descend if you have any aspirations to scale. Good products shuffle bad users away from the spotlight, a “clean handling” that keeps the quality of the core community intact. Great products extract business value from bad users all the same.

Once your product and your community jibe, user acquisition marketing can be a powerful tool to grow the business. But user acquisition at scale is a messy game, and without smart product decisions it can end in disaster. MySpace had brilliant email acquisition marketing run by some of the best in the business, but users acquired through these channels are the kind of users that will set fifteen music videos to auto-play on page load if you let them. By letting them — that is, by not making the product decisions that elegantly handled these types of users — MySpace blew it.

Don’t blow it. Choose your community through your product; scale it through user acquisition marketing.

## A note: Some folks from the old-school advertising world may be a bit confused here. Some decisions traditionally in “marketing” — logo, colors, fonts, name — actually go in the “product” bucket in many web- and mobile-focused companies, and this is the way I classify them in my writings. Here, “marketing” is more specifically “user / customer acquisition”.

Written by Brad Hargreaves

August 12th, 2012 at 10:35 am

Posted in Uncategorized

Rent-Seeking Assets

with 3 comments

If you haven’t read Slate’s article on the economics of New York’s taxi medallion system, it’s worth the time. The tl;dr version is that the city government of New York has created an artificial set of rent-seeking assets — taxi medallions — where none should have existed in the first place. Specifically, the harm done to passengers and drivers by the economic rent extracted by medallion owners outweighs any possible benefit to putting an artificial quota on the number of cabs. Anyone who has tried to hail a cab in NYC between 4 and 5pm can attest to this.

Rent-seeking assets — explained quite well here — can be positive or negative for society. Many would agree that taxi medallions are a net negative. Carbon credits, on the other hand, are an artificial class of rent-seeking assets that could be quite beneficial. If economic theory is right, carbon credits will use an auction system to allocate the right to pollute to the firms producing the most economic value per unit of pollution.

So what makes a rent-seeking asset a good thing? I would argue that three criteria need to be hit to justify this kind of asset:

  • Negative externality of supply (the costs of pollution are not naturally born by the polluter)
  • Lack of differentiation among supply (your ton of carbon is just as bad as mine)
  • Destructive Nash equilibrium (naturally, all profit-seeking firms will pollute as much as they can get away with)

Unlike carbon credits, most products do not meet one or more of these criteria. I would argue that taxis do not meet the second and third criteria. While there may be a negative externality of supply — namely, pollution and traffic — there is certainly supply differentiation (some cabs are comfier, cleaner and friendlier than others), and there should not be a destructive Nash equilibrium, as in a free market taxis should only work the streets as long as it is financially sensible for their owners and drivers. Without these three criteria met, it’s not appropriate for government to create a rent-seeking asset class like taxi medallions.

Supply quotas and rent-seeking behavior go hand-in-hand. The City of New York limits the supply of taxi medallions to 13,000, which leads to a fair market price for a medallions of just over $1 million. But limiting supply doesn’t have to mean a quota. Professional certifications — such as the bar and CFA credentials — are great examples of meritocratic supply limitations. This kind of limitation is appropriate when supply is highly differentiated. In this case, not all lawyers and financial analysts are the same. These merit-based systems attempt to differentiate those truly able to provide services from the unqualified.

This kind of merit-based qualification can also work within a quota system. High-end college admissions is one example of this. There are only approximately 1,500 spots in each class at Yale, but those 1,500 are newly allocated each year on a (mostly*) merit-based system.

But even if those spots were simply sold to the highest bidders, it wouldn’t be a very good example of a rent-seeking asset. Specifically, Yale acceptances are missing a key characteristic of this kind of asset: transferability.

Imagine, for a moment, that a group of 1,500 people were given the right to admit to Yale anyone of their choosing per year back when the university was founded in 1701. Yale never built an admissions office -- it simply filled each class by taking one name from each of these "Admission medallion" holders every year. Of course, since the University would greatly outlive the medallion holders, the holders were given the right to transfer or sell their Admission medallion to anyone of their choosing.

Over the course of time, the vast majority of Admission medallions were acquired by various investment funds and private equity firms, such as Admission Financial. Each year, these huge firms auction the 1,500 spots off to the highest bidders. With some parents willing to pay north of $5 million to get their kids into Yale, the Yale admissions business brings in north of $7 billion per year in economic rent to massive holding companies -- essentially a transfer payment from aspiring families to wealthy investors and funds.

This is a silly example, but it paints a picture of how insane this kind of rent-seeking asset model would seem for more obviously differentiated products. Medical licenses are an even more extreme example: obviously, medical licenses should not be assets simply to be auctioned off to the highest bidder and subject to transfer or sale without restriction like carbon credits or taxi medallions.

Unfortunately, higher education accreditation often functions as a rent-seeking asset. While accreditation should in theory be a quality assurance mechanic — licensing schools as Bar Associations license lawyers, for instance — in practice accreditation is an asset that can be bought and sold, accumulating rent for its owners. Extremely challenging to acquire and just as difficult to lose, the treatment of accreditation as an asset rather than a qualification has played a significant role in the sad state of higher education today. Much like Medallion Financial buys and sells taxi medallions, financial institutions have seen an opportunity doing the same with colleges and their accreditations.

I do not believe that economic rents are always bad things. Rent-seeking assets have their place. But identifying that place — and where merit may be a more appropriate filter — is more critical than ever.

*From an economic perspective, things like athletic recruitment and legacy status count as “merit”, as the spots aren’t simply allocated to the highest bidder as it would be in a rent-seeking economic model. Although I’m sure there is a certain level of donation beyond which my kid is pretty much guaranteed to get in, so “mostly” applies.

Written by Brad Hargreaves

July 10th, 2012 at 11:18 am

Posted in Uncategorized

Tagged with

Accuracy and Precision

with 5 comments

In conversation, the terms accuracy and precision are used interchangeably. But they mean different things, and the difference can play a big role in the growth of a business. Before getting into early stage companies I spent most of my time in a science lab, which couldn’t have put me in a worse position to understand how accuracy and precision affect startups.

Courtesy Jekel 1996

In science, precision is valued above accuracy. In this case it’s called repeatability, and being able to run the same experiment multiple times with the same results is a good thing. After all, most fields of science expect results with 95% confidence, which means that your error rate can be no higher than one in twenty. So controlling for all possible variables and demonstrating repeatability are of utmost importance.

In startups, this kind of thinking will get you killed on two fronts. First, achieving 95% confidence is impossible in business. If you can collect enough data to be right 60% of the time you’ll get buildings named after you. You can’t possibly control for all variables; you’re lucky if you have the time and money to even understand what they are. This can be summarized with with the old business adage “it’s better to be generally correct than exactly wrong.”

Second, running a variety of experiments that yield different results is a positive thing. If all of your business experiments look similar and yield similar results, you haven’t learned very much, and you certainly haven’t explored the full set of possibilities. In all likelihood there is a better outcome elsewhere, open for a competitor to find and exploit. In other words, you don’t want to optimize toward a local maxima while missing a bigger opportunity.

Take marketing strategy, for instance. Good entrepreneurs usually try a number of diverse strategies — perhaps PPC, plus SEO, plus events or social media — to get a few data points around what works and what doesn’t. Thinking about this as a fractal and trying a few diverse strategies within each of these categories can pay dividends as well. While most of these experiments will likely fail, they can provide multiple starting points from which to drill down and test further, or provide guideposts to the right answer.

So while the precision of experiments is not so critical, accuracy is key. With all your experiments, you want to be close enough within range to triangulate the right answers through experiments. While precision without accuracy is dangerous, being neither precise nor accurate is useless. Trying five wildly different social media engagement strategies for a beta product may yield a false negative if social media isn’t the right acquisition channel; picking a wider variety of tactics may have generated more interesting results.

Business isn’t science, but you can be scientific about it. Having the right experimental framework can go a long way to saving time and money.

Written by Brad Hargreaves

May 13th, 2012 at 12:38 pm

Posted in Uncategorized

Tagged with , ,