[Vision2020] Shoshana Zuboff: Harvard Business School: Surveillance Capitalism's Threat to Democracy

[Ted Moffett]

Shoshana Zuboff on Surveillance Capitalism’s Threat to Democracy The
Harvard Business School professor discusses her new book.
---------------------------------------
Vision2020 Post: Ted Moffett
---------------------------------------
http://nymag.com/intelligencer/2019/02/shoshana-zuboff-q-and-a-the-age-of-surveillance-capital.html


By Noah Kulwin <http://nymag.com/author/Noah%20Kulwin/>
 FEB. 24, 2019

On February 4, Google proudly announced that its Nest Secure home
anti-intrusion device would now support its voice-activated Google
Assistant service. There was a catch, however: No one knew that the Nest
Secure actually had a microphone inside it
<http://nymag.com/intelligencer/2019/02/google-sorry-it-forgot-to-mention-secret-microphones.html>.
Google claims the microphone “was never intended to be a secret” and “has
never been turned on,” but it’s hard to shake the feeling that hidden mics
are a natural step for Silicon Valley giants intent on collecting as much
data as possible, no matter the cost to user privacy.

This episode is a perfect encapsulation of the digital threat outlined in a
new book by tech critic and Harvard Business School professor Shoshana
Zuboff, *The Age of Surveillance Capitalism: The Fight for a Human Future
at the New Frontier of Power*
<https://www.amazon.com/Age-Surveillance-Capitalism-Future-Frontier/dp/1610395697?ascsubtag=[]in[p]cjsgh5du1006680yep2gjq4lk[i]PPNlHp[d]D[z]m>.
First published as an essay in a German publication in 2014, what Zuboff
describes amounts to a new economic logic hatched in corporate America that
aims to extract staggering value from users’ private lives. Cataloguing a
dizzying array of sensors and invasive software, Zuboff sketches a vision
of the economic future in which companies race to collect data in pursuit
of Facebook- or Google-like profits.

Zuboff, who published her first book in 1989 on the future of technology
and data in the workplace, warns in *Surveillance Capitalism* of a “seventh
extinction” that threatens to eradicate “what has been held most precious
in human nature.” Given the fragility of the global political and economic
order, surveillance capitalism amounts to a “coup from above,” Zuboff
argues, an assault on democracy by way of subverting the very idea of what
it means to be an individual.

I recently spoke with Zuboff about her book, what constitutes surveillance
capitalism, how it operates, and what sets it apart from other historical
changes in the economy. The following interview has been edited and
condensed for clarity and length.

*I’m probably just gonna hold the recorder close to you because there’s a
lot of ambient noise in here. The first thing I wanted to ask was for a
definition of surveillance capitalism for the uninitiated. How would you
describe it?*
Do you want to come closer? Snuggle up, snuggle up. I would say that
surveillance capitalism in many respects diverges from the history of
market capitalism, but in one major respect it emulates the pattern of
market capitalism, and that goes like this: Historians have long recognized
that capitalism evolves by claiming things that live outside the market
dynamic and bringing them into the market dynamic so that they can be
turned into commodities for sale and purchase. So famously, for example,
industrial capitalism claimed nature for the market dynamic. Nature lives
in its own space and time, claiming nature for the market dynamic, reborn
as real estate, as land that could be sold and purchased. Similarly,
industrial capitalism claimed work for the market dynamic, so activities
that people did in their fields, in their gardens, in their homes, in their
cottages, now subordinated to the market, reborn as labor that could be
sold and purchased.

Surveillance capitalism follows this pattern but with what I would call a
kind of dark twist. And that is it claims private human experience as a
source of free raw material, subordinated to the market dynamic and reborn
as behavioral data. Those behavioral data are then combined with advanced
computational capabilities in order to produce predictions of human
behavior. So all of these behavioral data now streamed into our
21st-century dark Satanic mills, which are what we call machine
intelligence, machine learning, artificial intelligence, in order to spew
out predictions. These are a new kind of product.

*There’s data and information collected about people that is used
specifically to inform the particular products that are served back to
them, but there’s this other data, what you’ve termed behavioral surplus,
that’s information that doesn’t have an immediate use but is itself a kind
of control and power that these organizations possess that gives them an
advantage over other companies. Why is behavioral surplus something that’s
so critical in surveillance capitalism?*
The idea here is that what is being produced are predictions, predictions
of future human behavior that are then sold to markets of business
customers who have an interest in what people will do now, soon, and later.
So that’s the sequence, the mechanisms of surveillance capitalism. When I
say claiming private human experience and then translating it into
behavioral data, I’m talking specifically about aspects of private human
experience that aren’t what is required for product and service
improvement. So, for example, in the world of search, where these
mechanisms were first discovered and invented, it was clear that people
were searching and people were browsing and you could use the data to
improve the search engine, and create ancillary services like translation.
But there was collateral data that was also produced in these processes
that was not behavior that people understood they were sharing. It was an
offshoot of their search experience, of their search activity, but not
something they knew they were sharing. In the same way for example that you
might do a post on Facebook to meet your friends and family for dinner, and
what becomes interesting from the data point of view is whether you say,
“I’ll meet you later” or whether you say, “I’ll meet you at 6:45.”

So the point is that there’s a meta level of these data that have
tremendous predictive value that you don’t know you’re communicating when
you are posting or when you’re searching or when you’re browsing or all of
these things, and that’s data that is more than was needed for product and
service improvement. These extra data initially, at the beginning of all of
this, were lying around, unused, considered data exhaust, waste material.
Eventually it was discovered that they had significant predictive value and
that’s what was used to create the kind of prediction products of where
people would click that became the basis for these new online advertising
markets. So the idea here is that there is behavioral data that companies
are collecting about us, that are being used to improve what they give us,
but there is much more behavioral information that we are communicating
that we don’t know. And this is the surplus that they then take for its
predictive value, stream it through their production processes to create
these prediction products, and all of this is happening without our
permission, without our knowledge. It’s happening in a way that is designed
to bypass our awareness, it’s happening in a way that is engineered to keep
us ignorant.

This is what I call the shadow text. It’s what they can lift from these
behavioral flows that gives them tremendous predictive power, and it’s
different from what we knew that we were giving them. Which is why today
under some of the new regulations, for example GDPR, it says, well, you can
go to a company and you can ask them for the data they have on you. But
when you go to a company and ask them for the data they have on you, what
you’re really talking about is the data that you’ve already given them. But
the important data they have on you is this metadata, this stuff that
they’ve been able to pull from your data that you don’t even know about.
Like if you use exclamation points or if you said 6:45 or later, if you use
bullet points instead of just a general paragraph. And a million other
things. So it’s this surplus that is more than what is needed for products
and service improvement that became critical to the fabrication of these
prediction products and laid the basis for this new work.

*Just to pick an isolated strand of that data, how does whether or not I
prefer to use periods or ellipses or semicolons become something that has
predictive power that’s valuable?*
So, there’s something called the five-factor personality model where you
can pick behavioral cues from online material and you analyze it through
this five factor personality model and come out with very specific
personality assessments. Like, this is at the root of the Cambridge
Analytica work. You can take behavioral data, you can run it through that
model and make very fine-grain predictions. Like they can tell for example,
you know, if you’re gay, or if you’re likely to vote alt-right, or if
you’re likely to be a political malcontent. It’s correlated with all kinds
of other behavioral predictions. So, for example they can take all the
people who use ellipses, and they can cross correlate that with your
outcomes on these personality profiles, and in doing that, they can see
that, you know, people who tend to use ellipses are people who have
uncertainty or don’t like to finish things. These are huge databases that
correlate all these different tiny little signals. Many of them organized
by the five factor personality model and then there are other models built
on the five factor personality — like IBM’s got 12 factors, and somebody
else has 16 factors. There are these immense correlations that are running
all the time, so somebody who uses bullet points you know, they correlate
that with greater tendency toward precision.

*You could then theoretically arrange that data in such a way and get
something that’s useful to whatever the possessor of that data’s purpose is
or agenda is.*
You build up real detail about these profiles, in real psychological
detail, so that you can then begin to predict how a person with this
detailed profile is likely to react to stimulus, right? So this person with
this profile and these predilections and maybe this sense of confusion or
whatever, and then if I bring a stimulus that says “this thing is happening
and you should do this” — whether it’s “you should buy this” or “you should
vote this way,” or “this thing is happening and you should be against this”
— you’ve built a very robust prediction about how people with this profile
are likely to react to that kind of stimulus. And that gets very
fine-grained, very fine-tuned. This is happening with parallel processing
with millions and millions of data points and ends up being able to
predict, as we say with Cambridge Analytica, just you know pivoting these
methodologies which are the sort of normal “day in the life” methodologies
of any self-respecting surveillance capitalist, just pivoting those
slightly from commercial outcomes to political outcomes, and you get really
robust predictions about how people will react to certain kinds of
material, triggers, stimuli, and so on.

*Why is surveillance capitalism such a departure from just capitalism
before? It’s something you address in the book to some extent, and how it
departs from managerial capitalism and other traditions of capitalism but
I’m kind of curious about what’s the value of having particular
nomenclature to describe surveillance capitalists?*
Yeah, that’s a good question. So you know, theoreticians of capitalism have
talked over the years about capitalism as a sort of meta-economic process
of capital flows — certain necessities of scale, economic growth,
productivity, so forth. But it’s also been recognized that every era
produces its own unique market forms that are participating in these larger
principles of capitalism, but specific to a time and place. This is
historical materialism, if you will. So the idea is, for example, Adam
Smith contrasted mercantile capitalism, which was trading and ships between
countries and so forth, to what became industrial capitalism. He was
fascinated with the pin factory, because now the pin factory took these
large-scale processes of capital accumulation and growth and productivity
but now brought it into a different market form, which was specifically
this form of high volume and standardized production.

So even early on in the theorizing of capitalism it was understood that
capitalism takes on different market forms and different eras in the
context of different technologies. We’ve had mercantile capitalism, and
we’ve had factory capitalism, and mass-production capitalism, and
managerial capitalism, financial capitalism. And typically what happens in
these new concepts is that modifier, like “mass production,” or in my case
“surveillance” capitalism, what that modifier is doing is pinpointing the
pivot of value creation in this new market form. So, for example,
mass-production capitalism: What’s creating the value in this new form of
capitalism is the idea that you have a minute division of labor and
standardization of parts. The separation of execution and administration,
the high volume, low-unit cost which was the outcome, all depended upon
having these particular characteristics of mass production that produced
the value that ended up resulting in a new economic logic that yielded high
volume and low-unit cost.

*One of the things that big Silicon Valley companies arguably have in
common, more than how they’re able to make use of consumer data, is that
they’re able to construct software and hardware platforms to which entire
economies have had to accommodate themselves. They create their own sort of
protected gardens in which they can operate and in which they can
discriminate. I’m curious about the ways in which the nature of the
platform is actually what informs the economic power and ascendance in the
structure of this new form of capitalism.*
Yeah, well, the platform is a major technological innovation, but what I’m
distinguishing here is an economic logic. I’m really making an effort to
separate the technology from the economic logic And it is possible to
imagine platform capitalism without the surveillance market form, but just
as it is possible to imagine the digital technology without surveillance
capitalism, it is impossible to imagine surveillance capitalism without the
digital. So I’m making a distinction about a specific market form that has
grown up that is not a necessity of a platform even though it has grown up
on many platforms.

*What’s an example of a platform without surveillance capitalism? Are there
any current examples that come to mind for you?*
This is a very dynamic situation, and there’s nothing pure. Let’s take
Amazon as an example. Amazon has always been a very canny and even ruthless
capitalist. And it has used the platform for achieving monopoly in many
dimensions, and it has used huge amounts of behavioral data for really
improving its services. And so it has used the platform technology, the
platform system in a very powerful way. But it’s only more recently that
Amazon appears to have moved into surveillance capitalism as a market form,
as Amazon has moved into personalized services — the Alexa and all the
stuff that goes with the Alexa. It’s taken itself out of the realm of just
doing really canny, even ruthless platform capitalism, creating monopolies
and so forth, to seeking these supply chains of behavioral surplus for
personalizing, for creating predictions, for selling into these new
prediction markets, in ways that are not a necessary consequence of the
platform as a form.

This surveillance capitalism is an economic logic now that is travelling
not only across the tech sector, but traveling across sectors in the
regular economy where platforms are not the prominent constellation. So in
the insurance industry, in the health-care industry, insurance companies
are using telematics so they know how you’re driving in real time, and can
reward and punish you with higher and lower premiums in real time for
whether or not your driving costs them more or less money, or whether or
not your eating costs them more or less money, or whether or not your
exercise patterns cost them more or less money. Health-care providers who
are using telematics for not only feeding into these prediction markets but
also collecting all kinds of ancillary data to sell to third parties, and
being part of these whole ecosystems now, are behavioral surplus suppliers.
You download a diabetes app, it takes your phone, it takes your microphone,
it takes your camera, it takes your contacts. Maybe it helps you manage
your diabetes a little bit, but it’s also just a part of this whole
supply-chain dynamic for behavioral surplus flows.

*And this is an economic logic that is only now made possible because of
the technological developments of the last 20 years.*
There was a window where we had the internet, we had the World Wide Web,
before we had surveillance capitalism, where we imagined the smart home as
a closed loop. Devices in the home fed data to the occupant of the home,
the occupant of the home decided how the data were going to be used, if
they’re going to be shared, what they meant, all of that. Simple
closed-loop system. That was going to be the smart home. That was also
going to be telemedicine. Simple closed loops between the physician, a
server and the hospital, and the patient at home.

Fast-forward 20 years and now those simple closed loops are things like the
Nest thermostat, where scholarly analysis indicates that if you are a
vigilant consumer and you install a Nest thermostat in your bedroom, you
should review a minimum of 1,000 privacy contracts, because everything is
flowing to third parties which is flowing to third parties, which is
flowing to third parties. No one takes accountability for the third
parties, and by the way, if you don’t agree to each policy at each stage,
then you begin to lose the functionality of the product and the reason that
you bought it in the first place.

*In your first book, In the Age of the Smart Machine, you outline two
distinct paths for how the future might unfold, and the ways technology
might evolve. What do you see as having changed since that time — between
1989 and the 2000s — that led to this situation?*
In that book I talked about the dilemmas of knowledge, authority, and
power, and I talked about what I called at that time “the electronic text,”
the textualization of the world — the idea that everything was being
digitized, being translated into information, and that in situations where
there were no sanctions or constraints, that that information was being
used to survey and to control rather than to empower and to teach and to
learn and to grow together. But in the 20th century, when I wrote that
book, these issues of capital and digital were targeted on the economic
domain. They were targeted on our roles as workers and, you know, employees.

That was a story about how, if we don’t pay attention to the opportunities
in what I called an information workplace, where, you know, people need to
get educated to be included in this kind of work, and it’s a more
intellectual work, and it’s less hierarchical and everyone can make
contributions — if we don’t pay attention, it’s just going to be top-down
information, and it’s going to be used for control and surveillance, and
that’s what I call the information panopticon, which was the end of that
book.

And I talked about either going this way or that way, there’s neoliberalism
and re-engineering and shareholder value maximization, and no one really
giving a toss about educating the workforce and really taking advantage of
these new technologies. It was all about offshoring and outsourcing and
cost-cutting and down, down, down, down. So, of course it all went to the
surveillance things, so that now workplaces have really become the
laboratories for surveillance capitalism because that’s where we have
habituated human beings to perpetual surveillance.

*Amazon’s a good example of a pioneer in that, to your point earlier. They
may have been late to it with consumers but they were absolutely pioneers
of it in terms of their own workforce.*
Oh yeah, and just about everybody is. I mean, I’ve been on a book tour.
We’re just going in and out of buildings for interviews and you gotta —
you’ve got the ID, and you’ve got the thing, and then you have to go
through this thing and that thing and that thing and that thing, and all
these buildings are surveillance now. If you’re actually an employee there,
everything is surveilled. Everything is surveilled. And the depth of
surveillance in the workplace now exceeds anything I even imagined in the
end of the ‘80s when I published that book. So that’s all come true.

Whatever darkness I foresaw there has come through and much, much more. But
now, the thing is, we’re in the 21st century, and so now these same
capabilities, this textualization of the world, the idea that everything is
now information — well, this [has] spilled over the walls of the factories
and the offices and the workplaces. If those struggles were between capital
and labor, the struggles today are between capital and society. Now this
stuff is bearing down on all of us because we’re the users, you know. It’s
not about our roles, our economic roles as workers, as employees — now our
interface with these forces of capital is just as regular people in the
course of our daily lives. We’re called users.

*So our app user is like the new prole — I don’t want to be glib, but it
seems like what you’re describing is a major reconfiguration —*
It’s not like we’re the new proletariat, but we are the new, like, target
of capital, and that’s why I started out by saying private human
experience. It’s our human experience, our private experience that is the
new target of capital for the raw material that it’s now manufacturing into
products that it’s selling.

*So the capitalists came for the trees, then they came for the labor, and
now they’re coming for individual, private lives. Do you have any idea what
could possibly come next?*
So what I want you to understand about this — because I feel like I haven’t
answered a question that you asked earlier, which is why I call it
surveillance capitalism and not platform capitalism — in coming for our
human experience … So, let’s recognize the structure of the markets that
they’ve created, because they’re coming for our human experience, they’re
translating it into behavioral data, they’re creating predictive knowledge
out of that and they’re selling that predictive knowledge. But they’re
selling it to others, not to us. So it’s knowledge about us but not for us.
These are inherent asymmetries, right? The reciprocities now are between
the surveillance capitalists and their business customers who figure out
that their best way to make money now, where their margins are going to
come from, is knowing what we’re going to do in the future, not selling us
something that is really, really terrific and is going to make our lives
better.

*A counterpoint: These companies are really bad at predicting what their
consumers will do in the future. One of the biggest places Amazon’s
advertising dollars go is into retargeting — ads showing you what you’ve
already looked at. And they believe that showing you what you’ve already
looked at is a good way to get you to buy more of that thing. And that’s
where they get the most value now, it’s not a predictive thing, it’s just
literally regurgitating what you’ve already bought back at you. Are these
companies on the precipice of developing far more sophisticated predictive
powers that can make use of the data they’re collecting?*
Oh, well, look, online targeted advertising was the place where this
started, but this is not the place where it ends. It’s like saying that
mass production was only relevant to make you Model T’s. This logic is
spreading to all these different contexts, and all these different
predictive markets, so — let’s go back to the Model T, for example. Three
or four months ago, the CEO of Ford says there’s a global auto slump, it’s
really hard to sell cars, but we’re getting downgraded in the markets. He
says, we want price-to-earnings ratios like Google and Facebook. How do we
do that? We’re going to become a data company. We got 100 million people
driving around in Ford vehicles. And what we’re gonna do is, we’re gonna
now figure out how to get all the data out of this driving experience.

So this is the telematics, this is the stuff that, you know, we can not
only know how you’re driving and where you’re driving, we can know the gaze
of your eyes — and that’s really important for insurers, to know if you’re
driving safely. And we can know what you’re talking about in your car, and,
like, Amazon and Google and so forth are already in a contest for the car
dashboard, because that’s a way of hearing what you’re talking about and
knowing where you’re going. So they talk about, you know, shopping from the
driving wheel. So now the automobile itself becomes this little
surveillance bubble. We can get all of this information, from your
conversation to your shopping to where you’re going to what you’re doing to
how you’re driving. And this has predictive value for all kinds of business
customers.

*It also seems to me, though, to be contingent on the capacity of these
companies to make technological developments. How do we know that the
surveillance capitalist is going to actually have power going forward? It’s
scary, and they’re doing a lot of excessive data collection already that
violates any conceivable norm, but it also seems like these companies are
struggling to develop the power to get even the data that they claim to
want in the future.*
Well, like I said, this is dynamic. This is where we are now: that for a
product company to say our margins and our revenue growth are not going to
come from our product, it’s going to come from the data that we can scrape
from the people that use our product, because those data are going to be
valuable and lucrative in these secondary markets that want to know where
we’re going to shop, where we’re going to do, what we’re going to do … This
is now where capital is shifting to create these new capabilities, and it’s
the telematics, and you read about what’s going on in insurance and stuff
is very real. You’ve got all the consultancies lined up, advising the
insurance companies. This is what is dynamic, this is what is taking root
now and flourishing now. Began in the online world, it began in the tech
sector, it’s migrating through all these different sectors.

What you’re seeing is a shift to this idea of, it’s not a product, it’s a
smart product, it’s not a service, it’s a personalized service. This is the
shift to surveillance capitalism, it’s a shift to this sort of parasitic
form that instead of, “We’re putting all our effort into making the best
product for you,” or the best service for you, and really solving the
problems in your life and really making your health better, helping your
financial situation and creating more employment opportunities, you know,
it’s this …

*I guess part of what I find curious is that some businesses are really
good at this and others are really bad. You use these apps and they’re all
terrible. They’re not great. Do you think that these laggards in
surveillance capitalism pose any threat to the viability of that economic
logic?*
We’re not anywhere near the endgame, you know, and people say to me all the
time, well, it’s too late and we can’t do anything about it, how do we ever
undo this? And I say, no, no, no, that’s not where we are.

Mass production began as a very violent form. And there was no law to
constrain it or impede it, and, there were unsafe working conditions and
people were paid slave wages and children were working in factories. And it
took decades and public contest and mustering democratic resources and
eventually law and regulation that shaped and tamed it and made it
something that was actually palatable for society in a relative equilibrium
that we call market democracy. And I believe that surveillance capitalism
has gone very far in some sectors, and is moving across other sectors.
That’s why Ford to me is such an iconic example: This is what businesses
are looking to, here’s where the margins are gonna come from, this is gonna
be the new solution for how we make profit in the information age. The
dominant form of capitalism is actually something where the main thing that
we sell and buy are behavioral futures.

So this is why it’s so critical for us to understand this, because we’re at
the beginning of this art, not the end of it. This has developed over the
last 20 years with barely any impediment from law, with barely any
impediment from regulation. It has developed while democracy has slept. And
I give — I explore 16 reasons, in-depth, including some historical reasons
why that is the case, why it has had basically a 20-year free run to
develop.

But my argument is, this is why now, you know, the time has come where
enough of this is substantiated out there that we can understand it, that
we can see how it works. We can see its aims and its goals and its results.
When I write about the Facebook contagion experiments and Pokémon Go, I
write about these as sort of population-scale experiments in how to do
surveillance capitalism in the real world, at scale. So Pokémon Go is a
dry-run for the Google City. Right? Google City is a population-level
experiment in how you herd and tune populations, modify people’s behavior,
to steer them toward guaranteed commercial outcomes, so that the game is
really based on Pokémon Go’s Niantic Labs which, of course, is a
Google-incubated operation. It’s hosting its own behavioral futures
markets, it has business establishments, McDonald’s, and so forth, paying
it for footfall in exactly the same way that online advertisers pay for
click-through rates.

*More than Amazon marketplace customers pay to play on that platform —*
That’s right. But now we’re in the real world, and we’re learning how to
tune and herd people through the real world, through their real lives, in
their real cities, to the places where now Niantic Labs gets paid because
you showed up there, and you bought a pizza, or you bought a drink, or you
bought a burger, or you stayed for how ever many hours, and you played in
the Pokémon gym or whatever, in the toilet.

So this is the template for the Google City, the smart city, where the
whole digital architecture becomes a global means of behavioral
modification that we use to tune and herd populations toward guaranteed
commercial outcomes in the city. And this is why citizens in Toronto now
are contesting the idea of Google having the waterfront, because the whole
idea is now these computational systems replace politics, and the whole
idea is that they’re gonna gently, with a smile, tune and herd people
toward the outcomes that serve Google’s commercial goals and the commercial
goals of its business customers in these prediction markets.

*It sounds almost like it’s a means of governance, as an algorithm.*
It substitutes computation for politics, so it’s post-democracy. It
substitutes populations for societies, statistics for citizens, and
computation for politics, and so I read a lot about this experimental zone,
the Facebook contagion experiments, where another experimental zone where
when they wrote up those experiments in scholarly journals — very smart
data researchers from Facebook combined with very smart academic scholars
boasted that now we know that we can use the online world to create
contagion that changes behavior in the real world. The first case it was
voting, the second case it was emotional. And they bragged in their
articles that we can do this in a way that bypasses the awareness of the
user. Right? Always engineered for ignorance. Because you know, that’s the
surveillance essence of this, you can’t do this by asking permission. You
can only do this by taking it in a way that is secret, that is hidden, that
is backstage.

*I feel like I’d be remiss if I didn’t bring up China. To what extent do
you think China offers a model of what the future may look like or how it’s
distinct from it?*
You’ve asked the question at the perfect time in our conversation, because
we’re talking about the substitution of computation for politics, which is
to say, for democracy. I’ve just been talking to you about how surveillance
capitalism commandeers the digital infrastructure as a global means of
behavioral modification, and I ask the question, what is the power to
modify population behavior at scale? What is that power? And I answer the
question. This is not totalitarian power. No one’s coming after us to
murder us, to put us in concentration camps, to throw us in the gulag, to,
um, to control us through terror.

*No, they just want our money.*
Really, they just want our data. They don’t care what we believe. They
don’t care if we’re happy, they don’t care if we’re sad. They don’t care if
we’re in pain, they don’t care if we’re in love. They only care that
whatever we are and whatever we do, we do it in a way that interfaces with
their supply chains. So that they’re getting their data flows. They are in
this structure, fundamentally indifferent to the content of our behavior.
They just want to have the data from our behavior.

I call this instrumentarian power, in contrast to totalitarian power.
Instrumentarian power. Two reasons: One, it relies on the instrumentation
of the digital milieu because it’s through that medium that we are being
tuned and herded and shunted and coached and modified. No one’s coming up
to like, you know, slap you or kill you or hurt you …

*It sounds kind of totalitarian, in that people are just stripped of
agency.*
Well, that’s a piece of it. But “totalitarian” is a very specific thing.
It’s a centralized totalitarian power that is specifically understood as
functioning through the mechanisms of terror and murder. That’s what
totalitarianism is. Instrumentarian power wants to control you, but it
doesn’t care about hurting you. It wants to control you toward its
guaranteed commercial outcomes. It’s making you, you as Noah, are simply
instrumentalized toward the outcomes of its business customers, right? So
you’re a means to others’ commercial ends. You’re instrumentalized and
you’re surrounded by this milieu of instrumentation that is sort of
hands-off. It’s your dishwasher, and your television set, and your car and
the telematics, and your phone. It’s this whole digital surround that is
now the instrumented medium that is producing the knowledge that creates
the opportunity for the power to modify your behavior. So, um, all right,
got to go back to the question that you just asked that I was answering …

*China.*
Oh, China, let’s go to China! China, China, China. All right. So what I’ve
said is that this is a kind of power that has never existed before, just as
totalitarianism, when it emerged in the 20th century, was a kind of power
that scholars had never seen before. So we have this instrumentarian power,
and it’s coming out of the private sphere, the commercial sphere. It’s
extraordinary power, under the auspices of private capital. And one of the
things that it wants to do, that it aims to do, is to substitute
computation for politics.

That’s our pivot. Now look at China. What is China? China is an
authoritarian state, not a democratic state. So, authoritarian state.
Inside China, these internet companies which amassed also enormous,
enormous instrumentarian capabilities, enormous asymmetries of behavioral
knowledge, predictive power, analytical capability, able to reward and
punish behavior in these very finely grained ways. Buy this, discount that,
so and so forth. An authoritarian state sees these instrumentarian
capabilities and says, “This is perfect. We want to take these, not terror,
not murder, not the gulag. We wanna take these instrumentarian capabilities
and now pivot them to the political and social outcomes that we as an
authoritarian government seek. This is how we want our population to behave
and this is how we’re gonna discipline our population. And these are the
parameters we’re gonna give them. And this is how we’re gonna restore order
in a society that has fallen into chaos because of the complete destruction
of social trust in the wake of decades of the Maoist project.” So, what you
see in China is the marriage of authoritarian state with instrumentarian
power. And that is a very dark and dangerous endgame.

*You describe it as an endgame, and I don’t want to conflate things, but
having earlier described the moment we’re at as an initial stage of
surveillance capitalism —*
A dynamic stage.

*Dynamic.*
Not initial, necessarily, but dynamic.

*And China represents one place where the dynamic could change and where —*
And what’s the key variable? The key variable there is democracy. That’s
why democracy is such an important concept in this whole conversation. We
were talking about the smart city, the substitution of computation for
politics and ultimately for democracy, and then you brought up China. I
said, “You brought it up at the perfect time,” because this is so
important. This is where these pieces connect. Because if we allow
computation to substitute for politics, and we allow statistics to
substitute for citizens, and we allow populations to substitute for
societies, we are destroying democracy as we know it. And if we destroy
democracy, all we are left with is this sort of computational governance,
which is a new form of absolutism, Noah. It’s a new form of absolutism,
computational governance. And that absolutism becomes its own opening to
authoritarianism, because now we have eroded and weakened democracy, and
hollowed out the resources of our democratic institutions in favor of this
sort of vision of a totalistic, certain rationality that optimizes all
population behavior toward perfect outcomes, be they commercial or
functional or political.

So, this is what surveillance capitalism represents. And this where my book
ends up, and the final chapters represent a profound threat to democracy
that goes beyond the brief of just another capitalism that makes a lot of
money. It increases mass consumption, and employs a lot of people, and
gives consumers what they want. Now we’re talking about a form of
capitalism that in order to fulfill itself, its own imperatives of scale,
scope, and action, its own imperative of prediction, in order to fulfill
its own imperatives, ultimately it replaces society and replaces politics
with these computational principles, which right now are aimed toward
commercial outcomes.

*But those principles could very easily shift.*
And that’s what Cambridge Analytica taught us. These were the people who
worked out the five-factor personality model on Facebook profiles for years
and years, and understood the huge predictive value from behavioral surplus
drawn from Facebook profiles. Now, under the new regime of Robert Mercer,
the billionaire who bought Cambridge Analytica and owned the Trump
campaign, under these new auspices of plutocracy, Cambridge Analytica takes
a day in the life of any self-respecting surveillance capitalist, just the
ordinary mechanisms that are being used to do this work every day, and just
pivots them a few degrees from commercial outcomes to political outcomes,
and uses all the same methodologies to use the online milieu to affect
real-world behavior. But this time it’s real-world political behavior, not
real-world commercial behavior. So, that’s what Cambridge Analytica taught
us.

*The way this can be repurposed for political ends, not commercial ends.*
That these capabilities of mass behavioral modification, instrumentarian
power that can be used to tune and herd populations toward commercial
outcomes in the real world, in the service of growth of surveillance
revenues, can be repurposed for political ends. The Chinese are doing it in
the service of an authoritarian state, and what we saw in America already
is that anybody with enough money, any ambitious plutocrat, can buy the
skills and the data to use these same methodologies to influence political
outcomes.

And so, what is under siege here is democracy, from two directions. The
structure created under the auspices of private capital that is based on
unprecedented asymmetries of knowledge — knowledge that is about us but not
for us, that gives rise to unprecedented asymmetries of power, a power that
is able to shape our behavior but that operates outside of our awareness
and is designed to operate, always keeping us ignorant. Engineered to keep
us ignorant. We’re entering the 21st century in this institutional setting
that introduces a whole new axis of social inequality. Not just economic
inequality now, but these profound inequalities of knowledge and the power
that accrues to behavioral knowledge that can actually influence our
behavior, influence the behavior of our group, of our city, of our region,
of our country, of our society. That’s at the institutional level and that
is disfiguring 21st-century society, right from the beginning.

At the individual level, we are the so-called users whose behavior is being
intervened upon, touched, coaxed, modified, contaged, influenced in all of
these ways that are designed to be undetectable to us, to bypass our
awareness. So, this is an intervention at the level of human agency, at the
level of human autonomy, at the level of individual sovereignty, at the
level where we expect to have an idea about our future that we act on now —
I’m gonna meet Noah, so I get up early in the morning and I do a bunch of
things that are gonna get me to the place where eventually I’m gonna meet
Noah, because that’s what I choose to do. That’s my action. That’s what we
call free will, that’s what we call agency, and that’s what we call moral
autonomy. We can’t imagine a democratic society without human beings who
can act this way. It was part and parcel of the conception of a
self-regulating demos from the beginning.

Now we’re entering the 21st century with these threats to democratic
society coming both from the structural level and from the individual
level, the level of our intimate behavior. And when you put these two
things together, that becomes something that’s, you know, bigger than what
we normally associated with capitalism. As we said at the beginning of our
conversation, Noah, that [was a] capitalism that bore down on the economic
domain and it was in our workplaces and in our economic roles. Now, this
thing, because of its drive for totality, for total information, for total
certainty, for scale, for scope, for action, these imperatives that drive
it … It’s into society, it’s into every aspect. When Eric Schmidt heard
that the Toronto public officials had given Google the go-ahead to take the
part of the waterfront to turn it into this smart city thing, I might be
slightly paraphrasing, but Eric Schmidt’s words were, “Oh great, now it’s
our turn.”

*I remember this.*
I quoted that because that seemed so pregnant to me, because “our turn”
meant something different from democracy’s turn, like, democracy’s turn was
up and now it’s our turn. The turn of private capital, the turn of private
surveillance capital. What I want our readers to know is that in the arc of
human history, the idea of democracy, the autonomous individual, individual
sovereignty, moral judgment, these ideas are young ideas, they’re five
minutes old. And humanity sacrificed, for many millennia, for the
legitimacy, ultimately, of these ideas. And for us to say that, you know,
democracy’s time is up and now it’s surveillance capitalism’s turn, it’s,
to me, an abomination. That is an intolerable idea. What we should be doing
right now is doubling down on democracy. Every generation has a
responsibility for democracy. Every generation has a responsibility to keep
it going, to keep that wheel turning, to keep it flourishing, to keep it
from dying, from falling over, from breaking, from falling vulnerable to
the forces that always want to supplant democracy with certainty and
financial advantage and power and so forth.

Surveillance capitalism really, as we said before, it moves beyond the
economic domain, to the societal domain, into our lives, threatening the
very basis of a democratic social system. And that’s where, to me, we now
recognize that the time has come for us to name it, to understand it, to
recognize that it’s an economic logic that is not a necessary consequence
of the digital or of the platform mechanism and that this is the time when
we now, through understanding and through naming, we change our
consciousness, we turn to our democratic institutions and reclaim democracy
as a source of the law and the new regulatory institutions that are gonna
intervene upon, interrupt, and outlaw these mechanisms, so that the age of
surveillance capitalism turns out to be a short age. This is not
legitimate. And the choices with which it has left us, as 21st-century
citizens, are not legitimate choices. We should not be having to choose
between having our experience scraped for other’s purposes and the basic
requirements of effective social participation.

*What do you think of the emerging political currents aligned against these
kinds of companies pursuing this economic logic?*
I mean, the great news is that we now have a political discussion going on.
And we haven’t had that because there has been so much political capture.
Google is the largest lobbyist, Google made itself invaluable in the
election process, actually getting people elected.

*It also struck me that these were people who were popular in the public
imagination because they were paying people really well, and they were
offering services that were free, and so on. There was a cultural cachet
that’s now cracking. What do you make of the efforts to build political
agendas meant to further destabilize it?*
Politics is the realm where this is gonna get fixed. It’s not surprising
that the loudest political discussions are beginning on the left. I’ve just
come back from several weeks in Europe, and there are discussions in
Brussels where it’s clear that we’ve gotta confront surveillance capitalism
and that’s gonna go beyond the GDPR, it’s gonna go beyond privacy, and it’s
gonna go beyond antitrust, because, you know, you take a big surveillance
capitalist like Facebook or Google, you break it up, and then you’ve got
four smaller surveillance capitalists. Antitrust, privacy, these are
critically important, but they’re also 20th-century paradigms that aren’t
gonna take us all the way into intervening upon and interrupting and
outlawing these new operations. So the political discussion is key.

Ultimately, I see it as a political discussion that’s gotta be driven by a
sea change in public opinion. Because everywhere I go, people are fed up
with this, but they haven’t known how to name it. It’s a sense of unease,
of anxiety, of loss of control, a sense of being manipulated, a sense of
loss of freedom. A sense of powers that we don’t understand. And as they
are able to name it, the sense of its intolerability becomes truly
palpable. These shifts in public opinion are what ultimately are gonna
force elected officials to pay attention. This is going to be a new era of
law, a new era of regulation, and it’s got to be designed for this
21st-century situation, where we have unprecedented economic mechanisms
that are going to require unprecedented legal remedies.

At the same time, I also believe that once we begin to double down on
democracy and democracy sort of awakens, the sleeping giant of democracy
awakens to actually confront this stuff, we’re creating space for new
competitive solutions. Because we need different kinds of companies, and
different kinds of capitalists, and different kinds of ecosystems and
alliances that are actually capable of reclaiming the digital for the kind
of values and functionality that we wanted from it in the first place.
Every single piece of research, going all the way back to the early 2000s,
shows that whenever you expose people to what’s really going on behind the
scenes with surveillance capitalism, they don’t want anything to do [with]
it. The only reason we keep engaging with it is because we feel like we
have no choice. So, as soon as there is real choice for people that can
also provide effective action, these new competitors have the opportunity
to have literally every person on Earth as their customer. I mean, that’s
really what it’s about right now.

Another way to look at surveillance capitalism, from the market point of
view, is as a colossal market failure. Because it is not giving people what
people want. It’s giving business customers what they want, to be able to
manipulate people, but it’s not giving actual populations of people what we
want. So, it has severed the traditional reciprocities between capitalism
and its societies, where it counts on its society as a source of customers
and a source of employees, and reoriented itself toward these prediction
markets of business customers. But in the process, you know, our problems
are not being solved. Our health-care problems are not being solved. Our
climate problems are not being solved. Our employment problems are not
being solved. How to build a better car, how to build a better building,
how to make a better city that is democratic and still contains pollution
and all those other things. These problems are not getting solved,
surveillance capitalism is scraping off data for other purposes that are
not solving our core issues.

*The trade-off that we’re sort of baked into believing is that, “Well, the
price I pay is that I’m giving up a bunch of shit I don’t know about
because I can watch The Simpsons whenever I want with maximum convenience.”
We’ve sort of implicitly accepted that there’s a trade-off already. We
didn’t even have to be duped, we didn’t even have to be presented with a
choice because it was made for us and we seemed okay with it. I mean,
that’s the image that’s presented. I don’t quite buy that.*
That’s the image that’s presented, but that’s because everything that’s
inside that choice has been designed to keep us in ignorance.

We download a diabetes app. Just downloading it allows x percentage of the
diabetes apps to take all the contacts on your phone, x percentage take the
contacts and the mic, x percentage takes the contacts, the mic, and the
camera. The stuff that they’re taking from you has nothing to do with the
diabetes functionality for which you downloaded the app. Absolutely
nothing. It’s simply siphoning off data to third parties for other revenue
streams that are part of these surveillance capitalists’ ecosystems. This
is a hidden world that is constructed around us in these elaborate supply
chains. Everything that we’re touching that is internet-enabled is
essentially a supply chain interface, but it is all designed to be hidden.

This is a conversation that hasn’t happened yet. And if there’s a reason
why people are really digging this book I’ve written [for which] I’ve spent
so long trying to discern these mechanisms, name them, and bring them forth
so that people can have them to think about and talk about, it’s that all
of a sudden, here’s this thing I’ve been feeling, this problem I’ve been
feeling that I didn’t really know what it was and I didn’t have a name for
it, and now I can name it. And when I can name it, I can think about it,
and there’s a language. And once you have that, really, the landscape
changes and the power dynamics change. And I don’t mean just my book. There
are many other inputs to this process. But as we begin to name, this whole
power dynamic is gonna change, because this stuff can no longer get away
with being secret, it can no longer get away with being surveillance. And
when the jig is up on that, then we’re cutting at the core of its economic
umbilical cord.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.fsr.com/pipermail/vision2020/attachments/20190407/5c56aeab/attachment-0001.html>