As the token European among the Roll For Enterprise hosts, I'm the one who is always raising the topic of privacy. My interest in privacy is partly scarring from an early career as a sysadmin, when I saw just how much information is easily available to the people who run the networks and systems we rely on, without them even being particularly nosy.

Because of that history, I am always instantly suspicious of talk of "personalising the customer experience", even if we make the charitable assumption that the reality of this profiling is more than just raising prices until enough people balk. I know that the data is unquestionably out there; my doubts are about the motivations of the people analysing it, and about their competence to do so correctly.

Let's take a step back to explain what I mean. I used to be a big fan of Amazon's various recommendations, for products often bought with the product you are looking at, or by the people who looked at the same product. Back in the antediluvian days when Amazon was still all about (physical) books, I discovered many a new book or author through these mechanisms.

One of my favourite aspects of Amazon's recommendation engine was that it didn't try to do it all. If I bought a book for my then-girlfriend, who had (and indeed still has, although she is now my wife) rather different tastes from me, this would throw the recommendations all out of whack. However, the system was transparent and user-serviceable. Amazon would show me transparently why it had recommended Book X, usually because I had purchased Book Y. Beyond showing me, it would also let me go back into my purchase history and tell it not to use Book Y for recommendations (because it was not actually bought for me), thereby restoring balance to my feed. This made us both happy: I got higher-quality recommendations, and Amazon got a more accurate profile of me, that it could use to sell me more books — something it did very successfully.

Forget doing anything like that nowadays! If you watch Netflix on more than one device, especially if you ever watch anything offline, you'll have hit that situation where you've watched something but Netflix doesn't realise it or won't admit it. And can you mark it as watched, like we used to do with local files? (insert hollow laughter here) No, you'll have that "unwatched" episode cluttering up your "Up next" queue forever.

This is an example of the sort of behaviour that John Siracusa decried in his recent blog post, Streaming App Sentiments. This post gathers responses to his earlier unsolicited streaming app spec, where he discussed people's reactions to these sorts of "helpful" features.

People don’t feel like they are in control of their "data," such as it is. The apps make bad guesses or forget things they should remember, and the user has no way to correct them.

We see the same problem with Twitter's plans for ever greater personalisation. Twitter defaulted to an algorithmic timeline a long time ago, justifying the switch away from a simple chronological feed with the entirely true fact that there was too much volume for anyone to be a Twitter completist any more, so bringing popular tweets to the surface was actually a better experience for people. To repeat myself, this is all true; the problem is that Twitter did not give users any input into the process. Also, sometimes I actually do want to take the temperature of the Twitter hive mind right now, in this moment, without random twenty-hour-old tweets popping up out of sequence. The obvious solution of giving users actual choice was of course rejected out of hand, forcing Twitter into ever more ridiculous gyrations.

The latest turn is that for a brief shining moment they got it mostly right, but hilariously and ironically, completely misinterpreted user feedback and reversed course. So much for learning from the data… What happened is that Twitter briefly gave users the option of adding a "Latest Tweets" tab with chronological listing alongside the algorithmic default "Home" tab. Of course such an obviously sensible solution could not last, for the dispiriting reason that unless you used lists, the tabbed interface was new and (apparently) confusing. Another update therefore followed rapidly on the heels of the good one, which forced users to choose between "Latest Tweets" or "Home", instead of simply being able to have both options one tap apart.

Here's what it boils down to: to build one of these "personalisation" systems, you have to believe one of two things (okay, or maybe some combination):

• You can deliver a better experience than (most) users can achieve for themselves
• Controlling your users' experience benefits you in some way that is sufficiently important to outweigh the aggravation they might experience

The first is simply not true. It is true that it is important to deliver a high-quality default that works well for most users, and I am not opposed in principle to that default being algorithmically-generated. Back when, Twitter used to have "While you were away" section which would show you the most relevant tweets since you last checked the app. I found it a very valuable feature — except for the fact that I could not access it at will. It would appear at random in my timeline, or then again, perhaps not. There was no way to trigger it manually, or any place where it would appear reliably and predictably. You just had to hope — and then, instead of making it easier to access on demand, Twitter killed the entire feature in an update. The algorithmic default was promising, but it needed just a bit more control to make it actually good.

On the other hand, I quite like “while you were away” and wish I could access it on demand, instead of being reduced to hoping it shows up, like a peasant praying for rain.

— Dominic 🇪🇺🇺🇦🏳️‍🌈 (@dwellington) June 29, 2021

This leads us directly to the second problem: why not show the "While you were away" section on demand? Why would Netflix not give me an easy way to resume watching what I was watching before? They don't say, but the assumption is that the operators of these services have metrics showing higher engagement with their apps when they deny users control. Presumably what they fear is that, if users can just go straight to the tweets they missed or the show they were watching, they will not spend as much time exploring the app, discovering other tweets or videos that they might enjoy.

What is forgotten is that "engagement" just happens to be one metric that is easy to measure — but the ease of measurement does not necessarily make it the most important dimension, especially in isolation. If that engagement is me scrolling irritably around Twitter or Netflix, getting increasingly frustrated because I can't find what I want, my opinion of those platforms is actually becoming more corroded with every additional second of "engagement".

There is a common unstated assumption behind both of the factors above, which is that whatever system is driving the personalisation is perfect, both unbreakable in its functioning and without corner cases that may deliver sub-optimal results even when the algorithm is working as designed. One of the problems with black-box systems is that when (not if!) they break, users have no way to understand why they broke, nor to prevent them breaking again in the future. If the Twitter algorithm keeps recommending something to me, I can (for now) still go into my settings, find the list of interests that Twitter has somehow assembled for me, and delete entries until I get back to more sensible recommendations. With Netflix, there is no way for me to tell it to stop recommending something — presumably because they have determined that a sufficient proportion of their users will be worn down over time, and, I don't know, whatever the end goal is — watch Netflix original content instead of something they have to pay to license from outside.

All of this comes back to my oft-repeated point about privacy: what is it that I am giving up my personal data in exchange for, in the end? The promise is that all these systems will deliver content (and ads)(really it's the ads) that are relevant to my interests. Defenders of surveillance capitalism will point out that profiling as a concept is hardly new. The reason you find different ads in Top Gear Magazine, in Home & Garden, and in Monocle, is that the profile for the readership is different for each publication. But the results speak for themselves: when I read Monocle, I find the ads relevant, and (given only the budget) I would like to buy the products featured. The sort of ads that follow me around online, despite a wealth of profile information generated at every click, correlated across the entire internet, and going back *mumble* years or more, are utterly, risibly, incomprehensibly irrelevant. Why? Some combination of that "we know better" attitude, algorithmic profiling systems delivering less than perfect results, and of course, good old fraud in the adtech ecosystem.

So why are we doing this, exactly?

It comes back to the same issue as with engagement: because something is easy to measure and chart, it will have goals set against it. Our lives online generate stupendous volumes of data; it seems incredible that the profiles created from those megabytes if not gigabytes of tracking data have worse results than the single-bit signal of "is reading the Financial Times". There is also the ever-present spectre of "I know half of my ad spending is wasted, I just don't know which half". Online advertising with its built-in surveillance mechanisms holds out the promise of perfect attribution, of knowing precisely which ad it was which caused the customer to buy.

And yet, here we are. Now, legislators in the EU, in China, and elsewhere around the world are taking issue with these systems, and either banning them outright or demanding they be made transparent in their operation. Me, I'm hoping for the control that Amazon used to give me. My dream is to be able to tell YouTube that I have no interest in crypto, and then never see a crypto ad again. Here, advertisers, I'll give you a freebie: I'm in the market for some nice winter socks. Show me some ads for those sometime, and I might even buy yours. Or, if you keep pushing stuff in my face that I don't want, I'll go read a (paper) book instead. See what that does for engagement.

🖼️ Photos by Hyoshin Choi and Susan Q Yin on Unsplash

Whatever will they think of next?

Last week was AWS re:Invent, and I’m still dealing with the email hangover.¹ AWS always announce a thousand and one new offerings and services at their show, and this year was no exception. However, there is one announcement that I wanted to reflect upon briefly, out of however many there were during the week.

AWS Outposts are billed as letting users "Run AWS infrastructure on-premises for a truly consistent hybrid experience". This of course provoked a certain amount of hilarity in the parts of Twitter that have been earnestly debating the existence of hybrid cloud since the term was first coined.

It could just be a big empty box with an LED on it and an internet connection inside, that way you get all the services 🙂

— James Tomkins 💙 (@jamestomkins) November 29, 2018

On the surface, it might indeed seem somewhat strange for AWS, the archetypal public cloud in most people’s minds, to start offering hardware to be deployed on customers’ premises. However, to me it makes perfect sense.

Pace some ten-year-old marketing slogans which have not aged well, most companies do not start out with a hybrid cloud strategy. Instead, they find themselves forced by circumstances to formulate one in order to deal with all of the various departments that are out there doing their own thing. In this situation, the hybrid cloud strategy is simply recognition that different teams have different requirements and have made their own decisions based on those. All that corporate IT can do is try to gain overall visibility and attempt to ensure that all the various flavours of compute infrastructure are at least being used in ways which are sane, secure, and fiscally responsible (the order of the priorities may change, but that’s the list).

Some of the more wild-eyed predictions around hybrid cloud instead expected that workloads would be easily moved, not only between on- and off-premises compute infrastructure, but even between different cloud providers. In fact, it would be so easy that it would be possible to make minute-by-minute assessments of the cost of running workloads with different providers, and move them from one to another in order to take advantage of lower prices.

Obviously, that did not happen.

For the cloud broker model to work, several laws of both economics and physics would have to be suspended or circumvented, and nobody seems to have made the requisite breakthroughs.

To take just a few of the more obvious objections:

The Speed Of Light

Moving any meaningful amount of data around the public internet still takes time. If you are used to your local 100 Gb-E LAN, it can be easy to forget this, but it is going to be a factor out there in the wild wild Web. This objection was obvious when we were talking about moving monolithic VMs around, but even if you assume truly immutable infrastructure, you are still going to have to shift at least some snapshot of the application state, and that adds up fast – let alone the rate of configuration drift of your "immutable" infrastructure with each new micro-release.

Transparent Pricing

The units of measure of different cloud providers are not easily comparable. How does the performance of an AWS M5 instance compare to an Azure Dv2-series? Well, you’d better know before you move production over there… And AWS has 24 instance types, whereas Azure has 7 different series, each with sub-types and options – and let’s not even talk about all the weird and wonderful single-use configurations in your local VMware or Openstack service catalogue! How portable is your workload, really?

Leaving Money On The Table

Or let’s take it from the other side: assume you have carefully architected your thing to use only minimum-common-denominator components that are, if not identical, at least similar enough across all of the various substrates they might find themselves running on. By definition, this means that you are not taking full advantage of the more advanced capabilities of each of those platforms. This limitation is not only at the ingredient level; you also have to make worst-case assumptions about the sorts of network bandwidth and latency you might have access to, or the sort of regulatory and policy compliance environment that you might find yourself operating within.

For all of these reasons and more, the dream of real-time cloud pricing arbitrage died a quick death, regardless of whether individual companies might use different cloud providers in various parts of their business.

Amazon Outposts is not that. For a start, despite running physically on the customer’s premises, it is driven entirely from the (remote) AWS control plane. Instead, it has the potential to address concerns about physical location together with associated concerns about latency and legal jurisdiction. Being AWS (with some help from VMware) it avoids the concern about different units of measure. For now, it only goes part of the way to resolving the final question about minimum common denominator ingredients, since at launch it only supports EC2. Additional features are expected shortly, however, especially including various storage options.

So yes, hybrid cloud. Turns out, it’s not only still a thing, but you can even get it from AWS. Who’d have thunk it?

This week’s AI-powered scandal is the news that Amazon scrapped a "secret" AI recruiting tool that showed bias against women.

Amazon.com Inc’s machine-learning specialists uncovered a big problem: their new recruiting engine did not like women.

The AI, of course, has no opinion about one women one way or the other. Amazon HR's recruitment tool was not "biased" in the sense that a human recruiter might be; it was simply unearthing existing bias:

That is because Amazon’s computer models were trained to vet applicants by observing patterns in resumes submitted to the company over a 10-year period. Most came from men, a reflection of male dominance across the tech industry.

If you train your neural networks with biased data, you are just re-encoding and reinforcing the bias! This was not news even in 2015 when Amazon started this experiment. However, it illustrates a general problem with AI, namely its users’ naive tendency to take the algorithms’ output at face value. As with any automated system, human experts will be needed to make intelligent evaluations of the output of the system.

Train The Guardians

The need for secondary evaluation is only going to increase as these systems proliferate. For instance, Canada now plans to use AI to decide immigration cases. If your request for a visa is rejected by an expert system, what recourse do you have? What if the basis of rejection is simply that claims like yours were usually denied in the past?

These concerns will become more and more critical as AI tools continue to become more mainstream.

"The computer says no" has always been the jobsworth’s go-to excuse. But who programmed the computer to say "no"? The fact that the computers are now programming themselves to say "no" does not absolve organisations of responsibility. The neural networks are simply reflecting the inputs they are given. It is on us to give them good inputs.

After all, we are educating our own children.

There is a widespread assumption in tech circles that privacy and (useful) AI are mutually exclusive. Apple is assumed to be behind Amazon and Google in this race because of its choice to do most data processing locally on the phone, instead of uploading users’ private data in bulk to the cloud.

A recent example of this attitude comes courtesy of The Register:

Predicting an eventual upturn in the sagging smartphone market, [Gartner] research director Ranjit Atwal told The Reg that while artificial intelligence has proven key to making phones more useful by removing friction from transactions, AI required more permissive use of data to deliver. An example he cited was Uber "knowing" from your calendar that you needed a lift from the airport.

I really, really resent this assumption that connecting these services requires each and every one of them to have access to everything about me. I might not want information about my upcoming flight shared with Uber – where it can be accessed improperly, leading to someone knowing I am away from home and planning a burglary at my house. Instead, I want my phone to know that I have an upcoming flight, and offer to call me an Uber to the airport. At that point, of course I am sharing information with Uber, but I am also getting value out of it. Otherwise, the only one getting value is Uber. They get to see how many people in a particular geographical area received a suggestion to take an Uber and declined it, so they can then target those people with special offers or other marketing to persuade them to use Uber next time they have to get to the airport.

I might be happy sharing a monthly aggregate of my trips with the government – so many by car, so many on foot, or by bicycle, public transport, or ride sharing service – which they could use for better planning. I would absolutely not be okay with sharing details of every trip in real time, or giving every busybody the right to query my location in real time.

The fact that so much of the debate is taken up with unproductive discussions is what is preventing progress here. I have written about this concept of granular privacy controls before:

The government sets up an IDDB which has all of everyone's information in it; so far, so icky. But here's the thing: set it up so that individuals can grant access to specific data in that DB - such as the address. Instead of telling various credit card companies, utilities, magazine companies, Amazon, and everyone else my new address, I just update it in the IDDB, and bam, those companies' tokens automatically update too - assuming I don't revoke access in the mean time.

This could also be useful for all sorts of other things, like marital status, insurance, healthcare, and so on. Segregated, granular access to the information is the name of the game. Instead of letting government agencies and private companies read all the data, users each get access only to those data they need to do their jobs.

Unfortunately, we are stuck in an stale all-or-nothing discussion: either you surround yourself with always-on internet-connected microphones and cameras, or you might as well retreat to a shack in the woods. There is a middle ground, and I wish more people (besides Apple) recognised that.

Photo by Kyle Glenn on Unsplash

There has been this interesting shift going on in coverage of Silicon Valley companies, with increasing scepticism informing what had previously been reliable hero-worshipping. Case in point: this fascinating polemic by John Battelle about the oft-ignored human externalities of "disruption" (scare quotes definitely intended).

Battelle starts from a critique of Amazon Go, the new cashier-less stores Amazon is trialling. I think it’s safe to say that he’s not a fan:

My first take on Amazon Go is this: F*cking A, do we really want eggplants and cuts of meat reduced to parameterized choices spit onto algorithmized shelves? Ick. I like the human confidence I get when a butcher considers a particular rib eye, then explains the best way to cook that one cut of meat. Sure, technology could probably deliver me a defensibly "better" steak, perhaps even one tailored to my preferences as expressed through reams of data collected through means I’ll probably never understand.
But come on.
Sometimes you just want to look a guy in the eye and sense, at that moment, that THIS rib eye is perfect for ME, because I trust that butcher across the counter. We don’t need meat informed by data and butchered by bloodless algorithms. We want our steak with a side of humanity. We lose that, we lose our own narrative.

Battelle then goes on to extrapolate that "ick" out to a critique of the whole Silicon Valley model:

It’s this question that dogs me as I think about how Facebook comports itself : We know what’s best for you, better than you do in fact, so trust us, we’ll roll the code, you consume what we put in front of you.
But… all interactions of humanity should not be seen as a decision tree waiting to be modeled, as data sets that can be scanned for patterns to inform algorithms.

Cut Down The Decision Tree For Firewood

I do think there is some merit to this critique. Charlie Stross has previously characterised corporations as immortal hive organisms which pursue the three corporate objectives of growth, profitability, and pain avoidance:

We are now living in a global state that has been structured for the benefit of non-human entities with non-human goals. They have enormous media reach, which they use to distract attention from threats to their own survival. They also have an enormous ability to support litigation against public participation, except in the very limited circumstances where such action is forbidden. Individual atomized humans are thus either co-opted by these entities (you can live very nicely as a CEO or a politician, as long as you don't bite the feeding hand) or steamrollered if they try to resist.
In short, we are living in the aftermath of an alien invasion.

These alien beings do not quite understand our human reactions and relations, and they try pin them down and quantify them in their models. Searching for understanding through modelling is value-neutral in general, but problems start to appear when the model is taken as authoritative, with any real-life deviation from the model considered as an error to be rectified – by correcting the real-life discrepancy.

Fred Turner describes the echo chamber these corporations inhabit, and the circular reasoning it leads to, in this interview:

About ten years back, I spent a lot of time inside Google. What I saw there was an interesting loop. It started with, "Don't be evil." So then the question became, "Okay, what's good?" Well, information is good. Information empowers people. So providing information is good. Okay, great. Who provides information? Oh, right: Google provides information. So you end up in this loop where what's good for people is what's good for Google, and vice versa. And that is a challenging space to live in.

We all live in Google’s space, and it can indeed be challenging, especially if you disagree with Google about how information should be gathered and disseminated. We are all grist for its mighty Algorithm.

This presumption of infallibility on the part of the Algorithm, and of the world view that it implements is dangerous, as I have written before. Machines simply do not see the world as we do. Building our entire financial and governance systems around them risks some very unwelcome consequences.

But What About The Supermarket?

Back to Battelle for a moment, zooming back in on Amazon and its supermarket efforts:

But as they pursue the crack cocaine of capitalism — unmitigated growth — are technology platforms pushing into markets where perhaps they simply don’t belong? When a tech startup called Bodega launched with a business plan nearly identical to Amazon’s, it was laughed off the pages of TechCrunch. Why do we accept the same idea from Amazon? Because Amazon can actually pull it off?

The simple answer is that Bodega falls into the uncanny valley of AI assistance, trying to mimic a human interaction instead of embracing its new medium. A smart vending machine that learns what to stock? That has value - for the sorts of products that people like to buy from vending machines.

This is Amazon’s home turf, where the Everything Store got its start, shipping the ultimate undifferentiated good. A book is a book is a book; it doesn’t really get any less fresh, at least not once it has undergone its metamorphosis from newborn hardback to long-lived paperback.

In this context, nappies/diapers¹ or bottled water are a perfect fit, and something that Amazon Prime has already been selling for a long time, albeit at a larger remove. Witness those ridiculous Dash buttons, those single-purpose IoT devices that you can place around your home so that when you see you’re low on laundry powder or toilet paper you can press the button and the product will appear miraculously on your next Amazon order.

Steaks or fresh vegetables are a different story entirely. I have yet to see the combination of sensors and algorithms that can figure out that a) these avocados are close to over-ripe, but b) that’s okay because I need them for guacamole tonight, or c) these bananas are too green to eat any time soon, and d) that’s exactly what I need because they’re for the kids’ after-school snack all next week.

People Curate, Algorithms Deliver

Why get rid of the produce guy in the first place?

Why indeed? But why make me deal with a guy for my bottled water?²

I already do cashier-less shopping; I use a hand-held scanner, scan products as I go, and swipe my credit card (or these days, my phone) on my way out. The interaction with the cashier was not the valuable one. The valuable interaction was with the people behind the various counters - fish, meat, deli - who really were, and still are, giving me personalised service. If I want even more personalised service, I go to the actual greengrocer, where the staff all know me and my kids, and will actively recommend produce for us and our tastes.

All of that personalisation would be overkill, though, if all I needed were to stock up on kitchen rolls, bottled milk, and breakfast cereal. These are routine, undifferentiated transactions, and the more human effort we can remove from those, the better. Interactions with humans are costly activities, in time (that I spend dealing with a person instead of just taking a product off the shelf) and in money (someone has to pay that person’s salary, healthcare, taxes, and so on). They should be reserved for situations where there is a proportionate payoff: the assurance that my avos will be ripe, my cut of beef will be right for the dish I am making, and my kids’ bananas will not have gone off by the time they are ready to eat them.

We are cyborgs, every day a little bit more: humans augmented by machine intelligence, with new abilities that we are only just learning to deal with. The idea of a cashier-less supermarket does not worry me that much. In fact, I suspect that if anything, by taking the friction out of shopping for undifferentiated goods, we will actually create more demand for, and appreciation of, the sort of "curated" (sorry) experience that only human experts can provide.

Photos by Julian Hanslmaier and Anurag Arora on Unsplash

The Internet is outraged by… well, a whole lot of things, as usual, but in particular by Apple. For once, however, the issue is not phones that are both unexciting and unavailable, lacking innovation and wilfully discarding convention, and also both over- and under-priced. No, this time the issue is apps, and in particular VPN apps.

Authoritarian regimes around the world (Russia, "Saudi" Arabia, China, North Korea, etc) have long sought to control their populations' access to information in general, and to the Internet in particular. Of course anyone with a modicum of technical savvy - or a friend, relative, or passing acquaintance willing to do the simple setup - can keep unfettered access to the Internet by going through a Virtual Private Network, or VPN.

A VPN does what it says on the tin: it creates a virtual network that connects directly with an endpoint somewhere else; importantly, somewhere outside the authoritarian regime's control. As such, VPNs have always existed in something of a grey area, but now China (the People's Republic, not that other China) has gone ahead and formally banned their use.

In turn, Apple have responded by removing unregistered VPN apps (which in practical terms means all of them) from their App Store in China. In the face of the Internet's predictable outrage, Apple provided this bald statement (via TechChrunch):

Earlier this year China’s MIIT announced that all developers offering VPNs must obtain a license from the government. We have been required to remove some VPN apps in China that do not meet the new regulations. These apps remain available in all other markets where they do business.

Now Apple do have a point; the law is indeed the law, and because they operate in China, they need to enforce it, just as they would with laws in any other country.

Here's the rub, though. By the regionalised way they have set up their App Store service, they have made themselves unnecessarily vulnerable to this sort of arm-twisting by unfriendly governments. Last time I wrote about geo-fencing and its consequences, the cause of the day was Russia demanding removal of the LinkedIn app, and China (them again!) demanding removal of the New York Times app. As I wrote at the time, companies like Apple originally set up the infrastructure for these geographic restrictions to enable IP protection, but the same tools are being repurposed for censorship:

This sort of restriction used to be "just" hostile to consumers. Now, it is turning into a weapon that authoritarian regimes can wield against Apple, Google, and whoever else. Nobody would allow Russia to ban LinkedIn around the world, or China to remove the New York Times app everywhere - but because dedicated App Stores exist for .ru and .cn, they are able to demand these bans as local exceptions, and even defend them as respecting local laws and sensibilities. If there were one worldwide App Store, this gambit would not work.

The argument against the infrastructure of laws and regulations that was put in place to enable (ineffective) IP restrictions was always that it could be, and would be, repurposed to enable repression by authoritarian regimes. People scoffed at these privacy concerns, saying "if you have nothing to hide, you have nothing to fear". But what if your government is the next to decide that reading the NYT or having a LinkedIn profile is against the law? How scared should you be then?

If you are designing a social network or other system with the expectation of widespread adoption, these days this has to be part of your threat model. Otherwise, one day the government may come knocking, demanding your user database for any reason or no reason at all - and what seemed like a good idea at the time will end up messing up a lot of people's lives.

Product designers by and large do not think of such things, as we saw when Amazon decided that it would be perfectly reasonable to give everyone in your address book access to your Alexa device - and make it so users could not turn off this feature without a telephone call to Amazon support.

How well do you think that would go down if you were a dissident, or just in the social circle of one?

Our instinctive attitude to data is to hoard them, but this instinct is obsolete, forged in a time when data were hard to gather, store, and access. It took something on the scale of the Stasi to build and maintain profiles on even six million citizens (out of a population of sixteen million), and the effort and expense was part of what broke the East German regime in the end. These days, it's trivial to build and access such a profile for pretty much anyone, so we need to change our thinking about data - how we gather them, and how we treat them once we have them.

Personal data are more akin to toxic waste, generated as a byproduct of valuable activity and needing to be stored with extreme care because of the dire consequences of any leaks. Luckily, data are different from toxic waste in one key respect: they can be deleted, or better, never gathered in the first place. The same goes for many other choices, such as restricting users to one particular geographical App Store, or making it easy to share your entire contact list (including by mistake), but very difficult to take that decision back.

What other design decisions are being made today based on obsolete assumptions that will come back to bite users in the future?

UPDATE: And there we go, now Russia is following China’s example and banning VPNs as well. The idea of a technical fix to social and legal problems is always a short-term illusion.

Image by Sean DuBois via Unsplash

In light of Amazon's latest moves - first the acquisition of Whole Foods, and now a cooperation with Sears - the stock market is trying to work out what will be the next area of retail to be hit.

Here's a sample of the latest Finimize newsletter:

Investors should be asking, who’s next in Amazon’s line of fire?
One of the biggest threats is likely to come from Amazon making more of its own-label products. For example, why would people pay a premium for Colgate toothpaste or Dove soap if Amazon basics are cheaper and arrive at our front door on the same day? Procter & Gamble, Unilever and other consumer goods giants could experience serious upheaval to their business models. Similarly, apparel companies are threatened: Amazon’s own-label clothing sales grew 25% last year, far outstripping the industry’s 3% growth. Do you care if your downward dog is performed in Amazon yoga pants rather than Lululemon? Perhaps not.

This analysis is not wrong, but is incomplete. Amazon will indeed take over a large chunk of the bottom end of the market, which is almost undifferentiated. The top end will be largely unaffected, as it is driven by completely different mechanisms.

The vulnerable actors are the ones in the middle of the market, who offered neither exceptionally high quality, nor particularly low prices. Historically, these brands succeeded by controlling distribution, especially outside major population centres.

Once an alternative becomes available that offers lower prices for equivalent quality (or the perception thereof), those middle-of-the-road brands and distribution outlets get squeezed hard.

I have noticed this over the last few years on the ski slopes, where a number of relatively undifferentiated brands have disappeared under a rising tide of previously unkown logos. All of these - Wed'ze, Quechua, and the like - are the house brands of Decathlon, the French sports superstore.

This is the Amazon Basics formula: good quality, and very reasonable prices. What Decathlon adds to the mix is that the designs are usually attractive, if somewhat non-descript, and that the clothes are available to try on in their vast network of retail outlets.

That last factor is key: to conserve differentiation and therefore ability to compete against a determined behemoth like Amazon, offer something they don't. Even buying something as simple as a T-shirt online is a fraught experience, with the differences between European and US sizing. I can't imagine shopping online for anything more complicated, like a shirt or a suit. On the other hand, I buy stuff from Decathlon every season because I can check it out in store, try it on, feel the material.

The same goes for Zara, part of the giant Inditex group, which benefits from a joined-up online/offline strategy. Customers can try on clothes in store, but if the precise colour and size combination is not available, they can order it online through Zara's own retail site.

Another factor that might cause shoppers to hesitate before making an online purchase is returns. The cost of shipping can outweigh the cost of the item, and in any case requires shoppers to deal with packaging and labelling, before taking the item to an out-of-town shipper. Zara on the other hand allows online shoppers to manage exchanges or returns in-store, thereby further monetising their investment in real estate even for online-first shoppers.

Of course not every operator has to take this route - which is good, because by definition there is not much room at this end of the market, where margins are razor-thin.

Sticking with winter sportswear, for anything I care deeply about, I shop from named brands that I have built up trust with - the likes of Level or Hestra for gloves, Under Armor for mid-layers, Burton or Arcteryx for outerwear, and so on. These are fairly big-ticket items, certainly in proportion to the base layers I pick up by the three-pack at Decathlon, so I shop around and get all picky about them.

Part of what I buy into is of course the experience. Going into a North Face store, you feel like you are participating in strenuous and exciting activities, even if all you are doing is bumbling about in-bounds trying to get your kids to graduate from their preferred "Snowplow Everywhere!!!" technique. Neither Amazon nor Decathlon can offer this, but then again, they have staked out the price-sensitive end of the market.

There is not much room in the middle any more, especially when you consider that consumers are quite happy to cross back and forth, wearing base layers from no-name brands under much more expensive outer layers. On the other hand, it's not a foregone conclusion that Amazon will own everything below the top of the market. Price-focused outlets still have a role to play, if they capitalise on their strengths.

As with most aspects of the digital revolution, it's the middlemen that are in trouble. The trick is to have a unique value proposition and stick to it. Operators whose only proposition was ubiquity and convenience cannot match the actual ubiquity and convenience of web-scale operators.

Coming full circle, groceries will experience a similar transition over time. Amazon is good at undifferentiated goods - which is why they started out with books: a book is the same regardless of when and where it is bought. I fully expect them to take over a big chunk of the market for packaged goods. On the other hand, there will always be a need to pick out fruit and groceries in person, to feel whether this avocado is ripe to make guacamole tonight or whether that bunch of bananas is green enough to last out the week. By combining online scale with Whole Foods' local presence, Amazon is going for the Zara play: check out a few products locally, buy many products online, and have both parts of the experience supported by the same seamless massive back-end.

Anyone wanting to compete with Amazon should choose their terrain very carefully indeed. After all, your margin is their opportunity.

Why does Apple hate wish lists so much?

The wish list is the main thing I miss since I fell out with Amazon and moved all of my media buying over to iTunes. Amazon not only has great management of its wish list, allowing you to sort it any way you like and highlighting deals, or sharing it with friends and family as suggestions; it also uses the contents of your wish list as inputs to its recommendation engine.

Over the decade or so that I used Amazon regularly, its recommendations grew to be uncannily accurate, alerting me to new books or albums that I might be interested in. The algorithm involved was clever enough to recommend not only new works by artists I had already bought from in the past, but also works by other artists I had not previously encountered. This was driven by their ability to identify that "other people who bought X also bought Y", based on their insight into all of our purchasing histories.

Of course this is a critical feature for Amazon, which explains why they spend so much time and effort on refining it. In fact, it was only when they messed with my wish list that I left in a huff.

I had continued to buy from Amazon’s UK site after leaving the UK, because with free shipping within the EU, it made no difference, while it allowed me to keep that all-important wish list history. A few years later, however, Amazon in their wisdom decided that many items would no longer be made available to ship outside the UK. Instead of simply tagging the items with a notice, they simply removed the items from users’ stored wish lists. In my case, this meant I lost nearly half of my wish list items.

I use wish lists as a way to spread out purchases or remind me of items that are due to come out in the future but that I am not committed enough to pre-order right away (or which may not yet be available to pre-order). Deleting half of my wish list in this high-handed way was enough for me to quit a triple-figure-per-month Amazon habit cold-turkey.

This coincided with the move to a new house, where even our existing media collections were overflowing the shelves once we had finished unpacking. The time was therefore ripe for a move to electronic content only, and given that I was cross with Amazon, Apple was the only real alternative.

It’s been a couple of years now, and I have not regretted it in any way. I adapted very quickly to reading on the iPad, and music and the occasional film are of course super-easy. There is only one glaring problem, and that is the utterly inconsistent handling of wish lists on the part of the Apple store apps.

iBooks app on iPhone - note lack of wish list button

The fact that it’s plural "apps" is a bit of a problem in its own right, actually. I have a Music app to listen to music, that I buy in the iTunes Store app. That is where I also buy videos, that I then watch in the Videos app. But if I want to buy books, I have to do that in a special tab of the iBooks app.

Historically this makes sense - iBooks came along much later than the rest of iTunes. But why the weird inconsistencies in when I can add something to my iTunes/iBooks wish lists? iBooks on iOS won’t allow this, but iBooks on the Mac will. On the other hand, iTunes on the Mac won’t let me add an album to my wish list, but the iTunes Store app on iOS will.

Same screen in iTunes Store app on iPhone - note "Add to Wish List" button

This is why I have a file in Notes with iTunes Store links to items that I wanted to add to my wish list, but couldn’t because I didn’t have access to the specific device that would let me do that at the time.

Workaround

This is admittedly a pretty minor niggle in the grand scheme of things, but I think it’s philosophically important for Apple to fix this inconsistency. It lies right at the heart of the iTunes ecosystem, and creates an unexpected and annoying discrepancy between MacOS and iOS platforms, and even between different devices on iOS.