What we wish the Internet were

Doc Searls and David Weinberger have apparently written a Cluetrainish hallucination about the Internet called the World of Ends. I haven’t read it [ed: now I have, see updates below] because their server’s down, so I’ll critique it without the unnecessary distraction of actually knowing what they have to say.

They’ve got it all wrong because they confuse what the net is – an extremely complex network of computers, routers, and links that barely works most of the time and falls far short of what it needs to be – with what they wish it were – a magic wand for bringing about a Utopian paradise in which everything is free, toil and trouble are abolished, men are all smart and women are all good-looking. We see this kind of rubbish over and over, etc, etc, etc.

Seriously though, it’s No. 1 on Blogdex so it’s worth a read when the net can serve it up, most likely. And I hope to be able to read it someday.

UPDATE: Their server stayed up long enough for me to read it, and it’s pretty much as I expected, a rehash of Weinberger’s previous claims that we’ve already dealt with.

The reactions to the article are interesting, and fall into two categories: most are laudatory and quite brief, statements like “brilliant explanation of how the Internet works”, although some quote the entire page. The others, written by people who really do understand how the Internet works, tend to go point-by-point showing how the authors have got it dead wrong. See Soundbitten, or Brett Glass on Seppuku, or Empty Bottle or Way.Nu or Marc Canter or Russell Beattie (my favorite) for a proper analysis.

Bottom line: Searls and Weinberger are hippies, and they see a Free and Open Internet as a key building block of a free and open world, in much the way that Timothy Leary saw LSD in a previous generation. They’re concerned that commercial interests will spoil their metaphor through misunderstanding, and somehow pervert it into a tool of control, oppression, and Big Brother. But they fall into the trap that they warn about, by romanticizing the ‘Net and making it more metaphor than reality.

Probably their most egregious error is the failure to understand that the Internet itself – the plumbing – is, like most technical things, dynamic. When it was deployed in the early 80s (as an improvement on ARPANet), it was intended to support three rudimentary applications (ftp, telnet, and e-mail), none of which was real-time, and its own maintenance needs, which in the early days were simply routing table updates. As the applications of that day were simple, the plumbing to carry them was also simple, and as the community of hosts was small and trusted, there was no need for security and no worries about viruses and attacks. So it was as simple as it needed to be, and no simpler.

Things have changed. DNS has been created (in the old days name-to-address mapping was accomplished by local files), the routing table update protocols have become more sophisticated, the application set has grown to encompass media, real-time, commerce, and web surfing, and we’ve had to cobble together a host of retrofits to deal with Denial of Service attacks, spam, and viruses. Needless to say, the basic transfer engine – the IP protocol – has gone through several enhancements, and in its latest form (IPv6) is in fact quite complicated. The trend over time is clearly toward greater complexity, but still no more complexity than the applications themselves require. One could say the same of cars, telephones, toasters, or any other technical thing.

One specific area where the boys are soaking wet is the matter of traffic priorities. From the beginning, the Internet has recognized that some streams are more time-sensitive than others, and from the beginning it’s had mechanisms to assign higher priority to them. In the early days, this was the “Urgent Data” flag in the IP header, and in IPv6 it’s four levels of priority. Higher priority streams are network maintenance streams updating routing tables and applications like digitized voice (VoIP or Real Audio), while lower-priority streams are things like e-mail and Usenet. Priority isn’t a judgment about value or importance, its a fact of life in the application domain. And while it’s true that giving priority to voice makes e-mail move slower, it’s nothing to get upset about. In the fullness of time, users of high-priority streams will pay for the load they put on the net, and others will get cheaper service as a side-effect, essentially drawing a subsidy. Think Priority Mail vs. bulk mail: the Postal Service collects money from Priority Mail users, which it uses to enhance its infrastructure; this enables all forms to mail to actually move faster.

As to the three crazy rules (nobody owns it, everybody can use it, anybody can improve it) there’s one more they’ve left out: everybody who uses it has to pay for it. Nothing’s free, guys, and when you crank economics into your metaphor, the whole thing changes.

Let’s get a clue here, OK? The Internet will continue to evolve, and it must evolve in order to support video streaming and the Semantic Web. That doesn’t mean there’s going to be an All Controlling Intelligence at the center of it, but it does mean that the pipes are going to get smoother and faster. Today, you push bits into the Internet and they come out the other end bearing little resemblence in terms of sequence to the way you put them in. This kind of behavior would be completely unacceptable for the phone nework, so it’s got some relatively simple mechanisms to ensure that delivery sequence matches up with transmit sequence. That’s all we need from the ‘Net to support video and voice – for it to act as if every endpoint-to-endpoint connection is a wire. It’s not so much a matter of network intelligence as network transparency. The ‘Net, in other words, needs to stop reminding us that it’s there and just carry the data faithfully. Is that too much to ask? I didn’t think so.

So let’s discard the hippie dogma and treat the ‘Net for what it is: a technical creation supporting a certain range of applications, not a politico-religious symbol that wants to keep us locked into Woodstock.

Another update: Weinberger links some of the pushback here. While most people are trying real hard to be polite, there’s no ignoring the fact that the World of Ends is based on outdated dogma.

ANOTHER UPDATE: See further commentary here.