Perhaps you’ve heard that Cox Cable is blocking Craig’s List; the Net Neutrality militias tout this as an example of the kind of discriminatory behavior they’re going to protect us from. Their leader, the self-described know-nothing Matt Stoller says:
There’s a pervasive myth that there has been no discrimination on the internet against content companies. That is simply untrue. For one, Craigslist has been blocked for three months from Cox customers because of security software malfunctions.
Back on February 23rd Authentium acknowledged that their software is blocking Craigslist but it still hasn’t fixed the problem, more than three months later. That’s a heck of long time to delete some text from their blacklist. And this company also supplies security software to other large ISPs.
Without net neutrality protections, cable and telecom companies will have no incentive to fix these kinds of problems. Already, it’s quite difficult to even know that this is happening because they are quite easy to disguise.
The telcos are of course lying about this, claiming that no web sites have been blocked. And gullible reporters are falling for the lies.
But the real story is that Craig Newmark’s administrators don’t know how to set up their system. Here’s a comment I found on Save the Internet that will probably be deleted pretty soon:
Has anyone here actually read the response from Authentium? Far from “opaque,” it pretty clearly (if technically) explains the problem and why this has nothing to do with blacklists:
“The network packets coming from the Craigslist.org web site were unusual in that they contained a zero-length TCP window that usually indicates a server is too busy to handle more data. The Authentium firewall driver responded by sending data only one byte at a time. This slowed down the web request and made the Craigslist.org web page load very slowly or not at all.”
From RFC 793 (which defines TCP/IP):
” Flow Control:
TCP provides a means for the receiver to govern the amount of data sent by the sender. This is achieved by returning a “window” with every ACK indicating a range of acceptable sequence numbers beyond the last segment successfully received. The window indicates an allowed number of octets that the sender may transmit before receiving further permission.”
Returning a 0 means “please talk to me very slowly.” Literally it means “don’t talk to me at all” but because that’s nonsense, sites generally interpret it as “I’m overloaded; slow down.”
I’ve verified this response myself by connecting to craigslist:
15:52:00.751836 IP www.craigslist.org.http > lemming.ranjan.org.47734: S 1639327951:1639327951(0) ack 3799817961 win 0
Note the final “win 0? that confirms exactly the problem that Authentium claims.
Summary: craigslist told Cox to please speak to it very slowly. Cox did, but for longer than craigslist explicitly requested. Fixing this for craigslist could break other sites, so some caution in shipping a fix is justified.
The fact that SaveTheInternet posted this as an “opaque” response without further comment raises a question of how much STI actually knows about how the Internet works.
Somebody’s lying here, and it’s not Cox Cable or Authentium.
PS: I did my own inspection of Craig’s List’s TCP packets and found the same thing: their initial ACK advertises a Window Size of 0. By comparison, my blog advertises one of 5792, and so does Technorati.
Craig Newmark’s site is screwed up and he’s blaming Cox for it – and seeking a new law. That’s taking Internet retardation to a whole new level.
UPDATE: See Jim Lippard’s blog for a fuller explanation.
UPDATE 2: George Ou at ZDNet is on the case. This story originated with Tom Foremski at ZDNet, and getting him to issue a correction is very important.
UPDATE 3: It’s worth noting that Matt Stoller blogs on myDD.com, half of the Kosola pay-to-blog scandal. Read more about that here or here. Some people will say anything for money. Net neutrality advocate Glenn Reynolds says blogs are a “low trust environment.” He doesn’t speak for this one.
UPDATE 4: Welcome Instapundit readers. Tom Foremski and Save the Internet refuse to own up to misrepresentation of the story.
Here are the facts:
1. Craig’s List isn’t blacklisted by Cox Cable and never has been.
2. Craig’s List puts out an improper TCP window size; other sites don’t.
3. Improper TCP causes some personal firewalls grief, and Cox used to distribute one, from Authentium.
4. As soon as the Craig’s List bug came to Authentium’s attention they created a patch, which you can get from Cox today. This patch probably ignores the initial window size Craig requests.
5. Craig’s List still puts out an improper TCP window size.
So how about a little honesty, Craig, Matt, Tim, and Tom?
UPDATE 5: Craig Newmark still refuses to acknowledge his bug. All he has to do is correct his TCP settings and the whole problem goes away. Why won’t he?
UPDATE 6: Go look at the system status page at Craig’s List and you’ll see some interesting problems with all sorts of other firewalls, including their own. And you’ll also see that their problem with the personal firewall Cox Cable gives away has had a known workabound since Feb. 23th. Why all the misdirection from Craig, Save the Internet, and Matt Stoller?
Incidentally, eBay is a minority shareholder in Craig’s List, and the sole owner of Skype. Is Craig doing his master’s bidding?
UPDATE 7: Authentium responds to Craig’s lying post. Their story is verifiable, Craig’s is fabricated.
I posted this on the Save the Internet blog:
I don’t see a way to characterize this one as simple ignorance. Matt Stoller falsely repeated the claim that Authentium had craigslist on a “blacklist” five days after Authentium posted the explanation as a comment on his own blog. Timothy Karr has pushed this on this blog, the Free Press Action HQ blog, and his own Media Citizen blog, also with the “blacklist” wording and also well after having the correct explanation in hand.
This is dishonest, plain and simple. If Karr and Stoller have any integrity, they will retract and apologize for lying.
From the summary:
I’d say both sites are screwed up, and “what we have here is a failure to communicate,” and real live human administrators at both ends of the connection need to look at their configuration and work together.
Let’s look at this in context. Its not the “superhighway” that is at issue here, its the “offramps,” at which a free marketplace doesn’t or barely exists. (“…only 2 percent of Americans get high-speed Internet access from someone other than their local phone company or cable provider–Findings section of S2360).
These offramps are controlled (at the telcos) by organizations who have decimated their staffs, often retiring or firing their most comptent and experienced employees because this represents the highest cost savings.
So no matter whom is at fault here, it is important that there be some way to compel these monopolies to respond when their systems are broken or even appear to be. Now that net neutrality agreements are expiring at the new mega-telcos and forced competition has been discarded by the FCC, the easy answer for the understaffed telcos who block access to sites (even inadvertantly), is: “Maybe we are, so what?”
The new megatelcos were quite happy with net neutrality when the carrot they were offered in exchange for it was the ability to eat other telcos. There’s no reason they can’t swallow it now through direct legislation, even though they’re not being given a prize for it.
A few curious twists here:
One, I followed up with Tom Foremski, who was the prime amplifier of the story up to Stoller. I felt that as a blogger riding on his decades of his journalistic experience, he ought to get around to doing a follow-up. He told me a week ago that he’d like to, but was focusing on the Scoble announcement. So I would press Foremski for a follow-up.
Two, MyDD is running BlogAds for “What is the Future of the Internet” — aka the non-network-neutrality camp. I wonder if anyone’s noticed.
In the first place, the reclassification of DSL as an Information Service doesn’t affect Cox, they’re a cable provider, not a telco or a mega-telco.
In the second place, the problem is strictly at the Craig’s List side and Craig’s List has the ability to fix it anytime they want with no help from anybody.
Forcing the “monopolies” to respond in one way or another to Craig’s incompetence isn’t going to happen, no matter how many “neutrality” laws we pass.
So, Richard, let me make sure I understand this. You say:
Alexa today rates Craig’s List #27 of around 96,000,000 urls. The reason Craig’s list is so highly rated is that most of the Internet is able to contact his list, despite his misconfiguration. Yet somehow that misconfiguration creates a three-month hiatus at a cable company that just happens to have a competing service.
The way a free market would solve this, is that Cox subscribers would have a littany of alternatives to choose from so there would be real business pressure on it to work with Craig to solve this. By claiming it is Craig’s problem alone, you’re contravening the very “authoritative” summary you posted, and raising the thorny question of why enough other subnets have so little problem taking Craig’s packets that he’s #27, yet cox.net is #1,182 and coc.com is #603.
I’ll take it as agreement with me when you say the monopolies aren’t going to respond to this issue no matter how many neutrality laws we pass. They’ll find some way to circumvent the issue (how many years was AT&T able to perpetuate the myth that hooking an additional telephone to your home service would cause the immediate collapse of the public switched network)?
Therefore, net neutrality is only a first step until we can get some real competition at the last mile level.
PBCLiberal:
“The way a free market would solve this, is that Cox subscribers would have a littany of alternatives to choose from so there would be real business pressure on it to work with Craig to solve this. By claiming it is Craig’s problem alone, you’re contravening the very “authoritative” summary you posted, and raising the thorny question of why enough other subnets have so little problem taking Craig’s packets that he’s #27, yet cox.net is #1,182 and coc.com is #603.”
1. There already are solutions for Cox subscribers: (a) deinstall the software firewall (and as a security professional I’d recommend turning on an alternative rather than going without one) or (b) install the free beta that addresses the problem. The first solution has always been available and the second has been available since a couple of weeks after the problem was first reported. I’m a Cox customer, but never experienced this issue because I’ve never used the Cox-provided Authentium software.
2. There already are alternative providers for many, if not most Cox subscribers. You’re right that last-mile competition is a concern, but it’s not as big a concern in many metropolitan areas as some seem to think.
3. Comparing the popularity of the craigslist.org website to the popularity of the cox.net website isn’t a good measure of the number of respective users–there are a lot more Cox eyeball customers using the Internet than there are users of Cox’s web site.
Jon Garfunkel: “I followed up with Tom Foremski, who was the prime amplifier of the story up to Stoller. I felt that as a blogger riding on his decades of his journalistic experience, he ought to get around to doing a follow-up. He told me a week ago that he’d like to, but was focusing on the Scoble announcement. So I would press Foremski for a follow-up.”
Given that the complaint against Cox/Authentium is for taking months to fix the problem, it seems like there’s a valid complaint against Foremski for taking months to correct the record. It’s easier to post an update to a blog than it is to fix, test, and release a piece of widely-used consumer software for production.
I wrote “it seems like there’s a valid complaint against Foremski for taking months to correct the record.” Sorry, had the February date of the first report of the problem in mind. Foremski first posted about this on June 6, 12 days ago, and followed up on June 8 with some reader comments that included the correct explanation amongst them.
So there’s no valid complaint against Foremski for “taking months to correct the record.” Let my record stand corrected…
Absolutely. This speaks to the level of customer service, which comes back to why we need free market pressures to movtiate these megaproviders to act responsibly. Since Cox and BellSouth’s coverage area coincide, I know that a lot of the “alternatives” to BellSouth service are nothing but BellSouth dressed us another way. For instance, I am on a network that is not BellSouth, where BellSouth leases my provider far more than just access to the phone lines. I suspect this will markedly change now that they need not even project the pretense of providing competition. Of course, there’s satellite, and even (now don’t go a callin’ me socialist) the spectre of municipal wifi.
As a matter of good customer service,Cox should be highly motivated to solve problems that affect popular websites. The contrast in the popularity of the website was included to show how popular Craigslist is compared with Cox, which offers a competing service. This is the classic case of content providers vs. content providers who also own the edge routers.
As a security professional, I assume you’re used to the blank look on the faces of users to whom you say the word “Firewall,” and the head shaking that usually comes tossing out: “Network Address Translation.” The only people who should ever disable their firewalls are those experienced enough to raise their systems from the ashes if they are successfully attacked. I think most of today’s internet users aren’t in that category.
Nonetheless, the bug is Craig’s, and he could have fixed it months ago. But instead of doing that, he helped organize a poltical movement to shackle ISPs. He’s running around the world whining about “level playing fields” and still hasn’t fixed his bug. He’s a regular Mary Mapes.
Who’s responsible for Craig Newmark’s tech support?
“As a matter of good customer service,Cox should be highly motivated to solve problems that affect popular websites….”
Since when does Craig’s list get to offload it’s technical support duties for it’s users onto third party telcos or ISPs?
One of the claims being made by Lippard in this thread, is that one of the solutions offered to Cox users was to
So I’d say they’ve done that already. The duopoly boosters are talking out of both sides of their mouthes. They are simultaneously arguing that the blame lays squarely with Craigslist and that Cox is fixing it.
Actually, it was probably the users who wanted to get to Craiglist who offloaded their concerns onto their telcos and ISPs regarding a third-party site they believed they were paying to access.
Be careful with this line of argument, because it undercuts the “power of the marketplace” line being peddled as a reason we net neutrality supporters are trying to fix a problem that doesn’t exist.
If the #27 website can’t get Cox to act when its undoubtely being on its best behavior even though it is not compelled, one can only imagine the range of possibilities when the restrictions expire everywhere.
install the free beta that addresses the problem.
So I’d say they’ve done that already. The duopoly boosters are talking out of both sides of their mouthes. They are simultaneously arguing that the blame lays squarely with Craigslist and that Cox is fixing it.
Not exactly. Offering customers advice on installing a beta doesn’t mean that the beta is fully supported by Cox, what Cox is offering is a “hack” or “work around” to a problem outside of their control. It’s a common problem, but I don’t go screaming to my congressman about it.
From a technical perspective, the blame and *correct* solution to the *problem* (Technically defined as Hosts violating RFCs) is to have Craigslist fix their content director/loadbalancer or server to be less RFC ignorant.
If the #27 website can’t get Cox to act when its undoubtely being on its best behavior even though it is not compelled, one can only imagine the range of possibilities when the restrictions expire everywhere.
BTW: the popularity of a website is no excuse for having it violate basic protocol behaviour… that arguement sounds suspiciously Microsoft-esque.
There is no question that everyone should follow the RFCs as closely as possible. Have you read RFC793? Authentium has, and their response was:
I’ve read RFC793, and I’m not sure Craigslist is technically outside the specification, its just bad practice, unless its being used as a “cue” to send a 1-octet packet, which is what Authentium assumes it is. So, strictly speaking, Authentium is broken in that it responds with a single octet when none was allowed.
From what I’ve read, Authentium’s Achillies Heel is that it doesn’t respond correctly on subsequent packets when it is asked increase the window size. How this has been characterized as Craigslist’s sole problem speaks more to the desire to deflect this issue than to comment on good protocol practice.
A lot of these arguments are taking the form that Craigslist is a fault because “they were the last one who could avoid the accident.”
How can Cox “fix” the problem, anyway? Cox can’t force its customers to upgrade their free firewall. Cox was already offering the upgraded beta version no?
And why, PBC, do you keep pushing at Cox, when the easy solution to the entire problem is for the Craigslist people to fix their broken window size?
You argument seems to take the form that Cox is at fault because they’re not Craigslist. (And what “competing service” do they have? I mean, seriously? Nobody, figuratively speaking, has ever even heard of this service, which means it’s not competing at all, as the only strength of something like Craigslist is that many people use it.
Hell, their website doesn’t even mention such a service. The idea that they’re somehow deliberately preventing a fix from going out to bolster their own competitor to Craigslist is… I don’t even have words for what that is.)
Like a lot of networking problems that I see in my day job, this one involves the interaction of two bugs, one on Craig’s List and the other in the Authentium firewall. Authentium didn’t test their software with all possible buggy servers, and Craig’s List hadn’t encountered a firewall that didn’t update the window size the second time it was sent. It’s probably a “multiply by zero” issue.
So the bottom line is this: Authentium has fixed its bug, but Craig hasn’t fixed his. Craig is still complaining about “discrimination” and Authentium is being gracious and taking full responsibility for the whole issue, even Craig’s part.
And meanwhile, the Save The Internet/Kosola Krowd are still saying this is proof that we need harsh regulations against ISPs.
Cox Cable delivers Craig’s List’s packets to its customers computers just fine, so the problem can’t remotely be attributed to malice on the part of Cox.
Craig Newmark has no credibility.
I’ve read RFC793, and I’m not sure Craigslist is technically outside the specification, its just bad practice, unless its being used as a “cue” to send a 1-octet packet, which is what Authentium assumes it is.
A zero window simply means that the client will need to recieve an ACK from $server after every attempt to communicate with $server. The result is an incredibly slow TCP conversation that requires additional overhead (chatty) communication that may exceed timeout thresholds of higher level protocols (E.g. HTTP) There is no rule against the client continuing the conversation with a server advertising a 0 Window. It just needs more ACKnowledgements from the server before requesting additional data.
From what I’ve read, Authentium’s Achillies Heel is that it doesn’t respond correctly on subsequent packets when it is asked increase the window size.
This is only a problem when dealing with systems that don’t properly negotiate window sizes in the first place, and it’s not just an Authentium problem.. Many stateful firewalls (especially hostbased ones) only use the window sizes negotiated during the 3 way handshake, especially for stateless protocols like HTTP.
Craiglist’s webserver appears to only boost the window after the 3 way handshake occurs. I’ve just confirmed that myself via tcpdump.
A lot of these arguments are taking the form that Craigslist is a fault because “they were the last one who could avoid the accident.”
I disagree. Craigslist is the only people in charge of what Windows their hardware/servers are advertising, and since running a 24/7 web infrastructure requires technically more clue than installing and operating a host based firewall, it seems that Craig’s list could easily resolve all of the various problems they have with *MANY* firewalls (See their system status page for details) by fixing things on their end.
Richard said:
UPDATE 5: Craig Newmark still refuses to acknowledge his bug. All he has to do is correct his TCP settings and the whole problem goes away. Why won’t he?
I’m not sure… but reading some of the issues Craigslist is having related to it’s own firewall (see their system status page) I think they may be waiting on a vendor fix as well 😉