New York Wants Congestion Pricing

If the New York City Council can approve this plan, why can’t we have the same thing on our broadband networks?

Updated, 9:24 p.m. | Shortly before 7:30 p.m., the New York City Council approved a measure urging state lawmakers to vote in favor of Mayor Michael R. Bloomberg’s congestion pricing proposal. The vote was 30 to 20, with one member absent. (See the full vote tally below.) Mayor Bloomberg immediately scheduled a news conference for later this evening with the Council speaker, Christine C. Quinn, to celebrate the passing of his plan, which has left city and state lawmakers sharply divided.

It seems like common sense, more or less, that drivers would have to pay for the privilege of clogging up the downtown areas. Singapore was doing this in the 1980s when I lived there. And yes, I know that the little island nation isn’t a model of free speech or human rights, but it’s in nobodys interest to have so many cars downtown that none of them actually moves.

Technorati Tags:

Comcast’s and BitTorrent’s Rodney King Moment

Check my latest piece in The Register on the Comcast/BitTorrent detente.

Analysis So Comcast will stop shaping peer-to-peer seeding sessions with spoofed TCP RST commands. I caught up with the cable giant’s CTO Tony Werner on Thursday for more details.

The move should delight the company’s critics. These innocent control packets have been compared to identity theft, to collusion with dictatorial regimes, and outright hacking of customer systems. BitTorrent may not be an IETF-sanctioned RFC protocol in its own right, but Comcast has decided to bind itself to traffic management practices that don’t produce obviously non-standard packets. Instead, they’re going to install additional equipment that will do real-time traffic accounting by user, feeding back information to the cable infrastructure that will equitably distribute opportunities to offer upstream traffic. In essence, this system makes the cable standard DOCSIS much more sophisticated; now it will behave just like DSL, only faster.

In DSL systems, each customer typically has a dedicated physical cable to a DSLAM, a type of concentrator. The DSLAM aggregates customer data onto a common upstream link according to a fairness algorithm that picks frames for from buffers at the heads of these dedicated links in a manner that minimizes bandwidth hogging. In the new Comcast system, the fairness algorithm is deployed in the CMTS (the cable equivalent of a DSLAM), and acts on buffers in each customer’s cable modem. The CMTS is able to do this because DOCSIS data transfers from customer to network are preceded with brief requests for bandwidth. Armed with intelligence about each user’s recent traffic history and the state of the network generally, the newly-intelligent CMTS will schedule bandwidth by customer according to a fairness algorithm of its own, with the same range of choices that exist for DSLAMs.

This was fun to write, and the reaction has so far been very positive, which is unusual among the crowd that comments at The Reg.

WordPress 2.3.3

This is for all you bloggers who use WordPress

WordPress 2.3.3 is an urgent security release. If you have registration enabled a flaw was found in the XML-RPC implementation such that a specially crafted request would allow a user to edit posts of other users on that blog. In addition to fixing this security flaw, 2.3.3 fixes a few minor bugs. If you are interested only in the security fix, download the fixed version of xmlrpc.php and copy it over your existing xmlrpc.php. Otherwise, you can get the entire release here.

The security hole allows spammers to infect your site with their crappy ads. When doing my backup I found 40 directories full of images and spam pages in a directory called “img” in my pictures directory and in another place within by wp-content. These freeloaders are a scourge.

On the plus side, my code’s up-to-date.

UPDATE: WordPress 2.5 is now released, and it’s very pretty, but it seems to be much slower.

New and Improved Traffic Shaping

Comcast’s CTO Tony Werner was kind enough to give me a few minutes today on the changes afoot in the cable giant’s Internet access network, and I like what I learned. I’ll do a longer post on this later with some diagrams, but for now I’d like to sketch out the high points. This is just from the Comcast side of the agreement, BitTorrent is also committed to making some changes on their end and I don’t have the details on those yet. BitTorrent will be making a presentation at the P4P Forum on its commitments.

Here’s what Comcast is going to do, pending how well it shakes out in the lab:

* Stop injecting TCP RSTs. This technique has been maligned way more than it deserves to be, because it has such a long history. Middleware devices (of which Sandvine is only one) have been doing this for at least a decade, drawing the ire of the IETF for it all along. It’s not necessary in a DOCSIS network for technical reasons, so they’re going to stop doing it. This should make the “Comcast is Impersonating You and Stealing Your Credit Card Numbers!!!” crowd happy.

* Start using CMTS scheduling to allocate bandwidth fairly among all users of a first-hop segment when the network is heavily loaded. The DOCSIS protocol permits packet scheduling, since every user has to request bandwidth for every upstream transfer, so all the CMTS has to do is implement Fair Scheduling to prevent bandwidth hogs from getting more than a fair share. There may be some limits to the delay the scheduler can impose (my conjecture, not Tony’s,) and that’s why field testing is important.

* Investigate longer-term solutions that will allow users to control how different traffic streams are handled. There are a number of IETF standards that relate to this problem, and their evaluation will be long-term work items for the industry forums.

CMTS scheduling puts Comcast on the same footing as the DSL providers. While Comcast customers share a first hop and DSL doesn’t (most of the time, they actually do if repeaters are used), all of them share a second hop, access to which is mediated by a fair queuing discipline. So Comcast is simply implementing their queuing discipline on the first hop, which makes good sense for their technology. So there’s no need to look at protocols and headers, it’s all just traffic and traffic opportunities can be managed with per-user fairness.

So the bottom line is this: the IETF protocols failed to deliver a scheme for per-user fairness, so Comcast will implement one on their first hop network. That’s what we call progress, and the only question is why it took them so long to do it.

Comcast & BitTorrent Announcement

You may have noticed that BitTorrent and Comcast have agreed to work together to improve co-existence. The Wall St. Journal is reporting the significant detail:

Rather than slow traffic by certain types of applications — such as file-sharing software or companies like BitTorrent — Comcast will slow traffic for those users who consume the most bandwidth, said Comcast’s Mr. Warner. Comcast hopes to be able to switch to a new policy based on this model as soon as the end of the year, he added. The company’s push to add additional data capacity to its network also will play a role, he said. Comcast will start with lab tests to determine if the model is feasible.

No details are out on the changes to be made on the BitTorrent side so far.

This is a huge announcement as it cuts off FCC chairman Kevin Martin at the knees, and does so in advance of his scheduled lynching in Palo Alto. If Comcast and BitTorrent can work together to resolve their problems with technical solutions (as I advocated at the FCC hearing at Harvard), there’ no predicate for new regulations or enforcement actions.

It’s a setback for the public interest lobby, and they’re taking it pretty hard. Public Knowledge in particular still calls for heavy regulation and increased FCC involvement in network management practices. It’s almost laughable:

Even in the best-case scenario for a Comcast/BitTorrent partnership, Comast is not the only one engaging in this sort of behavior. The FCC must make it clear that these types of practices are unlawful and against public policy, and that they will not be tolerated, now or in the future.

Free Press is also quite upset:

This agreement does nothing to protect the many other peer-to-peer companies from blocking, nor does it protect future innovative applications and services. Finally, it does nothing to prevent other phone and cable companies from blocking. Innovators should not have to negotiate side deals with phone and cable companies to operate without discrimination. The Internet has always been a level playing field, and we need to keep it that way.

You’d think BitTorrent had been bought off by Comcast, and no longer deserves to be adored as an “innovative new application.”

BitTorrent, Inc. is willing to modify their code to make it more manageable, but the pirates who use BitTorrent open source and Vuze won’t be, so the cat-and-mouse game between legal and illegal uses of P2P will continue. BitTorrent, Inc. won’t be affected, however, and that’s progress.

The agreement shows once again that technical solutions to technical problems are better than political ones. It’s unfortunate for the public interest lobby that this issue is no longer a cause for grandstanding, but they’ll recover. And in the meantime, the 60% of American broadband consumers who use cable to connect to the Internet will have a faster pipe that they’ll be able to use without being hogged out by their neighbors.

It’s a win for everybody.

Go to CNet for Declan McCullagh’s detailed interview with Comcast’s Joe Waz.

The FCC commissioners have all issued statements, the best of which is this one from Commissioner McDowell:

The private sector is the best forum to resolve such disputes. Today’s announcement obviates the need for any further government intrusion into this matter.


UPDATE: See this post for some detail on what’s afoot.

Harden and Brown Crush the Sox

Game 2 of the budding major league season was quite an enjoyable affair, as the Mighty A’s sent the Bosnia Red Sox home in disgrace. Harden no-hit the pretenders through 3, and struck ’em out at will. Emil Brown redeemed himself with a game-winning three-run homer and Crosby continued his hitting tear.

The A’s aren’t a bad team this year, but if I were a member of Bosnia Nation I’d be worried about the pitching. The Sox look a lot like some Yankee teams of recent memory, all bats and no curveballs.

Beckett’s injured, Schilling’s out for the season, Matsuzaka is still erratic, and Lester looks like a batting practice patsy. The middle of the Sox lineup is as strong as anyones, but it’s going to take a lot of runs to overcome the help Sox pitching will give opposing teams this year. The A’s aren’t a strong hitting team, but they scored 5 runs in each of the two games in the Tokyo Dome.

The Yankees are dealing with their pitching woes with a youth movement, and the Sox response is something like an injury movement. Not good, Boston Nation.

Tough loss on Opening Day

As odd as it may seem, the A’s opened their season in Tokyo last night, losing to the Bosnians 6-5 in 10. There were two new faces in the A’s lineup, Ryan Sweeney in center and Emil Brown in left, and one new old face, Keith Foulke in the bullpen.

The A’s lost this game the old-fashioned way, through a blown save by MLB blown saves over-achiever Huston Street in the ninth followed by some equally dismal Street pitching in the tenth. Jack Kust struck out every time he came to the plate and didn’t get hit, so there’s a lot of continuity with this squad from last year’s team.

Crosby, Ellis, Barton, and Hannahan all did well at the plate, so for now the infield isn’t holding the team back. With a little production from the over-stocked outfield, and some re-arranging of the deck chairs in the bullpen this could be a solid second-place team.

Nagle’s Answer

Slashdot picked up George Ou’s latest piece on the problems with TCP and Peer-to-Peer congestion that I’ve been writing about lo these many months, attracting one interesting comment in a sea of chaff:

As the one who devised much of this congestion control strategy (see my RFC 896 and RFC 970, years before Van Jacobson), I suppose should say something.

The way this was supposed to work is that TCP needs to be well-behaved because it is to the advantage of the endpoint to be well-behaved. What makes this work is enforcement of fair queuing at the first router entering the network. Fair queuing balances load by IP address, not TCP connection, and “weighted fair queueing” allows quality of service controls to be imposed at the entry router.

The problem now is that the DOCSIS approach to cable modems, at least in its earlier versions, doesn’t impose fair queuing at entry to the network from the subscriber side. So congestion occurs further upstream, near the cable headend, in the “middle” of the network. By then, there are too many flows through the routers to do anything intelligent on a per-flow basis.

We still don’t know how to handle congestion in the middle of an IP network. The best we have is “random early drop”, but that’s a hack. The whole Internet depends on stopping congestion near the entry point of the network. The cable guys didn’t get this right in the upstream direction, and now they’re hurting.

I’d argue for weighted fair queuing and QOS in the cable box. Try hard to push the congestion control out to the first router. DOCSIS 3 is a step in the right direction, if configured properly. But DOCSIS 3 is a huge collection of tuning parameters in search of a policy, and is likely to be grossly misconfigured.

The trick with quality of service is to offer either high-bandwidth or low latency service, but not both together. If you request low latency, your packets go into a per-IP queue with a high priority but a low queue length. Send too much and you lose packets. Send a little, and they get through fast. If you request high bandwidth, you get lower priority but a longer queue length, so you can fill up the pipe and wait for an ACK.

But I have no idea what to do about streaming video on demand, other than heavy buffering. Multicast works for broadcast (non-on-demand) video, but other than for sports fans who want to watch in real time, it doesn’t help much. (I’ve previously suggested, sort of as a joke, that when a stream runs low on buffered content, the player should insert a pre-stored commercial while allowing the stream to catch up. Someone will probably try that.)

John Nagle

I actually suggested the technique John proposes directly to Comcast engineering: drop packets before the first hop. They didn’t appear to have considered it before, but it actually is the answer. Unfortunately, the cable modem is not an IP device so it doesn’t understand when and how to do this presently, so it becomes a piece of housekeeping for the DOCSIS 3.0 upgrade.

Fine Kettle of Links

Some interesting reading for you:

George Ou tells the story of the unfairness of TCP and offers some suggestions.

Adam Thierer takes on Jonathon Zittrain’s paen to programmability The Future of the Internet: And How to Stop It.

And finally, Harold Feld explains the 700 MHz auction and what it does and doesn’t mean. Harold is a communist, but he’s a much smarter communist than his comrade Susan Crawford, who tries to read the same tea leaves and doesn’t say much.