No Bandwidth Hogs?

Some would argue there is no "exaflood" and no such thing as a "bandwidth hog." 

I have no more detailed data from any Internet service provider than anybody else does, so I doubt anybody can prove or disprove the thesis definitively. But I also have no reason to think the usage curve will be anything other than a Pareto distribution, since so many common distributions in the physical and business world conform to such a distribution.

Vilfredo Pareto, an Italian economist, was studying the distribution of wealth in1906. What he found was a distribution most people would commonly understand as the "80/20 rule," where a disproportionate share of results come from 20 percent of actions. The Pareto distribution has been found widely in the physical and human worlds. It applies, for example, to the sizes of human settlements (few cities, many hamlets/villages). It fits the file size of Internet traffic (many smaller files, few larger ones).

It describes the distribution of oil reserves (a few large fields, many small fields) and jobs assigned supercomputers (a few large ones, many small ones). It describes the price returns on individual stocks. It likely holds for total returns from stock investments over a span of several years, as most observers point out that most of the gain, and most of the loss in a typical portfolio comes from changes on just a few days a year.

The Pareto distribution is what one finds when examining the sizes of sand particles, meteorites or numbers of species per genus, areas burnt in forest fires, casualty losses: general liability, commercial auto, and workers compensation.

The Pareto distribution also fits sales of music from online music stores and mass market retailer market share. The viewership of a single video over time fits the Pareto curve. Pareto describes the distribution of social networking sites. It describes the readership of books and the lifecycle value of telecom customers.

So knowing nothing else than that the Pareto distribution is so widely represented in the physical world and in business, I would expect to see the same sort of distribution in bandwidth consumption. As applied to users of bandwidth, Pareto would predict that a small number of users in fact do consumer a disproportionate share of bandwidth.

I certainly can't say for sure, but would be highly surprised if in fact a Pareto distribution does not precisely describe bandwidth consumption.

DPI Raises Consumer Ire, Should it?

"Network bandwidth is a finite resource, especially so in wireless networks, so it is reasonable and indeed expected that carriers will manage their network bandwidth to assure sufficient quality of service for all subscribers," says Brian Wood, Continuous Computing's VP. That tends to mean use of deep packet inspection, and that tends to raise hackles in some quarters.

The problem is that Internet access, and Internet backbones and servers, are shared resources. There is a "tragedy of the commons" problem if a few users have behavior patterns dramatically different from those of the typical user, because all networks are engineered statistically.

Nobody builds a network that provisions bandwidth on a "nailed up" basis, because nobody could afford to do so. Instead, bandwidth is "underprovisioned," on purpose. Network architects assume that not every user will be putting load on the network, all at the same time.

That works remarkably well most of the time. What causes problems are unexpected loads that haven't been engineered into the network.

"Without traffic management, a few 'bandwidth hogs' can easily degrade the user experience of many other users on the same network," says Wood. That might especially be true in the access network, but the entire Internet transmission  network, including all the servers, are shared resources.

"For consumers, DPI-based traffic management ensures that subscribers get the quality of service that they expect, or at least that they pay for," says Wood.  So, for example, a business user might opt to pay a slight premium for a guaranteed level of service (e.g., guaranteed minimum bandwidth) while a frugal college student might go for a cheaper "best effort" rate plan.

Basically, DPI or other traffic shaping mechanisms are about fairness: making sure most users get reasonable performance most of the time. The other advantage is the ability to learn or be instructed by a user on what activities are most important, so those activities get the highest priority during congested periods.

DPI can be viewed as an automated away, or a self-learning kind of way, for the network to provide those kinds of benefits, says Wood. "It's all a matter of filtering out the stuff that, based on past behavior or the behavior of similarly-profiled individuals, is deemed to not be of value and, instead, prioritizing the stuff that is deemed to be of value."

Behavioral tracking is an issue, though. "Cookie-based tracking seems to be a generally-accepted practice with web sites these days, but there was great concern when cookies were first introduced," says Wood. There are end user advantages, of course, such as sites "remembering" who you are and what your preferences are.

Behavior-based tracking has raised more concern. Deep packet inspection is deemed by some as intrusive and too personal, says Wood. The same sorts of concerns are raised about DPI-based ad insertion.

"What's interesting to me, though, is that Google has been offering Gmail for free to users in exchange for content-based advertisements being displayed next to their emails, and I haven't heard of any uprising against Gmail," he says.

Subscriber notification, how subscribers are notified, and whether those subscribers have any say in the matter, seem to be the key sticking points here, he muses. "Nobody likes the idea of being monitored without their consent, especially if they feel like information gathered through such monitoring will be used in an attempt to profile or manipulate them in the future."

But behavior-based marketing seems to work well for Netflix and Amazon, Wood notes. The difference seems to be one of perception. Lots of people are afraid technology will be used "on" them, rather than "for" them.