|
Per-flow policing and marking is performed by the first trusted leaf
router downstream from the sending host. When a local admissions control
decision has been made by the sender's cloud, the leaf router is
configured with the contracted per-flow service profile. Downstream from
the first leaf router, all traffic is handled as aggregates.
On cloud ingress, incoming traffic is classified by the PHB bits into
aggregates, which are policed according to the aggregate profiles in
place. Depending on the particular diffserv service model in question,
out-of-profile packets are either dropped at the edge or are remarked with
a different PHB. As in-profile traffic traverses a cloud, it may
experience induced burstiness caused by queuing effects or increased
aggregation. Consequently, clouds may need to shape on egress to prevent
otherwise conforming traffic from being unfairly policed at the next
downstream cloud.
Finally, to make appropriate internal and external admissions control
decisions and to configure leaf and edge device policers correctly, each
cloud is outfitted with a bandwidth broker (BB). When a sender
signals its local bandwidth broker to initiate a connection, the user is
authenticated and subject to a local policy-based admissions control
decision. On behalf of the sender, the BB then initiates an end-to-end
call setup along the chain of bandwidth brokers representing the clouds to
be traversed by the flow. The bandwidth broker abstraction is critically
important because it allows separately administered network clouds
(possibly implemented with very different underlying technologies and
polices) to manage their network resources as they see fit.
The reader is strongly encouraged to consult the references for a more
thorough discussion of the evolving differentiated services framework. In
particular, [TwoBit]
and [OpDef]
make good introductory reading.
2.1 Non-Relative Differentiated
Services
Much of what is driving the current push for differentiated services is
the perception by providers that there is a sizable and immediate
commercial market for differentiated classes of best-effort IP service. In
the precedence-based differentiated services proposed in [IPPrec],
flows are aggregated into ranked classes that are given relative
transmission guarantees. As congestion increases, traffic in a given class
will experience loss due to congestion later than traffic in a lower
class. Additionally, high precedence traffic may be guaranteed to
experience less queuing delay than low precedence traffic. Users may still
contract for a specified profile at a specified precedence level, but
there is no way to characterize the service that a flow receives in an
absolute sense (or even in an a priori statistical sense).
Such class-of-service (CoS) solutions will almost certainly play an
important role within Internet2, since when viewed as a production
network, Internet2 will face demands similar to those faced by the
commercial networks. CoS is very useful for providing better service to
largely static classes of users, institutions, or applications. CoS,
however, is unlikely to meet the need of the most advanced Internet2
applications. To function correctly, applications such as collaborative
virtual environments or remote instrument control, require explicit,
non-relative (i.e. absolute) transmission guarantees.
3. Evaluating Differentiated Services
In this section we evaluate the appropriateness of the profile-based
differentiated services approach to meeting Internet2's QoS requirements.
Each requirement set forth in [I2Reqs]
is considered separately below. It is argued that diffserv appears to be a
remarkably good fit to Internet2's needs and that Internet2 should explore
diffserv in a wide-area testbed.
3.1 Enables Advanced Applications
3.11 Non-Relative and High-Performance Requirements
Many advanced applications have "absolute" transmission
requirements in the sense defined in [I2Reqs].
For example, remote collaborative tools usually have medium hard
requirements based in human factors results that translate into very
specific, non-relative bandwidth and latency needs. Failure to meet these
needs results in the application being unusable. Moreover, some important
advanced applications, such as remote surgery or remote instrument
control, have requirements that, if not met, result in unacceptable real
world costs (i.e. loss of life or damage to expensive remote
instruments). The family of differentiated services that supports
well-defined, non-relative service profiles offers a promising path
towards providing such assurances in a scalable manner.
Additionally, it is assumed that there will always exist advanced
applications that will stretch the capabilities of the network to the
breaking point. Such applications may have bandwidth needs nearly on the
order of the aggregate available bottleneck bandwidth -- making a
"reservation free" approach impossible. This translates to a requirement
within Internet2 for a dynamic call setup mechanism that can pin and unpin
coarse allocations of network bandwidth. Note that the BB-to-BB call setup
within diffserv results in resource reservations that are made on a
per-cloud basis; but, that a cloud may choose to keep dynamic reservation
state at finer detail (i.e. per ingress-egress pair or even per-hop) if
the increased complexity is merited.
3.12 Busy Signals
It is important to observe that the advent of a QoS-enabled network
will result in an expectational paradigm shift for applications
developers, users, and network planners. There are essentially two models
for how the network can respond when congestion occurs:
- New connections can be admitted and everyone given degraded
performance (this is characteristic of today's Internet and of
precedence-based differentiated services schemes).
- New connections can be refused and the quality of existing
connections maintained (this is characteristic of the phone system and
of QoS approaches that offer non-relative transmission guarantees).
For many applications, it is far more desirable to experience an
occasional busy signal during call setup than to experience unexpected
degradation of service. For example, researchers visualizing a
computational simulation on a super computer using a distributed
collaborative visualization tool cannot afford to have their technical
debate cut short because of unexpected congestion in the network. By
explicitly reserving resources end-to-end during call setup time, a
diffserv architecture is able to support this need and generate a busy
signal when the required network resources cannot be guaranteed.
3.13 Scheduling
Even more demanding are applications that require scheduled
reservations. Examples include: a distance learning course that is
scheduled for a certain time period each day; or remote access to a
one-of-a-kind scientific instrument. As bandwidth brokers are standardized
and their feature sets become more robust, it is certainly within the
scope of the architecture to support scheduled resource reservations.
Bandwidth brokers would be required to make admissions control decisions
not only on the basis of the network resources currently available, but
also on the basis of the previously scheduled reservations. One
implication of scheduled reservations is that all reservations would have
to be granted in terms of "leases" with explicit expiration times.
3.14 Multicast
Several important classes of application have requirements that go
beyond how a particular flow's packets are forwarded. For example, in
addition to needing QoS assurances, interactive applications with multiple
users often need multicast capability to scale gracefully. None of the
work on differentiated services has been applied to solving the multicast
QoS problem. However, there is broad consensus that nothing has been done
within diffserv that makes solving the multicast problem any more
difficult.
3.15 Example Services: Premium and Assured
The relevance of a QoS approach to enabling advanced applications must
ultimately be judged on the effectiveness of the end-to-end transmission
guarantee that can be made to the advanced application. The general family
of approaches advocated in this document, non-relative profile-based
differentiated services, includes specific instances that must be
evaluated with respect to application requirements. Examples services
include: Premium service [TwoBit]
and Assured service [Assured].
The Premium service model emulates a leased line service at a given
peak rate and requires a per-hop behavior equivalent to strict priority
forwarding. Premium users contract for a service profile defined by a peak
rate and a token bucket with a depth of one or two packets. Premium is
highly appropriate for applications that have strict bandwidth, latency,
and jitter requirements. An analysis of how Premium could be used in
support of Internet2's needs is given in [I2Premium].
Clark and Wroclawski have proposed a statistically assured service
known as "Assured" [Assured]
that emulates a lightly loaded network. Assured users contract for a
service profile that is characterized by a bit rate and token bucket burst
filter and are assured that in-profile traffic will experience service
consistent with a "lightly loaded network" even in the presence of
congestion. Out-of-profile traffic is re-marked at the network edge with a
special PHB indicating drop preference and, in the presence of congestion,
out-of-profile packets are dropped preferentially over in-profile packets,
without reordering the traffic within a flow. Statistically assured
services like Assured are likely to be much less expensive to provision
than Premium and may be highly appropriate for applications that require
transmission assurances that are averaged over long periods of time or for
certain classes of adaptive applications that can tolerate some packet
loss. Statistically assured differentiated services are, however, more
difficult to characterize precisely than Premium and have several
difficult technical problems that remain to be solved. Further study of
these services is an important research area where Internet2 should
contribute.
3.2 Interoperable
Interoperability is one of the key strengths of the diffserv approach.
Much of the diffserv work within the IETF has striven to specify PHBs and
functional components that admit multiple, interoperable implementations.
Additionally, supporting end-to-end services across multiple, separately
administered network clouds has been a primary design goal since the
beginning.
3.21 Interoperable Network Equipment
The ongoing work in the IETF Integrated Services working group to
standardize a small number of per hop behaviors (PHBs) that are
independent of the underlying mechanisms that implement them should lead
to a situation where multiple vendors can implement diffserv-enabled
packet forwarders. Although much of the work to date has been on
specifying PHBs and on finding agreement on how to divide the DS byte,
additional work will soon be needed to define standard protocols for:
- Hosts to signal their requests to their local bandwidth broker
- Bandwidth brokers to signal other bandwidth brokers for end-to-end
call setup
- Bandwidth brokers to signal leaf routers with the per-flow policing
parameters they should enforce
- These protocols should also evolve to admit multiple, interoperable
implementations.
The Internet Draft [OpDef]
gives an overview of several example instances of differentiated services
-- including Assured, Premium, and a precedence service called Olympic.
This draft describes the per-hop behaviors (PHB), as well as the location
and role of key functional components required to implement them. Although
the services discussed vary radically from one another, there is
significant overlap in the location of functional components and the
mechanisms required in them.
3.22 Concatenatable Network Clouds
The need to concatenate the separately administered and engineered
services of multiple network clouds into a meaningful end-to-end service
has been a primary motivater of the diffserv architecture since its
inception. The diffserv framework meets this need by defining services
only at cloud edges, and allowing for flexibility in how they implemented
across each cloud. Bandwidth brokers abstract the differences between how
clouds are engineered internally and provide a common meeting point for
admissions control negotiation. Within a diffserv cloud, network engineers
have considerable flexibility in choosing among competing technologies,
network equipment vendors, and provisioning alternatives. In particular,
it is believed that diffserv will be implementable either completely at
the IP-level (for an IP-over-SONET cloud) or at the ATM level (for an
IP-over-ATM diffserv cloud) using techniques that map diffserv PHBs to ATM
service classes.
The differentiated services approach is particularly well-suited to
meeting the administrative challenges of extending a QoS service across
separately administrative clouds. The framework results in simple,
bilateral agreements between service providers that are easy to
comprehend, enforce, and audit. Additionally, by concealing the internal
provisioning and engineering details behind the bandwidth broker
abstraction, network service providers are afforded the privacy and
flexibility that they need to operate independently and competitively. The
separation within diffserv between the interdomain problem and the
intradomain problem has been compared with the historically successful
separation of interdomain and intradomain internet routing.
3.3 Scales Well
Scalability is one of the most serious engineering requirements but,
happily, is also the paramount concern driving the design of the diffserv
architecture. The main facets to the QoS scalability problem are:
- Scaling to large numbers of flows with different assurances
- Scaling to high speeds
3.31 Scaling to Large Numbers of Flows
Core routers are routinely burdened with forwarding thousands of flows
and, consequently, any QoS approach that requires substantial per-flow
state or computational overhead in the forwarding engines will not scale
well. The diffserv approach attempts to push per-flow complexity away from
the network core and towards the network periphery where both the
forwarding speeds and the fan-in of flows are smaller. The aim is to
provide per-flow service assurances by policing flows at leaf routers and
servicing only a small number of traffic aggregates in the core. As has
been discussed, the aggregates are characterized by only a small number of
simple per-hop forwarding behaviors, which greatly reduces the per-packet
forwarding complexity for core routers.
3.32 Scaling to High Speeds
In addition to the need to keep PHBs simple so that aggregates may be
handled at high speeds, there will be an increasing need to handle
individual flows at high speeds. This is true especially in an environment
where increases in available network bandwidth create a perception that
bandwidth is plentiful. Dramatic increases in available bandwidth will
inspire and enable new kinds of applications that increasingly will depend
on the existence of fat pipes; when these pipes eventually become
congested, the new applications will require QoS assurances that scale to
very high speeds. The differentiated services approach of pushing per-flow
complexity as close to the application as possible represents the best
hope for scaling per-flow QoS assurances to very high speeds.
3.4 Administratable
In the differentiated services framework, each cloud is free to set its
own policies and administrative procedures as long as the bilateral
agreements that it makes with other clouds are met. Since resource
allocation policies are often sensitive, it is appealing that the
bandwidth broker admissions control model handles external admissions
requests without extensive sharing of policy information.
Because internal policies and procedures vary so widely among domains,
the administrative flexibility of diffserv is especially important for
clouds that contain end-hosts. The advent of production quality QoS will
inevitably be accompanied by a need to control access to it according to
locally appropriate policies. The negotiation between a host and its local
bandwidth broker is the ideal place to enforce such local policies. Within
a domain, the bandwidth broker is the centralized point for performing
authentication, authorization, and accounting (AAA) and as such is the
centralized point where policies are administered.
Some campuses within Internet2 may have very decentralized or
hierarchical administrative structures that would be better served by a
distributed bandwidth broker architecture. Distributed bandwidth broker
architectures are fully consistent with the model and may also have
desirable performance and fault tolerance characteristics. Other campuses
may choose to avoid the complexities of highly dynamic AAA by selling
statically configured, subscription-based services to users. This approach
is explored further in [EntCamp].
3.5 Provides a Measurable Service
Unlike class-of-service approaches that define relative priorities
among traffic classes, the family of non-relative differentiated services
allows a contracted performance guarantee to be compared against a
well-defined metric. Hence, a user with suitable measurement tools can
audit the service that is being received and paid for. Measurement tools
could also be used by network service providers to audit their aggregate
profile contracts.
Additionally, network service providers may rely on measurement tools
to facilitate network engineering and provisioning. For example,
measurements might reveal unexpected congestion points that either need to
be relieved or managed carefully by the local bandwidth broker. The
diffserv framework is especially conducive to measurement in that a large
amount of useful measurement data falls naturally out of the packet
classifiers and policers, which are an inherent part of the
architecture.
Having a good measurement infrastructure is always important, but it is
particularly important during the initial deployment of a new service.
Therefore, Internet2 should make measurement an integral part of its
testbed exploration of differentiated services.
3.6 Deploys Incrementally
An attractive quality of the diffserv framework is that it may be
deployed incrementally and offer useful pre-production services before a
complete system is in place. There are three main elements to a
full-fledged system: diffserv-aware routers, bandwidth brokers, and a
community of diffserv-aware applications and users. Fostering the
development of an early community of users is critically important and is
a prime motivation for making early iterations of a diffserv service
available to end users.
Diffserv-aware routers must be able to recognize the PHB bits and
perform the required forwarding. The queuing disciplines required to
implement currently proposed diffserv forwarding behaviors either already
exist in current routers or will be available very soon. High-speed
implementations of packet classifiers and shapers are also expect in next
generation routers. ATM implementation techniques based on mapping
diffserv to ATM service classes may also be used.
Network clouds may evolve to support full-fledged differentiated
services through a combination of over-provisioning, "lying", and
deploying diffserv-aware routers incrementally starting at known
congestion points. As an example of when it might be useful for a cloud to
lie, consider a cloud that is prepared to offer a service that it knows is
not a true Premium service, but that it is committed to evolving to a true
Premium service.
Because interprovider interoperability is so important to both
Internet2's needs and to the diffserv framework, early testbed trials of
diffserv within Internet2 should work to demonstrate the meaningful
concatenation of services across multiple clouds. Initial bandwidth broker
functionality could consist of very static contracts and manual
administration of router settings for a few routers, but it would be
better to begin to automate this key functionality from the start. Early
development and testing of bandwidth broker prototypes should therefore be
made a priority.
An example Phase 0 system might implement a phony Premium service and
consist of two clouds with a handful of diffserv-enabled routers and two
prototype bandwidth brokers. Users might make Premium reservations via
e-mail to their local BB, giving the required peak rate and the time
period desired. The BB would need to authenticate the request, insure the
peak rate is within the limits authorized for that user, and determine
whether sufficient bandwidth is available to grant the request. If the
request is granted, the user might simply be trusted to stay within
profile. (If leaf router policers or diffserv-enabled edge routers are
present, they would be appropriately configured by the BBs.) Initially it
may not be necessary to have all core routers act on the PHB bits. Traffic
measurements might reveal congestion points or "hot spots" where the
associated router needs to participate in PHB bit recognition and action.
Other core routers may be so lightly loaded that they give near Premium
service to all flows. Diffserv traffic would essentially "tunnel" through
these routers.
Eventually, bandwidth brokers may need to evolve in sophistication to
cope with routing changes and to route call setup requests based on their
QoS requirements. It is assumed, however, that Internet2's initially
hierarchical topology will result in a network with very few route changes
and that forgoing these additional complexities for some time will not
unduly affect service.
3.7 Works with End-Host Operating Systems and
Middle-Ware
As discussed above, a differentiated services architecture could be
deployed initially using today's QoS-blind applications and simple e-mail
requests. It is expected, however, that there will quickly develop a need
for applications to be able to signal their QoS requirements dynamically
and with the support of appropriate APIs and middleware.
RSVP is increasingly available in modern operating systems and could be
used as the signaling protocol between hosts and their local bandwidth
broker. An architecture for mating RSVP to diffserv has been outlined in a
recent Internet Draft [IntDiff].
To facilitate the development of diffserv-enabled applications, it has
been proposed that differences among existing APIs and among operating
system QoS signaling protocols be abstracted by a new API and middleware
[QoSAPI]. This approach also advocates that the new API provide access to
a library of pre-defined service profiles, indexed by application type or
codec.
4. Conclusion
The Internet2 QoS working group believes that the evolving
differentiated services framework offers the most promising approach for
meeting Internet2's QoS requirements. Diffserv represents the very best
current thinking on how to provide a flexible array of QoS services in the
Internet in a way that scales to the high packet-per-second rates and
large number of flows that will be required. Moreover, particular diffserv
services that have been proposed appear to be very appropriate for meeting
the demanding transmission requirements of advanced Internet2
applications.
Although important components of the diffserv framework have been
implemented and understood in other contexts, many aspects have yet to be
standardized and the overall approach has yet to be evaluated empirically.
As has been observed above, there is currently significant commercial
interest in selling precedence-based differentiated services and in
designing the network equipment to implement them. The QoS working group
recommends that Internet2 deploy precedence-based solutions they become
available, while simultaneously exploring one or two experimental
differentiated services in a testbed environment.
Acknowledgments
The authors wish to acknowledge the members of the Internet2 QoS
working group and the many members of the Internet2 community that have
participated in the ongoing discussions on the appropriateness of
differentiated services to Internet2.
References
[Assured] D. Clark and J. Wroclawski, "An Approach to Service
Allocation in the Internet", <draft-clark-diff-svc-alloc-00.txt>,
August, 1997.
[DSByte] K. Nichols and S. Blake, "Definition of the Differentiated
Services Field (DS Byte) in the IPv4 and IPv6 Headers", Internet Draft
<draft-ietf-diffserv-00.txt>, May, 1998.
[EntCamp] "T. Gray "Enterprise/Campus QoS: Managing Administrative
Complexity", Internet2 Technical Paper, May, 1998.
[I2Premium] K. Nichols, "Using Premium Service to Provide Internet2
QoS", Internet2 Technical Paper, May, 1998.
[I2Reqs] B. Teitlebaum, "QoS Requirements for Internet2", Internet2
Technical Paper, 1998.
[IntDiff] Y. Bernet et. al., "A Framework for End-to-End QoS Combining
RSVP/Intserv and Differentiated Services",
draft-bernet-intdiff-00.txt,
ftp://ds.internic.net/internet-drafts/draft-bernet-intdiff-00.txt,
March 1998.
[IPPrec] F. Baker, S. Brim, T. Li, F. Kastenholz, S. Jagannath, and J.
Renwick, "IP Precedence in Differentiated Services Using the Assured
Service", Internet Draft <draft-diff-serv-precedence-00.txt>, April
1998.
[OpDef] K. Nichols and S. Blake, "Differentiated Services Operational
Model and Definitions", Internet Draft,
<draft-nichols-dsopdef-00.txt>, February 1998.
[QoSAPI] B. Riddle & A. Adamson, "A QoS API Proposal", Internet2
Technical Paper, May, 1998.
[TwoBit] K. Nichols, V. Jacobson, L. Zhang, "A Two-bit Differentiated
Services Architecture for the Internet", Internet Draft
<draft-nichols-diff-svc>,
<http://www-nrg.ee.lbl.gov/papers/2bitarch.pdf>, November
1997.
Author's Addresses
John Sikora
Technical Manager, Broadband and Advanced
Networking
AT&T Labs, rm 1J-307
101 Crawfords Corner
Rd
Holmdel, NJ 07733
voice: 732-949-1828
email:
jjs@arch4.att.com
Ben Teitelbaum
Internet Engineer
200 Business Park
Drive
Armonk, NY 10504
voice: 914-765-1118
fax:
914-273-1809
e-mail: ben@internet2.edu | 
