Ivip - a new scalable routing and addressing architecture for the Internet

Robin Whittle rw@firstpr.com.au 2009-04-23

To the main IP page


Multihoming service restoration with Ivip


Introduction

Ivip - Internet Vastly improved Plumbing (or your choice of several other meanings, such as "Internet Versatile redIrection of Packets" etc.) is a  proposal I began in June 2007 to help solve the crisis in routing and addressing (AKA the" routing scaling problem"), to provide a new form of mobility for both IPv4 and IPv6, and to help improve IPv4 address utilization - and thereby combat the IPv4 address depletion problem.

For people who are new to the RRG discussion of the Internet's routing scaling problem, please see this RRG List message msg00623.html for a description of some of the terminology and for an explanation of the above diagram, which depicts multi-homing service restoration and (ITR1) Ivip's "anycast ITRs in the core/DFZ", which are now known as OITRDs (Open ITRs in the DFZ).  Please search the RRG list 2008 archives for "Newbies" for other messages which may help to familiarize you with this discussion.

Please see RRG-2009/ for links to the RRG and to other resources regarding scalable routing and addressing.

Recent developments

2009-04-23
After an extensive discussion with Bill Herrin about anycast and "disaster recovery" arrangements, I suggested an additional function for core-edge separation schemes such as LISP, APT, Ivip and TRRP. (It could also work with the host-based core-edge elimination scheme, but I think core-edge separation is better.)  This is a "Distance Server", which enables an ITR to choose, or have chosen for it, the ETR from a list of ETRs which is "closest" in BGP terms.   Please see this RRG message and discuss it there:  

Adding a Distance Server for anycast / disaster recovery  
http://www.ietf.org/mail-archive/web/rrg/current/msg04907.html
 
Currently, and with existing core-edge separation proposals, the best way of doing "anycast" is the current, conventional, BGP-based approach.  This works well, but is highly unscalable. With "Distance Servers", Ivip etc. could do much the same thing in a highly scalable fashion.

2009-04-19
A new page RRG-2009/constraints/ has a draft list of constraints on any scalable routing solution which result from our reliance on widespread voluntary adoption.  Please comment on this in the RRG list.


2009-04-15
Updated the Summary and Analysis document #summary with a more detailed discussion of how a Multihoming Monitoring Company's system would be the authoritative source of mapping when there was a failure of one ETR or the other, but when both ETRs were working fine, would accept real-time mapping changes from the end-user network itself for the purpose of real-time adjustment of the load sharing between the two ETRs and their associated ISPs and links.


2009-03-21
Added a page namespace/ as part of trying to determine the meaning of the noun "namespace".  This is part of a debate on the LISP list about whether or not statements about LISP involving separate namespaces for EIDs and RLOCs are accurate or not.  This is probably more than anyone ever wanted to know about the meaning of the term, but since the debate has involved the suggestion that "namespace" may have "a lot of other definitions floating around" I tried to find them - without success.

2009-03-01
Added a page RRG-2009/rec/ listing what I think is missing from the current version of the RRG's draft recommendations, which are to be completed in March 2009.

This paper:

Towards a Future Internet Architecture: Arguments for Separating Edges from Transit Core
Dan Jen, Lixia Zhang, Lan Wang, Beichuan Zhang
http://conferences.sigcomm.org/hotnets/2008/papers/18.pdf

the authors argue that "core-edge separation" approaches  is a better way of achieving scalable routing and addressing than  "core-edge elimination" schemes.  Core-edge separation schemes are "Strategy A" in Bill Herrin's taxonomy (below) and in the 3.3 taxonomy of the draft RRG recommendations.  Core-edge separation proposals include LISP, APT, Ivip, TRRP and Six/One Router.
2009-01-03
Added a page rrgarch/ which attempts to match actual scalable routing proposals to the strategies in Bill Herrin's Summary of Routing Architectures: http://bill.herrin.us/network/rrgarchitectures.html .  Bill's taxonomy is now the basis of much of the RRG draft recommendations mentioned above.
2008-12-21
Updated the Summary and Analysis document (#summary) to include mentioning of "Forwarding with modified IP headers" as an alternative to encapsulation.
2008-08-25
Steve Russert and I have finished a paper on the TTR Mobility extensions to Ivip, or to other core-edge separation solutions to the routing scaling problem.  See below: #mobile.

2008-08-12
 (updated a little 2008-08-23)
Forwarding with modified headers: I have devised two systems for getting packets from ITRs to ETRs without encapsulation or address translation.  Both systems require a modest upgrade to most or all core (DFZ) routers and for some or many internal routers - but would make for a much simpler, more efficient system, with no PMTUD problems and simpler ITRs and ETRs.  An overview of these systems is now in the #summary document.

ETR Address Forwarding (EAF) - for IPv4

This is fully documented in tools.ietf.org/html/draft-whittle-ivip4-etr-addr-forw .  It puts the 30 most significant bits of the ETR address in the header bits for:

More Fragments bit.
Fragment Offset.
Checksum.
and marks the packet as being in this EAF format by setting bit 48, the "Evil Bit" tools.ietf.org/html/rfc3514.

I have a page ipv4-bits/actual-packets.html documenting some Google servers sending DF=0 packets as long as 1470 bytes.


Prefix Label Forwarding (PLF) - for IPv6

I don't yet have an Internet draft for this, but the proposal is described in detail here: ivip6/ .

The current plan is to use a 19 bit value in the proposed new Forwarding Label to get the packet from the ITR to one of 524,287 provider networks to which the new kind of end-user networks connect.  Each such recipient provider network can have effectively unlimited numbers of ETRs.

There is a another page exploring extending this system to be used by edge networks, with each such network, including the just-mentioned recipient provider networks, able to use 2^19 values of the proposed Forwarding Label to get the packet from its border router to any one of 524,287 ETRs, again without any encapsulation, PMTUD problems etc. ivip6/lfe/

Also, a little research page on which upper layer protocols look at bits in the IPv4 or IPv6 headers, in terms of their checksums - and which fields the IPsec Authentication Header looks at:  checksums/ .

The Forwarding approaches are more difficult to introduce, since they require modest upgrades to most or all core routers, and to some, many or all internal routers of networks which have ITRs or ETRs.  However, there are profound benefits:
  1. No encapsulation overheads.
  2. No PMTUD problems.
  3. Simpler ITRs and ETRs.
In the long term, the cost of upgrading core routers is insignificant, since the extra functions can be built into new gear.  So the long-term goal of Ivip is to run entirely with EAF for IPv4 and PLF for IPv6.

There will be three potential realisations for each of the Internets that Ivip could be applied to.  (Unfortunately, the IPv6 Internet is a different Internet from the IPv4 Internet.  Ideally they would be one thing, but they are not.)

The term Ivip refers to the overall architecture of both systems: Ivip4 and Ivip6.

For IPv4For IPv6Notes
Ivip4eIvip6eEncapsulation only.  It would make no sense to deploy ITRs and ETRs which could only do encapsulation, when in the long-term the system should work with Forwarding.  So I would never recommend Ivip or anything like it be deployed only with an encapsulation capability.
Ivip4efIvip6efAdvantages:
Easy to deploy initially, since it uses encapsulation and does not depend on upgraded core and internal routers.

Can be switched over to Forwarding when the routers are upgraded.

Disadvantages:
Packet overhead when using encapsulation.  More complex ITRs and ETRs to cope as best as possible with the PMTUD problems inherent in encapsulation
Ivip4fIvip6fAdvantages:
Go straight to the most efficient and elegant mode of operation - simpler ITRs and ETRs and no encapsulation overhead.

No need to figure out the PMTUD stuff and create ITRs and ETRs which can handle it.

Disadvantage:
Can only be deployed once most routers are upgraded - however, maybe that won't be too hard, so we should consider this as a viable option.  This is especially the case for IPv6, where there is not much urgency and there are few deployed routers.




2008-06-11


In the last few days there has been a big debate on the RRG list about how rapidly IPv6 will be adopted, which is a vital question when the RRG is deciding whether to recommend a solution for IPv4 and IPv6, or just for IPv6.  

There is a similar debate in the ARIN PPML list under the threads: "IPv6 in the Economist".  This includes some interesting critiques of the IPv6 DNS arrangements by Leo Bicknell and Dean Anderson.  Also, the thread: "Portable address space vs. IPv6 auto-numbering".  That thread starts here and someone who liked my initial message suggested I "blog or wiki it".  

There was an article about IPv6, dated June 5th 2008 in The Economist Technology Quarterly by an unidentified author (Monitor): Your number's up .  Mine is the fourth comment from the bottom here.

See also my July 2008 discussion psg.com/lists/rrg/2008/msg02045.html  responding to David Conrad, regarding IPv6 adoption and "Dual Stack Lite".

In late 2008, some more discussion of the difficulties in adopting IPv6 can be found in an IETF list discussion, especially these messages from Thomas Narten and Keith Moore:  TN1 , KM1 and TN2.

Summary and Analysis

Conceptual Summary and Analysis of Ivip 2008-04-15

Ivip-summary.pdf  

This is the most concise introduction to Ivip.  It is written primarily for people who are already up to speed with RRG list discussions.  It contains its own changelog pointing to earlier versions.


Comparison with other proposals

Comparing LISP-NERD/ALT, APT, Ivip and TRRP (All new 2008-04-21 - but does not cover Ivip's new "Forwarding" arrangements.)
See also the rrgarch/ page which attempts to match actual proposals to Bill Herrin's summary of strategies.


Internet Drafts

ivip-arch

In the future I will update the long ivip-arch ID to become a shorter, purely architectural description to be accompanied by a bunch of new, more specific IDs.  For now, it may be best to refer to the Summary and Analysis mentioned above.

The Ivip Architecture Internet Draft (I-D) version 00 of 2007-07-15 is:


ivip-fast-push

This was completed on 2008-02-19 and concerns the fast push database distribution system.  Version 01 is the same.

etr-addr-forw

2008-08-21: An IPv4 alternative to encapsulation and forwarding - no encapsulation overhead, no PMTUD problems - just needs a little upgrade to all the DFZ routers . . .

Overall plan for Internet Drafts


See the notes above #2008-08-12 for my overall Ivip plans.

ivip-arch-02This will be a fresh version, not directly based on the current ivip-arch-01 - to introduce the basic concepts and overall structure of both Ivip4 and Ivip6.

For now, please refer to Ivip-summary.pdf (above) for an architectural overview, to ivip-arch-00 and to the documents listed below.
ivip-db-fast-push-01
Please see  ivip-db-fast-push-00 linked to above.

Fast push of mapping data to hundreds of thousands or millions of ITRDs and QSCs all over the Net. Not yet updated to OITRD terminology.  In the next update I intend to go into more detail of the data formats from the RUAS to the Launch system, and then for the packets fanned out by the Replicator system,
ivip-qsQuery Servers, their internal functions and how to use them.
ivip4-itrivip6-itr
ITRs - caching ITRCs, ITR functions in sending hosts and OITRDs - Open ITRs in the DFZ.

Most of the material will be in the ivip4 draft, with only IPv6-specific material in the ivip6 draft.
ivip4-etrivip6-etrETR internal functions and how to use them.  Also how Ivip's encapsulation approach involves the outer header source address being the sending host's address.  This enables the ETR to automatically respect the provider network's filtering of packets with spoofed source addresses.  

For Ivip6, using ETRs and other techniques to get the packet from the recipient provider's network border router (where the core Forwarding Label system ends) to the destination end-user networks.
ivip-encapsulationNot needed for Ivip4f or Ivip6f.

The map-encap aspect of Ivip4ef and Ivip6ef.  
ivip4-etr-addr-forwarding
d-w-ivip4-etr-addr-forw
ivip6-pre-lab-forwardingSee mention above of ETR Address Forwarding (EAF)  Prefix Label Forwarding (PLF) .
ivip-ttr-mobilityTranslating Tunnel Router internal functions and how to use them for the new kind of mobility provided by Ivip.  Please see messages and diagrams in the Mobility section below.
ivip-pmtud-fragNot needed for Ivip4f or Ivip6f.

Ivip4ef and Ivip6ef will be the only map-encap scheme to integrate mechanisms for handling the problems of encapsulated packets exceeding path MTU limits - apart from some recent additions to LISP, which I think are problematic.  (See critique at: lisp-links/.)

For now, see details at pmtud-frag/ .  Ivip6ef (encapsulation and forwarding) would have similar PMTUD problems to all other encapsulation schemes, but Ivip6f (forwarding only) would have none.  
ivip-deploymentWhy ISPs and end-users will want what Ivip provides, and how to get the ball rolling.  See the section below on RRG list discussions for a message regarding the business case for the mapping system and OITRDs.

 

Ivip's approach to mobility

On 2008-08-25 Steve Russert and I completed a paper:

TTR Mobility Extensions for Core-Edge Separation Solutions to the Internet's Routing Scaling Problem
Robin Whittle, First Principles, Rosanna, Vic, Australia
Steven Russert Boeing Phantom Works, Seattle, WA, USA
2008-08-25 http://www.firstpr.com.au/ip/ivip/#mobile

TTR-Mobility.pdf

Some illustrations:

TTR mobility extensions for Ivip or other core-edge separation solutions to the Internet's routing scaling problem

TTR mobility extensions for Ivip or other core-edge separation solutions to the Internet's routing scaling problem


Path MTU Discovery and fragmentation problems


On 2008-04-21 I devised a new approach to the PMTUD (Path Maximum Transmission Unit Discovery) problems which all map-encap schemes need to deal with.  Please see:

pmtud-frag/ 

the RRG list discussion of this, and Fred Templin's SEAL ID.


Notable RRG list discussions

To-do, but see LISP critiques at  lisp-links/ and the Ivip business models message: psg.com/lists/rrg/2008/msg01158.html .


IRTF Routing Research Group


See the Internet Research Task Force Routing Research Group wiki page for similar "summary and analysis" documents for other proposals, according to instructions on 2007-12-20.

The RRG homepage www.irtf.org/charter?gtype=rg&group=rrg contains the charter, instructions for joining the mailing list and a link to the list archives.

Here are some more RRG links:

tools.ietf.org/html/draft-irtf-rrg-design-goals  Design goals (my critique of version 01, 2007-07-08 2007/msg00199.html)

psg.com/lists/rrg/2008/msg01175.html (2008-04-18) Design goals are still open for discussion.

psg.com/lists/rrg/2007/msg00686.html (2007-12-02) Schedule to produce recommendations to IESG in March 2009.

psg.com/lists/rrg/2008/msg00794.html (2008-03-15) Intended results: the form of recommendation for a new architecture.

psg.com/lists/rrg/2008/msg01159.html (2008-04-18) More from the co-chairs about the RRG process.

There is also a RADIR Design Goals document: tools.ietf.org/html/draft-narten-radir-problem-statement (my critique of version 01, 2007-08-09: www.ietf.org/mail-archive/web/ram/current/msg01809.html).

The major map-encap proposals are LISP, APT, Ivip and TRRP. Since the LISP links at the wiki page are incomplete, I have made a page to try to track the latest LISP Internet drafts: lisp-links/ and to link so some critiques of LISP.  




Some older material

Ivip documentation is pretty detailed - this is about a big topic: the first major change to the routing and addressing architecture of the Net since the early 1980s.  If you only have a few minutes, there is always the script for the Ivip TV advertisement: tv-ad/ .

Here is a Quick Introduction (mid 2007) to the Internet's routing and addressing problems, and to the most significant currently proposed solutions: LISP, eFIT-APT and Ivip.
Slides: LISP-NERD/CONS, eFIT-APT and Ivip - and some challenges common to them all
(Color diagram above added 2007-07-26, pointer to Errata added 2007-07-28.)Two tables from these slides:

General features
   SHIM6Mobile
IPv6
LISP-
NERD
LISP-
CONS or ALT
APTIvip
Address portability      YYYY
MultihomingY   YYYY
Mobility    Y         Y
IPv4 too      YYYY


Multihoming and Traffic Engineering functionality
   LISP-
NERD
LISP-
CONS or ALT
APTIvip
Address ITRs do complex communications, accept and forward ICMP etc.? Yes  Yes  Yes +
Default Mappers
No
ETRs communicate with ITRs? Yes  Yes  Yes +
Default Mappers  
No
Real time decisions for multihoming service restoration and traffic engineering. Functionality fixed in protocols, controlled by mapping data and implemented in real-time by ITRs etc.No built-in MH or load sharing functions. Open ended - relies on external systems chosen by end-user, and fast replication of mapping changes to all ITRs.
Complex communications, responding to ICMP etc.
= security problems and heavy load for router's CPU.

Since BGP's limitations are driving the need for a new architecture, please also see also the discussion of BGP's path hunting, the MRAI timer etc. in the RRG and IDR mailing lists, beginning around 2007-06-20.  I have a little guide to this discussion at:  ../sram-ip-forwarding/##BGP_hunting_MRAI_disc .

A list of messages on the RAM list regarding LISP, between 15 June and 15 July 2007 is available here: ram-messages/ .

(These pages are created with KompoZer www.kompozer.net , apart from the Internet Drafts, which were created with xml2rfc xml.resource.org . The slides were made with Microsoft PowerPoint, the diagrams with Adobe Illustrator 8 and the summary and analysis PDF with MS Word and an old version of Adobe Acrobat Distiller.)