25 years ago, I worked for a small ISP (back when there was such a thing.) When I started there, we had one upstream ISP. I was charged with getting us multihomed. I found some tutorials written by Avi Freedman (1). I don't know what I would have done without him. He made an intimidating topic approachable. Thanks to him, I got us a /20 from ARIN and advertised our routes to two different peers. It was fascinating to learn how it all worked. And the more I learned about it, the more amazed I was that it worked at all.
Like the human body the more you study the Internet the more amazing it is not that it sometimes breaks, but that it works at all. Especially for video/phone/etc.
Glad the content was helpful, I have links to some of them at avi.net (tutorials and old Boardwatch articles).
I swear my motive was pure (frustration with the content out there) but it was easy to see back then that helping people out with good content yields rewards ("Can I buy a T1?") or ("Come run my big global network"). So I still encourage everyone to write about what's confusing and frustrating...
I think I sent you an e-mail back then, but let me take this opportunity to heap a bunch of praise and gratitude upon you in front of God and everybody. I bought several books on BGP at the time and slogged through them, but it was intimidating and I had a lot of trouble putting the pieces together. Your tutorials were simple, straightforward and helped me an awful lot. And the other thing about the way you wrote them that I really appreciated was that you didn't make me feel inferior for not knowing WTF I was doing. Thanks Avi. From time to time, I think back on the challenges I had in my life/career and I'm grateful for the people who helped me. You're definitely on that list, sir.
That flapping from EpicUp 140.99.244.0/23 prefix should have been subject to route dampening. This is per peer or per prefix rate limiting typically enforced on all peers by ISPs to prevent this exact issue of a single prefix making up a significant portion of the global BGP churn.
I’m unconvinced of the correlation between the updates that the author attributed to knock on effects. It would be pretty janky to have your advertisements be based on the path to other autonomous systems’ prefixes, especially unstable ones.
I don’t think there is a 40 minute periodicity either (at least there wasn’t 8 years ago when I was deep in the BGP world). Smells like what this dataset happened to show either by luck or because of the network the author was getting the BGP feed from.
If you dig into the data and look at which AS’s and prefixes are experiencing changes, you’ll find it’s all over the place and there isn’t really any bigger pattern.
On any given day there are usually a few noisy ISPs because of bad circuits or misconfigurations. Then there are new prefixes flapping in and out as a new thing is brought online for the first time, etc. Then sprinkle in path changes for regular draining maintenance, etc.
It’s simultaneously both fascinating and a little horrifying how a little ISP in Kansas experiencing a fiber consuming backhoe shows up on routers in Perth. Yet the frequency of updates is kept to <10hz globally through tons of hand tuned policies.
Route dampening has mostly fallen out of fashion with networks these days.
Most setups were horribly misconfigured and (most) routers are no longer extremely CPU starved as they once were, That doesn't mean that it does not still exist of course, when I did bgp battleships ( https://blog.benjojo.co.uk/post/bgp-battleships ) I found that 3356 (at the time) was doing route dampening, so play had to be paused for a while.
The direct peering to the router is likely going to have a bad time, but route advertisement interval I mention in the article is going to coalesce all of those updates together. Downstream peers would only see the one update every 30 seconds (or so).
That’s only true if they can be coalesced. Even with RPKI an intermediate transit router can path length flap 100,000 routes every 30 second interval.
Depending on the RA interval alone is negligence and if you encountered a small ISP that isn’t dampening your updates directly, their peering session is at risk with any of the major transit providers.
Route dampening guardrails were super common 7 years ago and there isn’t any technological development that fixes what they did so I highly doubt they fell out of favor.
See my adjacent comment to yours. I would like to see why you think dampening is out of favor. Interval batching is not an equivalent protection. If you were playing BGP battleships you were likely playing at a rate where a single prefix was not updating more than once per minute.
That wouldn’t land in the dampening levels that were normally configured that encountered with all of the transit providers.
If anyone wants to learn about BGP (especially day-to-day stuff for peering scenarios), the Network Startup Resource Center out of U.Oregon has a good series of videos going through things:
Based on some quick sleuthing, I would assume that the 0xff reserved BGP attribute is likely a huawei quirk. Almost all of the 0xff's visible to bgp.tools (hi) follow the same format as the one in the post, and some of those networks with them seem to be running huawei kit.
I once designed and configured a satellite-microwave hybrid network for a large US customer with field offices around Borneo. I’ll never forget making the leased line handoffs back in Jakarta. I had exactly zero experience doing this so I googled around and read that BGP is what you would use to connect between our OSPF/UBNT net and their IGRP/Cisco corporate WAN. When asked to configure BGP on their routers, the guys from Tata were like “what do you think you are AT&T or something?” We did kind of feel that way until one season of lightning strikes took out most of our AirFibers.
I wrote a python script to extract data from an MRT file containing BGP routes from [1] and importing it into Neo4j for exploration. This file contained about 56 million extremely redundant routes and Neo4j is great for "merging" this type of data.
I have had a project on the back burner for about a year now to offer a BGP feed via a websocket to facilitate people playing around and doing research without allowing them to accidently spew crap into the DFZ. Shoot me an email if you are interested and I'll try to get it spun up this week.
Generally, your upstream won't allow you to spew crap. Route filters are in place. I run a small AS for hobbyist purposes and all my upstreams are locked down.
It would be nice for the Memory Safety initiative[1] which tackles core internet infrastructure security and safety via re-implementing it in Rust to also work on implementing the BGP server implementation.
25 years ago, I worked for a small ISP (back when there was such a thing.) When I started there, we had one upstream ISP. I was charged with getting us multihomed. I found some tutorials written by Avi Freedman (1). I don't know what I would have done without him. He made an intimidating topic approachable. Thanks to him, I got us a /20 from ARIN and advertised our routes to two different peers. It was fascinating to learn how it all worked. And the more I learned about it, the more amazed I was that it worked at all.
(1) http://avi.freedman.net/
Avi
Thanks!
Like the human body the more you study the Internet the more amazing it is not that it sometimes breaks, but that it works at all. Especially for video/phone/etc.
Glad the content was helpful, I have links to some of them at avi.net (tutorials and old Boardwatch articles).
I swear my motive was pure (frustration with the content out there) but it was easy to see back then that helping people out with good content yields rewards ("Can I buy a T1?") or ("Come run my big global network"). So I still encourage everyone to write about what's confusing and frustrating...
I think I sent you an e-mail back then, but let me take this opportunity to heap a bunch of praise and gratitude upon you in front of God and everybody. I bought several books on BGP at the time and slogged through them, but it was intimidating and I had a lot of trouble putting the pieces together. Your tutorials were simple, straightforward and helped me an awful lot. And the other thing about the way you wrote them that I really appreciated was that you didn't make me feel inferior for not knowing WTF I was doing. Thanks Avi. From time to time, I think back on the challenges I had in my life/career and I'm grateful for the people who helped me. You're definitely on that list, sir.
I used to work for Avi at Kentik. He is a smart, nice person and remembered writing these articles fondly to help people out!
> Avi
Somewhat off-topic, I’ve noticed on HN that sometimes comments end with single word on a newline, like you have done here.
Did your comment get truncated somehow, or did you mess up a copy/paste, or what?
Has anyone else noticed this? Is this a telltale sign of generated comments, or other tool usage?
I’ve only noticed this on HN, and maybe Reddit once? It happens somewhat more regularly than I can attribute to mere coincidence or accident.
Interesting. Not sure how that happened. I thought I ended the post with the URL. You may be on to something.
Nice article.
That flapping from EpicUp 140.99.244.0/23 prefix should have been subject to route dampening. This is per peer or per prefix rate limiting typically enforced on all peers by ISPs to prevent this exact issue of a single prefix making up a significant portion of the global BGP churn.
I’m unconvinced of the correlation between the updates that the author attributed to knock on effects. It would be pretty janky to have your advertisements be based on the path to other autonomous systems’ prefixes, especially unstable ones.
I don’t think there is a 40 minute periodicity either (at least there wasn’t 8 years ago when I was deep in the BGP world). Smells like what this dataset happened to show either by luck or because of the network the author was getting the BGP feed from.
If you dig into the data and look at which AS’s and prefixes are experiencing changes, you’ll find it’s all over the place and there isn’t really any bigger pattern.
On any given day there are usually a few noisy ISPs because of bad circuits or misconfigurations. Then there are new prefixes flapping in and out as a new thing is brought online for the first time, etc. Then sprinkle in path changes for regular draining maintenance, etc.
It’s simultaneously both fascinating and a little horrifying how a little ISP in Kansas experiencing a fiber consuming backhoe shows up on routers in Perth. Yet the frequency of updates is kept to <10hz globally through tons of hand tuned policies.
Route dampening has mostly fallen out of fashion with networks these days.
Most setups were horribly misconfigured and (most) routers are no longer extremely CPU starved as they once were, That doesn't mean that it does not still exist of course, when I did bgp battleships ( https://blog.benjojo.co.uk/post/bgp-battleships ) I found that 3356 (at the time) was doing route dampening, so play had to be paused for a while.
That seems crazy to me. What guardrails are there against a single hacked router pumping 10000 path changes/sec?
The direct peering to the router is likely going to have a bad time, but route advertisement interval I mention in the article is going to coalesce all of those updates together. Downstream peers would only see the one update every 30 seconds (or so).
That’s only true if they can be coalesced. Even with RPKI an intermediate transit router can path length flap 100,000 routes every 30 second interval.
Depending on the RA interval alone is negligence and if you encountered a small ISP that isn’t dampening your updates directly, their peering session is at risk with any of the major transit providers.
Route dampening guardrails were super common 7 years ago and there isn’t any technological development that fixes what they did so I highly doubt they fell out of favor.
Yup, unless that component has been disabled (which is quite rare) or the other side is bird, a bgpd that doesn't buffer anything !
See my adjacent comment to yours. I would like to see why you think dampening is out of favor. Interval batching is not an equivalent protection. If you were playing BGP battleships you were likely playing at a rate where a single prefix was not updating more than once per minute.
That wouldn’t land in the dampening levels that were normally configured that encountered with all of the transit providers.
If anyone wants to learn about BGP (especially day-to-day stuff for peering scenarios), the Network Startup Resource Center out of U.Oregon has a good series of videos going through things:
* https://learn.nsrc.org/bgp
Based on some quick sleuthing, I would assume that the 0xff reserved BGP attribute is likely a huawei quirk. Almost all of the 0xff's visible to bgp.tools (hi) follow the same format as the one in the post, and some of those networks with them seem to be running huawei kit.
I learnt quite a few things I didn’t know about BGP from this article, probably most interesting how chaotic it is!
I’d definitely be interested to read some follow-ups, diving into more details.
I once designed and configured a satellite-microwave hybrid network for a large US customer with field offices around Borneo. I’ll never forget making the leased line handoffs back in Jakarta. I had exactly zero experience doing this so I googled around and read that BGP is what you would use to connect between our OSPF/UBNT net and their IGRP/Cisco corporate WAN. When asked to configure BGP on their routers, the guys from Tata were like “what do you think you are AT&T or something?” We did kind of feel that way until one season of lightning strikes took out most of our AirFibers.
I wrote a python script to extract data from an MRT file containing BGP routes from [1] and importing it into Neo4j for exploration. This file contained about 56 million extremely redundant routes and Neo4j is great for "merging" this type of data.
[1] https://data.ris.ripe.net/rrc00/
What is the easiest way for average joe to get hands on BGP data? If I wanted to try do similar analysis and don't happen to have a friend at ISP.
* http://archive.routeviews.org
* https://www.ripe.net/analyse/internet-measurements/routing-i...
* https://lukasz.bromirski.net/post/bgp-w-labie-3/
https://ris.ripe.net/docs/route-collectors/
I have had a project on the back burner for about a year now to offer a BGP feed via a websocket to facilitate people playing around and doing research without allowing them to accidently spew crap into the DFZ. Shoot me an email if you are interested and I'll try to get it spun up this week.
Generally, your upstream won't allow you to spew crap. Route filters are in place. I run a small AS for hobbyist purposes and all my upstreams are locked down.
> What is apparent in both the path and IP space changes over time is that there is some sort of cyclic behaviour in the IPv4 updates.
does this imply the internet has something akin to tides ?
It would be nice for the Memory Safety initiative[1] which tackles core internet infrastructure security and safety via re-implementing it in Rust to also work on implementing the BGP server implementation.
[1] https://www.memorysafety.org/
as8772 NetAssist routes some pretty bad traffic
AS15626 GREEN FLOID LLC (russian propaganda) AS34224 Neterra Ltd. (bullet proof) AS44477 STARK INDUSTRIES SOLUTIONS LTD (bullet proof) AS207656 Epinatura LLC (bullet proof)
/me thinks to himself : real nice font on that page