I feel like you were done dirty. When I was in grad school 12 years ago, our networking classes used mininet to simulate networks on a single host. It's mostly meant for developing SDN systems, but probably would have met your needs and supports way more.
On the other hand, building even a tiny subset but doing it yourself from scratch is a great way to learn. I made a very poor man's VM image builder for HyperV years back because Packer didn't have a builder for it at the time and that was a pretty interesting experience. Finally grokked the Windows object model and even though I still don't use it, I at least no longer jeer at PowerShell.
I'm interested in the answer to your question, too, but as a customer of an ISP. I don't work for one. I was the first owner of my house and when they hooked me into their network, whoever did messed up my neighbors badly, putting them on the wrong circuit and bleeding noise into adjacent neighborhoods. For three years, complaint calls would get our network cut by third-party contractors with no warning, then we'd have to call and get it reconnected. I don't know how they're supposed to do it, but know it can cause quite a mess when they do it wrong.
Mininet did help me a lot during the initial phases. The main reason I made the switch to containernet(mininet-fork with containers ) and then to containerlab was because I wanted to run an actual NOS image as part of my topology. That was really what pushed me to try and switch to other options.
Yup, its a different experience. Sometimes, you end up learning something you never even intended to.
As for the circuit planning, I'd guess that they have circuits map on something like Netbox and using a intermediary system that maps a customer's location to the nearest circuit. Though, I don't know how they handle the optimization side of it to prevent cases like what happened to your neighbour.
Forgive my ignorance, this isn't my strong suit. Am I correct in understanding that this is mostly a simulation layer for the actual physical network, but that you're mostly(?) running off-the-shelf software on top? So this is running the same software that you'd use for a real ISP network, just without having to actually provision all the hardare? Or is part of the actual network management custom as well?
Hello. Containerlab gives me the virtual network topology ( links through veth pairs, containers etc.). The actual BNG's Control plane ( authentication, authorization, session handling, traffic shaping, events streaming etc. ) is written by me. So it's less running off-the-shelf software running on virtualized hardware, and more writing the software and running it on a virtualized hardware.
At some point, I did use Nokia SR Linux as my access node + relay, but had issues with configuration and Option 82. Later, I wrote one myself.
> - Currently, the circuit where the user connects is arbitrarily decided by the demo user. In a real system with thousands of circuits, it'd be very difficult to properly assess which circuit the customer might connect to. When adding a new customer to a service, how does the operator decide, based on customer's location, which circuit to provide the service to ?
I'm not exactly sure what you're asking, but port allocation is, depending on the ISP's deployment model, either going to be fixed at the time the infrastructure was built, or whoever is doing the last metre install will choose a random available port on the switch. The subscriber will be assigned to that port in the RADIUS or equivalent database, and the BNG will query the subscriber based on DHCP Option 82 port information added by the switch. You could also map the subscriber based on MAC address, but this doesn't really work unless you don't support customer provided equipment on their end.
My access edge is injecting DHCP Option 82, and I'm mapping customers based on (bng_id + circuit_id + remote_id ). Say, a customer on Oakwood Drive ABC wants a service. What is the process of finding the right circuit between storing the customer's desired address and finding the best circuit to connect it to? Since, as mentioned in this thread, having connected to a wrong circuit can cause network noise for other customers too, how is the "cleanest" circuit + port assigned to a customer in a location ?
Depends on the access technology and environment. But usually there is not much choice to be made, by design. The cable or equivalent from the customer prem will go to exactly one aggregation location, and in that location, the choice of port generally doesn't matter. Among the potentially multiple cables or ports, they're all meant to be functionally the same. Maybe something is wrong with a cable or port, and that will hopefully come out in post-install testing, but there's not meant to be much of a decision to be made for commodity service like DSL or GPON (anything that'd use BNG). It's typically just going to be up to the last metre installer.
Metro ethernet services will be designed by an engineering team on a case-by-case basis, but they very rarely if ever use BNG.
I reached out to the maintainer of osvbng, and he was really receptive about it. I may contribute to the project in a few months.
Meanwhile, I also found another project that works with eBPF-accelerated BNGs for k8s edge deployments. https://github.com/codelaboratoryltd/bng (BUSL-1.1 license though)
Thank you for listing me the projects. Really helped me out!!
That is awesome! I was walking almost blind folded here, making decisions based on observations, intuition, some blogs, and RFC. These projects give me something to look against as I further develop my skills.
I recently found out about Apache Netbox that would act as the authoritative source of truth for the network topology and replace majority of aether.config.yaml. In Splynx, I did not see any mention of an external solution. It seems they have their own stack for that.
A better and UX-friendly implementation would have been Netbox + aether.config.yaml -> configuration pipeline -> topology.yaml + <other generated files>.
this looks pretty interesting! i plan to take a closer look after work, but thought i would mention it now: it may be worth a look through the NANOG (north american network operators group) archives (https://nanog.org/nanog-mailing-list/list-archives/) for information around your question if you havent, and/or posting your question to the NANOG mailing list. there are many very friendly people who have experience running ISPs of all sizes.
(or whichever operators group best fits your area. i only subscribe to NANOG, so cant speak to the activity/friendliness of the other groups. you can find a pretty comprehensive list here: https://nanog.org/resources/organizations-our-community/)
Subscribed! I am very new to things here since it's barely been a 1.5 months since I touched this area, so many of my questions have probably been answered. I will search around and post a question if can't find an answer.
I plan to take all the feedback I can this week, and work on them on spring break.
This brings back fond memories of my first job real job in IT, as the sysadmin for a small boutique mom-n-pop ISP. This was dialup/ISDN days though (back in the late 90's).
I appreciate the compliment. I wish this type of knowledge was more easily available for the general public since it represents an integral part of modern day internet. A comment on this thread mentioned other similar ongoing project which I'm very happy about and excited to explore.
I am surprised the author did not mention or uses Software Defined Networking (SDN), Openflow or P4 (programming language for programmable switches) or the mininet simulator. He must have skipped reading the scientific literature even though he is a computer science sophomore?
I programmed and build one of the very first ISP hardware and software systems in 1987-1997 when we connected the first submarine link between the US and Europe in Amsterdam.
Google switched 50% of the internet that they owned in 2012 to SDN and Openflow [1]. I'm sure they progressed to P4 and more recent SDN controllers since then. They build the Google Fiber ISP[5] with SDN. Cloudflare also uses SDN when last I checked. A majority of the internet has moved to SDN (there are many versions.
The author built his simulation on legacy systems mostly from the Telecom world, an alternate reality distinct from the real internet and acces providers we call ISPs. Telecom systems are about surveillance and monetizing the free internet.
You can query the US ISPs on the Nanog mailing list, there are similar social media for the European, Asian and other ISPs on other continents. Beware that those are biased to Telecom as well as Tier 1 network operators and less to ISP access providers.
I do not think we should continue with the current implementation of the internet. I think we should start deploying the true internet (decentralized, peer to peer) standard and expand it to the Enernet standards of the near future: every building a router (switch) and fiber optic and electricity cables to their peers; their closest neighbors. If every building has peer connections than you are connected all the way to the internet exchanges without need for Tech Bros, Government, Telecom, ISP or Tier 1 network oligopolies. True internet [3], true Enernet [4].
well.. openflow is pretty much dead, too inflexible, too slow. The whole control/user plane split is an attempt of the classical router vendors to keep their proprietary boxes. It adds complexity as it requires to synchronize the state of some controller with some data plane box.
P4 was a great idea, but there's not much hardware that supports it.
fd.io / vpp is an impressive stack for software-only routing. Like all SW-only solutions, it suffers from high power consumption and packet rate variability. At today's packet rates, you always have to ask 'how many CPU instructions / cycles are required to perform this or that function per packet'.
Thank you for the context. I did start out with mininet, then moved to containernet->containerlab. Mininet could not model subscriber session lifecycle in how I wanted it. P4/Openflow is on the radar, thanks for the pointer.
I feel like you were done dirty. When I was in grad school 12 years ago, our networking classes used mininet to simulate networks on a single host. It's mostly meant for developing SDN systems, but probably would have met your needs and supports way more.
On the other hand, building even a tiny subset but doing it yourself from scratch is a great way to learn. I made a very poor man's VM image builder for HyperV years back because Packer didn't have a builder for it at the time and that was a pretty interesting experience. Finally grokked the Windows object model and even though I still don't use it, I at least no longer jeer at PowerShell.
I'm interested in the answer to your question, too, but as a customer of an ISP. I don't work for one. I was the first owner of my house and when they hooked me into their network, whoever did messed up my neighbors badly, putting them on the wrong circuit and bleeding noise into adjacent neighborhoods. For three years, complaint calls would get our network cut by third-party contractors with no warning, then we'd have to call and get it reconnected. I don't know how they're supposed to do it, but know it can cause quite a mess when they do it wrong.
Thank you for the comment.
Mininet did help me a lot during the initial phases. The main reason I made the switch to containernet(mininet-fork with containers ) and then to containerlab was because I wanted to run an actual NOS image as part of my topology. That was really what pushed me to try and switch to other options.
Yup, its a different experience. Sometimes, you end up learning something you never even intended to.
As for the circuit planning, I'd guess that they have circuits map on something like Netbox and using a intermediary system that maps a customer's location to the nearest circuit. Though, I don't know how they handle the optimization side of it to prevent cases like what happened to your neighbour.
Forgive my ignorance, this isn't my strong suit. Am I correct in understanding that this is mostly a simulation layer for the actual physical network, but that you're mostly(?) running off-the-shelf software on top? So this is running the same software that you'd use for a real ISP network, just without having to actually provision all the hardare? Or is part of the actual network management custom as well?
Hello. Containerlab gives me the virtual network topology ( links through veth pairs, containers etc.). The actual BNG's Control plane ( authentication, authorization, session handling, traffic shaping, events streaming etc. ) is written by me. So it's less running off-the-shelf software running on virtualized hardware, and more writing the software and running it on a virtualized hardware.
At some point, I did use Nokia SR Linux as my access node + relay, but had issues with configuration and Option 82. Later, I wrote one myself.
> - Currently, the circuit where the user connects is arbitrarily decided by the demo user. In a real system with thousands of circuits, it'd be very difficult to properly assess which circuit the customer might connect to. When adding a new customer to a service, how does the operator decide, based on customer's location, which circuit to provide the service to ?
I'm not exactly sure what you're asking, but port allocation is, depending on the ISP's deployment model, either going to be fixed at the time the infrastructure was built, or whoever is doing the last metre install will choose a random available port on the switch. The subscriber will be assigned to that port in the RADIUS or equivalent database, and the BNG will query the subscriber based on DHCP Option 82 port information added by the switch. You could also map the subscriber based on MAC address, but this doesn't really work unless you don't support customer provided equipment on their end.
My access edge is injecting DHCP Option 82, and I'm mapping customers based on (bng_id + circuit_id + remote_id ). Say, a customer on Oakwood Drive ABC wants a service. What is the process of finding the right circuit between storing the customer's desired address and finding the best circuit to connect it to? Since, as mentioned in this thread, having connected to a wrong circuit can cause network noise for other customers too, how is the "cleanest" circuit + port assigned to a customer in a location ?
Depends on the access technology and environment. But usually there is not much choice to be made, by design. The cable or equivalent from the customer prem will go to exactly one aggregation location, and in that location, the choice of port generally doesn't matter. Among the potentially multiple cables or ports, they're all meant to be functionally the same. Maybe something is wrong with a cable or port, and that will hopefully come out in post-install testing, but there's not meant to be much of a decision to be made for commodity service like DSL or GPON (anything that'd use BNG). It's typically just going to be up to the last metre installer.
Metro ethernet services will be designed by an engineering team on a case-by-case basis, but they very rarely if ever use BNG.
Thank you! That answers my question. I appreciate your feedback.
Thanks for sharing! I am happy to see open-source BNG projects taking off in the last few months. These are a couple of others to look at:-
https://github.com/codelaboratoryltd/bng-edge-infra
https://github.com/veesix-networks/osvbng
I reached out to the maintainer of osvbng, and he was really receptive about it. I may contribute to the project in a few months.
Meanwhile, I also found another project that works with eBPF-accelerated BNGs for k8s edge deployments. https://github.com/codelaboratoryltd/bng (BUSL-1.1 license though)
Thank you for listing me the projects. Really helped me out!!
That is awesome! I was walking almost blind folded here, making decisions based on observations, intuition, some blogs, and RFC. These projects give me something to look against as I further develop my skills.
I recently found out about Apache Netbox that would act as the authoritative source of truth for the network topology and replace majority of aether.config.yaml. In Splynx, I did not see any mention of an external solution. It seems they have their own stack for that.
A better and UX-friendly implementation would have been Netbox + aether.config.yaml -> configuration pipeline -> topology.yaml + <other generated files>.
IMO doing yourself a disservice by not prioritizing competency in IPv6
this looks pretty interesting! i plan to take a closer look after work, but thought i would mention it now: it may be worth a look through the NANOG (north american network operators group) archives (https://nanog.org/nanog-mailing-list/list-archives/) for information around your question if you havent, and/or posting your question to the NANOG mailing list. there are many very friendly people who have experience running ISPs of all sizes.
(or whichever operators group best fits your area. i only subscribe to NANOG, so cant speak to the activity/friendliness of the other groups. you can find a pretty comprehensive list here: https://nanog.org/resources/organizations-our-community/)
Subscribed! I am very new to things here since it's barely been a 1.5 months since I touched this area, so many of my questions have probably been answered. I will search around and post a question if can't find an answer.
I plan to take all the feedback I can this week, and work on them on spring break.
Thank you for sharing. This is really cool and way more than I accomplished as a sophomore in CS. Keep it up!
Thank you. I'm exploring more on this. Getting feedback from people here has hyped me up even more for the next step.
This brings back fond memories of my first job real job in IT, as the sysadmin for a small boutique mom-n-pop ISP. This was dialup/ISDN days though (back in the late 90's).
Good job!
<sniff>
I appreciate the compliment. I wish this type of knowledge was more easily available for the general public since it represents an integral part of modern day internet. A comment on this thread mentioned other similar ongoing project which I'm very happy about and excited to explore.
You ask for feedback:
I am surprised the author did not mention or uses Software Defined Networking (SDN), Openflow or P4 (programming language for programmable switches) or the mininet simulator. He must have skipped reading the scientific literature even though he is a computer science sophomore?
I programmed and build one of the very first ISP hardware and software systems in 1987-1997 when we connected the first submarine link between the US and Europe in Amsterdam.
Google switched 50% of the internet that they owned in 2012 to SDN and Openflow [1]. I'm sure they progressed to P4 and more recent SDN controllers since then. They build the Google Fiber ISP[5] with SDN. Cloudflare also uses SDN when last I checked. A majority of the internet has moved to SDN (there are many versions.
The author built his simulation on legacy systems mostly from the Telecom world, an alternate reality distinct from the real internet and acces providers we call ISPs. Telecom systems are about surveillance and monetizing the free internet.
You can query the US ISPs on the Nanog mailing list, there are similar social media for the European, Asian and other ISPs on other continents. Beware that those are biased to Telecom as well as Tier 1 network operators and less to ISP access providers.
I do not think we should continue with the current implementation of the internet. I think we should start deploying the true internet (decentralized, peer to peer) standard and expand it to the Enernet standards of the near future: every building a router (switch) and fiber optic and electricity cables to their peers; their closest neighbors. If every building has peer connections than you are connected all the way to the internet exchanges without need for Tech Bros, Government, Telecom, ISP or Tier 1 network oligopolies. True internet [3], true Enernet [4].
[1] OpenFlow @ Google - Urs Hoelzle, https://www.youtube.com/watch?v=VLHJUfgxEO4
[2] The Future of Networking, and the Past of Protocols - Scott Shenker https://www.youtube.com/watch?v=YHeyuD89n1Y
[3] Fiberhood White Paper https://www.researchgate.net/profile/Merik-Voswinkel/publica...
[4] Enernet - Bob Metcalfe https://www.youtube.com/watch?v=axfsqdpHVFU
[5] Google Fiber build "Fiberhoods" but my own Enernet ISP Fiberhood had trademarked that name before in 2011.
well.. openflow is pretty much dead, too inflexible, too slow. The whole control/user plane split is an attempt of the classical router vendors to keep their proprietary boxes. It adds complexity as it requires to synchronize the state of some controller with some data plane box.
P4 was a great idea, but there's not much hardware that supports it.
fd.io / vpp is an impressive stack for software-only routing. Like all SW-only solutions, it suffers from high power consumption and packet rate variability. At today's packet rates, you always have to ask 'how many CPU instructions / cycles are required to perform this or that function per packet'.
Thank you for the context. I did start out with mininet, then moved to containernet->containerlab. Mininet could not model subscriber session lifecycle in how I wanted it. P4/Openflow is on the radar, thanks for the pointer.
[dead]