• shorter links
• the need for more wired or wireless backhaul
• unpredictable service due to the large interference domain

To provide longer range communications, a PtMP or PtP system can be used. This is normally used for applications such as fixed wireless access to homes or businesses, and smart grid backhaul, especially in medium density to rural areas. But problems with this model include:

• a lack of redundancy due to each client normally seeing only a single base station
• many base stations are required due to single hop
• the need to engineer each link
• incomplete coverage (each client must have a direct path to a base station, so some installations may be completely obstructed, and the network must be built very densely to minimize this)

To address some of these issues with each type of system, an architecture started to emerge a few years ago which combined PtMP backhaul along with omni-directional mesh. However, not only does this require two different solutions and sets of equipment, but the many of the issues are inherited from each type of system. For instance, while a subscriber connecting to the short-range mesh may benefit from the many mesh nodes available to choose from, there is still a need to engineer the backhaul links and there are still issues around interference with the short-range mesh. The PtMP system would still need to be built very densely to provide sufficient coverage and in order to compensate for the single hop links, and there still may be coverage holes due to obstructions. Also, the PtMP system lacks redundancy. And although it may be possible to use multiple meshes to heal around back-haul outages, this requires complex dynamic routing to be run between the backhaul network and the short-range mesh, and requires multiple adjacent short-range meshes which may not be present in situations where the meshes are islands within a larger sparse network.

One of the reasons that we chose to implement dynamic antenna pointing was to address both of these network architecture issues by providing a single system that can do both long-range backhaul and short-range meshing. In fact, while our first internal testbed ran over 7 hops that ranged from 10 yards to 300 yards, our first customer deployment connected mountain tops across 20 mile links. Over the shorter links the dynamically switched antennas have the isolation needed to avoid interference and to provide spectral reuse, while over longer links the antennas provide the gain needed to close the links at a decent modulation. These very different deployments use the same hardware, same protocol and exactly the same configuration – the only difference is the deployment locations.

Below are snapshots of two live deployments, one mostly PtMP (with one SkyExtender relay) and one dense mesh. The PtMP system has a mixture of links, from short to several miles, while the dense mesh has links of mostly under 100 yards. These systems are running equivalent hardware (although DualBands are used in the dense mesh), the same software, with the same basic configuration.

In some rural areas there is even a hybrid model that some customers use with the SkyPilot equipment where pockets of dense subscribers, connected to each other using short links, are interconnected using long distance links. For example, there are areas of rural Germany where a single SkyGateway connects over long links to SkyExtenders in different villages, which then mesh over shorter links with other SkyExtenders and SkyConnectors within the villages.

Bookmark to:

Issues with layer 3 (IP):

• Only IP is transported: no AppleTalk, PPPoE, broadcast device discovery, or legacy Ethernet devices such as serial/Ethernet telemetry…
• No virtual LAN services: offering layer 2 pipes for virtual LAN services has become an extremely important offering for many service providers. IP networks do not inherently support this service, and additional equipment or protocols need to be layered on top in order to support it.
• Many layer 3 systems only support IPv4: IPv6 is very different from IPv4, so support needs to be explicitly added, and IPv6 is critical for large networks such as smart grids.
• IP demarcation issues: the interface from the wireless access equipment must support whatever dynamic routing protocol the operator has chosen (RIP, OSPF, BGP-4, …). And, the operator may need to run a dynamic IP routing protocol in order to support client mobility (while an Ethernet system would allow learning switches to interconnect gateways for fast, transparent roaming.)
• IP multicast support (for some types of video streaming) needs to be explicitly supported. IP Multicast forwarding is very different than regular IP forwarding, and involves different protocols.
• Slower re-routing: compared to Ethernet table learning, IP dynamic routing is slow.

Issues with layer 2 (Ethernet):

• Scalability limitations due to a large broadcast domain and Ethernet learning table size restrictions of external switches.
• Inter-subscriber security concerns due to layer 2 attacks directly between subscribers (ARP poisoning, rogue DHCP servers, …).
• Subscriber-to-network security concerns from Ethernet MAC address spoofing and ARP poisoning.

Since layer 3 is a higher layer protocol than layer 2, it seems to become a question of limitations versus problems. Is it better to live with the limitations of an IP transport or with the problems of an Ethernet transport?

To deal with the issue of MAC address scalability, fortunately switch learning tables have greatly increased in size. And even if an Ethernet learning table overflows, the standard behavior is to replace the oldest entry, which is often from an inactive device. And data is still forwarded in any case, so the total number of devices supported on a network is much larger than the size of the switches’ Ethernet learning tables.

And to deal with both the large broadcast domain issue and lack of security between subscribers due to potential layer 2 attacks, many switches have a feature called “protected ports” (which SkyPilot has implemented as “Peer to Peer Control”). This feature can selectively block layer 2 forwarding between ports and VLANs of an Ethernet switch, or between subscribers within a virtual LAN within the SkyPilot system, in cases where the users of those ports or VLANs are not from the same administrative domain (for example, not employees of the same company). And since this control can be done on a VLAN basis, an operator can use this control to provide some groups of subscribers direct layer 2 access while limiting the layer 2 access of other users, such as home Internet subscribers, to only the router that leads to the Internet.

And even if users of different protected ports or VLANs need to communicate at layer 3 (for some cases of VoIP, gaming, file sharing, …) then several simple methods are available to allow that communication at layer 3 or above, such as the “local proxy ARP” feature of most routers or /30 IP subnetting at the subscriber level.

So with the control of protected ports and VLANs in both the wireless system and any external network switches, the potential of attacks between subscribers (such as ARP poisoning and rogue DHCP servers) can be completely avoided, and the only attacks left are attacks directly from subscribers to the network (such as to the first hop router). These layer 2 attacks fall into two specific cases: MAC spoofing and ARP poisoning. In both of these attacks one user intentionally mimics the Ethernet MAC address of another user, which causes a temporary Denial of Service (DoS). These can not effectively be used as data intercept attacks, so data is not compromised. And the denials of service are extremely short, especially in the case of MAC address spoofing where the attack only lasts until the real user sends a single Ethernet frame. And since the attacker is easily identified and their access can simply disabled, these attacks are not actually very common, and are ineffective.

And an alternative or supplementary tool that an operator can use to address many of these issues is filtering. SkyPilot devices support filters that range from the Ethernet MAC layer up to the IP port level. For example, instead of (or in addition to) disabling peer to peer communication using protected ports, an operator can simply configure UDP port filters to prevent rogue DHCP servers.

So by having addressed these Ethernet scalability and security concerns, the edge network can take advantage of the benefits of an Ethernet transport, including:

• Simple IP address management: IP addresses can be handed out in a number of ways (DHCP, static, PPPoE, …) and they can be assigned independently of the point of attachment.
• Support for any Ethernet device, such as IPv4, IPv6, IP multicast, NetBIOS, AppleTalk,  and legacy Ethernet devices.
• Seamless, fast intra-network mobility.
• Virtual LAN services (private LANs can be configured across the network).
• Simple layer 2 demarcation at the base-station (no IP routing protocol requirements).

An important aspect of Ethernet is that using it as a transport method does not mean a lack of IP services. IPv4, IPv6 and virtually every layer 3 protocol has an Ethernet convergence function, so if a device talks Ethernet then it can run over an Ethernet transport system without any special support from the network devices. And, even if a device such as a wireless mesh node provides Ethernet transport, it can also include an IP stack for its own communication, such as remote management. And IP-aware filters can be added to devices that are providing only an Ethernet transport service. So, Ethernet transport does not mean “no IP”.

Bookmark to:

This diagram shows how a traditional single radio mesh system has its bandwidth reduced due to a large interference domain allowing only a single device to transmit at a time (note: the circles show the communication range, while the interference range will usually have a radius many times larger).

A multi-radio system could use several frequencies to allow multiple transmissions to take place at the same, reducing some of these interference conditions (however, not only does this require multiple clean channels, but there are some pitfalls that will be analyzed in a future post).

So an obvious question is, “How does SkyPilot’s dynamic antenna switching affect system capacity?” The answer is that even though the SkyPilot system uses a single backhaul radio, it can still provide 1/n the channel capacity simultaneously to each device due to the dynamic antenna switching.

In addition to all of the previously discussed benefits of dynamic antenna switching, such as higher link budget, interference avoidance and point-to-point power levels, the largest benefit is probably from something called “spectral re-use”. Basically, spectral re-use is a benefit of using dynamically switched high-gain antennas where multiple transmissions can take place simultaneously, on the same frequency, in very close proximity.

For example, the dynamic point-to-point link formed by the high-gain antennas allows a first-hop transmission to not interfere with a third-hop reception, even on the same channel. And while one first-hop device is relaying, spectral re-use allows many other devices to simultaneously communicate, such as allowing the gateway to transmit to another first-hop device. That is why we always recommend at least 2 first-hop devices. This allows the gateway, and most other devices within the mesh, to be continuously active, so the capacity of the overall system is equal to the capacity of the gateway radio.  This allows at least 1/n to be delivered to each device simultaneously, equivalent to the multi-radio mesh system and much higher than traditional single radio systems.

And by only consuming a single channel, additional channels can be employed in order to multiply overall system capacity (plus, it is often difficult to find the multiple clean channels that multi-radio architectures require). But, the use of multiple radios in context of traditional mesh networks and the SkyPilot system will be explored in a future post.

Bookmark to: