Gateway bridge on gateway vs in cloud (latency and overhead)

ChirpStack Gateway Bridge
1

Hello, to begin, I’ve already read and understand the benefits of running the gateway bridge on each gateway as highlighted by this excellent page https://www.loraserver.io/lora-gateway-bridge/install/deployment/

I was curious if anyone on this forum had experience with a particular scenario: when the gateway is backhauled over cellular. There are two primary issues I’m concerned about.

  • Data usage: my assumption is that switching to JSON over MQTT introduces extra overhead, I don’t have enough knowledge of this topic to have a ballpark feeling about how much this would impact my cellular data consumption (billed per MB).

  • Latency and reliability: cellular is high latency and may have a higher rate of packet loss. Maybe the extra data overhead is worth it to reduce issues/worries related to high latency and potential packet loss over the “raw” JSON over UDP strategy

If anyone has any input it would be greatly appreciated!

1 Like
2

I’ve been thinking in also offering a binary format (next to JSON) for this purpose (e.g. Protobuf). This hasn’t materialized yet, but could reduce the overhead for this use case.

1 Like
3

At work, we have lot of lora devices communicating.

I have set a gateway with lora-gateway-bridge, vpn and ethernet and I have the following counters :slight_smile:

root@mtgateway0007:/var/config/lora-gateway-bridge# uptime
15:02:56 up 11 days, 6:40, 1 user, load average: 1.11, 0.91, 0.74

root@mtgateway0007:/var/config/lora-gateway-bridge# ifconfig
eth0 Link encap:Ethernet HWaddr 00:08:00:4A:0B:24
inet addr:192.168.2.23 Bcast:0.0.0.0 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2307510 errors:2 dropped:994 overruns:1 frame:2
TX packets:1283774 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:193650519 (184.6 MiB) TX bytes:365270614 (348.3 MiB)
Interrupt:23 Base address:0xc000

for me it’s totally reasonable and more efficient than plugin the lora-packet-forwarder to a cloud-based gateway bridge.

1 Like
4

Great, thanks for the numbers! With ethernet or wifi it makes total sense to have the gateway-bridge at the gateway itself, but if there’s no internet connection and you have to rely on a Bam with a sim card to get 3G or 4G, hundreds of MiB per months can be really expensive (and worse if you need an international one). So a binary format for the gateway-bridge could be quite helpful.

5

Thanks for the info. I guess to an even further extreme is satellite backhaul. I remember hearing at The Things Conference that Inmarsat (a satellite data provider) was working with Kerlink and TTN on an even lighter data exchange format between gateway and network server. Maybe @brocaar has thoughts on that?

@gpilloud do you have a rough number of devices/how often they’re transmitting? 350MB over 11 days seems like a lot!

6

@gpilloud do you have a rough number of devices/how often they’re transmitting? 350MB over 11 days seems like a lot!

We are in our office, with many legit and un legit lorawan messages on the 868 ! we are tunelling all informations inside a openvpn tunnel. In all cases, execepting in a known environment, you can have messages from others devices transmitted thru your gateway, and ignored after, by the loraserver.

7

Is there any update on the potential to binary encode meta data?

We’ve been looking at cellular backhaul usage. Presently, we’re using the UDP semtech packet forwarder with GW’s from: laird, tektelic, and gemteks. The semtech packet forwarder seems to be the most universally available for use with the server stack.

In going through the protocol details and analyzing data using tcpdump I’ve found that there is quite a bit of overhead from GW to/from Server (i.e. cellular portion of the link). I’ve seen that a 9 byte application UL message can turn into a 235 byte UDP packet. Similarly a DevStat request with 0 bytes application payload (i.e. mac only) can turn into a 218 byte UDP packet. Then, the PULL_DATA/PULL_ACK are 40/32 bytes a piece.

Basically, as a GW scales to 1000’s of devices we could be looking at 2-5Gbytes/month for backhaul data for a modest 1 hour UL rate.

So, trying to refresh this link to see if there’s been any further look at compression of metadata.

Thanks. for reference here are some examples for PULL_DATA, PULL_ACK, PULL_RESP, PUSH_DATA:

===============================================================================
17:25:59.824721 IP (tos 0x0, ttl 41, id 19041, offset 0, flags [DF], proto UDP (17), length 40)
XXXXXX.lightspeed.sndgca.sbcglobal.net.44706 > ip-XXXXXX.us-west-2.compute.internal.1700: [udp sum ok] UDP, length 12
0x0000: 4500 0028 4a61 4000 2911 b31d 6327 4013 E…(Ja@.)…c’@.
0x0010: ac1a 04f2 aea2 06a4 0014 1bed 026b f402 …k…
0x0020: 0000 1c49 7bb4 4be0 …I{.K.

17:25:59.825030 IP (tos 0x0, ttl 64, id 17922, offset 0, flags [DF], proto UDP (17), length 32)
ip-XXXXXX.us-west-2.compute.internal.1700 > XXXXXX.lightspeed.sndgca.sbcglobal.net.44706: [bad udp cksum 0x5464 -> 0xffd8!] UDP, length 4
0x0000: 4500 0020 4602 4000 4011 a084 ac1a 04f2 E…F.@.@…
0x0010: 6327 4013 06a4 aea2 000c 5464 026b f404 c’@…Td.k…

17:26:00.453341 IP (tos 0x0, ttl 232, id 48687, offset 0, flags [none], proto UDP (17), length 235)
XXXXXXXX.lightspeed.sndgca.sbcglobal.net.49155 > ip-XXXXXXXXX.us-west-2.compute.internal.1700: [udp sum ok] UDP, length 207
0x0000: 4500 00eb be2f 0000 e811 bf8b 6327 4013 E…/…c’@.
0x0010: ac1a 04f2 c003 06a4 00d7 47ba 02eb 8900 …G…
0x0020: 647f daff fe00 5183 7b22 7278 706b 223a d…Q.{“rxpk”:
0x0030: 5b7b 2274 6d73 7422 3a31 3734 3930 3230 [{“tmst”:1749020
0x0040: 3735 352c 2263 6861 6e22 3a31 2c22 7266 755,“chan”:1,“rf
0x0050: 6368 223a 302c 2266 7265 7122 3a39 3032 ch”:0,“freq”:902
0x0060: 2e35 3030 3030 302c 2273 7461 7422 3a31 .500000,“stat”:1
0x0070: 2c22 6d6f 6475 223a 224c 4f52 4122 2c22 ,“modu”:“LORA”,"
0x0080: 6461 7472 223a 2253 4637 4257 3132 3522 datr":“SF7BW125”
0x0090: 2c22 636f 6472 223a 2234 2f35 222c 226c ,“codr”:“4/5”,“l
0x00a0: 736e 7222 3a31 302e 302c 2272 7373 6922 snr”:10.0,“rssi”
0x00b0: 3a2d 3436 2c22 7369 7a65 223a 3232 2c22 :-46,“size”:22,"
0x00c0: 6461 7461 223a 2251 5062 2f47 6636 4147 data":“QPb/Gf6AG
0x00d0: 7834 4348 734b 5a2b 4c72 597a 4753 4544 x4CHsKZ+LrYzGSED
0x00e0: 3067 4d47 673d 3d22 7d5d 7d 0gMGg==”}]}

17:56:45.783850 IP (tos 0x0, ttl 64, id 54665, offset 0, flags [DF], proto UDP (17), length 218)
ip-XXXXXXXX.us-west-2.compute.internal.1700 > XXXXXXXX.lightspeed.sndgca.sbcglobal.net.46586: [bad udp cksum 0x551e -> 0xaf8e!] UDP, length 190
0x0000: 4500 00da d589 4000 4011 1043 ac1a 04f2 E…@.@…C…
0x0010: 6327 4013 06a4 b5fa 00c6 551e 024b c203 c’@…U…K…
0x0020: 7b22 7478 706b 223a 7b22 696d 6d65 223a {“txpk”:{“imme”:
0x0030: 6661 6c73 652c 2274 6d73 7422 3a31 3433 false,“tmst”:143
0x0040: 3330 3236 3231 322c 2266 7265 7122 3a39 3026212,“freq”:9
0x0050: 3236 2e33 2c22 7266 6368 223a 302c 2270 26.3,“rfch”:0,“p
0x0060: 6f77 6522 3a32 302c 226d 6f64 7522 3a22 owe”:20,“modu”:"
0x0070: 4c4f 5241 222c 2264 6174 7222 3a22 5346 LORA",“datr”:“SF
0x0080: 3742 5735 3030 222c 2263 6f64 7222 3a22 7BW500”,“codr”:"
0x0090: 342f 3522 2c22 6970 6f6c 223a 7472 7565 4/5",“ipol”:true
0x00a0: 2c22 7369 7a65 223a 3133 2c22 6461 7461 ,“size”:13,"data
0x00b0: 223a 2259 487a 4579 2f2b 426c 6745 474a ":“YHzEy/+BlgEGJ
0x00c0: 644b 464e 673d 3d22 2c22 6272 6422 3a30 dKFNg==”,“brd”:0
0x00d0: 2c22 616e 7422 3a30 7d7d ,“ant”:0}}

8

@patricksingler @elliott to give you an update:

I’m currently working on a refactored LoRa Gateway Bridge version which will be backwards compatible with LoRa Server v2. In this version it will be possible to select the serialization format. For LoRa Gateway Bridge this will be:

# Payload marshaler.
#
# This defines how the MQTT payloads are encoded. Valid options are:
# * v2_json:   The default LoRa Gateway Bridge v2 encoding (will be deprecated and removed in LoRa Gateway Bridge v3)
# * protobuf:  Protobuf encoding (this will become the LoRa Gateway Bridge v3 default)
# * json:      JSON encoding (easier for debugging, but less compact than 'protobuf')

As you can see it will be possible to switch to protobuf serialization, which should reduce the used bandwidth a lot! To still keep the possibility to debug what the gateway is sending and receiving, it will be possible to also select json. Basically protobuf and json share the same Protocol Buffers message structure, the latter one uses the Protocol Buffers JSON serialization. The “old” v2_json will the probably be removed from LoRa Gateway Bridge v2 (I think it makes sense to keep LoRa Server backwards compatible a bit longer to give people more time to upgrade their LoRa Gateway Bridge instances).

As upgrading all the gateways at once is almost impossible, LoRa Server will be able to detect the used serialization format (as this is quite easy to detect) and will use the same serialization format for downlink.

I don’t have an ETA for this yet, but I do have something working already :slight_smile: You can find the Protobuf definition (WIP) here: https://github.com/brocaar/loraserver/blob/protobuf/api/gw/gw.proto

2 Likes
9

I’ve just pushed the Protocol Buffer based gateway messages to the master branches of lora-gateway-bridge and loraserver :slight_smile:

Any of you would like to help testing this? I could push a test-release to the testing deb repository today or tomorrow.

10

I expect protobuf will help a lot. Unfortunately, I’m not in a position to test immediately. I expect I’ll be able to take a deep look at this the next time I’m able to update my infrastructure which will likely be in a month or so. I’ll definitely plan to follow up and confirm any savings that I see. Thanks for the release!