Packet Forwarder not sending packets back to node in LoRa Gateway OS

Hey folks,

I’m running the latest version of LoRa-Gateway-OS on a Pi 3 B+ with an IC880A radio board.

The data for a node join currently flows as expected node -> lora_pkt_fwd -> lora-gateway-bridge -> loraserver -> lora-app-server and I see the JoinRequest and JoinAccept in the control panel on the lora app server.

I then see the JoinAccept head back the other way lora-app-server -> loraserver -> lora-gateway-bridge but that’s where it stops. I don’t see any packets being sent from lora_pkt_fwd back to the gateway, the logs from the packet forwarder when I run it manually are as follows:

SON up: {"stat":{"time":"2019-03-26 06:50:45 GMT","rxnb":0,"rxok":0,"rxfw":0,"ackr":100.0,"dwnb":0,"txnb":0}}
INFO: [up] PUSH_ACK received in 1 ms
INFO: [down] PULL_ACK received in 1 ms
INFO: [down] PULL_ACK received in 0 ms
^C
##### 2019-03-26 06:51:03 GMT #####
### [UPSTREAM] ###
# RF packets received by concentrator: 0
# CRC_OK: 0.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00%
# RF packets forwarded: 0 (0 bytes)
# PUSH_DATA datagrams sent: 1 (113 bytes)
# PUSH_DATA acknowledged: 100.00%
### [DOWNSTREAM] ###
# PULL_DATA sent: 2 (100.00% acknowledged)
# PULL_RESP(onse) datagrams received: 0 (0 bytes)
# RF packets sent to concentrator: 0 (0 bytes)
# TX errors: 0
# BEACON queued: 0
# BEACON sent so far: 0
# BEACON rejected: 0
### [JIT] ###
# SX1301 time (PPS): 542402159
src/jitqueue.c:448:jit_print_queue(): INFO: [jit] queue is empty
### [GPS] ###
# GPS sync is disabled
##### END #####

With this being LoRa Gateway OS, I’m unable to use TCPDump without building my own image, so the steps recorded at https://www.loraserver.io/guides/troubleshooting/gateway/ aren’t an option for me.

The gateway-bridge logs for the same time as the above packet forwarder logs show the following:

INFO[0395] gateway: rxpk packet received                 addr="127.0.0.1:54426" data="AAAAAAAAAAAA6zwiwpGkCwByPK2yoHs=" mac=b827ebfffeedc724
INFO[0395] backend: publishing packet                    qos=0 topic=gateway/b827ebfffeedc724/rx
DEBU[0395] gateway: sending udp packet to gateway        addr="127.0.0.1:54426" protocol_version=2 type=PushACK
DEBU[0400] gateway: received udp packet from gateway     addr="127.0.0.1:54426" protocol_version=2 type=PushData
INFO[0400] gateway: stat packet received                 addr="127.0.0.1:54426" mac=b827ebfffeedc724
DEBU[0400] gateway: sending udp packet to gateway        addr="127.0.0.1:54426" protocol_version=2 type=PushACK
INFO[0400] backend: publishing packet                    qos=0 topic=gateway/b827ebfffeedc724/stats
DEBU[0404] gateway: received udp packet from gateway     addr="127.0.0.1:47169" protocol_version=2 type=PullData
DEBU[0404] gateway: sending udp packet to gateway        addr="127.0.0.1:47169" protocol_version=2 type=PullACK
DEBU[0414] gateway: received udp packet from gateway     addr="127.0.0.1:47169" protocol_version=2 type=PullData
DEBU[0414] gateway: sending udp packet to gateway        addr="127.0.0.1:47169" protocol_version=2 type=PullACK
DEBU[0425] gateway: received udp packet from gateway     addr="127.0.0.1:47169" protocol_version=2 type=PullData
DEBU[0425] gateway: sending udp packet to gateway        addr="127.0.0.1:47169" protocol_version=2 type=PullACK
DEBU[0430] gateway: received udp packet from gateway     addr="127.0.0.1:54426" protocol_version=2 type=PushData
INFO[0430] gateway: stat packet received                 addr="127.0.0.1:54426" mac=b827ebfffeedc724
DEBU[0430] gateway: sending udp packet to gateway        addr="127.0.0.1:54426" protocol_version=2 type=PushACK
INFO[0430] backend: publishing packet                    qos=0 topic=gateway/b827ebfffeedc724/stats
DEBU[0435] gateway: received udp packet from gateway     addr="127.0.0.1:47169" protocol_version=2 type=PullData
DEBU[0435] gateway: sending udp packet to gateway        addr="127.0.0.1:47169" protocol_version=2 type=PullACK
DEBU[0445] gateway: received udp packet from gateway     addr="127.0.0.1:47169" protocol_version=2 type=PullData
DEBU[0445] gateway: sending udp packet to gateway        addr="127.0.0.1:47169" protocol_version=2 type=PullACK

I’m reasonably sure this is a configuration issue somewhere, so here’s the relevant config files:

[general]
# debug=5, info=4, warning=3, error=2, fatal=1, panic=0
log_level = 5


# Configuration which relates to the packet-forwarder.
[packet_forwarder]
# ip:port to bind the UDP listener to
#
# Example: 0.0.0.0:1700 to listen on port 1700 for all network interfaces.
# This is the listeren to which the packet-forwarder forwards its data
# so make sure the 'serv_port_up' and 'serv_port_down' from your
# packet-forwarder matches this port.
udp_bind = "0.0.0.0:1700"

# Skip the CRC status-check of received packets
#
# This is only has effect when the packet-forwarder is configured to forward
# LoRa frames with CRC errors.
skip_crc_check = false


  # # Managed packet-forwarder configuration.
  # #
  # # By configuring one or multiple managed packet-forwarder sections, the
  # # LoRa Gateway Bridge updates the configuration when the backend receives
  # # a configuration change, after which it will restart the packet-forwarder.
  # [[packet_forwarder.configuration]]
  # # Gateway MAC.
  # #
  # # The LoRa Gateway Bridge will only apply the configuration updates for this
  # # gateway MAC.
  # mac="0102030405060708"

  # # Base configuration file.
  # #
  # # This file will be used as base-configuration and will not be overwritten on
  # # a configuration update. This file needs to exist and contains the base
  # # configuration and vendor specific
  # base_file="/etc/lora-packet-forwarder/global_conf.json"

  # # Output configuration file.
  # #
  # # This will be the final configuration for the packet-forwarder, containing
  # # a merged version of the base configuration + the requested configuration
  # # update.
  # # Warning: this file will be overwritten on a configuration update!
  # output_file="/etc/lora-packet-forwarder/local_conf.json"

  # # Restart command.
  # #
  # # This command is issued by the LoRa Gateway Bridge on a configuration
  # # change. Make sure the LoRa Gateway Bridge process has sufficient
  # # permissions to execute this command.
  # restart_command="/etc/init.d/lora-packet-forwarder restart"


# Configuration for the MQTT backend.
[backend.mqtt]
# MQTT topic templates for the different MQTT topics.
#
# The meaning of these topics are documented at:
# https://docs.loraserver.io/lora-gateway-bridge/use/data/
#
# The default values match the default expected configuration of the
# LoRa Server MQTT backend. Therefore only change these values when
# absolutely needed.
# Use "{{ .MAC }}" as an substitution for the LoRa gateway MAC.
uplink_topic_template="gateway/{{ .MAC }}/rx"
downlink_topic_template="gateway/{{ .MAC }}/tx"
stats_topic_template="gateway/{{ .MAC }}/stats"
ack_topic_template="gateway/{{ .MAC }}/ack"
config_topic_template="gateway/{{ .MAC }}/config"

# Payload marshaler.
#
# This defines how the MQTT payloads are encoded. Valid options are:
# * v2_json:   The default LoRa Gateway Bridge v2 encoding (will be deprecated and removed in LoRa Gateway Bridge v3)
# * protobuf:  Protobuf encoding (this will become the LoRa Gateway Bridge v3 default)
# * json:      JSON encoding (easier for debugging, but less compact than 'protobuf')
marshaler="v2_json"

  # MQTT authentication.
  [backend.mqtt.auth]
  # Type defines the MQTT authentication type to use.
  #
  # Set this to the name of one of the sections below.
  # Note: when the 'v2_json marhaler' is configured, the generic backend will
  # always be used.
  type="generic"

    # Generic MQTT authentication.
    [backend.mqtt.auth.generic]
    # MQTT server (e.g. scheme://host:port where scheme is tcp, ssl or ws)
    server="tcp://<MY MQTT SERVER>:1883"

    # Connect with the given username (optional)
    username="<MY USERNAME>"

    # Connect with the given password (optional)
    password="<MY PASSWORD>"

    # Quality of service level
    #
    # 0: at most once
    # 1: at least once
    # 2: exactly once
    #
    # Note: an increase of this value will decrease the performance.
    # For more information: https://www.hivemq.com/blog/mqtt-essentials-part-6-mqtt-quality-of-service-levels
    qos=0

    # Clean session
    #
    # Set the "clean session" flag in the connect message when this client
    # connects to an MQTT broker. By setting this flag you are indicating
    # that no messages saved by the broker for this client should be delivered.
    clean_session=true

    # Client ID
    #
    # Set the client id to be used by this client when connecting to the MQTT
    # broker. A client id must be no longer than 23 characters. When left blank,
    # a random id will be generated. This requires clean_session=true.
    client_id="ilsgwos1"

    # CA certificate file (optional)
    #
    # Use this when setting up a secure connection (when server uses ssl://...)
    # but the certificate used by the server is not trusted by any CA certificate
    # on the server (e.g. when self generated).
    ca_cert=""

    # mqtt TLS certificate file (optional)
    tls_cert=""

    # mqtt TLS key file (optional)
    tls_key=""

    # Maximum interval that will be waited between reconnection attempts when connection is lost.
    # Valid units are 'ms', 's', 'm', 'h'. Note that these values can be combined, e.g. '24h30m15s'.
    max_reconnect_interval="10m0s"


# Metrics configuration.
[metrics]

  # Metrics stored in Prometheus.
  #
  # These metrics expose information about the state of the LoRa Gateway Bridge
  # instance like number of messages processed, number of function calls, etc.
  [metrics.prometheus]
  # Expose Prometheus metrics endpoint.
  endpoint_enabled=true

  # The ip:port to bind the Prometheus metrics server to for serving the
  # metrics endpoint.
  bind=""
{
    "SX1301_conf": {
        "lorawan_public": true,
        "clksrc": 1,
        "antenna_gain": 0,
        "radio_0": {
            "enable": true,
            "type": "SX1257",
            "freq": 867500000,
            "rssi_offset": -166.0,
            "tx_enable": true,
            "tx_freq_min": 863000000,
            "tx_freq_max": 870000000
        },
        "radio_1": {
            "enable": true,
            "type": "SX1257",
            "freq": 868500000,
            "rssi_offset": -166.0,
            "tx_enable": false
        },
        "chan_multiSF_0": {
            "enable": true,
            "radio": 1,
            "if": -400000
        },
        "chan_multiSF_1": {
            "enable": true,
            "radio": 1,
            "if": -200000
        },
        "chan_multiSF_2": {
            "enable": true,
            "radio": 1,
            "if": 0
        },
        "chan_multiSF_3": {
            "enable": true,
            "radio": 0,
            "if": -400000
        },
        "chan_multiSF_4": {
            "enable": true,
            "radio": 0,
            "if": -200000
        },
        "chan_multiSF_5": {
            "enable": true,
            "radio": 0,
            "if": 0
        },
        "chan_multiSF_6": {
            "enable": true,
            "radio": 0,
            "if": 200000
        },
        "chan_multiSF_7": {
            "enable": true,
            "radio": 0,
            "if": 400000
        },
        "chan_Lora_std": {
            "enable": true,
            "radio": 1,
            "if": -200000,
            "bandwidth": 250000,
            "spread_factor": 7
        },
        "chan_FSK": {
            "enable": true,
            "radio": 1,
            "if": 300000,
            "bandwidth": 125000,
            "datarate": 50000
        },
        "tx_lut_0": {
            "pa_gain": 0,
            "mix_gain": 8,
            "rf_power": -6,
            "dig_gain": 0
        },
        "tx_lut_1": {
            "pa_gain": 0,
            "mix_gain": 10,
            "rf_power": -3,
            "dig_gain": 0
        },
        "tx_lut_2": {
            "pa_gain": 0,
            "mix_gain": 12,
            "rf_power": 0,
            "dig_gain": 0
        },
        "tx_lut_3": {
            "pa_gain": 1,
            "mix_gain": 8,
            "rf_power": 3,
            "dig_gain": 0
        },
        "tx_lut_4": {
            "pa_gain": 1,
            "mix_gain": 10,
            "rf_power": 6,
            "dig_gain": 0
        },
        "tx_lut_5": {
            "pa_gain": 1,
            "mix_gain": 12,
            "rf_power": 10,
            "dig_gain": 0
        },
        "tx_lut_6": {
            "pa_gain": 1,
            "mix_gain": 13,
            "rf_power": 11,
            "dig_gain": 0
        },
        "tx_lut_7": {
            "pa_gain": 2,
            "mix_gain": 9,
            "rf_power": 12,
            "dig_gain": 0
        },
        "tx_lut_8": {
            "pa_gain": 1,
            "mix_gain": 15,
            "rf_power": 13,
            "dig_gain": 0
        },
        "tx_lut_9": {
            "pa_gain": 2,
            "mix_gain": 10,
            "rf_power": 14,
            "dig_gain": 0
        },
        "tx_lut_10": {
            "pa_gain": 2,
            "mix_gain": 11,
            "rf_power": 16,
            "dig_gain": 0
        },
        "tx_lut_11": {
            "pa_gain": 3,
            "mix_gain": 9,
            "rf_power": 20,
            "dig_gain": 0
        }
    },

    "gateway_conf": {
        "gateway_ID": "b827ebFFFEedc724",
        "server_address": "localhost",
        "serv_port_up": 1700,
        "serv_port_down": 1700,
        "keepalive_interval": 10,
        "stat_interval": 30,
        "push_timeout_ms": 100,
        "forward_crc_valid": true,
        "forward_crc_error": false,
        "forward_crc_disabled": false
    }
}

If anyone can shed some light on this (or even just tell me how to install tcpdump on the gateway!) that would be very much appreciated.

This is now fixed.

The configuration on the application server had gateway/{{ .MAC }}tx and not gateway/{{ .MAC }}/tx (note the missing /)