Failed to start LoRa Server

Daniel_Lundell · August 25, 2017, 1:07pm

So we tried out the stack on a Raspberry Pi running Stretch.
The gateway-bridge runs fine without error and the lora-app-server seems to do as well.

But the loraserver randomly crash at startup everytime. It appears to be random according to the log files. The connection to postgres, redis and MQTT is double and triple cheched. In the log it more often crash after applying migrations but the migrations are done correct upon inspection of the database.

Any thoughts?

brocaar · August 25, 2017, 1:37pm

Could you try to start LoRa Server manually, it looks like the error is not in your logs? Trying to start LoRa Server manually might give you more information. The easiest is to use the commands below:

# set the loraserver configuration
set -a
source /etc/default/loraserver

# start loraserver
loraserver

Daniel_Lundell · August 25, 2017, 1:43pm

I tried, resulted in

FATA[0000] NetID parse error: lorawan: exactly 3 bytes are expected

Checked in the config file and its set to the standard 010203

Daniel_Lundell · August 25, 2017, 1:48pm

So I found it!
Turned out there was some whitespace hiding in the configuration file.

Thanks for the help and swift response!

ThangDH · October 23, 2017, 3:32am

I’m getting the same error. I trying to start lora-app-server and loraserver manually, the output is:

INFO[0000] starting LoRa Server                          band="AU_915_928" docs="https://docs.loraserver.io/" net_id=010203 version=0.21.0
INFO[0000] setup redis connection pool                   url="redis://localhost:6379"
INFO[0000] connecting to postgresql                     
INFO[0000] backend/gateway: connecting to mqtt broker    server="tcp://localhost:1883"
INFO[0000] connecting to application-server              ca-cert= server="127.0.0.1:8001" tls-cert= tls-key=
INFO[0000] no network-controller configured             
INFO[0000] applying database migrations                 
INFO[0000] backend/gateway: connected to mqtt server    
INFO[0000] backend/gateway: subscribing to rx topic      topic="gateway/+/rx"
INFO[0000] backend/gateway: subscribing to stats topic   topic="gateway/+/stats"
INFO[0000] grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:8001: getsockopt: connection refused"; Reconnecting to {127.0.0.1:8001 <nil>} 
INFO[0000] migrations applied                            count=0
INFO[0000] starting api server                           bind="0.0.0.0:8000" ca-cert= tls-cert= tls-key=
INFO[0000] starting gateway api server                   bind="0.0.0.0:8002" ca-cert= tls-cert= tls-key=
FATA[0000] --gw-server-jwt-secret must be set

lora-app-server

INFO[0000] starting LoRa App Server                      docs="https://docs.loraserver.io/" version=0.13.1
INFO[0000] connecting to postgresql                     
INFO[0000] setup redis connection pool                  
INFO[0000] handler/mqtt: connecting to mqtt broker       server="tcp://localhost:1883"
INFO[0000] connecting to network-server api              ca-cert= server="127.0.0.1:8000" tls-cert= tls-key=
INFO[0000] applying database migrations                 
INFO[0000] handler/mqtt: connected to mqtt broker       
INFO[0000] handler/mqtt: subscribling to tx topic        topic="application/+/node/+/tx"
INFO[0000] grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:8000: getsockopt: connection refused"; Reconnecting to {127.0.0.1:8000 <nil>} 
INFO[0000] migrations applied                            count=0
INFO[0000] gwmigrate: migrating gateway data from LoRa Server 
ERRO[0000] gwmigrate: list gateways error (will retry): rpc error: code = Unavailable desc = grpc: the connection is unavailable 
INFO[0001] grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:8000: getsockopt: connection refused"; Reconnecting to {127.0.0.1:8000 <nil>} 
ERRO[0002] gwmigrate: list gateways error (will retry): rpc error: code = Unavailable desc = grpc: the connection is unavailable 
INFO[0002] grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:8000: getsockopt: connection refused"; Reconnecting to {127.0.0.1:8000 <nil>}

I check port :8000, :8001 are not use

vmlinux@ubuntu:~$ netstat -at
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 ubuntu:domain           *:*                     LISTEN     
tcp        0      0 *:ssh                   *:*                     LISTEN     
tcp        0      0 localhost:postgresql    *:*                     LISTEN     
tcp        0      0 *:1883                  *:*                     LISTEN     
tcp        0      0 localhost:6379          *:*                     LISTEN     
tcp        0      0 localhost:1883          localhost:52140         ESTABLISHED
tcp        0      0 192.168.0.152:ssh       192.168.0.151:55426     ESTABLISHED
tcp        0      0 localhost:52140         localhost:1883          ESTABLISHED
tcp        0      0 192.168.0.152:52742     174.138.99.199:https    ESTABLISHED
tcp6       0      0 [::]:ssh                [::]:*                  LISTEN     
tcp6       0      0 [::]:1883               [::]:*                  LISTEN

Please help me to solve my error. Thanks

aparticle · October 23, 2017, 5:11am

Looking at the last line of the error-log:

"FATA[0000] --gw-server-jwt-secret must be set "

You might have forgotten to set the value of GW_SERVER_JWT_SECRET in /etc/default/loraserver .

Hope this helps.

ThangDH · October 23, 2017, 6:23am

thanks aparticle, it’s work.

Jan · October 25, 2017, 7:25am

How did you find and remove the whitespace?
vim and nano cant detect it…

Jim · November 9, 2017, 8:09pm

@brocaar
I was seeing the same issue described here - the SECRETS were missing from my /etc/default/loraserver and /etc/default/lora-app-server

I have done a fresh install of the server via the ansible play book pointing it to a fresh AWS instance with nothing loaded except Python.

the hosts_vars/vagrant.yml contained:

loraserver:
[…]
GW_SERVER_JWT_SECRET: VX0DfnL3QyUHUNUeETjPBvKPJ3GOdnxtsRDakIPnXdY=

lora_app_server:
[…]
JWT_SECRET: Y301pq2kbuKbZHHlrGCsIoiMf7UJAnlEerv8OHYD93Q=
MQTT_USERNAME: loraserver_as
MQTT_PASSWORD: loraserver_as

I have edited the /etc/default/lora-app-server and /etc/default/loraserver on the remote server after the install - and replaced the null entries for the secrets with the original ones from the vagrant.yml file.
Running the server manually now results in this:

INFO[0000] starting LoRa Server band=“EU_863_870” docs=“https://docs.loraserver.io/” net_id=010203 version=0.21.0
INFO[0000] setup redis connection pool url=“redis://localhost:6379”
INFO[0000] connecting to postgresql
INFO[0000] backend/gateway: connecting to mqtt broker server=“tcp://localhost:1883”
ERRO[0000] backend/gateway: connecting to mqtt broker failed, will retry in 2s: Network Error : dial tcp 127.0.0.1:1883: getsockopt: connection refused
ERRO[0002] backend/gateway: connecting to mqtt broker failed, will retry in 2s: Network Error : dial tcp 127.0.0.1:1883: getsockopt: connection refused

Looking at Mosquitto - gives me this:

1510256857: mosquitto version 1.4.8 (build date Mon, 26 Jun 2017 09:31:02 +0100) starting
1510256857: Config loaded from /etc/mosquitto/mosquitto.conf.
1510256857: *** auth-plug: startup
1510256857: ** Configured order: files,postgres

1510256857: }}}} POSTGRES
1510256857: HERE: select count(*) from “user” where username = $1 and is_admin = true
1510256857: HERE: select distinct ‘application/’ || a.id || ‘/node/+/+’ from “user” u left join organization_user ou on u.id = ou.user_id left join organization o on o.id = ou.organization_id left join application_user au on u.id = au.user_id left join application a on au.application_id = a.id or a.organization_id = o.id left join node n on a.id = n.application_id where a.id is not null and u.username = $1 and $2 = $2

It seems like Mosquitto should be running - unless the Postgres stuff is failing. (Not familiar with this sql server)

However - Mosquitto isn’t running - so that breaks everything.

Suggestions on what to check next?

brocaar · November 10, 2017, 9:12am

ERRO[0000] backend/gateway: connecting to mqtt broker failed, will retry in 2s: Network Error : dial tcp 127.0.0.1:1883: getsockopt: connection refused
ERRO[0002] backend/gateway: connecting to mqtt broker failed, will retry in 2s: Network Error : dial tcp 127.0.0.1:1883: getsockopt: connection refused

I think that indicates the issue is with the MQTT / Mosquitto server. Two things that could cause this:

PostgreSQL backend should attempt to reconnect to the database · Issue #269 · jpmens/mosquitto-auth-plug · GitHub
Set SUPPORT_DJANGO_HASHERS=no. · chirpstack/chirpstack-ansible-playbook@2408f50 · GitHub (make sure you have the latest loraserver-setup!)

Jim · November 10, 2017, 11:37am

Yes,
I had done a “git pull” prior to the installation attempt and updated my local copy of the repo.

It does look like PSQL wasn’t running in time when my instance booted - and “sudo systemctl restart mosquitto” seems to have fixed things.

I say seems since even though it appears it should accepting connections on 443 (https)

Nov 10 11:27:12 ip-172-31-25-123 lora-app-server[1362]: time=“2017-11-10T11:27:12Z” level=info msg=“gwmigrate: migrating gateway data from LoRa Server”
Nov 10 11:27:12 ip-172-31-25-123 lora-app-server[1362]: time=“2017-11-10T11:27:12Z” level=info msg=“starting application-server api” bind=“0.0.0.0:8001” ca-cert= tls-cert= tls-key=
Nov 10 11:27:12 ip-172-31-25-123 lora-app-server[1362]: time=“2017-11-10T11:27:12Z” level=info msg=“starting client api server” bind=“0.0.0.0:443” tls-cert="/etc/lora-app-server/certs/http.pem" tls-key="/etc/lora-app-server/certs/http-key.pem"
Nov 10 11:27:12 ip-172-31-25-123 lora-app-server[1362]: time=“2017-11-10T11:27:12Z” level=info msg=“registering rest api handler and documentation endpoint” path="/api"
Nov 10 11:27:12 ip-172-31-25-123 lora-gateway-bridge[1307]: time=“2017-11-10T11:27:12Z” level=info msg=“backend: connecting to mqtt broker” server=“tcp://127.0.0.1:1883”
Nov 10 11:27:12 ip-172-31-25-123 lora-gateway-bridge[1307]: time=“2017-11-10T11:27:12Z” level=info msg=“gateway: starting gateway udp listener” addr=0.0.0.0:1700
Nov 10 11:27:12 ip-172-31-25-123 lora-gateway-bridge[1307]: time=“2017-11-10T11:27:12Z” level=info msg=“backend: connected to mqtt broker”

Connection attempts to https via my browser just timeout eventually.

I am using the same security settings for this server as another one on AWS - and the iptables shouldn’t be a factor since I have 443 wide open iptables wise (relying on AWS security settings)

-A INPUT -s 0.0.0.0/32 -p tcp -m tcp --dport 443 -j ACCEPT

Manually starting mosquitto is an OK work-around for that issue, but now I’m stuck being unable to manage it…

brocaar · November 11, 2017, 6:12pm

Please note:

github.com

brocaar/chirpstack-setup/blob/master/Vagrantfile#L10


# -*- mode: ruby -*-
# vi: set ft=ruby :


Vagrant.configure("2") do |config|
  config.vm.define "vagrant" do |box|
    box.vm.box = "debian/contrib-buster64"
    # box.vm.box = "debian/contrib-stretch64"
    # box.vm.box = "ubuntu/bionic64"
    # box.vm.box = "ubuntu/xenial64"


    box.vm.network "forwarded_port", guest: 80,   host: 8080, protocol: "tcp"
    box.vm.network "forwarded_port", guest: 1700, host: 1700, protocol: "udp"
    box.vm.network "forwarded_port", guest: 1883, host: 1883, protocol: "tcp"
    box.vm.network "forwarded_port", guest: 1884, host: 1884, protocol: "tcp"


    box.vm.provision "ansible_local" do |ansible|
      ansible.install         = true
      ansible.playbook        = "full_deploy.yml"
      ansible.config_file     = "ansible.cfg"
    end

The port you should use in your browser is 8080. This is also mentioned here: https://github.com/brocaar/loraserver-setup#vagrant-local-environment-using-virtualbox.

Jim · November 13, 2017, 7:46pm

I ended up cloning my original US server install which I had setup back in May - and modifying the clones for use as an EU and an AU server.

I’ll try a new install with those port redirects in mind next time I need a new server.

Ryan · December 21, 2017, 10:58am

Hi @brocaar I meet the same error,when I try to compile from the source
make build
cd build
./loraserver
FATA[0000] NetID parse error: lorawan: exactly 3 bytes are expected

when I run make test
Given a clean database and api instance ERRO[0000] ping database error, will retry in 2s: pq: password authentication failed for user “ryan”
ERRO[0002] ping database error, will retry in 2s: pq: password authentication failed for user “ryan”
ERRO[0004] ping database error, will retry in 2s: pq: password authentication failed for user “ryan”
…
I have set the GW_SERVER_JWT_SECRET in /github.com/brocaar/loraserver/packaging/deb/default ,but it doesn’t work

brocaar · December 21, 2017, 11:00am

FATA[0000] NetID parse error: lorawan: exactly 3 bytes are expected

and

ERRO[0002] ping database error, will retry in 2s: pq: password authentication failed for user “Ryan”

I think the error message is quite clear Please see also: https://docs.loraserver.io/loraserver/install/config/

Ryan · December 21, 2017, 11:15am

thanks for swift reply ,it work after i type below

./loraserver --band CN_470_510 --net-id 010203 --postgres-dsn postgres://loraserver_as:password@localhost/loraserver_as?sslmode=disable

but I have set these values in the /loraserver/packaging/deb/default and then make build ,how can I make my config effect

iegomez · December 21, 2017, 12:39pm

When you start loarserver manually, you need to explicitly set either the environment variables or the cli flags. Normally, the service would do this for you (if you check packaging/deb/init.sh, you’ll see that the default file is getting set) on sudo service loraserver start, but manually you need to do it. This can be done with:

set -a
source /loraserver/packaging/deb/default (or /path/to/file)
./loraserver

Ryan · December 22, 2017, 1:27am

thanks for your help ,that’s great,I solve my problem

iegomez · December 22, 2017, 1:42pm

No problem. It can be a pain remembering to source the env vars before starting the binaries, so if it’s of any help, there’s a dropdown terminal called Guake that’ll let you hook an init script before launching (and a lot more). Of course, you could do the same with an init script that launches 5 terminals, but I prefer using Guake.

I usually work with 5 tabs, for loraserver, lora-app-server, npm, some simulation scripts and a multi-purpose one (connecting to psql, ssh-ing to a remote server, etc.). So I just adapted a script that on startup opens 5 tabs, cds to the proper directories and executes anything else necessary, like setting the env vars. I’ll leave it here as an example in case anyone wants to set something similar:

#!/bin/bash

/usr/bin/guake &
sleep 2 # let main guake process start and initialize D-Bus session
guake -r "Appserver" -e "cd /home/iegomez/go/src/github.com/brocaar/lora-app-server/ && set -a && source ../appserver_env_vars"
guake -n " " -r "Loraserver" -e "cd /home/iegomez/go/src/github.com/brocaar/loraserver/ && set -a && source ../loraserver_env_vars"
guake -n " " -r "React UI" -e "cd /home/iegomez/go/src/github.com/brocaar/lora-app-server/ui"
guake -n " " -r "Lora scripts" -e "cd /home/iegomez/go/src/github.com/lora-scripts/"
guake -n " " -r "Terminal" -e "cd /home/iegomez"
guake -s 0

brocaar · December 22, 2017, 2:17pm

An alternative could also be https://direnv.net. It allows you to create .envrc files in each directory with the environment variables that must be set within the context of that directory.