Errors received in the LOG after adding another organzation

VikhyatChandra · October 6, 2017, 9:07am

Hi brocaar, I am currently facing a server issue. I had added another organization in the web interface GUI , thereafter when we are passing the data to the newly created organization’s application node we are receiving data only once. There after the JSON_down stops in the log gateway. The logs of loraserver and lora-app-server are as follow,Please revert back asap
root@ubuntu-1gb-blr1-01:~# sudo journalctl -u loraserver -f -n 50
Journal file /run/log/journal/eef1323751bc4eb984f724371fde07bb/system.journal is truncated, ignoring file.
– Logs begin at Fri 2017-10-06 06:44:35 UTC. –
Oct 06 08:36:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:36:15Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:36:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:36:15Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:36:32 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:36:32Z” level=info msg=“backend/gateway: rx packet received”
Oct 06 08:36:32 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:36:32Z” level=error msg=“processing rx packet error: get node-session error: node-session does not exist or invalid fcnt or mic” data_base64=“gENlhwmAAAAC+MDT1QI=”
Oct 06 08:36:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:36:45Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:36:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:36:45Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:36:50 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:36:50Z” level=info msg=“backend/gateway: rx packet received”
Oct 06 08:36:50 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:36:50Z” level=error msg=“processing rx packet error: get node-session error: node-session does not exist or invalid fcnt or mic” data_base64=“gENlhwmAAAAC+MDT1QI=”
Oct 06 08:37:09 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:37:09Z” level=info msg=“backend/gateway: rx packet received”
Oct 06 08:37:09 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:37:09Z” level=error msg=“processing rx packet error: get node-session error: node-session does not exist or invalid fcnt or mic” data_base64=“gENlhwmAAAAC+MDT1QI=”
Oct 06 08:37:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:37:15Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:37:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:37:15Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:37:28 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:37:28Z” level=info msg=“backend/gateway: rx packet received”
Oct 06 08:37:28 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:37:28Z” level=error msg=“processing rx packet error: get node-session error: node-session does not exist or invalid fcnt or mic” data_base64=“gENlhwmAAAAC+MDT1QI=”
Oct 06 08:37:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:37:45Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:37:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:37:45Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:38:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:38:15Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:38:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:38:15Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:38:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:38:45Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:38:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:38:45Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:39:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:39:15Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:39:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:39:15Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:39:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:39:45Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:39:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:39:45Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:40:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:40:15Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:40:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:40:15Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:40:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:40:45Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:40:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:40:45Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:41:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:41:15Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:41:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:41:15Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:41:28 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:41:28Z” level=info msg=“node-session deleted” dev_eui=4786e6ed00490090
Oct 06 08:41:28 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:41:28Z” level=info msg=“node-session saved” dev_addr=013e6354 dev_eui=4786e6ed00490090
Oct 06 08:41:39 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:41:39Z” level=info msg=“node-session deleted” dev_eui=4786e6ed00490105
Oct 06 08:41:39 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:41:39Z” level=info msg=“node-session saved” dev_addr=09876543 dev_eui=4786e6ed00490105
Oct 06 08:41:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:41:45Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:41:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:41:45Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:42:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:42:15Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:42:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:42:15Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:42:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:42:45Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:42:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:42:45Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:43:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:43:15Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:43:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:43:15Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:43:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:43:45Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:43:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:43:45Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:44:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:44:15Z” level=error msg=“backend/gateway: acquire stats lock error: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.”
Oct 06 08:44:21 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:44:21Z” level=info msg=“backend/gateway: rx packet received”
Oct 06 08:44:21 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:44:21Z” level=error msg=“backend/gateway: acquire uplink payload lock error: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.”
Oct 06 08:44:39 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:44:39Z” level=info msg=“backend/gateway: rx packet received”
Oct 06 08:44:40 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:44:40Z” level=error msg=“backend/gateway: acquire uplink payload lock error: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.”
Oct 06 08:44:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:44:45Z” level=error msg=“backend/gateway: acquire stats lock error: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.”
backend/gateway: acquire uplink payload lock error: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.^C
root@ubuntu-1gb-blr1-01:~# sudo journalctl -u loraserver -f -n 50
Journal file /run/log/journal/eef1323751bc4eb984f724371fde07bb/system.journal is truncated, ignoring file.
– Logs begin at Fri 2017-10-06 06:44:35 UTC. –
Oct 06 08:36:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:36:15Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:36:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:36:15Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:36:32 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:36:32Z” level=info msg=“backend/gateway: rx packet received”
Oct 06 08:36:32 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:36:32Z” level=error msg=“processing rx packet error: get node-session error: node-session does not exist or invalid fcnt or mic” data_base64=“gENlhwmAAAAC+MDT1QI=”
Oct 06 08:36:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:36:45Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:36:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:36:45Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:36:50 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:36:50Z” level=info msg=“backend/gateway: rx packet received”
Oct 06 08:36:50 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:36:50Z” level=error msg=“processing rx packet error: get node-session error: node-session does not exist or invalid fcnt or mic” data_base64=“gENlhwmAAAAC+MDT1QI=”
Oct 06 08:37:09 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:37:09Z” level=info msg=“backend/gateway: rx packet received”
Oct 06 08:37:09 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:37:09Z” level=error msg=“processing rx packet error: get node-session error: node-session does not exist or invalid fcnt or mic” data_base64=“gENlhwmAAAAC+MDT1QI=”
Oct 06 08:37:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:37:15Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:37:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:37:15Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:37:28 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:37:28Z” level=info msg=“backend/gateway: rx packet received”
Oct 06 08:37:28 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:37:28Z” level=error msg=“processing rx packet error: get node-session error: node-session does not exist or invalid fcnt or mic” data_base64=“gENlhwmAAAAC+MDT1QI=”
Oct 06 08:37:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:37:45Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:37:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:37:45Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:38:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:38:15Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:38:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:38:15Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:38:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:38:45Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:38:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:38:45Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:39:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:39:15Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:39:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:39:15Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:39:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:39:45Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:39:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:39:45Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:40:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:40:15Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:40:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:40:15Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:40:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:40:45Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:40:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:40:45Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:41:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:41:15Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:41:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:41:15Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:41:28 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:41:28Z” level=info msg=“node-session deleted” dev_eui=4786e6ed00490090
Oct 06 08:41:28 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:41:28Z” level=info msg=“node-session saved” dev_addr=013e6354 dev_eui=4786e6ed00490090
Oct 06 08:41:39 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:41:39Z” level=info msg=“node-session deleted” dev_eui=4786e6ed00490105
Oct 06 08:41:39 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:41:39Z” level=info msg=“node-session saved” dev_addr=09876543 dev_eui=4786e6ed00490105
Oct 06 08:41:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:41:45Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:41:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:41:45Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:42:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:42:15Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:42:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:42:15Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:42:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:42:45Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:42:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:42:45Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:43:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:43:15Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:43:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:43:15Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:43:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:43:45Z” level=info msg=“backend/gateway: gateway stats packet received” mac=b827ebfffe9ccc85
Oct 06 08:43:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:43:45Z” level=info msg=“gateway updated” mac=b827ebfffe9ccc85
Oct 06 08:44:15 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:44:15Z” level=error msg=“backend/gateway: acquire stats lock error: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.”
Oct 06 08:44:21 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:44:21Z” level=info msg=“backend/gateway: rx packet received”
Oct 06 08:44:21 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:44:21Z” level=error msg=“backend/gateway: acquire uplink payload lock error: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.”
Oct 06 08:44:39 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:44:39Z” level=info msg=“backend/gateway: rx packet received”
Oct 06 08:44:40 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:44:40Z” level=error msg=“backend/gateway: acquire uplink payload lock error: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.”
Oct 06 08:44:45 ubuntu-1gb-blr1-01 loraserver[25931]: time=“2017-10-06T08:44:45Z” level=error msg=“backend/gateway: acquire stats lock error: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.”
^C

root@ubuntu-1gb-blr1-01:~# sudo journalctl -u lora-app-server -f -n 50
Journal file /run/log/journal/eef1323751bc4eb984f724371fde07bb/system.journal is truncated, ignoring file.
– Logs begin at Fri 2017-10-06 06:44:35 UTC. –
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: /home/travis/.gimme/versions/go1.9.linux.amd64/src/net/fd_unix.go:202 +0x52
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: net.(*conn).Read(0xc42000c1b0, 0xc42028f800, 0x800, 0x800, 0x0, 0x0, 0x0)
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: /home/travis/.gimme/versions/go1.9.linux.amd64/src/net/net.go:176 +0x6d
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: crypto/tls.(*block).readFromUntil(0xc4206c2f30, 0x7fe8a9b900f0, 0xc42000c1b0, 0x5, 0xc42000c1b0, 0x1)
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: /home/travis/.gimme/versions/go1.9.linux.amd64/src/crypto/tls/conn.go:488 +0x95
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: crypto/tls.(*Conn).readRecord(0xc4206f4a80, 0xb0e917, 0xc4206f4ba0, 0xc42042bc68)
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: /home/travis/.gimme/versions/go1.9.linux.amd64/src/crypto/tls/conn.go:590 +0xe0
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: crypto/tls.(*Conn).Read(0xc4206f4a80, 0xc4206aa000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: /home/travis/.gimme/versions/go1.9.linux.amd64/src/crypto/tls/conn.go:1134 +0x110
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: bufio.(*Reader).Read(0xc4200dd260, 0xc4202f42d8, 0x9, 0x9, 0xc42042bd78, 0x4413f4, 0xc420ed3bf0)
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: /home/travis/.gimme/versions/go1.9.linux.amd64/src/bufio/bufio.go:213 +0x30b
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: io.ReadAtLeast(0xe9dd80, 0xc4200dd260, 0xc4202f42d8, 0x9, 0x9, 0x9, 0x7fdfbf, 0xc4208a2788, 0x0)
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: /home/travis/.gimme/versions/go1.9.linux.amd64/src/io/io.go:309 +0x86
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: io.ReadFull(0xe9dd80, 0xc4200dd260, 0xc4202f42d8, 0x9, 0x9, 0x410cd8, 0xc42042be28, 0x4295e6)
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: /home/travis/.gimme/versions/go1.9.linux.amd64/src/io/io.go:327 +0x58
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: github.com/brocaar/lora-app-server/vendor/golang.org/x/net/http2.readFrameHeader(0xc4202f42d8, 0x9, 0x9, 0xe9dd80, 0xc4200dd260, 0x0, 0xc400000000, 0xc42042be6d, 0xc4208a33e0)
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: /home/travis/gopath/src/github.com/brocaar/lora-app-server/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: github.com/brocaar/lora-app-server/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc4202f42a0, 0xea3140, 0xc4208a33e0, 0x0, 0x0)
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: /home/travis/gopath/src/github.com/brocaar/lora-app-server/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: github.com/brocaar/lora-app-server/vendor/google.golang.org/grpc/transport.(*framer).readFrame(0xc420183d70, 0xc4208a33e0, 0xc4208a33e0, 0x0, 0x0)
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: /home/travis/gopath/src/github.com/brocaar/lora-app-server/vendor/google.golang.org/grpc/transport/http_util.go:593 +0x2f
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: github.com/brocaar/lora-app-server/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc4203c8f00)
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: /home/travis/gopath/src/github.com/brocaar/lora-app-server/vendor/google.golang.org/grpc/transport/http2_client.go:1104 +0xc0
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: created by github.com/brocaar/lora-app-server/vendor/google.golang.org/grpc/transport.newHTTP2Client
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: /home/travis/gopath/src/github.com/brocaar/lora-app-server/vendor/google.golang.org/grpc/transport/http2_client.go:252 +0xbd8
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: goroutine 122 [IO wait]:
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: internal/poll.runtime_pollWait(0x7fe8a9b8b730, 0x72, 0x0)
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: /home/travis/.gimme/versions/go1.9.linux.amd64/src/runtime/netpoll.go:173 +0x57
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: internal/poll.(*pollDesc).wait(0xc4200db518, 0x72, 0xffffffffffffff00, 0xea1ac0, 0xe9c400)
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: /home/travis/.gimme/versions/go1.9.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0xae
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: internal/poll.(*pollDesc).waitRead(0xc4200db518, 0xc4206fe400, 0x400, 0x400)
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: /home/travis/.gimme/versions/go1.9.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3d
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: internal/poll.(*FD).Read(0xc4200db500, 0xc4206fe400, 0x400, 0x400, 0x0, 0x0, 0x0)
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: /home/travis/.gimme/versions/go1.9.linux.amd64/src/internal/poll/fd_unix.go:125 +0x18a
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: net.(*netFD).Read(0xc4200db500, 0xc4206fe400, 0x400, 0x400, 0x54, 0x8, 0x0)
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: /home/travis/.gimme/versions/go1.9.linux.amd64/src/net/fd_unix.go:202 +0x52
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: net.(*conn).Read(0xc42000c1a8, 0xc4206fe400, 0x400, 0x400, 0x0, 0x0, 0x0)
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: /home/travis/.gimme/versions/go1.9.linux.amd64/src/net/net.go:176 +0x6d
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: crypto/tls.(*block).readFromUntil(0xc420309aa0, 0x7fe8a9b900f0, 0xc42000c1a8, 0x5, 0xc42000c1a8, 0x0)
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: /home/travis/.gimme/versions/go1.9.linux.amd64/src/crypto/tls/conn.go:488 +0x95
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: crypto/tls.(*Conn).readRecord(0xc4206f4700, 0xb0e917, 0xc4206f4820, 0x0)
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: /home/travis/.gimme/versions/go1.9.linux.amd64/src/crypto/tls/conn.go:590 +0xe0
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: crypto/tls.(*Conn).Read(0xc4206f4700, 0xc420138000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: /home/travis/.gimme/versions/go1.9.linux.amd64/src/crypto/tls/conn.go:1134 +0x110
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: bufio.(*Reader).Read(0xc4200ddc80, 0xc4202f4658, 0x9, 0x9, 0x0, 0x10, 0xc42060ae80)
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: /home/travis/.gimme/versions/go1.9.linux.amd64/src/bufio/bufio.go:213 +0x30b
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: io.ReadAtLeast(0xe9dd80, 0xc4200ddc80, 0xc4202f4658, 0x9, 0x9, 0x9, 0xc420046a00, 0xc420426ea8, 0x8)
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: /home/travis/.gimme/versions/go1.9.linux.amd64/src/io/io.go:309 +0x86
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: io.ReadFull(0xe9dd80, 0xc4200ddc80, 0xc4202f4658, 0x9, 0x9, 0xc7076751634a201, 0x700000004, 0x0)
Oct 06 08:45:15 ubuntu-1gb-blr1-01 lora-app-server[8393]: /home/travis/.gimme/versions/go1.9.linux.amd64/src/io/io.go:327 +0x58

brocaar · October 6, 2017, 9:14am

Seems to me like you have a storage or I/O related issue. As Redis is unable to operate and is a dependency of LoRa (App) Server, these services can’t operate either. In your logs is a lora-app-server crash too, which could be related to the same issue.

VikhyatChandra · October 6, 2017, 10:27am

This shows in the log of redis server.

1411:M 06 Oct 08:44:55.045 * 10 changes in 300 seconds. Saving…
1411:M 06 Oct 08:44:55.048 # Can’t save in background: fork: Cannot allocate memory
1411:M 06 Oct 08:45:01.103 * 10 changes in 300 seconds. Saving…
1411:M 06 Oct 08:45:01.106 # Can’t save in background: fork: Cannot allocate memory
1411:M 06 Oct 08:45:07.013 * 10 changes in 300 seconds. Saving…
1411:M 06 Oct 08:45:07.015 # Can’t save in background: fork: Cannot allocate memory
1411:M 06 Oct 08:45:13.083 * 10 changes in 300 seconds. Saving…
1411:M 06 Oct 08:45:13.096 # Can’t save in background: fork: Cannot allocate memory
1411:M 06 Oct 08:45:19.212 * 10 changes in 300 seconds. Saving…
1411:M 06 Oct 08:45:19.224 # Can’t save in background: fork: Cannot allocate memory
1411:M 06 Oct 08:45:25.055 * 10 changes in 300 seconds. Saving…
1411:M 06 Oct 08:45:25.055 * Background saving started by pid 20788
20788:C 06 Oct 08:45:25.064 * DB saved on disk
20788:C 06 Oct 08:45:25.065 * RDB: 2 MB of memory used by copy-on-write
1411:M 06 Oct 08:45:25.156 * Background saving terminated with success
1411:M 06 Oct 08:50:26.057 * 10 changes in 300 seconds. Saving…
1411:M 06 Oct 08:50:26.059 * Background saving started by pid 20926
20926:C 06 Oct 08:50:26.065 * DB saved on disk
20926:C 06 Oct 08:50:26.065 * RDB: 6 MB of memory used by copy-on-write
1411:M 06 Oct 08:50:26.159 * Background saving terminated with success
1411:M 06 Oct 08:55:27.057 * 10 changes in 300 seconds. Saving…
1411:M 06 Oct 08:55:27.093 * Background saving started by pid 21360
21360:C 06 Oct 08:55:27.098 * DB saved on disk
21360:C 06 Oct 08:55:27.099 * RDB: 6 MB of memory used by copy-on-write
1411:M 06 Oct 08:55:27.194 * Background saving terminated with success
1411:M 06 Oct 09:00:28.098 * 10 changes in 300 seconds. Saving…
1411:M 06 Oct 09:00:28.099 * Background saving started by pid 21535
21535:C 06 Oct 09:00:28.107 * DB saved on disk
21535:C 06 Oct 09:00:28.108 * RDB: 2 MB of memory used by copy-on-write
1411:M 06 Oct 09:00:28.199 * Background saving terminated with success
1411:M 06 Oct 09:05:29.102 * 10 changes in 300 seconds. Saving…
1411:M 06 Oct 09:05:29.102 * Background saving started by pid 21581
21581:C 06 Oct 09:05:29.107 * DB saved on disk
21581:C 06 Oct 09:05:29.108 * RDB: 2 MB of memory used by copy-on-write
1411:M 06 Oct 09:05:29.202 * Background saving terminated with success
1411:M 06 Oct 09:10:30.011 * 10 changes in 300 seconds. Saving…
1411:M 06 Oct 09:10:30.011 * Background saving started by pid 21661
21661:C 06 Oct 09:10:30.018 * DB saved on disk
21661:C 06 Oct 09:10:30.018 * RDB: 2 MB of memory used by copy-on-write
1411:M 06 Oct 09:10:30.111 * Background saving terminated with success
1411:M 06 Oct 09:15:31.034 * 10 changes in 300 seconds. Saving…
1411:M 06 Oct 09:15:31.034 * Background saving started by pid 21704
21704:C 06 Oct 09:15:31.039 * DB saved on disk
21704:C 06 Oct 09:15:31.039 * RDB: 2 MB of memory used by copy-on-write
1411:M 06 Oct 09:15:31.135 * Background saving terminated with success
1411:M 06 Oct 09:20:32.041 * 10 changes in 300 seconds. Saving…
1411:M 06 Oct 09:20:32.042 * Background saving started by pid 21814
21814:C 06 Oct 09:20:32.057 * DB saved on disk
21814:C 06 Oct 09:20:32.058 * RDB: 2 MB of memory used by copy-on-write
1411:M 06 Oct 09:20:32.142 * Background saving terminated with success
1411:M 06 Oct 09:25:33.095 * 10 changes in 300 seconds. Saving…
1411:M 06 Oct 09:25:33.096 * Background saving started by pid 22007
22007:C 06 Oct 09:25:33.102 * DB saved on disk
22007:C 06 Oct 09:25:33.102 * RDB: 2 MB of memory used by copy-on-write
1411:M 06 Oct 09:25:33.196 * Background saving terminated with success
1411:M 06 Oct 09:30:34.009 * 10 changes in 300 seconds. Saving…
1411:M 06 Oct 09:30:34.021 * Background saving started by pid 22340
22340:C 06 Oct 09:30:34.030 * DB saved on disk
22340:C 06 Oct 09:30:34.031 * RDB: 6 MB of memory used by copy-on-write
1411:M 06 Oct 09:30:34.121 * Background saving terminated with success
1411:M 06 Oct 09:35:35.005 * 10 changes in 300 seconds. Saving…
1411:M 06 Oct 09:35:35.006 * Background saving started by pid 22384
22384:C 06 Oct 09:35:35.010 * DB saved on disk
22384:C 06 Oct 09:35:35.011 * RDB: 4 MB of memory used by copy-on-write
1411:M 06 Oct 09:35:35.106 * Background saving terminated with success
1411:M 06 Oct 09:46:04.982 * 10 changes in 300 seconds. Saving…
1411:M 06 Oct 09:46:04.982 * Background saving started by pid 22498
22498:C 06 Oct 09:46:04.988 * DB saved on disk
22498:C 06 Oct 09:46:04.989 * RDB: 2 MB of memory used by copy-on-write
1411:M 06 Oct 09:46:05.083 * Background saving terminated with success
1411:M 06 Oct 09:51:06.101 * 10 changes in 300 seconds. Saving…
1411:M 06 Oct 09:51:06.101 * Background saving started by pid 22559
22559:C 06 Oct 09:51:06.106 * DB saved on disk
22559:C 06 Oct 09:51:06.107 * RDB: 2 MB of memory used by copy-on-write
1411:M 06 Oct 09:51:06.202 * Background saving terminated with success
1411:M 06 Oct 09:56:07.004 * 10 changes in 300 seconds. Saving…
1411:M 06 Oct 09:56:07.004 * Background saving started by pid 22604
22604:C 06 Oct 09:56:07.009 * DB saved on disk
22604:C 06 Oct 09:56:07.010 * RDB: 2 MB of memory used by copy-on-write
1411:M 06 Oct 09:56:07.105 * Background saving terminated with success
1411:M 06 Oct 10:01:08.006 * 10 changes in 300 seconds. Saving…
1411:M 06 Oct 10:01:08.007 * Background saving started by pid 22684
22684:C 06 Oct 10:01:08.013 * DB saved on disk
22684:C 06 Oct 10:01:08.013 * RDB: 2 MB of memory used by copy-on-write
1411:M 06 Oct 10:01:08.108 * Background saving terminated with success
1411:M 06 Oct 10:06:09.018 * 10 changes in 300 seconds. Saving…
1411:M 06 Oct 10:06:09.019 * Background saving started by pid 22836
22836:C 06 Oct 10:06:09.030 * DB saved on disk
22836:C 06 Oct 10:06:09.031 * RDB: 2 MB of memory used by copy-on-write
1411:M 06 Oct 10:06:09.119 * Background saving terminated with success
1411:M 06 Oct 10:12:45.175 * 10 changes in 300 seconds. Saving…
1411:M 06 Oct 10:12:45.176 * Background saving started by pid 22893
22893:C 06 Oct 10:12:45.182 * DB saved on disk
22893:C 06 Oct 10:12:45.183 * RDB: 2 MB of memory used by copy-on-write
1411:M 06 Oct 10:12:45.276 * Background saving terminated with success
1411:M 06 Oct 10:17:46.093 * 10 changes in 300 seconds. Saving…
1411:M 06 Oct 10:17:46.094 * Background saving started by pid 23115
23115:C 06 Oct 10:17:46.105 * DB saved on disk
23115:C 06 Oct 10:17:46.105 * RDB: 2 MB of memory used by copy-on-write
1411:M 06 Oct 10:17:46.194 * Background saving terminated with success

I see for some reason its not able to save in background and cannot allocate memory…

brocaar · October 6, 2017, 10:30am

Looks like you have a memory related issue then

VikhyatChandra · October 6, 2017, 10:34am

How can this be resolved any idea?? DO i have to buy a server with more Memory capacity??

brocaar · October 6, 2017, 10:55am

What did you try by yourself to find out what the issue is?

VikhyatChandra · October 6, 2017, 10:59am

Okay… I will check the logs.Thanks for the help