Http integration fails after hour or two

We have a server running Brocaar App Server:

lora-app-server 0.13.2
loraserver 0.21.0

From ubuntu packages.
Linux ip-10-0-0-242 4.4.0-1039-aws #48-Ubuntu SMP Wed Oct 11 15:15:01 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Since our application server has CGI end points - I have used that method to integrate brocaar with it. However, after couple hundred messages, I get these sorts of errors in my logs:

Nov 5 12:19:24 ip-10-0-0-242 lora-app-server[19493]: time=“2017-11-05T12:19:24Z” level=error msg=“handler *httphandler.Handler error: http request error: Post http://xyz.amazonaws.com:8080/foobar: dial tcp: lookup xyz.amazonaws.com on 10.0.0.2:53: no such host”

This requires me to restart the processes every few hours, which is not desirable.
The alternative error I do get is saying “too my files opened”. I wonder if there’s some sort of a memory leak, resource leak issue in this version.

too my files opened

Interesting, I need to do a bit more debugging before I can give any feedback on this. But please note that this could also be caused by your system configuration or any other service running on the same machine. See also https://www.cyberciti.biz/tips/linux-procfs-file-descriptors.html

Post http://xyz.amazonaws.com:8080/foobar: dial tcp: lookup xyz.amazonaws.com on 10.0.0.2:53: no such host

I don’t think that this is a LoRa App Server issue. The problem here is that the DNS server you’ve configured does not know how to translate the requested hostname into an IP address.

@4a70023fa655b29e995d have you looked into testing the number of file descriptors per running process?

Hi, I have set the max-file descriptor value to 99999999.
It doesn’t seem to be an issue if I get a message every few seconds, but getting a message a second, or more frequent seems to trigger this issue.

That is a workaround, setting the max-file descriptors to a high value. However, it would be of great help to know if this is caused by LoRa (App) Server or an other process running on your server :slight_smile:

Put it this way, I am killing the app-server process every two hours in crontab - and that “fixes” the problem.
Looks to me like a resource (memory, file descriptor) that isn’t closed when done with.

If this was written in any other language than Go, I’d probably have a go myself … (pun unintentional) .

So do I. It likes memory or http connection that isn’t closed when done with…

I’ll look into this this today / tomorrow and issue an update if needed :slight_smile: I might have forgotten to close a reader.

I think I’ve found the issue. There was indeed a response body that I forgot to close. I’m adding it now and issue a new bug fix release :slight_smile:

Could you please test again with the latest LoRa App Server release (0.14.2). I indeed forgot to close the HTTP response body on each call:

1 Like