Help testing the next LoRa (App) Server release with LoRaWAN 1.1 support

[brocaar]

LoRa Server v1 has been released not too long ago and it marked a stable version after many v0.x releases with some internal code-cleanup (migration code).

Meanwhile I’ve also been working on what will become the LoRa (App) Server release v2. The two main features / changes that this release will include are:

• LoRaWAN 1.1 support
• A cleaner API

LoRaWAN 1.1

Together with Stackforce (the guys behind https://github.com/Lora-net/LoRaMac-node) I have been working on adding LoRaWAN 1.1 support to LoRa Server. There are still a couple of areas that have been untested at the LoRa Server side, but in general this is fully functional Note that this means that both LoRaWAN 1.0 and LoRaWAN 1.1 will be supported.

API cleanup

Over the years many API changes were made, also introducing some inconsistencies. As the introduction of LoRaWAN 1.1 already introduces some backwards incompatible changes to the API with regards to the root keys I also took this as an opportunity to go through the entire API and make sure all methods, messages and fields are consistent.

When using the LoRa App Server REST API, probably the biggest change you will see that a lot of objects / messages have been made re-usable. Example to demonstrate this:

Old API

POST /api/devices

{
  "name": "test-device",
  "devEUI": "0102030405060708",
  "applicationID": "123"
  ...
}

New API

POST /api/devices

{
  "device": {
    "name": "test-device",
    "devEUI": "0102030405060708",
    "applicationID": "123"
    ...
  }
}

This means that the device object / gRPC message (when using the gRPC API) is now the same for create, get and update. So after a GET you can update the fields of the device object and use this device object in your POST.

As before, the REST API is documented at the LoRa App Server /api endpoint

---

This is still work in progress, but I would love your feedback! When you’re using Docker, you can grab the :latest tag instead of :1 for both LoRa App Server as LoRa Server. You can also use the code from the master branch of both repositories. Soon I will also publish a first test release to the 2.x debian repository (testing channel).

Looking forward to your feedback!

[iegomez]

This is great!

We are looking to host an open newtork in Chile (something like TTN, but loraserver based), so I’ll be working and modifying a little (allow users to registrate, maybe a couple of integrations etc.) the latest version, which means I’ll always keep on sync with the master branch. So yeah, we’ll be testing it soon enough and I’ll let you know of any issues we encounter. On the same line, I’m curious if you are still interested on the ADR disabling option that I promised long ago (sorry, been really busy) as this would be a perfect time to add it.

Cheers!

[brocaar]

On the same line, I’m curious if you are still interested on the ADR disabling option that I promised long ago (sorry, been really busy) as this would be a perfect time to add it.

That would be great! Please note that in the master branch there is already a disable_mac_commands flag (which disables all mac-command processing). Something like that, but then for ADR only would be good to have too

[iegomez]

Great, I’ll take a look at it this or next week.

[brocaar]

I’ve just tagged a first version for testing!

See https://github.com/brocaar/lora-app-server/releases/tag/v2.0.0-alpha.1 and https://github.com/brocaar/loraserver/releases/tag/v2.0.0-alpha.1.

Debian / Ubuntu

For testing with Ubuntu / Debian, you can use the 2.x testing repository:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 1CE2AFD36DBCCA00

sudo echo "deb https://artifacts.loraserver.io/packages/2.x/deb testing main" | sudo tee /etc/apt/sources.list.d/loraserver.list
sudo apt-get update

Docker (Compose)

Use the Docker Compose example from: https://github.com/brocaar/loraserver-docker. And update the docker-compose.yml file so that loraserver/loraserver:2.0.0-alpha.1 and loraserver/lora-app-server:2.0.0-alpha1 images are used.

Looking forward to your feedback!

[Alexandre_Pelagaggi]

This means that the API is now more flexible, regarding the properties of the device?

In other Words, can I set any device properties like fCntDown or nwkSKey by just posting an update of what i expect to be my device Object in LoRa App Server runtime enviroment? Is there any unsetable properties?

Thank you Brocaar.

[brocaar]

Flexible as in objects are re-usable so less code is needed when you do for example a get and then an update, not as in that more properties are available to set. When you install the test release using the above explained steps, the full API interface documentation is available when you navigate in your browser to the /api endpoint (as explained by: https://www.loraserver.io/lora-app-server/integrate/rest/).

[Alexandre_Pelagaggi]

@brocaar

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 1CE2AFD36DBCCA00 sudo echo “deb Index of /packages/2.x/deb/ testing main” | sudo tee /etc/apt/sources.list.d/loraserver.list sudo apt-get update

After running those commands I got the Following Error from App Server:

lora-app-server[13794]: time="2018-07-04T16:08:21-03:00" level=fatal msg="applying migrations error: pq: could not create unique index \"idx_device_name_application_id\" handling 0032_fix_table_constraints.sql"

Have I done something Wrong?

[Alexandre_Pelagaggi]

@Alexandre_Pelagaggi

lora-app-server[13794]: time="2018-07-04T16:08:21-03:00" level=fatal msg="applying migrations error: pq: could not create unique index \"idx_device_name_application_id\" handling 0032_fix_table_constraints.sql"

Apparently it was an postgre restriction added in this app_server version, I fixed by doing the following.

sudo -u postgres psql
\c loraserver_as
\dt
select * from devices;
# Check the device with repeated Name and take note of devEUI like \xaf515f237a0cb927 of one repeated device.
delete from devices where dev_eui = 'One_of_the_devices_dev_eui';

I am leaving my question here because it may help someone some day.

[Alexandre_Pelagaggi]

I notice That in this new app Server Version in the Device activation Tab, the frame-counter validation does not shown marked. But when checking via REST API it does. as Shown.

• app Server UI
  
  

• app Server REST UI

Output has.

Also noticed That Using the REST API to add messages in deviceQueue I end up eith data null field regardless of input data,and every field except devEUI seems wrong.

POST/api/devices/611441a962f80636/queue

{
	"confirmed": true,
	"data": "FwAAAAAAAAAAAAA=",
	"devEUI": "611441a962f80636",
	"fPort": 0,
	"reference": "abcde1234"
}

Always geting the following output.

{
      "devEUI": "611441a962f80636",
      "reference": "",
      "confirmed": false,
      "fPort": 0,
      "data": null,
      "jsonObject": ""
}

Also Would be nice to get back the fCnt field back from GET/api/devices/{devEUI}/queue as shown in Example Value.

"items": [
    {
      "confirmed": true,
      "data": "string",
      "devEUI": "string",
      "fCnt": 0,
      "fPort": 0,
      "reference": "string"
    }

Also noticed that in the example in GET/api/devices/{devEUI}/queue it show “items” as array of queue objects and in output i get “deviceQueueItems”. I do not know if this will have any implication.

Any ideia how to fix this? Should I rollback My LoRa App Server?

[Felipe]

@Alexandre_Pelagaggi

Always geting the following output.

{
      "devEUI": "611441a962f80636",
      "reference": "",
      "confirmed": false,
      "fPort": 0,
      "data": null,
      "jsonObject": ""
}

I’ve also got the same problem here

Upon making a POST call to /api/devices/{devEUI}/queue with pretty much anything such as

{
"Anything":"BLABLABLA"
}

the result is the same output as stated by Alexandre.

{
      "devEUI": "600a8c29b2ce8277",
      "reference": "",
      "confirmed": false,
      "fPort": 0,
      "data": null,
      "jsonObject": ""
} 

[brocaar]

Thanks @Alexandre_Pelagaggi and @Felipe this is really helpful feedback I’ll make sure this gets fixed (probably today) and will end up in the next testing release (probably next week). If you would need a new .deb between the fix and the next testing release, please let me know and I can create a snapshot for you.

Update: I have checked. The Skip frame-counter field was removed from the activation API as this was moved to the device object. In 1.x this was still at two places to be backwards compatible. In v2 this will be available only on configuration of the device.

With regards to your enqueue issue, please see the /api interface for examples (and maybe make sure your cache is cleared?). Note that between the latest testing version I have made some additional changes: https://github.com/brocaar/lora-app-server/commit/effa41b016169cd89f5a1fcddefb99dc52d412ca. This means that you’ll now get the fCnt back (which maps to the fCnt in case of an acknowledgement) and the reference has been removed.

This works as expected:

(I do see that this doesn’t expose the fCnt, I’ll add this).

[brocaar]

A preview of what will probably be in the next testing release (this is still wip):

This doesn’t add new features to the web-interface, but it should make the whole UI easier to use as the main components are now located in the sidebar

(@iegomez this is what I was referring to in my DM to you)

[iegomez]

That looks pretty nice!

On a side note, yesterday we found a bug in the UI where when entering as an admin you’d see the gateways link, but after pressing some other tab it would disappear and you’d had to refresh to get the gateways link back. I didn’t have time to fix it and knew you were preparing a new UI, but thought I’d mention it just in case it remained.

[brocaar]

Yes, I noticed the same (I think the state update isn’t handled correctly when the organization is loaded). However I’m planning to merge the new UI into master hopefully somewhere next week

[brocaar]

I’ve just released a new testing release of LoRa App Server (v2.0.0-alpha.2). This includes the new UI. I hope you all like it Happy testing and please share your feedback!

Please see the above Debian / Ubuntu and Docker (Compose) instructions.

[iegomez]

Hi, I’ve merged the changes and am testing everything right now. I still need to reimplement our modifications, but so far I’ve encountered a couple of little bugs:

Gateway details fails on getting stats because of nil timestamps. Just changing the timestamps fields names from camel to snake case fixed it:

start_timestamp: start,
end_timestamp: end,

Surely it’s something related to how they are defined at the proto, but i’m just getting familiar with new protobuf notations, new ways of calling the swagger client instead of fetching from the uris, etc.

After deleting an organization, it is fetched when returning to the organizations list, resulting in an “object does not exist” warning. I see that getPage is passed as a prop to DataTable, but again, being new to Material UI, I prefer just to let you know.

One suggestion: maybe service profile list and edit views could mention the organization id (which is already fetched), or better, the organization name (though it involves adding the message field and having to get the organization when getting the service profile). I noticed it because after you explained what’s the service profile “role” I created default ones programatically when users register, all named the same. Of course one could be careful with the name selection, but it wouldn’t hurt to be able to link the SP with the organziation visually.

I’ll keep digging during the day as I add our implementations and let you know of any other detail that I may find, but it’s looking really promising and we like the feeling of the new UI very much. Great work!

[brocaar]

Thanks @iegomez

Gateway details fails on getting stats because of nil timestamps. Just changing the timestamps fields names from camel to snake case fixed it:

That is odd, when testing using the v2.0.0-alpha.2 tag (git checkout v2.0.0-alpha.2, which maps to commit b127a33ccd92fc44364cc92083de127d34ec636e) I don’t have any issues with getting the gateway stats. You mentioned you have merged your changes. Could it be that your version is not exactly the same as the v2.0.0-alpha.2 tag?

After deleting an organization, it is fetched when returning to the organizations list, resulting in an “object does not exist” warning. I see that getPage is passed as a prop to DataTable, but again, being new to Material UI, I prefer just to let you know.

Will look into this. Thanks!

One suggestion: maybe service profile list and edit views could mention the organization id (which is already fetched), or better, the organization name

Note that this is already visible. The selected organization (to which the service-provide, device-profile, …) is shown in the sidebar:

Or do I misunderstand you?

Thanks for all the feedback

Update: I already know what you mean with the start_ and stop_timestamp. I made a local change to the Swagger API generator which is called when calling make api. See also:

gen-swagger: use GetJsonName for struct member names

diff --git a/protoc-gen-swagger/genswagger/template.go b/protoc-gen-swagger/genswagger/template.go
index c4f2b8d..0d206b7 100644
--- a/protoc-gen-swagger/genswagger/template.go
+++ b/protoc-gen-swagger/genswagger/template.go
@@ -107,7 +107,7 @@ func queryParams(message *descriptor.Message, field *descriptor.Field, prefix st
                        desc = strings.TrimSpace(schema.Title + ". " + schema.Description)
                }
                param := swaggerParameterObject{
-                       Name:        prefix + field.GetName(),
+                       Name:        prefix + field.GetJsonName(),
                        Description: desc,
                        In:          "query",
                        Type:        schema.Type,
@@ -143,7 +143,7 @@ func queryParams(message *descriptor.Message, field *descriptor.Field, prefix st
                return nil, fmt.Errorf("unknown message type %s", fieldType)
        }
        for _, nestedField := range msg.Fields {
-               p, err := queryParams(msg, nestedField, prefix+field.GetName()+".", reg, pathParams)
+               p, err := queryParams(msg, nestedField, prefix+field.GetJsonName()+".", reg, pathParams)
                if err != nil {
                        return nil, err
                }
@@ -251,7 +251,7 @@ func renderMessagesAsDefinition(messages messageMap, d swaggerDefinitionsObject,
                                panic(err)
                        }

-                       schema.Properties = append(schema.Properties, keyVal{f.GetName(), fieldValue})
+                       schema.Properties = append(schema.Properties, keyVal{f.GetJsonName(), fieldValue})
                }
                d[fullyQualifiedNameToSwaggerName(msg.FQMN(), reg)] = schema
        }

I assume you re-generated the API code which was causing a mismatch of names because of the above change I made. What I did was kind of a local quickfix to work around this issue, then I forgot I made it as the swagger code is comitted in the repo, so the build server doesn’t complain I will make a fork of the code generator and will add my fork as a requirement so when you run make requirements and then make api you will end up with the same result.

[iegomez]

Yeah, I regenerated the API with make api after merging, so that was it.

As for the service profiles, I mean when I enter as an admin, I may fully list all service profiles:

There are 2 profiles belonging to 2 different organizations with id 1 (for organization"loraserver") and 23 (for organization “org-1”), but with no way to know about that other than checking with the developer tools. Moreover, if I hit the “loraserver service profile” one, which belongs to the “loraserver” organization, “org-1” remains selected at the side bar: