Display data in ThingsBoard

Hello. I’m looking for some guidance on how best to publish data to a separate app server. I have an instance of LoraServer running and ThingsBoard running on a different computer. At first I thought I could get Thingsboard to subscribe to an mtqq topic but it seems its an mtqq server as well. I tried the http integration but couldn’t get that to work I think because ThingsBoard is expecting a flat name pair json format and Loraserver is sending the whole lorawan datapacket:

https://thingsboard.io/docs/reference/protocols/

{“applicationID”:“1”,“applicationName”:“testapp”,“deviceName”:“seeeduino”,“devEUI”:“xxxxf6064bfdxxxx”,“rxInfo”:[{“mac”:“xxxx40ffff29xxxx”,“rssi”:-49,“loRaSNR”:7.2,“name”:“LairdRG191”,“latitude”:xx.xxxx,“longitude”:-xxx.xxxx,“altitude”:0}],“txInfo”:{“frequency”:904300000,“dataRate”:{“modulation”:“LORA”,“bandwidth”:125,“spreadFactor”:7},“adr”:true,“codeRate”:“4/5”},“fCnt”:358,“fPort”:8,“data”:“A4gHybTumY0BrIQ=”,“object”:{“gpsLocation”:{“3”:{“latitude”:51.0388,“longitude”:-114.0339,“altitude”:1097}}}}

I finally got Thingsboard to accept data by forwarding mtqq messages with a python script. Notice I had to pull out just the geolocation data before publishing to Thingsboard:

def on_connect(client, userdata, flags, rc):
print("Connected with result code "+str(rc))
client.subscribe(“application/1/node/xxxxf6064bfdxxxx/#”)

def on_message(client, userdata, msg):
print(msg.payload.decode())
a=json.loads(msg.payload.decode())
print(json.dumps(a[‘object’][‘gpsLocation’][‘3’]))
pclient.publish(“v1/devices/me/telemetry”,json.dumps(a[‘object’][‘gpsLocation’][‘3’]))

I feel like I’m publishing to Thingsboard the hard way. What is the proper way of doing this? Can I use the http integration by formatting the data first? What have other people done? Thanks.

1 Like

Hello @gradoj!

Do you have any authentication privilleges in your MQTT Server?

I have to login to the LoraServer mtqq server but the Thingsboard mtqq server is open. But it is working and updating Thingsboard just fine. I really just want to know what the design intention is for LoraServer to get the payload to an AppServer. Is there a better way to do this? More complete code:

import paho.mqtt.client as mqtt
import json

def on_connect(client, userdata, flags, rc):
  print("Connected with result code "+str(rc))
  client.subscribe("application/1/node/xxxxf6064bfdxxxx/#")

def on_message(client, userdata, msg):
  print(msg.payload.decode())
  a=json.loads(msg.payload.decode())
  print(json.dumps(a['object']['gpsLocation']['3']))
  pclient.publish("v1/devices/me/telemetry",json.dumps(a['object']['gpsLocation']['3']))

client = mqtt.Client()
pclient = mqtt.Client()
client.username_pw_set('xxxxxx', password='xxxxxx')
pclient.username_pw_set('xxxxIU0fw0CmcLnbxxxx', password=None)
client.connect("x.x.x.x",1883,60)
pclient.connect("x.x.x.x",1883,60)
 
client.on_connect = on_connect
client.on_message = on_message

client.loop_forever()

Using the above code, you received data or you issued any-connection-problem.

Again code works fine here is the output of a new message:

Connected with result code 0

{“applicationID”:“1”,“applicationName”:“testapp”,“deviceName”:“seeeduino”,“devEUI”:“xxxxf6064bfdxxxx”,“rxInfo”:[{“mac”:“xxxx40ffff29xxxx”,“rssi”:-55,“loRaSNR”:9.8,“name”:“LairdRG191”,“latitude”:xx.xxxx,“longitude”:-xxx.xxxx,“altitude”:x}],“txInfo”:{“frequency”:904100000,“dataRate”:{“modulation”:“LORA”,“bandwidth”:125,“spreadFactor”:7},“adr”:true,“codeRate”:“4/5”},“fCnt”:5,“fPort”:8,“data”:“A4gAAAAAAAAAAAA=”,“object”:{“gpsLocation”:{“3”:{“latitude”:x,“longitude”:x,“altitude”:x}}}}
{“latitude”: x, “longitude”: x, “altitude”: x}

So, you receive rc=0 (Connection established) and you parse json data, as in mqtt broker or http integration, right?

Yes, I connect to LoraServer mtqq server(client) rc=0 and register a new message callback. When a new message is received I pull out the GPS coordinates and publish them to the ThingsBoard mtqq server(pclient). The ThingsBoard displays the location data properly as it is supposed to.

As i’ve stated this works fine just looking for the ‘proper’ way to do this. I assume LoraServer is designed to work with other AppServers such as ThingsBoard or The Things Network. I just don’t understand how I would use the HTTP integration feature as it is. Has anyone used TTN or ThingsBoard as an AppServer with LoraServer?

I think you are a bit confused, as you are actually running loraserver and lora-app-server right now. The latter offers ways of publishing your data to other parties, through an mqtt topic or an http integration, but it will send the whole json format. Also, a custom codec will operate over the byte array data and add it to the payload, but will keep the rest of the schema.

So, what you are really looking for is to change lora-app-server’s behaviour, or at least extend it, and then publish your data. This may be achieved in 2 ways:

The first and most straightforward is the one that you are attempting, i.e. having a middleman subscribe to the data topics (or to an http integration), reformat the data to your needs and then send it over to your final endpoint.

The other way is to modify lora-app-server from source to either change the existing functionality or extend it. For example, in my lora-app-server I wrote a custom function that publishes a custom mqtt message to my frontend. It goes like this:

// processUpData creates an rxMessage and notifies the websocket channel.
func (h *MQTTHandler) processUpData(b []byte, payload handler.DataUpPayload) {
	var iface interface{}

	json.Unmarshal(b, &iface)
	ifaceMap := iface.(map[string]interface{})
	dataStr, _ := ifaceMap["data"].(string)

	rxMessage := storage.RxMessage{
		DevEUI:     payload.DevEUI,
		Message:    dataStr,
		Rssi:       int32(payload.RXInfo[0].RSSI),
		Snr:        payload.RXInfo[0].LoRaSNR,
		GatewayMAC: payload.RXInfo[0].MAC,
	}

	//Decode base64 message
	_, decodeErr := base64.StdEncoding.DecodeString(rxMessage.Message)
	if decodeErr != nil {
		log.Errorf("handler/mqtt: rx_message create error: decode error %s", decodeErr)
		return
	}

	var clientData storage.ClientData

	clientData, sErr := storage.CreateRxMessage(config.C.PostgreSQL.DB, config.C.Sparkpost.Client, &rxMessage)
	if sErr != nil {
		log.Errorf("handler/mqtt: rx_message create error: %s", sErr)
		return
	}

	jsonClientData, _ := json.Marshal(clientData)
	jsonMessage := string(jsonClientData)

	topic := fmt.Sprintf("application/%d/device/%s/data", payload.ApplicationID, payload.DevEUI)
	fmt.Printf("Sending to frontend topic: %s\n", topic)
	if token := h.conn.Publish(topic, 0, false, []byte(jsonMessage)); token.Wait() && token.Error() != nil {
		log.Errorf("handler/mqtt: publish data to frontend error: %s", token.Error())
	}

	//Send alarms

	for _, v := range clientData.Alarms {
		if v.Show && v.Type != "none" {
			jsonAlarm, _ := json.Marshal(v)
			alarmMessage := string(jsonAlarm)
			alarmTopic := fmt.Sprintf("application/%d/device/%s/alarm", payload.ApplicationID, payload.DevEUI)

			fmt.Printf("Sending alarm topic: %s\n", alarmTopic)
			if token := h.conn.Publish(alarmTopic, 0, false, []byte(alarmMessage)); token.Wait() && token.Error() != nil {
				log.Errorf("handler/mqtt: publish alarm to frontend error: %s", token.Error())
			}
		}
	}

}

So then I just extend the SendDataUp function from the mqtt handler, firing a goroutine that executes my function:

// SendDataUp sends a DataUpPayload.
func (h *MQTTHandler) SendDataUp(payload handler.DataUpPayload) error {
	b, err := json.Marshal(payload)
	if err != nil {
		return fmt.Errorf("handler/mqtt: data-up payload marshal error: %s", err)
	}

	topic := fmt.Sprintf("application/%d/node/%s/rx", payload.ApplicationID, payload.DevEUI)
	log.WithField("topic", topic).Info("handler/mqtt: publishing data-up payload")

	if token := h.conn.Publish(topic, 0, false, b); token.Wait() && token.Error() != nil {
		return fmt.Errorf("handler/mqtt: publish data-up payload error: %s", err)
	}

	//Everything's fine, process the data
	go h.processUpData(b, payload)

	return nil
}

Thanks I appreciate it. I understand but may have been using the wrong terminology. Yes, I am running lora-app-server but I was thinking of ThingsBoard as the app-server but I really should just call it the front-end I guess. Time for me to get the toolchain setup.

Or you could just use something like node-red to filter, reformat and publish your data from one MQTT server to the other. Depending on the scope of your project this might be much quicker.

Please note that LoRa App Server now integrates with ThingsBoard directly. This was released as LoRa App Server 3.1.0: [release] LoRa App Server v3.1

We have also made a guide to set this up: https://www.loraserver.io/guides/thingsboard/.

1 Like

@brocaar

Does it measn we can integrate Thingsboard with community edition ? or we need to have Professional Edition to integrate with LoRa-App-Server ?

Regards
Jayesh

I guess it works with both :slight_smile: I have tested with the open-source version / community edition.

1 Like