How the user input values-text fields, in lora-app-server, post that value to golang at backend?

Hello, Everyone & @brocaar !

I am trying to look for the backend code (written in golang) which accepts or gets value from the LoRa-App-Server’s text fields (frontend: React.js) and later on inserts that data into the database (postgres & Redis).

For Example: As LoRa-App-Server is connected to LoRa Server over gRPC, when we create a Gateway on LoRa-App-Server, we have to fill this GatewayForm having GatewayDetails, like GatewayName, GatewayID…,etc.

My question is, that, after clicking on the ‘CREAT GATEWAY’ button, where does all of this data is ‘posted’ or ‘stored’ in Golang, such that Golang inserts all of this data, into the database??

It would be great if anybody can guide me to that file, where all the data is posted from React.js to Golang.

Thank You!

Handlers for the available API methods are located at /internal/api/external, so in this case you’d want to look at https://github.com/brocaar/lora-app-server/blob/master/internal/api/external/gateway.go#L36.

There you’ll see that -among other things- the gateway is created by calling the storage.CreateGateway function from the storage package, and calling the external gRPC method nsClient.CreateGateway, from the network server (i.e. loraserver).

The former is defined at https://github.com/brocaar/lora-app-server/blob/master/internal/storage/gateway.go#L95.

The latter at https://github.com/brocaar/loraserver/blob/master/internal/api/network_server.go#L796.

Hope that helps.

2 Likes

Hi, @iegomez! Thank you for taking the time to reply!

Correct me if I’m wrong, but as per my understanding you mean to say is, that ;

  1. we fetch the user-input value from the Lora-App-Server, through CreateGatewayRequest struct (https://github.com/brocaar/lora-app-server/blob/cf02d43439f0a5c863316e75d9bdd58a1652cb94/api/gateway.pb.go#L194 ),
  1. Then, it calls ns.CreateGatewayRequest (https://github.com/brocaar/lora-app-server/blob/cf02d43439f0a5c863316e75d9bdd58a1652cb94/internal/api/external/gateway.go#L61), copies mac[:] into Id.

  2. Then calls the storage.CreateGateway to store the value of mac into MAC:, in the database, (https://github.com/brocaar/lora-app-server/blob/cf02d43439f0a5c863316e75d9bdd58a1652cb94/internal/api/external/gateway.go#L99).

  3. After storing it, it calls the external gRPC method of loraserver, using nsClient.CreateGateway. (but for what?) (possibly for sending the copied Id from mac[:] to loraserver? : To store it in gw.GatewayID[:](https://github.com/brocaar/loraserver/blob/253bb9e6d96fac97d07cdda6912c1d178d99cc0d/internal/api/network_server.go#L814)).

Did I get all these points right?

Also, @iegomez, I know for a fact that Lora-app-server posts and gets the data from the application(React.js) in protobuf format through gRPC API.

But what if I want to get and post data in JSON format over HTTP? Is it possible?

Don’t bother checking *pb.go files, those are autogenerated when you compile the protobuf services (i.e., when calling make api). Instead, check the proto definition and the actual implementation, e.g., /api/gateway.proto and the mentioned files.

Both lora-app-server and loraserver store gateway information, as the first needs to allow the user to manage them, while the second does all the LoRaWAN network related work and thus needs to be aware of gateways.

That’s not right. The frontend application communicates with lora-app-server through HTTP using the exposed REST API built by the grpc-gateway. In short, lora-app-server defines it’s API as gRPC services, but it also exposes a REST one by autogenerating HTTP endpoints from your proto definitions. That’s why services methods define an HTTP verb and an endpoint uri, e.g. a PUT request for a gateway update:

// Update updates the gateway matching the given mac address.
	rpc Update(UpdateGatewayRequest) returns (google.protobuf.Empty) {
		option (google.api.http) = {
			put: "/api/gateways/{gateway.id}"
			body: "*"
		};
	}

When calling make api, not only will the proto definitions get compiled to Go code for those services (/api/*.pb.go files), it’ll also call the grpc-gateway plugin to generate HTTP endpoints for those services too (/api/*.pb.gw.go files) and the swagger plugin to generate swagger definitions for your API (/api/swagger/*.swagger.json files).

The React app that’s shipped with lora-app-server imports the swagger-client package to make use of the swagger definitions to implement the clients, but if you open up your inspector at the browser you’ll see regular HTTP requests going to the server and you may try those in any HTTP client.

So yeah, that’s exactly what happens and you can do it yourself too.

1 Like

Thanks for your reply, @iegomez! This explanation cleared a lot of things for me.

However, One more question,in reference to this:

This point is clear that to support RESTful JSON API, HTTP endpoints are generated inside the .proto files.

But since the react-ui posts JSON data over HTTP, and then how does the grpc-gateway plugin of protoc convert this JSON data posted by HTTP endpoints into protobuf? So that the data in protobuf format can be posted over gRPC.

The autogenerated /api/*.pb.gw.go files do it (please check one of them for more details), basically by getting the json data from the *http.Request and decoding it to the correct proto request (depending on the method), calling the gRPC service method with this proto request and then forwarding back the response to the client by translating it back to json.

For details about registering services for either the gRPC or the HTTP APIs, check the /internal/api/external/external.go file from lora-app-server.

1 Like

Thank you @iegomez, your explanation helped me get better understanding of the code. I had one more doubt in my mind:
That React.js runs in localhost:8080, if we make our own Golang API to get the JSON encoded data, how can we get the data from the same port i.e. 8080, so that it doesn’t show the error : port:8080 already in use?

Sorry, I don’t think I understand your question. What do you mean with your own Go API to get the json encoded data? What data? From where? I thought you wanted to extend lora-app-server's API, not create a separate one.

1 Like

Sorry, I wasn’t clear with my question. Yes, I wanted to do that only. So aren’t the lora-app-server/api made on Golang? Can you guide me to some files which point to the API code, such as, for e.g., sending downlink to the device through the api?

Well, then this doesn’t make much sense:

You don’t want to run a separate API or service in that port, you want to modify lora-app-server's source, build it and then run it as usual. As mentioned in earlier posts, that usually means modifying your gRPC services definitions and implementations, which are located at /api/service-name.proto and /internal/api/external/service-name.go respectively. You don’t need to worry about the REST conversion, that’s taken care for you by the grpc-gateway plugin when you build the proto definitions.

1 Like

Hi there!

You mean to say that I just change the .proto files, and another section will be created in the localhost:8080/api ?

It depends on what you mean by another section. If you mean a new endpoint in some service, e.g. /api/devices/your-new-endpoint, then you only need to modify the proto definitions, run make api and implement the endpoint at /internal/api/external/device.go.

If instead you want to create a new service, then you need to create the new your-service.proto file, register it at /api/gen.sh to be picked up by protoc and the plugins that generate the REST gateway and Swagger definitions, then run make api and finally implement the service at /internal/api/external/your.service.go.

1 Like

Hi, @iegomez, Thank you for such quick replies!

Yes, I want to create a new end point in the same service. For example, from your answer, I got that if I want to add a new end-point in localhost:8080/api/DeviceQueueService, then I have to go to api/deviceQueue.proto , add a new end-point, then the lora-app-server/api/deviceQueue.pb.go file will be generated by protoc, after that I have to add the necessary changes in the , device_queue.go file.

It would be great if you could tell me if I am correct?

If I am correct, then can you tell me how to store the new variables introduced in the new end-point in the database?

Also, these changes are only for lora-app-server, do we have to make the similar changes in the loraserver/api also?

Yes, that’s what I said. For storing data you’ll have to call to functions from the storage package that you’ll have to implement too. Check some endpoint implementations for examples on logic and which things they call.

If you need to change anything at loraserver, then yes.

1 Like

Hey, @iegomez! Thank you for your extremely helpful insights!

I wanted to ask, how are these protoc generated swagger definitions, deployed on localhost:8080/api ?

I mean, I know that all the filename.swagger.json files, are merged in the main.go file. But after that how is it integrated with react such that it is deployed on localhost??

They are put at static/swagger when building. From the Makefile:

static/swagger/api.swagger.json:
	@echo "Generating combined Swagger JSON"
	@GOOS="" GOARCH="" go run api/swagger/main.go api/swagger > static/swagger/api.swagger.json
	@cp api/swagger/*.json static/swagger

So everything UI related ends up at the static dir (check ui/build at that Makefile, you’ll see after built it gets copied to static too), which is served by lora-app-server.

As for how the React UI uses the definitions, they are called from the stores using the swagger-client package:

import Swagger from "swagger-client";

After imported, each store loads the specific client:

class ApplicationStore extends EventEmitter {
  constructor() {
    super();
    this.swagger = new Swagger("/swagger/application.swagger.json", sessionStore.getClientOpts());
  }
...
}
2 Likes

So, after adding an endpoint in the .proto file, and running make api command, will I have to make changes to the respective .js file also, which is importing that respective auto-generated .swagger.json file? or those changes will be automatically added?

Hi! I just tried editing the .proto file and ran make api after setting the GOPATH, but it is showing various errors. Then, I removed the changes made by me and ran make api on the existing original code, it still showed error.

Attached below is the screenshot:


It would be great if you could tell me, where I went wrong.

You are probably using a different protoc version. Try uninstalling it, checking what versions of the tools are being used in the projects (check go.mod) and then install the correct version of protoc for those tools. You may encounter some more of these problems, you’ll need to debug them yourself because they may vary depending on your configurations. Also check the forum for already asked questions and other resources.

You’ll have to do the changes yourself, i.e. implement the methods that call the new endpoints.

2 Likes