Looking for feedback on configuration files

Since the beginning the LoRa Server components have been using cli flags (e.g. --mqtt-server) and environment variables (e.g. MQTT_SERVER) for configuration.

As the set of features is expanding, this key / value approach has its limits. Take for example this feature: https://github.com/brocaar/loraserver/issues/271. Ideally that would be represented as an array where each element has a frequency and a set of valid data-rates. This is currently doable but you would get config values like 868700000:X-Y,868900000:X-Y.

There are also cases where environment variables are not so practical. My idea is to start using more structured configuration files. E.g. for the LoRa Gateway Bridge this could look like:

[general]
udp_bind = "0.0.0.0:1700"
log_level = 4
skip_crc_check = false

[backend.mqtt]
server="tcp://127.0.0.1:1883"
username="loraserver_gw"
password="password"
ca_cert="/etc/lora-gateway-bridge/certs/ca.pem"

Then when starting lora-gateway-bridge it would look for a lora-gateway-bridge.toml file at some standard locations (e.g. /etc/lora-gateway-bridge/lora-gateway-bridge.toml, $HOME/.config/lora-gateway-bridge/lora-gateway-bridge.toml and ./lora-gateway-bridge.toml) or you could specify a --config flag, pointing to a configuration file.

My idea is to initially make this fully backwards compatible, so when no config file is present, it will still keep working with the already configured environment variables and flags, but it will print a warning that you should update your configuration. Then after some versions / time, the flags and environment variable parsing will be fully removed.

What do you think?

Just some quick thoughts:

There are also cases where environment variables are not so practical.

What cases are you thinking about?

I agree that it’s better structured in the proposed way. The con is that (when cli options are fully removed) you lose the ability to quickly start loraserver or other with just a few tweaks to test something. A common case of course being log level modification when something goes wrong and you need to check debug messages, but any other has the same con. To deal with it, in a simple approach you’d end up writing several conf files for different use cases, and then giving the path to the --config path.

But maybe you could add something like a --conf-env flag, so you could write loraserver.prod.toml, loraserver.dev.toml, loraserver.stage.toml, loraserver.some_custom_env.toml, etc., store them in the standard locations and then just run ./loraserver --conf-env dev and the loraserver.dev.toml would be used. Just an idea, there’s probably some better way of doing this, but it was the first that came to my mind.

Finally, if you are going to deprecate it later, why keep the temporal compatibility? It’ll break sooner or later. I know there are some that update through their software manager and don’t get to see the changes, but warning that it’ll be deprecated versus warning/erroring that it already changed and you should write a new toml conf file should work pretty much the same.

To aid on the transition, I was thinking that it should be fairly easy to write a “conf file to toml” (maybe using GitHub - BurntSushi/toml: TOML parser for Golang with reflection.) so that when someone meets the conf error, they can just download the “translator” and run it against their conf file to get new toml ones. If you are interested, I’d be happy to write something up to get this done.

What cases are you thinking about?

E.g. when you want to update the configuration, without restarting the full process for example. But my main painpoint is that is works great for key=value store, but with more complex configuration with for example map structures, or arrays of objects, it doesn’t work well.

A common case of course being log level modification

Actually, I could leave a couple of basic flags still available so you are still able to override e.g. the log-level from the cli, without the need to update the config and go back and forward.

Finally, if you are going to deprecate it later, why keep the temporal compatibility? It’ll break sooner or later. I know there are some that update through their software manager and don’t get to see the changes, but warning that it’ll be deprecated versus warning/erroring that it already changed and you should write a new toml conf file should work pretty much the same.

It is not much of an effort to support both and it will be appreciated by some of my clients / contacts / sponsors :slight_smile: Actually the framework that I have tried (GitHub - spf13/viper: Go configuration with fangs + GitHub - spf13/cobra: A Commander for modern Go CLI interactions) has great support for this.

To aid on the transition, I was thinking that it should be fairly easy to write a “conf file to toml”

I was thinking the same! Even starting with a warning instead of a error directly makes sense, as it could render the environment variables in a text template and display that to the user. E.g. by:

./lora-gateway-bridge migrate-config > /etc/lora-gateway-bridge/lora-gateway-bridge.toml`

If you’re interested, this is a working WIP which I created today for testing: https://github.com/brocaar/lora-gateway-bridge/compare/config_files

Update: the current WIP includes a command to generate a config file, by simply calling ./lora-gateway-bridge config. As this is generated from a template and the current config object is used to populate the fields, it will either print all the defaults (when no previous config is set) or can be used to migrate from env. variables to the config file. This can also be used to migrate to a newer config file version. E.g. when new options gets added, you could do ./lora-gateway-bridge config --config old-config.toml > new-config.toml (as it will render the template with the old config options populated and the new fields to their default values).

@iegomez actually I think I’m going to script the migration also inside the .deb post-install hook. That way you can just do an apt upgrade and the migration is automatically done for you :slight_smile:

Update:

root@iot-test:~# dpkg -i lora-gateway-bridge_2.2.0-9-g336dbf2_amd64.deb
(Reading database ... 94027 files and directories currently installed.)
Preparing to unpack lora-gateway-bridge_2.2.0-9-g336dbf2_amd64.deb ...
Unpacking lora-gateway-bridge (2.2.0-9-g336dbf2) over (2.2.0) ...
Removed symlink /etc/systemd/system/multi-user.target.wants/lora-gateway-bridge.service.
Setting up lora-gateway-bridge (2.2.0-9-g336dbf2) ...




-----------------------------------------------------------------------------------------
Your configuration file has been migrated to a new location and format!
Path: /etc/lora-gateway-bridge/lora-gateway-bridge.toml
-----------------------------------------------------------------------------------------
1 Like

Well, it’s settled then. Great work!

This is how the LoRa App Server configuration would look like. Does this layout seem logical? (/cc @iegomez)

[general]
# debug=5, info=4, warning=3, error=2, fatal=1, panic=0
log_level=4

# The number of times passwords must be hashed. A higher number is safer as
# an attack takes more time to perform.
password_hash_iterations=100000


# PostgreSQL settings.
#
# Please note that PostgreSQL 9.5+ is required.
[postgresql]
# postgresql dsn (e.g.: postgres://user:password@hostname/database?sslmode=disable)
dsn="postgres://localhost/loraserver_as?sslmode=disable"

# automatically apply database migrations
automigrate=true


# Redis settings
#
# Please note that Redis 2.6.0+ is required.
[redis]
# redis url (e.g. redis://user:password@hostname/0) (default: "redis://localhost:6379")
url="redis://localhost:6379"


# Application-server settings.
[application_server]
# random uuid defining the id of the application-server installation (used by LoRa Server as routing-profile id)
id="6d5db27e-4ce2-4b2b-b5d7-91f069397978"


  # MQTT integration configuration used for publishing (data) events
  # and scheduling downlink application payloads.
  # Next to this integration which is always available, the user is able to
  # configure additional per-application integrations.
  [application_server.integration.mqtt]
  # MQTT server (e.g. scheme://host:port where scheme is tcp, ssl or ws)
  server="tcp://localhost:1883"

  # Connect with the given username (optional)
  username=""

  # Connect with the given password (optional)
  password=""

  # CA certificate file (optional)
  #
  # Use this when setting up a secure connection (when server uses ssl://...)
  # but the certificate used by the server is not trusted by any CA certificate
  # on the server (e.g. when self generated).
  ca_cert=""


  # Settings for the "internal api"
  #
  # This is the API used by LoRa Server to communicate with LoRa App Server
  # and should not be exposed to the end-user.
  [application_server.internal_api]
  # ip:port to bind the api server (default: "0.0.0.0:8001")
  bind="0.0.0.0:8001"

  # ca certificate used by the api server (optional)
  ca_cert="/Users/brocaar/go/src/github.com/brocaar/loraserver-certificates/certs/ca/ca.pem"

  # tls certificate used by the api server (optional)
  tls_cert="/Users/brocaar/go/src/github.com/brocaar/loraserver-certificates/certs/lora-app-server/api/server/lora-app-server-api-server.pem"

  # tls key used by the api server (optional)
  tls_key="/Users/brocaar/go/src/github.com/brocaar/loraserver-certificates/certs/lora-app-server/api/server/lora-app-server-api-server-key.pem"

  # public ip:port of the application-server api (used by LoRa Server to connect back to LoRa App Server)
  public_host="localhost:8001"


  # Settings for the "external api"
  #
  # This is the API and web-interface exposed to the end-user.
  [application_server.external_api]
  # ip:port to bind the (user facing) http server to (web-interface and REST / gRPC api) (default: "0.0.0.0:8080")
  bind="0.0.0.0:8080"

  # http server TLS certificate
  tls_cert="certs/as-http.pem"

  # http server TLS key
  tls_key="certs/as-http-key.pem"

  # JWT secret used for api authentication / authorization
  # You could generate this by executing 'openssl rand -base64 32' for example
  jwt_secret="verysecret"

  # when set, existing users can't be re-assigned (to avoid exposure of all users to an organization admin)"
  disable_assign_existing_users=false


  # Gateway discovery configuration.
  #
  # When enabled, each gateway will periodically broadcast a discovery "ping"
  # which other gateways in the network are able to receive and which is
  # presented in the web-interface as a map.
  [application_server.gateway_discovery]
  # Enable the gateway discovery feature.
  enabled=false

  # the interval used for each gateway to send a ping
  interval="24h0m0s"

  # the frequency used for transmitting the gateway ping (in Hz)
  frequency=868100000

  # the data-rate to use for transmitting the gateway ping
  dr=5


# Join-server configuration.
#
# LoRa App Server implements a (subset) of the join-api specified by the
# LoRaWAN Backend Interfaces specification. This API is used by LoRa Server
# to handle join-requests.
[join_server]
# ip:port to bind the join-server api interface to
bind="0.0.0.0:8003"

# ca certificate used by the join-server api server
ca_cert="/Users/brocaar/go/src/github.com/brocaar/loraserver-certificates/certs/ca/ca.pem"

# tls certificate used by the join-server api server (optional)
tls_cert="/Users/brocaar/go/src/github.com/brocaar/loraserver-certificates/certs/lora-app-server/join-api/server/lora-app-server-join-api-server.pem"

# tls key used by the join-server api server (optional)
tls_key="/Users/brocaar/go/src/github.com/brocaar/loraserver-certificates/certs/lora-app-server/join-api/server/lora-app-server-join-api-server-key.pem"


# Network-server configuration.
#
# This configuration is only used to migrate from older LoRa App Server.
[network_server]
server="127.0.0.1:8000"

The only weird thing is the integration.mqtt naming, which is inconsistent with other names under application_server. The rest looks fine.

You’re right. That is a typo and should have the same prefixes as the other blocks.

This is how the loraserver configuration is going to look like :slight_smile:

[general]
# Log level
#
# debug=5, info=4, warning=3, error=2, fatal=1, panic=0
log_level=4


# PostgreSQL settings.
#
# Please note that PostgreSQL 9.5+ is required.
[postgresql]
# PostgreSQL dsn (e.g.: postgres://user:password@hostname/database?sslmode=disable).
#
# Besides using an URL (e.g. 'postgres://user:password@hostname/database?sslmode=disable')
# it is also possible to use the following format:
# 'user=loraserver dbname=loraserver sslmode=disable'.
#
# The following connection parameters are supported:
#
# * dbname - The name of the database to connect to
# * user - The user to sign in as
# * password - The user's password
# * host - The host to connect to. Values that start with / are for unix domain sockets. (default is localhost)
# * port - The port to bind to. (default is 5432)
# * sslmode - Whether or not to use SSL (default is require, this is not the default for libpq)
# * fallback_application_name - An application_name to fall back to if one isn't provided.
# * connect_timeout - Maximum wait for connection, in seconds. Zero or not specified means wait indefinitely.
# * sslcert - Cert file location. The file must contain PEM encoded data.
# * sslkey - Key file location. The file must contain PEM encoded data.
# * sslrootcert - The location of the root certificate file. The file must contain PEM encoded data.
#
# Valid values for sslmode are:
#
# * disable - No SSL
# * require - Always SSL (skip verification)
# * verify-ca - Always SSL (verify that the certificate presented by the server was signed by a trusted CA)
# * verify-full - Always SSL (verify that the certification presented by the server was signed by a trusted CA and the server host name matches the one in the certificate)
dsn="postgres://localhost/loraserver_ns?sslmode=disable"

# Automatically apply database migrations.
#
# It is possible to apply the database-migrations by hand
# (see https://github.com/brocaar/loraserver/tree/master/migrations)
# or let LoRa App Server migrate to the latest state automatically, by using
# this setting. Make sure that you always make a backup when upgrading Lora
# App Server and / or applying migrations.
automigrate=true


# Redis settings
#
# Please note that Redis 2.6.0+ is required.
[redis]
# Redis url (e.g. redis://user:password@hostname/0)
#
# For more information about the Redis URL format, see:
# https://www.iana.org/assignments/uri-schemes/prov/redis
url="redis://localhost:6379"


# Network-server settings.
[network_server]
# Network identifier (NetID, 3 bytes) encoded as HEX (e.g. 010203)
net_id="010203"

# Time to wait for uplink de-duplication.
#
# This is the time that LoRa Server will wait for other gateways to receive
# the same uplink frame. Valid units are 'ms' or 's'.
deduplication_delay="200ms"

# Device session expiration.
#
# The TTL value defines the time after which a device-session expires
# after no activity. Valid units are 'ms', 's', 'm', 'h'. Note that these
# values can be combined, e.g. '24h30m15s'.
# Please note that this value has influence on the uplink / downlink
# roundtrip time. Setting this value too high means LoRa Server will be
# unable to respond to the device within its receive-window.
device_session_ttl="744h0m0s"

# Get downlink data delay.
#
# This is the time that LoRa Server waits between forwarding data to the
# application-server and reading data from the queue. A higher value
# means that the application-server has more time to schedule a downlink
# queue item which can be processed within the same uplink / downlink
# transaction.
# Please note that this value has influence on the uplink / downlink
# roundtrip time. Setting this value too high means LoRa Server will be
# unable to respond to the device within its receive-window.
get_downlink_data_delay="100ms"


  # LoRaWAN regional band configuration.
  #
  # Note that you might want to consult the LoRaWAN Regional Parameters
  # specification for valid values that apply to your region.
  # See: https://www.lora-alliance.org/lorawan-for-developers
  [network_server.band]
  # LoRaWAN band to use.
  #
  # Valid values are:
  # *   AS_923
  # * AU_915_928
  # * CN_470_510
  # * CN_779_787
  # * EU_433
  # * EU_863_870
  # * IN_865_867
  # * KR_920_923
  # * US_902_928),
  name="EU_863_870"

  # Enforce 400ms dwell time
  #
  # Some band configurations define the max payload size for both dwell-time
  # limitation enabled as disabled (e.g. AS 923). In this case the
  # dwell time setting must be set to enforce the max payload size
  # given the dwell-time limitation. For band configuration where the dwell-time is
  # always enforced, setting this flag is not required.
  dwell_time_400ms=false

  # Enforce repeater compatibility
  #
  # Most band configurations define the max payload size for both an optional
  # repeater encapsulation layer as for setups where a repeater will never
  # be used. The latter case increases the max payload size for some data-rates.
  # In case a repeater might used, set this flag to true.
  repeater_compatible=false


  # LoRaWAN network related settings.
  [network_server.network_settings]
  # Installation margin (dB) used by the ADR engine.
  #
  # A higher number means that the network-server will keep more margin,
  # resulting in a lower data-rate but decreasing the chance that the
  # device gets disconnected because it is unable to reach one of the
  # surrounded gateways.
  installation_margin=10

  # Class A RX1 delay
  #
  # 0=1sec, 1=1sec, ... 15=15sec. A higher value means LoRa Server has more
  # time to respond to the device as the delay between the uplink and the
  # first receive-window will be increased.
  rx1_delay=1

  # RX1 data-rate offset
  #
  # Please consult the LoRaWAN Regional Parameters specification for valid
  # options of the configured network_server.band.name.
  rx1_dr_offset=0

  # RX2 data-rate (when set to -1, the default rx2 data-rate will be used)
  #
  # Please consult the LoRaWAN Regional Parameters specification for valid
  # options of the configured network_server.band.name.
  rx2_dr=-1

  # Enable only a given sub-set of channels
  #
  # Use this when ony a sub-set of the by default enabled channels are being
  # used. For example when only using the first 8 channels of the US band.
  #
  # Example:
  # enabled_uplink_channels=[0, 1, 2, 3, 4, 5, 6, 7]
  enabled_uplink_channels=[]


  # Extra channels to use for ISM bands that implement the CFList
  #
  # Use this for LoRaWAN regions where it is possible to extend the by default
  # available channels with additional channels (e.g. the EU band).
  # Note: the min_dr and max_dr are currently informative, but will be enforced
  # in one of the next versions of LoRa Server!
  #
  # Example:
  # [[network_server.network_settings.extra_channels]]
  # frequency=867100000
  # min_dr=0
  # max_dr=5

  # [[network_server.network_settings.extra_channels]]
  # frequency=867300000
  # min_dr=0
  # max_dr=5

  # [[network_server.network_settings.extra_channels]]
  # frequency=867500000
  # min_dr=0
  # max_dr=5

  # [[network_server.network_settings.extra_channels]]
  # frequency=867700000
  # min_dr=0
  # max_dr=5

  # [[network_server.network_settings.extra_channels]]
  # frequency=867900000
  # min_dr=0
  # max_dr=5


  # Network-server API
  #
  # This is the network-server API that is used by LoRa App Server or other
  # custom components interacting with LoRa Server.
  [network_server.api]
  # ip:port to bind the api server
  bind="0.0.0.0:8000"

  # ca certificate used by the api server (optional)
  ca_cert=""

  # tls certificate used by the api server (optional)
  tls_cert=""

  # tls key used by the api server (optional)
  tls_key=""

  # Gateway API
  #
  # This API is used by the LoRa Channel Manager component to fetch
  # channel configuration.
  [network_server.gateway.api]
  # ip:port to bind the api server
  bind="0.0.0.0:8002"

  # CA certificate used by the api server (optional)
  ca_cert=""

  # tls certificate used by the api server (optional)
  tls_cert=""

  # tls key used by the api server (optional)
  tls_key=""

  # JWT secret used by the gateway api server for gateway authentication / authorization
  jwt_secret=""

  # Gateway statistics settings.
  [network_server.gateway.stats]
  # Create non-existing gateways on receiving of stats
  #
  # When set to true, LoRa Server will create the gateway when it receives
  # statistics for a gateway that does not yet exist.
  create_gateway_on_stats=false

  # Aggregation timezone
  #
  # This timezone is used for correctly aggregating the statistics (for example
  # 'Europe/Amsterdam').
  # To get the list of supported timezones by your PostgreSQL database,
  # execute the following SQL query:
  #   select * from pg_timezone_names;
  # When left blank, the default timezone of your database will be used.
  timezone=""

  # Aggregation intervals to use for aggregating the gateway stats
  #
  # Valid options: second, minute, hour, day, week, month, quarter, year.
  # When left empty, no statistics will be stored in the database.
  # Note, LoRa App Server expects at least "minute", "day", "hour"!
  aggregation_intervals=["minute", "hour", "day"]


  # MQTT gateway backend settings.
  #
  # This is the backend communicating with the LoRa gateways over a MQTT broker.
  [network_server.gateway.backend.mqtt]
  # MQTT server (e.g. scheme://host:port where scheme is tcp, ssl or ws)
  server="tcp://localhost:1883"

  # Connect with the given username (optional)
  username=""

  # Connect with the given password (optional)
  password=""

  # CA certificate file (optional)
  #
  # Use this when setting up a secure connection (when server uses ssl://...)
  # but the certificate used by the server is not trusted by any CA certificate
  # on the server (e.g. when self generated).
  ca_cert=""

  # TLS certificate file (optional)
  tls_cert=""

  # TLS key file (optional)
  tls_key=""


# Default join-server settings.
[join_server.default]
# hostname:port of the default join-server
#
# This API is provided by LoRa App Server.
server="http://localhost:8003"

# ca certificate used by the default join-server client (optional)
ca_cert=""

# tls certificate used by the default join-server client (optional)
tls_cert=""

# tls key used by the default join-server client (optional)
tls_key=""


# Network-controller configuration.
[network_contoller]
# hostname:port of the network-controller api server (optional)
server=""

# ca certificate used by the network-controller client (optional)
ca_cert=""

# tls certificate used by the network-controller client (optional)
tls_cert=""

# tls key used by the network-controller client (optional)
tls_key=""

Nice!

I think you mistakenly copy-pasted something here though:

# Device session expiration.
#
# The TTL value defines the time after which a device-session expires
# after no activity. Valid units are 'ms', 's', 'm', 'h'. Note that these
# values can be combined, e.g. '24h30m15s'.
# Please note that this value has influence on the uplink / downlink
# roundtrip time. Setting this value too high means LoRa Server will be
# unable to respond to the device within its receive-window.
device_session_ttl="744h0m0s"
1 Like

You’re right! Will fix that (it should have been under the key above the device_session_ttl).

Thanks for all the reviews @iegomez :slight_smile: I’ve merged all the config changes (LoRa Gateway Bridge, LoRa Server and LoRa App Server) into the master branches. I hope to release this later today, else tomorrow morning!

1 Like

Even though I’m still getting used to these configurations and parameters, I think your layouts make sense.

Thanks again for helping with the network-server issue (Can not register network server).
I believe this new configuration method shall help with running the servers from the cli.

1 Like

This has been released https://forum.loraserver.io/t/release-lora-gateway-bridge-2-3-0-lora-server-0-24-0-lora-app-server-0-18-0/805/2

I’ll close this topic. Please share your feedback in the above topic :slight_smile:

2 Likes