Overview

Prerequisite knowledge

To develop for Streambed, the following should be familiar to you:

• the Java Virtual Machine (JVM) and an associated language e.g. Java, Kotlin, Scala.
• an understanding of the problem that Reactive Streams solves
• optional, but helpful: an introduction to Akka Streams

The JVM has been targeted given its resilience, maturity and widespread adoption.

Reactive streams are leveraged primarily for resource control. These streams support back-pressure and provide an important characteristic for Streambed: resiliency.

Another important characteristic of reactive streams is that, when used with the right implementation (reactive streams are a specification, and not an implementation), blocking and synchronous operations are avoided. As your application will be deployed at the edge, resources will be limited and so being asynchronous and non-blocking is not simply desired; it is the law! Do not worry though, reactive streams and their associated libraries make all of this quite manageable for you.

Service gateways have limited resources; typically one processing core, less than 1GiB of RAM and non write-able file system.

Akka Streams is our preferred implementation of Reactive Streams given its maturity and widespread adoption.

Your Streambed applications

Here’s a diagram that will illustrate what your Streambed applications will be made up of including the application introduced here, “soilstate”.

Let’s break it down:

Public fdp repository

xDP provides a reference application that transforms LoRaWAN packets into soil moisture/temperature domain objects (soilstate-ui). soilstate-ui also provides a web frontend written using Angular and a backend using Java.

Public lora-sdk repository

The public lora-sdk repository includes a library known as the lora-control-plane. This library is for interfacing and controlling various LoRaWAN system components e.g. configuring the durable queue topic where the LoRaWAN Network Server will publish soilstate readings to. While the private lora-server repository (discussed next), includes applications to provision sensors, your applications have the ability to provide and leverage sensor metadata it e.g. add geo location data to describe a sensor’s position. Leveraging sensor metadata is actually something that the soilstate-ui application does.

Another notable library of this repository is lora-packet. This library provides all that is required to decode/encode and encrypt/decrypt LoRaWAN payloads. soilstate-ui is a consumer of this library so that it can transform LoRaWAN packets into soilstate domain objects. Each new type of sensor will typically require a soilstate-ui application.

Finally, there’s a lora-streams library that provides some conveniences for when dealing with streams and LoRaWAN packets.

Private lora-server repository

The private lora-server repository provides a proprietary Network Server that interfaces with the LoRaWAN network and integrates with Streambed’s durable queue. We mention it here only so that you can see the full picture in terms of production.

Public streambed repository

Streambed is an open source project located at https://github.com/streambed/streambed.

Developing applications for Streambed may feel familiar to you. Streambed avoids introducing proprietary components and consists broadly of the following:

• A durable queue abstraction backed by the open source Chronicle Queue from Chronicle Software
• An identity management abstraction backed by the IOx’s Secure Storage Service from Cisco
• Support of the popular Dropwizard Metrics toolkit from Yammer Inc, backed by statsd from Etsy
• Support of the Open Tracing standard backed by the open source Jaeger tracing system from Uber

There is also a Docker Compose suite of images providing a sandbox that can run locally on your machine. The sandbox includes some of the components that reside on a service gateway with the aim of supporting the ability to deploy your application and test it during development.

Your development lifecycle

Your development cycle will be something like:

1. Create a JVM based project using your favorite tools - the project should have a dependency on Streambed and any required LoRaWAN libraries
2. Write unit tests and verify your application’s behavior against Streambed and the LoRaWAN libraries
3. Use Docker Compose to launch an “Streambed sandbox” that provides common services e.g. Jaeger tracing, identity and the ability to send sensor observations
4. Run your program using your IDE or the java command, along with a debugger if required
5. Test your program
6. Fix problems and restart
7. Regularly profile your code and observe it with the Jaeger tracing UI provided by the sandbox - remember that resource usage is critical
8. Ship it! Form a Docker image of your project’s artifacts
9. Test running via Docker
10. You’re now ready to deploy the image to IOx via ioxclient

Shipping your software involves publishing your Docker image to our private Docker repository. We will then take care of its deployment after we verify that its runtime characteristics are within our constraints. The Streambed sandbox is configured to limit memory so that your application has the best chance of running well on a real service router.

An example: Soilstate UI

We have open sourced a project named “Soilstate UI” so that you can get started on writing your own applications. “Soilstate” refers to soil moisture/temperature sensor readings and their representation and consists of the following packages:

• soilstate
• soilstate.transformer
• soilstate.ui

All of Soilstate is written using Java 8 and uses Apache Maven as its build tool. In addition, the Java API of Akka Streams is used along with Akka HTTP as it integrates well with Akka Streams. Apache Maven is used as a build tool for the backend along with JUnit for testing.

soilstate

soilstate provides a domain object declaration representing soil moisture/temperature readings, the means to marshal them from a sequence of bytes and the means to marshal to and from JSON. The package is intended to be used by other packages, including soilstate.transformer and soilstate.ui.

soilstate.transformer

The soilstate.transformer package is a service that tails a queue that a LoRaWAN network server appends LoRaWAN packets to. The transformer decrypts and decodes the LoRaWAN application data bytes and forms a domain object. This object is then encoded as JSON, encrypted and then published to another queue.

soilstate.ui

Soilstate UI consists of two parts - the front end for running on the farmer’s phone or tablet and a back end that will run on a service router. The front end is written using Angular with the back end being written using Java.

The following pages will take you through the Soilstate UI project, explaining the backend first and then the front end.