Earlier this year, I made a decision with my BitFarm co-founder to close down the project. We turned off our servers on AWS and a two-year project came to an end.
Using data to help farmers make better decisions
In early 2014, I had been building drones for a few years and was investigating applicability of the technology for agriculture. My friend Ben was was interested in applying his data science skills to analyzing geospatial data acquired via drones, and another friend, Alex, was interested in helping write software to make the system work. We were all working full time (Ben Cook at Hudl, Alex DeBrie and I as corporate lawyers), but BitFarm began: we were going to create software to help farmers derive insight from their data.
Drone experimentation
I built a few fixed wing drones for data collection during the spring of 2014. We also spent considerable effort thinking through the cost of acquiring this data. When considering the benefit provided to the farmer, we quickly learned that drones simply were not cost effective.
We used drones to collect a ton of data that summer, and even developed our own imaging system based on a Raspberry Pi and an assortment of different sensors. After examining this data with growers to understand their needs, it was clear that the most immediate benefit we could give to farmers is information about when to irrigate their crops and in what quantity.
Drones simply were not a great tool for this job, primarily because they required an operator to drive to the location at least once a week to operate the drone and acquire the desired data. We concluded that drones were not the right tool for that job and that semi-permanent sensors made more sense.
Build our own sensors and data analytics application
Once we concluded that in-field sensors would capture the data our customers wanted, we began looking for off-the-shelf sensors solutions. We quickly learned one reason why these tools haven’t been adopted broadly: existing sensor solutions were very expensive and not connected wirelessly. We decided to build our own sensor nodes, connecting them to the trusted and widely available Watermark soil moisture sensor.
Our sensors had another benefit: they enabled us to wirelessly and automatically collect data from multiple points in the field. This data gave us information about the state of a field that was more representative of reality than existing commercial solutions that only sampled at one location.
Our sensors were refined throughout the summer of 2014 and 2015. By the end of 2015, these sensors collected hundreds of thousands of data points while deployed in the field for months at a time.
Timeline
All of the work done on BitFarm was done in our free time. After a busy summer in 2014 spending evenings and weekends deploying and maintaining sensor systems deployed all over Nebraska (and some in South Dakota), we took a few months off from the project. We ramped back up in January of 2015, redesigning our sensor nodes and creating a new web application for the 2015 growing season. We landed our first paying customers during the summer of 2015, including a grant-funded research project with the University of Nebraska.
How the BitFarm system worked
The BitFarm system consisted of sensor nodes to acquire the data in the field, base stations to aggregate data collected from all nodes in a field, backend servers to collect, process, and store the data, and a user-facing website to facilitate review of the data.
The sensors
As with most “Internet of Things” projects, the our sensors started as Arduino prototypes. I was primarily responsible for designing the sensors and writing the firmware.
Once I got the prototypes working with the Arduino, I began designing custom PCBs containing the components I needed. I went with the trusty ATMEGA328P chip in an SMT form factor, along with a clock, SD card reader, and other components.
The sensor node version we deployed in 2014 did not initially include wireless connectivity, but wrote the data to the SD card. Due to an unfortunate firmware bug combined with an underpowered clock battery, our power consumption on these nodes was higher than it should have been and many of the nodes we deployed died a month or so into testing.
The redesigned nodes we deployed during the summer of 2015 fared much better. All of the nodes deployed in 2015 included wireless connectivity.
These nodes could also accommodate up to 9 Watermark sensors, five more than the 2014 versions. We used both 900mhz and 2.4ghz radios on these nodes, with the 900mhz performing optimally in field conditions. These nodes also consumed far less power: they successfully read sensors every hour for about six months until they were removed from the field.
Each deployment of sensor nodes was accompanied by the deployment of a base station. The base station was composed of a solar powered Raspberry Pi with a custom PCB mounted to it and a cellular modem for connectivity. The base stations aggregated data from individual nodes and forwarded it to our server via cellular connection. We used Fluentd on the base station and backend server to reliably transfer data between these systems.
The server stack
BitFarm’s backend infrastructure on AWS using EC2, S3, and RDS. We had one EC2 instance running Fluentd that served as an aggregator for our basestations in the field. This instance would receive data points from the basestations and send them on for processing and archiving to S3. When processing the data, a background worker would validate and clean the data before saving it to a Postgres instance running in RDS. This processing helped transform the data from the raw electrical signals from the sensors into a soil moisture metric that was more understandable for our customers. Additionally, we did some rollup aggregations of the data to enable quicker display of summarized data to end users.
The data was displayed to end users through a simple web app. We used Django to create the app, and it was hosted on a few EC2 instances. We ran the app in Docker containers, which made it easy to deploy changes and roll back a faulty deploy. The app would communicate with our Postgres instance to grab and display readings for users. Django has a lot of great packages and enabled us to move quickly to add features to the app. For example, in the summer of 2015, we worked on a pilot project with a partner to fully automate an irrigation system. The partner needed to interact with our system via an API. We used the excellent Django Rest Framework package to quickly add an API to our app.
The interface
To display data to users, we created a front-end website using leveraging Django, Bootstrap, and Highcharts.js. Here’s what the website interface looked like:
The interface was designed to give the user the most important information first: do they need to water the crop? This information was derived by processing the raw data we captured in the field based on soil type and other data we collected. This radically simplified this process for the farmer. Instead of consulting an agronomy manual to determine what a sensor reading meant, they could simply log on to our website and quickly understand the status of their field.
By selecting the “field view” users were able to view more detailed time-series plots from their field. We also presented crop maturity and evapotranspiration to give users a better idea of the conditions impacting their crops.
Why we decided to shut it down
We ultimately decided to quit working on BitFarm because 1) the team simply didn’t have time to continue working on it during our free time, and 2) market forces (particularly low commodity prices) made the technology commercially less viable in the short term.
Building a technology business during evenings and weekends is extremely difficult. There were several times during 2014 and 2015 when the BitFarm team considered jumping in and making BitFarm a full-time business. However, the timing never worked out and we had our reservations about the market. Particularly with my move to California last year, sustaining BitFarm progress during our free time becoming futile.
What I learned
Building reliable hardware and software solutions is incredibly hard
I also discovered that building reliable software systems is incredibly difficult. When we first started BitFarm, we had ambitions of creating a data analytics platform for agriculture. We wanted to combine sensor data with drone data and other available information to provide synthesized answers for customers. We quickly learned that even doing a portion of this work was enormously difficult.
Even when the requirements, end goals, and technical implementation is clear, building technical products is extremely hard work. This difficulty is compounded when hardware is involved. The main lesson for me is that a successful technology product requires discipline and intense focus. A working, reliable system almost always takes more time to create than initially predicted.
Building software to solve problems is incredibly rewarding
Despite the challenges, the main lesson I learned from working on BitFarm is that building technology products that do useful things in the world is incredibly rewarding. My experience working on BitFarm – and the skills developed while doing so – were a main catalyst for leaving my previous career to pursue a career in software and technology.