An IoT project: Monitoring soil moisture (Phase 2) – Adding WiFi capability (Part 1)

In phase 1 of the project, we implemented a soil moisture monitoring device that was capable of only telling us the moisture level of the soil through an OLED display and buzzer.

For phase 2, we will be implementing WiFi connectivity for the Arduino so that it could start sending data to a central location. We can then build a dashboard with the data and give us a better picture of the soil status.

To achieve this, we need a WiFi module that will work with the Arduino. There are a few out there in the market but for this home project, we can go with the ESP-01 8266 WiFi module.

We will be splitting the topic on Adding WiFi capability into multiple articles to make it easier to follow through. In this article, we will be discussing the process of understanding and getting the WiFi module ready for the actual implementation.

Setting up the ESP-01 8266 WiFi module

In phase 1, we were using an Arduino Uno for soil moisture monitoring and we will leave it alone since it’s considered running in “production”. Instead, we will be using another Arduino Uno for the development of the WiFi connectivity feature.

We will upload a BareMinimum sketch to the Arduino to get rid of any pre-existing program that is running on the Arduino to test whether the WiFi module is working. Once that is completed, we can start hooking up the ESP-01 to the Arduino.

Let us first disconnect the Arduino from the computer to prevent any accidents such as short-circuiting.

Connecting the WiFi module

Then, we will get a breadboard and some male-male and female-male jumper wires. In my case, I got four male-male and five female-male jumper wires. You can also choose to go with colour-coding the wires so that you know which colour is used for what purpose. We connect the 3.3v and ground from the Arduino to the breadboard’s power rails with the jumper wires. The reason for 3.3V instead of 5V is because the module is designed to run on the former voltage level and has a max tolerance of 3.6V. Running the module on 5V will destroy it.

Next, we connect the Arduino’s TX/RX pins to the breadboard’s vertical rails with another pair of jumper wires. Then, connect the 3V3 and GND from the WiFi module to the power rails on the breadboard with the female-male wires. Following that, the EN (aka CH_PD) is connected to the 3V3 rail. This is the chip enable pin and pulling it high will enable the WiFi module.

This next part is where it gets a little tricky. Getting it wrong could prove fatal for the WiFi module.

This is where colour-coding the wires could prove very useful

Connect the TX pin from the WiFi module to the TX pin from the Arduino on the breadboard. In my case, I use blue wire to indicate this is the data receiving line from the perspective of the Arduino.

The next line to connect is the WiFi module’s RX pin. However, before we can connect it to the Arduino’s RX pin, we need to do something extra. Most of the Arduino digital pin output 5V instead of 3.3V when enabled. This is a problem for the WiFi module since this could destroy the chip. To get around this, we will use a voltage divider.

A voltage divider is a passive linear circuit that produces an output voltage that is a fraction of the input. In its most basic form, we could use two resistors connected in series with the input voltage applied across the resistor pair. The desired output voltage emerges from the connection between the resistor pair.

The output voltage is determined by the following formula:

Vout= (Vin x R2) / (R1   R2) =>
Where:
 R1 and R2 is the resistance value in Ohm
 Vin is the input voltage
 Vout is the output voltage

I used this website to help me make a quick determination of the resistance values I need.

For R1, I used 1.2 kiloohm (Resistor Colour: Brown, Red, Black, Brown, Brown) while R2 is 1.8 kiloohm (Resistor Colour: Brown, Grey, Black, Brown, Brown). This combination gives us a Vout of 3V which is more than sufficient for our purpose.

On the breadboard, R1 connects the Arduino’s RX pin to an empty vertical rail and R2 connects R1 to Ground. Then, we connect the WiFi module’s RX pin to the connection between the resistor pair. I use orange wires for the RX pins.

The final result looks like this:

Once the WiFi module is connected, we can power up the Arduino by connecting it to the computer over USB.

Testing the WiFi module

Once the Arduino is turned on, open the Arduino IDE (if you haven’t already) and then, open the Serial Monitor.

By default, the WiFi module comes with the AT firmware. The documentation for the available commands can be found here.

You can choose to use other type of firmware like the NodeMCU but for the project, we won’t be using that.

For the Serial Monitor, ensure the Baud rate is set to 1152000 and the dropdown option Both NL & CR is selected as WiFi AT firmware depends on the newline and carriage return to determine if a message is ready.

In the textfield, type “AT” and press Enter. This will send the AT command to the WiFi module. It should return an “Ok” response.

Note: All AT commands have to be uppercase.

Next, let’s check the version of the firmware running on the WiFi module by typing the command: AT GMR?

From the response, it appears that the WiFi module has a really old version of the firmware. We will need to update it to use newer commands.

If we go to the download section for the module on ExpressIf website, we can find the latest version of the AT firmware. However, that may not be the best firmware version for us to use at least according to the information found in this forum topic. Another user had attempted to flash the firmware (ESP8266 IDF AT Bin V2.0) to the latest as of the topic post but the WiFi module stopped responding to AT commands. Instead, v1.74 is the better version to use.

Since I’m using a Mac, I can’t run the default Flash Download Tool from ExpressIf website. It is for Windows only. What we can do is to use a python-based firmware flashing application call esptool. It was originally created by Fredik Ahlberg as a an unofficial community project and ExpressIf has since supported it.

We can install the esptool via pip command according to the README found here. After that, we will download the AT v1.74 firmware from here.

Unzip it to a folder of your choice and open Terminal on the Mac. Navigate to the folder containing the unzipped firmware and ensure you are in the ‘bin’ folder. For me, I’ve placed it in my Downloads folder.

Now, we are ready to start flashing the firmware.

Flashing ESP-01 8266 Firmware

Before the firmware flashing process, we need to do two things:

  1. Determine the port of the WiFi module.
  2. Get the module into a programmable state.

To determine the port, we can use the Arduino IDE. Navigate to Tools menu and under Ports, we should see the port the Arduino is connected to.

Take note of the port and then we can proceed to get the WiFi module into a programmable state.

The steps are as follows:

  1. Connect the RESET pin on the Arduino to Ground with a jumper wire.
  2. Connect the WiFi module’s GPIO_0 pin to Ground on the breadboard. Keep this pin connected throughout the whole flashing process.
  3. Connect the WiFi module’s RESET pin to Ground on the breadboard for exactly one second and disconnect it.

The WiFi module should now be ready for programming.

From the Terminal, type the following command. The value for the –port parameter should be the same port you determined earlier. Then, press Enter to start the flashing process.

esptool.py --port /dev/cu.usbmodem1432101 write_flash --flash_mode dout --flash_size 1MB 0x0 boot_v1.7.bin 0x01000 at/512 512/user1.1024.new.2.bin 0xfb000 blank.bin 0xfc000 esp_init_data_default_v08.bin 0xfe000 blank.bin 0x7e000 blank.bin

You should see the following during the flashing process.

After the flashing is done, let’s check the WiFi module to ensure it’s still working.

We will send the “AT” command and we should get an “Ok” response. After that, we send the AT GMR? command. We should get the following response:

Also, the baud rate by default runs at 115200 and that’s too fast. We will need to slow it down to 9600. To do so, we will need to use the AT UART_DEF command. Information about the command can be found here.

To set the baud rate, type the command AT UART_DEF=9600,8,1,0,0 and then press enter.

After running the command, change the baud rate on the Serial Monitor to 9600.

Then, run the AT UART_DEF? command.

The above looks good. The WiFi module is now updated and the baud rate is set. We are ready to go to the next stage.

Scanning for WiFi

In order for the program on the Arduino to communicate with the WiFi module, we will need to use another set of pins. By default, the TX and RX pins on the Arduino is used for Serial communication (e.g. USB). These are the pins that allow us to upload Sketch to the device and also used for the Serial Monitor. When other devices are connected to the Arduino via the TX and RX pins, we are using the Arduino as a USB to TTL serial adapter, which could interfere with the communication with the WiFi module in our case.

To use other Digital Pins on the Arduino for Serial Communication, we can use the SoftwareSerial library, which will emulate the remaining digital pins.

First, let’s disconnect the WiFi’s TX and RX pins from the Arduino. Then, create a new Sketch and define the following.

#include "SoftwareSerial.h"

const byte rxPin = 2;
const byte txPin = 3;

SoftwareSerial wifiSerial(2,3); //PIN 2 to receive data from WiFi, PIN 3 to transmit data to WiFi module

The above code is setting Pin 2 as the RX Pin and Pin 3 as the TX Pin from the perspective of the Arduino. What that means is that Pin 2 will be used to receive data from the WiFi module whereas Pin 3 will be used to send data to the WiFi module for transmission to the outside world.

Then, in the setup function of the Sketch, we will establish the USB serial connection to have a baud rate of 115200. After that, set the pinMode for the rxPin and txPin to INPUT and OUTPUT respectively. Then, we also initialise the SoftwareSerial library to run at 9600 baud.

void setup() {
  // put your setup code here, to run once:
  Serial.begin(115200);

  pinMode(rxPin, INPUT);
  pinMode(txPin, OUTPUT);
  
  wifiSerial.begin(9600);
}

Now, we are ready to connect the WiFi module to the Arduino. Unlike the earlier connection we did, the TX and RX pins are swapped around. We will connect the RX pin from the WiFi module to Pin 3 while the TX pin will go to Pin 2. As mentioned before, we have to be careful not to insert the WiFi TX pin to the Arduino Pin 3 since that pin will now be outputting 5V once the above Sketch is deployed to the Arduino.

To check that the WiFi module is working well with the SoftwareSerial, we will use it to query for available access points.

For this, I tried to send the AT CWLAP command via println function. Then, I will read the input. After several tries, I realised it’s actually very tedious and error-prone to work with serial communication.

I needed a better solution.

During my research, I found that the easiest library to use is the WiFiEsp. This library that can be found via the library manager of the Arduino IDE. After installing the latest version, we will proceed with implementing the WiFi access point scanning capability.

First, let’s modify the start of the Sketch to use the WiFiEsp library.

#include "WiFiEsp.h"

const byte rxPin = 2;
const byte txPin = 3;

#ifndef HAVE_HWSERIAL1
    #include "SoftwareSerial.h"
    SoftwareSerial wifiSerial(rxPin, txPin); //PIN 2 to receive data from WiFi, PIN 3 to transmit data to WiFi module
#endif

Then, modify the setup function as follows:


void setup() {
  // put your setup code here, to run once:
  Serial.begin(115200);

  pinMode(rxPin, INPUT);
  pinMode(txPin, OUTPUT);

  wifiSerial.begin(9600);
  WiFi.init(&wifiSerial);

}

Next, in the loop function, we do the following:

  Serial.println();
  Serial.println("Scanning available networks ... ");

  printNetworkScanResult();
  delay(10000);

For the printNetworkScanResult function, we do the following:


void printNetworkScanResult()
{
  int numSsid = WiFi.scanNetworks();
  while (numSsid == -1)
  {
    Serial.println("Couldn't get a WiFi connection");
    delay(3000);
    numSsid = WiFi.scanNetworks();
  }


  Serial.print("Number of available networks: ");
  Serial.println(numSsid);

  for (int i = 0; i < numSsid; i  )
  {
    Serial.print(i   1);
    Serial.print(")");
    Serial.print(WiFi.SSID(i));
    Serial.println();
  }
}

Once the sketch is deployed to the Arduino and it started running, look at the response returned in the Serial monitor. We should see a list of WiFi access point.

Finally, now we know that the WiFi module is working.

And, this is a good point to end this article before I run the risk of droning on. In the next article, we will look at implementing the actual WiFi communication and integrating it with our existing soil moisture monitoring program.

What is Wi-Fi 6 and why it is a game changer?

Wi-Fi 6 is the consumer friendly name for the IEEE 802.11ax, which is the current generation Wi-Fi specification standard and the successor to Wi-Fi 5, also known as IEEE 802.11ac.

This new standard comes with various improvements to efficiency and throughput. It is also backwards compatible with your older devices since it supports both 2.4 Ghz and 5 Ghz bands.

But, before we can dive deep into explaining what is Wi-Fi 6 and advantages it bring to the table, we need to be on the same page regarding certain terminologies. We shall also setup some analogies that we will use to explain technical details in simpler terms.

To connect to the internet, we will need to install either a modem or gateway provided to us by internet service providers. If it is the former, a router typically is connected to the modem to provide additional connection points such as ethernet ports and Wi-Fi for our devices to connect to in order to access the internet. If a gateway is used instead, the gateway itself is also a router and could also provide the same functionality as a standalone router.

With that in mind, let us imagine the router to be a large warehouse in a middle of a small city. It has a number of gates or main doors that represent antennas. In addition, there are buildings (or clients, if you will), which represents devices such as smartphones and laptops. Then, there are also couriers who represent radio waves. These couriers are the middlemen between the warehouse and the various buildings, and are responsible for transporting packages to the mall and back to their respective clients. These packages contain things, which are pieces of data, that the building need.

It is fast

Wi-Fi 6 has a theoretical maximum speed of 9.6 Gbps (gigabits per second) or 1.2 gigabyte per seconds. That is almost 2.6 times more than Wi-Fi 5. To put things in perspective, it is possible to transfer a 100GB 4K blu-ray video in just under two minutes with such speed.

There are three key technological updates that contribute to Wi-Fi 6’s improved performance over the previous generation: MU-MIMO, OFDMA and 1024-QAM

MU-MIMO

MU-MIMO stands for “multi-user, multiple input, multiple output”. It is a technology that help increase the number of antennas a router has and could be found implemented in most Wave 2 Wi-Fi 5 (IEEE 802.11ac) routers and devices.

Wave 1 Wi-Fi 5 and earlier routers could only communicate with one device one at a time and the others have to wait until it is their turn.

With MU-MIMO release for Wave 2 routers, they could communicate with up to four devices simultaneously.

The caveat here is that, the location of each client device matters. If two or marke devices are in the same general location, they will still need to wait in line to communicate with the router.

And you might wonder how does this technology help to improve performance?

Let us assume that a router will be placed in the center of the house or a room. Then, if there are multiple devices around that require Wi-Fi access, they could access the router via the nearest antenna pointed in their respective general direction. If two or more devices are in the same general location, then these devices will share the same antenna and need to wait for their turn.

For those who have a hard time understanding that, let us go back to the analogy we created earlier.

When the warehouse is first built, which represent the earlier version of Wi-Fi, there is only one gate. In order to enter or exit, couriers need to queue up by the gate and wait for their turn before they can proceed to either deliver or collect their respective packages.

Then, with MU-MIMO technology for Wave 2 Wi-Fi 5, the warehouse undergoes a renovation to have four gates installed according to the points on a compass instead of just the one. Now with four gates, more couriers can enter and exit albeit from four different directions at any given point in time. With this, the buildings are now able to get their requests serviced faster.

But if a particular gate has a long queue of couriers, they will still need to wait for their turn to enter or exit unless they choose to use other gates.

And for Wi-Fi 6, MU-MIMO is upgraded to support up to eight devices at the same time.

Going back to the warehouse example, renovation is done to add four more gates, for a total of eight. More couriers now can enter and exit the warehouse from eight directions at any given point in time, therefore servicing even more buildings.

OFDMA

Orthogonal frequency division multiple access (OFDMA)is an extension of Orthogonal frequency division multiplexing (OFDM) technology.

But, before we can dive deeper into OFDMA, we must first understand what is OFDM.

OFDM is a technology that takes a radio channel such as the 20 Mhz channel, which is often used for Wi-Fi, and sets a number of sub-carriers instead of having just one carrier. For Wi-Fi 5, 52 sub-carriers can be created from a single 20 Mhz channel using this technology while Wi-Fi 6 take that to the next level and can create 234 sub-carriers.

To the uninitiated, a carrier is a modulated wave that convey information.

Each sub-carrier is then modulated independently and simultaneously to form symbols, which are waveforms that represent information or data. And these symbols are separated in time by guard intervals to prevent interference cause by neighbouring sub-carriers. Finally, a single transmission to the receiving device will consist of a number of these simultaneous symbols spanning the sub-carriers.

The receiving device is able to track all these sub-carriers simultaneously and extract data from each sub-carrier independently. This contributes to the increase in Wi-Fi performance since multiple pieces of data are transmitted at the same time.

However, the problem with OFDM is that any device is free to transmit a signal whenever they are ready, creating a first-come-first serve situation. This may work in a home-setting where there are not a lot of devices but does not work in a high-density area such as a stadium and shopping mall. Too many devices will be fighting for a chance to send and receive data from the router, resulting in an inefficient use of the router.

For those who have trouble following the above explanation, let us go back to the warehouse example we used earlier.

OFDM can be thought of as multiple couriers who are dispatched to a specific building but at slightly different timings. They travelled on the same road but on a different lane, and each of them carries a different part of the final data. Once they arrived at their destination, the building’s manager will collect the different parts of the data and began the reconstruction process. Even if some of the couriers got lost, the manager is still able to reconstruct the data because each courier carries a nice little clipboard containing a detailed description of the content and its relation to the other couriers.

However, there could be couriers serving other buildings traveling on the same road. Some of these couriers could end up on a lane where other couriers are on. This can lead to a scenario where the couriers would fight amongst themselves in order gain access the warehouse first. During the fight, packages will be lost and when that happens, the buildings need to dispatch couriers with the same packages again in an attempt to gain access to the warehouse.

This is where OFDMA comes in.

OFDMA technology solves the network contention issue by grouping the sub-carriers into Resource Units (RU) to service one or more clients depending on their needs.

Therefore, if there is a client that need higher bandwidth because of the data it is downloading, then all the sub-carriers can be grouped as one Resource Unit to give the client the full bandwidth of the channel. Similarly, if multiple clients in the same area need a fraction of the bandwidth because of their small data requirement, then multiple Resource Units can be created to serve all of the clients.

And the change in the RU configuration is also done in real time, therefore enabling a consistent efficient use of the available network bandwidth.

To the layman, it is the equivalent of upgrading the warehouse to dispatch one truck per lane to serve a group of buildings that are close together. The purpose of the truck would be to carry as many couriers as it can hold and transport them to the group of buildings where they could then drop or pick up the packages.

When does the warehouse dispatches the trucks and decides whether the couriers on board all serve the same or different building are dependent on the requirements such as the service type, package size and total number of packages.

With that, the odds of couriers losing their packages and having to restart the transport process again, which is an overhead, is reduced. Furthermore, the different buildings (clients) in the same area get an equal amount of attention from the warehouse.

1024-QAM

Wi-Fi 6 improves on the amount of data transmitted per signal, allowing improvement in speed by up to 30%. This means that you can stream bandwidth hungry content such as 4K video with further reduction in loading times and have a smooth viewing experience.

But before we could dive deeper into how Wi-Fi 6 achieve this, we need to understand how Wi-Fi work in general.

Wi-Fi works by using radio waves. To transmit data so that the receiving device understand it, the sender need to modulate the signal to represent bits of binary code. This type of modulation is known as “Quadrature amplitude modulation” or QAM for short.

The better a device is at modulation, the more information it can transmit each time.

For example, a 2-QAM device means it capable of transmitting one bit (1 or 0) of information each time because it can modulate the signal in one of the two ways. A 4-QAM device can transmit 2 bits (00, 01, 10, 11) of information each time because it can modulate a signal four different ways.

With that in mind, current generation of Wi-Fi 5 devices are 256-QAM, which means eight bits of information can be transmitted each time. This is why most of us today do not spend a lot of time waiting for video to load and buffer. With Wi-Fi 6, devices are able to do 1024-QAM, which means 10 bits of information can now be transmitted each time.

To explain QAM much simply, let us go back our warehouse example.

2-QAM is the equivalent of the courier only having one hand. They can either carry one bag or nothing at all. 4-QAM give them another hand, so now they can carry up to two packages. 256-QAM for Wi-Fi 5 is the equivalent of giving a courier four pairs of hands, thereby enabling them to carry up to eight packages. With the upgrade to 1024-QAM for Wi-Fi 6, each courier now has five pairs of hands to carry up to ten packages.

On the surface it may not look like much. However, if a request is for a large amount of data such as those typically found during 4K movie streaming, having the ability to transfer more data per trip will mean less trips needed to download the full content. After all, less trips equals more time saving.

It could improve battery life

Other than being faster, Wi-Fi 6 also comes with a new feature call Target Wake Time. This allows certified Wi-Fi 6 routers to schedule check-in times with connected devices.

With scheduling, devices only activate their antennas at the right time instead of having to keep their antennas powered on to transmit or search for signals for an extended period of time, which can consume quite a fair amount of power.

For devices such as laptops or desktops which are connected to a power source and do need persistent internet connection, this feature may not be useful. But for IoT devices it could be a world of difference since they may not have access to consistent power and probably run on batteries.

It has better security

Since 2004, the Wi-Fi security revolves around WPA2. It is a protocol that encrypts the communication session between the router and the client device so that they could exchange information safely and privately.

WPA2 was considered to be very secure until 2017 when a weakness in the protocol was discovered that made it possible for attackers in range of the Wi-Fi router to steal sensitive information.

The Wi-Fi Alliance announced WPA3 in 2018 to be the replacement. WPA3 replaces the need for the 4-way handshake to authenticate a client in WPA2 with another method called Simultaneous Authentication of Equals (SAE).

SAE is a proven zero-knowledge method to establish a secret shared key that both the client and the router will use to generate the session key to encrypt and decrypt Wi-Fi transmissions. If another client wishes to connect to the network, the client will established its own secret shared key with the router.

The other important feature of WPA3 is Forward Secrecy, which is an indirect effect of implementing SAE. This ensure that even if an attacker managed to capture the encrypted Wi-Fi transmissions and then crack the session key, older data continue to remain unaccessible as the keys used to encrypt those data will be different.

WPA3 is optional for existing devices and many device manufacturers may choose not to patch these products via firmware update. But in order for these manufacturers to market their devices to be Wi-Fi 6 certified, the Wi-Fi Alliance mandated that WPA3 be implemented. Therefore, we can be sure that Wi-Fi 6 will be more secure.

Other than improvements made to the WPA protocol, the security and privacy of open Wi-Fi networks such as those we find in cafes, shopping malls and stadiums are also improved. Wi-Fi 6 will see the implementation of Opportunistic Wireless Encryption (OWE).

OWE is a security technique that is similar to SAE to encrypt the transmission channel between the device and the router but without the need for authentication. The established shared key is only known to the client device and the router.

Although it is not as secure since there is no way to tell who is connected to what, it is more secure than connecting to a public Wi-Fi secured by WPA2 and using a the pre-shared password, or connecting to a completely open Wi-Fi network.