Network connectivity issues, resource instabilities, and other system state changes can affect connectivity to the device and potentially lead to a test failure. It can be helpful to get ahead of unfavorable system state changes by implementing an alert system.

You should begin this section by creating a Gmail account that is only used for these alert systems. It is not recommended that you alert using a personal email account. Next, install necessary Python libraries:

sudo apt install msmtp msmtp-mta
sudo apt install mailutils

IP Address Notification

The following describes an example to alert a user about potential IP address changes. This alert system uses Gmail to send out a notification every time the RPi reboots containing the device’s IP address. This is not only useful to connect to a device plugged into a new network for the first time, but is also a useful indicator for a device’s unscheduled reboot. First, create a file named ip_mailer.py in your home directory and add the code below. Three lines need to be modified: the recipient, the gmail_user and the gmail_password.

import subprocess
import smtplib
from email.mime.text import MIMEText
import datetime

def connect_type(word_list):
    if 'wlan0' in word_list or 'wlan1' in word_list:
        con_type = 'wifi'
    elif 'eth0' in word_list:
        con_type = 'ethernet'
    else:
        con_type = 'current'

    return con_type

# Gmail Account Information
recipient = 'email to send to' # Recipient
gmail_user = '[email]@gmail.com' # Sender (MUST BE GMAIL)
gmail_password = 'password' # Gmail password.
smtpserver = smtplib.SMTP('smtp.gmail.com', 587) # Gmail smtp server

# Establish a connection to the gmail server
smtpserver.ehlo()  
smtpserver.starttls() 
smtpserver.ehlo()
smtpserver.login(gmail_user, gmail_password) 
today = datetime.date.today() 

# Get the unit's IP address
arg='ip route list'  
p=subprocess.Popen(arg, shell=True, stdout=subprocess.PIPE)
data = p.communicate()  # Get data from 'p terminal'.

# Split IP text block into containing IPs
ip_lines = data[0].splitlines()
split_line_a = ip_lines[1].split()
ipaddr_a = split_line_a[split_line_a.index('src')+1]

# Get the connection type, e.g. 'ethernet', 'wifi', ...
ip_type_a = connect_type(split_line_a)

# Creates the sentence sent via email stating the IP address
my_ip_a = 'Your %s ip is %s' % (ip_type_a, ipaddr_a)

# Creates the text, subject, 'from', and 'to' of the message.
msg = MIMEText(my_ip_a + "\\n") # + my_ip_b)
msg['Subject'] = 'IPs For 019 on %s' % today.strftime('%b %d %Y')
msg['From'] = gmail_user
msg['To'] = to

# Sends the message
smtpserver.sendmail(gmail_user, [recipient], msg.as_string())

# Closes the smtp server.
smtpserver.quit()

Then, modify /etc/rc.local to execute the script whenever the system boots up. Edit rc.local to look similar to this:

# Print the IP address
**sleep 30                                  # add this line**
_IP=$(hostname -I) || true
if [ "$_IP" ]; then
  printf "My IP address is %s\\n" "$_IP"
  **python /home/pi/ip_mailer.py            # add this line**
fi

exit 0

Adding the sleep parameter makes sure that a network connection has been established before the Python script is executed.

Disk Alert

This alert system uses Gmail again to send out an email when the RPi’s local storage reaches a set threshold. To configure the system, begin by opening /etc/msmtprc with an editor of your choice (the file is empty by default) and add the following:

# Generics
defaults
auth           on
tls            on
tls_trust_file /etc/ssl/certs/ca-certificates.crt

# user specific log location, otherwise use /var/log/msmtp.log, however,
# this will create an access violation if you are user pi, and 
# have not  changed the access rights
logfile        ~/.msmtp.log

# Gmail specifics
account        gmail
host           smtp.gmail.com
port           587

from           pi@raspberry
user           [alert email]@gmail.com
password       [alert email password]

# Default
account default : gmail

Edit the user and password lines. Note that providing a password lowers the likelihood of an alert being marked as spam even though in general it is not best practice. Only use a dedicated alert email for this system of alerting. Next, execute df -H in a terminal to see all filesystems and the name of the one that should be monitored (typically /dev/root). Create a new file named disk_alert.sh in your home directory (e.g. /home/pi) and add the following:

#!/bin/bash

# Bash script to watch local disk space and send an alert to $ADMIN
# if the percentage of occupied spice reaches $ALERT.
# -------------------------------------------------------------------------
# Set $ADMIN to the email the notification should be sent to.
ADMIN="[email protected]"

# Level at which an alert is sent out, e.g. 50%
ALERT=50

# Create a list of partitions that shouldn’t be monitored
# Use "|" to separate multiple partitions.
# An example: EXCLUDE_LIST="/dev/hdd1|/dev/hdc5"
EXCLUDE_LIST="/dev/mmcblk0p1"

#
#:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
#

function main_prog() {
while read output;
do
#echo $output
  usep=$(echo $output | awk '{ print $1}' | cut -d'%' -f1)
  partition=$(echo $output | awk '{print $2}')
  if [ $usep -ge $ALERT ] ; then
     echo "Running out of space \\"$partition ($usep%)\\" on server $(hostname), $(date)" | \\
     mail -s "Alert: Almost out of disk space $usep%" $ADMIN
  fi
done
}

if [ "$EXCLUDE_LIST" != "" ] ; then
  df -H | grep -vE "^Filesystem|tmpfs|cdrom|${EXCLUDE_LIST}" | awk '{print $5 " " $6}' | main_prog
else
  df -H | grep -vE "^Filesystem|tmpfs|cdrom" | awk '{print $5 " " $6}' | main_prog
fi

Specify the destination address for all storage alerts in the line beginning with ADMIN. The line beginning with ALERT specifies the percentage of storage occupied at which this alert system activates. To avoid checking filesystems that don’t have an effect on file storage, EXCLUDE_LIST can be used to create a list of partitions that will not be checked. Next, make the file executable with sudo chmod +x disk_alert.sh and create a cron job that periodically checks the disk and sends out an alert if necessary. Open crontab with crontab -e and add */5 * * * * /home/pi/disk_alert.sh. This cron job will check the filesystem every 5 minutes and send out an email if the occupied space passes the level specified in disk_alert.sh.

Health Monitor

The RPi’s health monitoring system focuses on reporting the device hardware’s health. This includes CPU usage, CPU temperature, and GPU temperature. Upon reaching a critical, predetermined level, a specified email address will be notified. In addition, every CPU and temperature check is logged in a local log file. Begin by creating health_monitor.sh in your home directory and making it executable:

cd ~
touch health_monitor.sh
sudo chmod +x disk_alert.sh

Then, open the file and add the following script: