Live Data Logging - Discovery Kit For Raspberry Pi Pico Extension Exp 3

This resource is part of a series to create a Data logger on the Pico. It assumes that the user has completed the Pico Discovery Kit, and both the additional ‘Pico Thermometer’ and ‘Storing Data on the Pico’ resourcesThis is the third of several extension experiments for the Discovery kit that will only be available online.

Aims:

    • To create a complete data logging and temperature display unit.
    • To understand a little about threads, thread synchronisation and inter-process communication.

Parts Used:

    • All of the parts required for this experiment can be found within the Kitronik Discovery Kit for Raspberry Pi Pico, they are;
      • 2 x Red LED.
      • 2 x Yellow LED.
      • 1 x Green LED.
      • 5 x Resistors.
      • 6 x M-M Jumper Wires

This tutorial uses the same circuit as the Pico Thermometer tutorial .

 

 Live Data Logging on the Raspberry Pi Pico:

In the previous two tutorials we have created a thermometer and seen how we could save data into a size-limited file on the Pico’s internal storage.

Writing to a file can take a long time (relatively, compared to the speed of the processor).

As the Pico has two cores, we can dedicate one of them to do the file handling, whilst the other is still taking and displaying the readings of the temperature. As well as making the code nicely partitioned, this allows the logging to be done at a different rate to the live update in a simple-to-follow manner.

This tutorial will also cover the basics of inter-process communication.

 

Build the following circuit on the discovery kit bread board:

 

As the Pico Discovery Kit explained, a thread is an independent execution of code. The Pico can run two threads, one on each core.

As we are creating a thermometer with data logging, we will have the main loop read the temperature and display it, and then write to a file using a separate thread to record the value periodically. Usually we don’t need to log as fast as possible because we are monitoring a variable that changes relatively slowly. The example code uses a multiple of 60 – so we display live data roughly every second, and log the value roughly once a minute. By setting a limit of 5000 bytes in the file this allows us to store around 7 hours of data in a rolling file.

Firstly, let’s look at how to have two loops that are synchronised, but operating at different speeds.

The following is a simple example of two different loops – one running fast, the second released in a multiple of the first (faster) loop:

Create the following code;

import _thread
import machine
import utime

# We will use the inbuilt LED as an activity light
LED_FileWrite = machine.Pin(25,machine.Pin.OUT) 

# A semaphore to sync and control the threading
ThreadRun = _thread.allocate_lock()

# Thread function for file writing
def ThreadFuncion():
    LEDValue = False
    while True:
        ThreadRun.acquire()
        LEDValue = not LEDValue
        LED_FileWrite.value(LEDValue)
        
        
# Kick off the LED thread
_thread.start_new_thread(ThreadFunction,())
count = 0
while True:
    print("Main loop: ",count)
    utime.sleep_ms(100)
    count += 1
    # Release the LED thread to flash once every 10 loops
    if(count == 10):
        # Do a release the thread thing
        ThreadRun.release()
        count = 0 

What's going on?

In this code, the main ‘while’ loop is one thread of execution, and the ‘ThreadFunction()’ is another. The two threads are synchronised by using a lock. These are also known as Semaphores or Mutexes (there are detail differences in these names, but for this tutorial they are not important).

Locks start off as ‘unlocked’, become ‘locked’ when they are ‘acquired’, and become unlocked again when ‘released’.

A lock object can only be ‘held’ by a single thread. The default acquire method will block the execution of a thread if the lock is already locked. Using a lock provides us with a method to ensure that threads are synchronised.

In the simple example, the main loop unlocks the ‘ThreadRun’ lock once every 10 loops. It never tries to acquire it – so the main loop always runs.

The ‘ThreadFunction()’ also has a loop that runs forever. In that loop, the thread tries to acquire the lock object. If the thread has already run, then the lock object will be locked and the thread will pause until it can acquire it again once it becomes unlocked by the main loop.

The next step is to pass the temperature we want logging to the thread to write to the file. Passing data between threads means we have to be careful, because one thread might be reading the value whilst the other one is trying to update it. We can also use a lock to protect against this. Unlike in the synchronising case we already have, in this case both threads will try to acquire the lock, and only if they have acquired it will they read/write to the shared variable.

Create the following code;

The following code modifies the simple synchronising example to pass the value to show on the LED from the main loop:

import _thread
import machine
import utime

# We will use the inbuilt LED as an activity light
LED_FileWrite = machine.Pin(25,machine.Pin.OUT) 

# A semaphore to sync and control the threading
RunThread = _thread.allocate_lock()
# A semaphore to allow safe sharing of a variable
ShareVariableLock = _thread.allocate_lock()
SharedVariable = False

# Thread function for file writing
def ThreadFunction():
    LEDValue = False
    while True:
        RunThread.acquire()
        # Take a local copy of the shared variable
        ShareVariableLock.acquire()
        LEDValue = SharedVariable
        ShareVariableLock.release()
        # Now use our local copy
        LED_FileWrite.value(LEDValue)
               
# Kick off the file logging thread
_thread.start_new_thread(ThreadFunction,())
count = 0
LEDState = False
while True:
    print("Main loop: ",count)
    utime.sleep_ms(100)
    count += 1
    # change the LED once every 5
    if(count == 5):
        ShareVariableLock.acquire()
        SharedVariable = LEDState
        ShareVariableLock.release()
        LEDState = not LEDState
        # Do a release the thread thing
        RunThread.release()
        count = 0

What's going on?

Because the execution will stop if the shared variable lock cannot be acquired, it is important to keep the operations completed when holding the lock as short as possible. This reduces the risk that one thread will prevent the other from running. In more complex systems, the use of inter-process communication needs to be handled with care, as it can be a cause of undesirable operation – where the system does not function as expected due to the inter-dependency of the various parts.

Now that the basic structure of a dual-threaded data logger is complete, we can expand the code by bringing in the previous tutorials – to allow us to read the temperature, and to write to a rolling log file.

There are currently some limitations in the Pico MicroPython implementation, which mean that the full logging code has to take a slightly different structure to the simple ideal examples above. If the thread function never exits, then it can cause a random crash of the Pico. To avoid this, the full code (below) does not use the lock to synchronise the thread, and instead it creates a new thread that writes to the file and then exits.

Create the following code;

Either copy and paste the code below or download the zipped Python code here.

import _thread
import machine
import utime
import os

tempSensor = machine.ADC(machine.ADC.CORE_TEMP)
ADC_count_to_volts = 3.3/(65535)

LED_Cold = machine.Pin(15,machine.Pin.OUT)
LED_Chilly = machine.Pin(14,machine.Pin.OUT)
LED_Nice = machine.Pin(13,machine.Pin.OUT)
LED_Warm = machine.Pin(12,machine.Pin.OUT)
LED_Hot = machine.Pin(11,machine.Pin.OUT)
# We will use the inbuilt LED as a file activity light
LED_FileWrite = machine.Pin(25,machine.Pin.OUT) 

# A semaphore to control access to the shared variable
ShareVariableLock = _thread.allocate_lock()
SharedTemperature = 0

LogFileName = "log.txt"
Max_File_Size = 5000 # Allow 5000 bytes. 
 # Each entry is about 12 bytes -> T: XX.XXX\r\n 
 # so this is enough space for 415 or so entries, 
 # or about 7 hours of rolling data @1 a min.
     
# This writes whatever is passed to it to the file     
def WriteFile(passed):
    # Indicate writing to file, so don’t power off
    LED_FileWrite.value(1) 
    # Open in append - creates if not existing, will append if it exists
    log = open(LogFileName,"a")
    log.write(passed)
    log.close()
    LED_FileWrite.value(0)

# This returns the size of the file, or 0 if the file does not exist
def CheckFileSize():
    # ‘f’ is a file-like object.
    try:
        f = open(LogFileName,"r") # Open read - this throws an error if file does not exist
# in that case the size is 0 f.seek(0, 2) size = f.tell() f.close() return size except: # If we wanted to know we could print some diagnostics here like: # print("Exception - File does not exist") return 0 # This removes one line from the file by copying the whole file
# except the first line to a new file and then renaming it def RemoveOneLine(): LED_FileWrite.value(1) # Indicate writing to file, so don’t power off tmpName = LogFileName + '.bak' readFrom = open(LogFileName, 'r') writeTo = open(tmpName, 'w') readFrom.readline() # Read the first line and throw it away.
# This moves the file handle on by 1 line. # Now Read the rest of the lines from original file one by one
# and write them to the dummy file for lines in readFrom: writeTo.write(lines) # Now close all the handles and swap the file names around. readFrom.close() writeTo.close() os.remove(LogFileName) os.rename(tmpName,LogFileName) LED_FileWrite.value(0) # Thread function for file writing: def LoggingThreadFunc(): FileValue = 0 # Take a local copy of the shared variable. ShareVariableLock.acquire() FileValue = SharedTemperature ShareVariableLock.release() # Now use our local copy, but first make sure the file is small enough. while(CheckFileSize() > Max_File_Size): RemoveOneLine() # Format to 3 decimal places, add a line feed/carriage # return to ensure each entry is on its own line in the file. stringToWrite = "T : "+ '{0:.3f}'.format(FileValue) + "\r\n" WriteFile(stringToWrite) # At the start of each sensing cycle we turn off all the LEDs, so make a function for it. def LED_AllOff(): LED_Cold.value(0) LED_Chilly.value(0) LED_Nice.value(0) LED_Warm.value(0) LED_Hot.value(0) # Indicate we are running by turning on all the LEDs in a one at a time manner LED_FileWrite.value(1) LED_Cold.value(1) utime.sleep_ms(500) LED_Chilly.value(1) utime.sleep_ms(500) LED_Nice.value(1) utime.sleep_ms(500) LED_Warm.value(1) utime.sleep_ms(500) LED_Hot.value(1) utime.sleep_ms(500) LED_FileWrite.value(0) count = 0 LEDState = False while True: # First read the temperature temperature = 27-((tempSensor.read_u16()*ADC_count_to_volts)-0.706)/0.001721 # uncomment the below line to print the temp value to the console so we can see it # print("Temp: ",temperature) # Now decide what to light up. # First turn off the last indication: LED_AllOff() if(temperature <= 17): LED_Cold.value(1) elif (temperature>17 and temperature<=19): LED_Chilly.value(1) elif (temperature>19 and temperature<=21): LED_Nice.value(1) elif (temperature>21 and temperature<=24): LED_Warm.value(1) else: # It must be hot! LED_Hot.value(1) count += 1 # Log the temperature approximately once a minute (60 times around the ~1 second loop) if(count == 60): ShareVariableLock.acquire() SharedTemperature = temperature ShareVariableLock.release() # Kick off a separate thread to write to the file _thread.start_new_thread(LoggingThreadFunc,()) count = 0 # We can have a sleep - there is no need to check the temp as fast as possible. # This will also help to prevent too much bias because the chip is running 'flat out' utime.sleep_ms(1000)

Conclusion:

The final code runs both temperature indication and file writing. It uses the onboard LED to show when the file is being accessed, and after running for a period of time, the code will automatically limit the size of the file. The file can be accessed through Thonny, and then saved to be analysed in Excel or other software.

The structure of the code is such that adding in other sensors – such as a phototransistor for light levels – is relatively straightforward. Writing additional values to the log file can be accomplished either through more shared variables, or by passing a more complex data type – such as an array.

 

Extension tasks to try:

  • Add a ‘Heading line’ to the file (Hint: read the first line into the copy file, then drop the second line and copy the rest of the file).
  • Log additional inputs – such as a light sensor connected to the ADC.

 

By
MEng (hons) DIS, CEng MIMarEST
Technical Director at Kitronik

 

 

Discovery Kit Extension Experiments:

The table below contains links to extension experiments for the Pico Discovery Kit. They have been listed in the order that they should be completed, and they should only be tackled once you have completed the 7 experiments supplied with the kit. Each extension experiment is thoroughly explained and code examples are provided. Although they cover quite challenging concepts, the information is provided in such a way as to walk you through the solution.

Exp No#. Experiment Name/Link.
1 Pico Thermometer.
2 Storing Data on the Pico.
3 Live Data Logging on the Pico.

 

 

If you enjoyed this guide, make sure you don't miss out on any other new free learning resources by signing up for our newsletter here

Leave a comment

All comments are moderated before being published