Storing Data - Discovery Kit For Raspberry Pi Pico Extension Exp 2

This resource is part of a series to create a Data logger on the Pico. It assumes that the user has completed the Kitronik Discovery Kit for Raspberry Pi Pico. It is intended to be completed after the Pico Thermometer resource, but this is not essential. This is the second of several extension experiments for the Discovery kit that will only be available online.

The aims of this experiment are; 

    • To understand how to write to a file on the Pico internal storage
    • To understand how to get the size of a file, and use that to limit the maximum size a file gets to.

Parts Used;

 

 Storing Data On The Pico:

The thermometer made with the Pico is great for indicating the current temperature. However, its indication is relatively coarse and it only shows an instantaneous reading. If we were investigating how the temperature of a room (for instance) varied over time as well as writing down which LED was on, we would not have a very accurate record of the actual temperature, only of the temperature band.

The Pico has on-board storage, and we can save data to that – thereby creating a log of the temperature. This can happen automatically, so we can set up the datalogger and then retrieve the values and analyse them later.

Before we store live data, this tutorial will look at files and how to handle them in MicroPython on the Pico.

File Handling on the Pico:

When using Thonny, the save dialog gives the option of saving to the Pico. This is the Pico's onboard storage, and is where log files can be written. Thonny can open files from the Pico and then save a copy to your computer to work on later – for analysis, for instance.

To open a file from the Pico, use Thonny’s ‘File -> Open’ command. Open from the Pico when asked. This will load the file into Thonny, and then the ‘File -> Save as’, or ‘File -> Save copy’ commands can be used to save the data to a file on your computer.

The Pico has quite a lot of storage, but it is not infinite. If we just write values to a file stored on it then the file will end up filling all the space, and that could cause other problems.

To prevent this, it is good practice to limit the log file to a certain number of entries, or a maximum size.

To do this, the software either needs to:

  • Keep track of how many entries there are, and when the maximum number is exceeded, delete some of them from the file.
Or:
  • Keep track of how big the file is, and when a certain size is exceeded, delete some entries to bring the file size back below that.

Both approaches have benefits and weaknesses.

Managing Space With Python;

Keeping track of entries is good when the entries are well defined and consistent in size. The resulting file size is predictable and can be calculated in advance for a certain number of entries. If there is a need to have a certain sized dataset then this approach can be the best to use. Keeping track of the number of entries is the main challenge in this method. The number of entries can be written into the file itself, but it is generally more convenient to keep in a variable.

If the entry count is kept in a variable, then a possible issue it that the variable gets lost if the processor resets. Writing it into the file –for instance as the first line in the file – means more complex file operations, where the number of entries needs to be read before we write to it, and a decision taken on clearing the excess entries.

Python has file handling functions built in to enable the size of a file to be determined. Rather than counting the number of entries, it is possible to see the size of the file before writing to it, and then make the decision about deleting one (or more) entries. This method does not directly keep track of how many entries are in the file,

In this tutorial we will keep track of the size of the file, as this is a more generally useful method to learn. To help keep things simple, the file will treat each entry as a new line. This means that we can just delete a line out of the file when it gets to maximum size.

See the Python code below; 

To establish the size of a file, we use some functions. The following snippet Python code gets the size of a file called ‘log.csv’:

file_handle = open(“log.csv”,"r")  #get a handle to the file
file_handle.seek(0, 2) #find the end
size = file_handle.tell() #tell us where the end is
file_handle.close() #close the handle

In this code snippet the file’s size ends up in the variable ‘size’.

There is quite a bit going on in those few lines of code; let’s break it down and explain a little more about how the internals of files work.

The ‘open’ command instructs the Python interpreter to find the file for us and give us an object which represents it. There are some options passed to open – one is the file name, the other is what mode to open the file in.
Commonly used modes are:

    • read (“r”) – Opens an already existing file for reading (from the beginning)
    • write (“w”) – Creates a file if it does not exist, and clears the file if it does exist, ready for writing to it
    • append (“a”) – Opens the file, but does not clear its contents, ready for writing to it after the current contents

There are other modes, but we will not cover them here. They include binary files.

The handle can be thought of as a signpost, pointing to a place in a file. When first opened, this ‘signpost’ points to somewhere in the file, depending on the mode option (r and w – the beginning of the file, a – the end of the file). Using the ‘seek’ function, we can look at any point in the file. The parameters for the seek function tell it to start from the beginning (0), and seek to the end (2) of the file.  Once the handle has bee positioned at the end we then  use the ‘tell’ function to tell us where the handle is. The end of the file is the same as the size of the file.

Now there is a method for determining file size, the next thing to tackle is removing the older entries and adding the new ones. When the file is smaller than the maximum size, the newer entries can just be appended to the end of the file, but when it is larger, then the oldest entries should be removed.

It is not possible to delete just part of a file. The technique to remove a portion is to copy only the parts that are needed into a new file, and then delete the older file. As we have entries that are complete lines, we can simply copy all but one line from the old file into the new file when the file has reached the maximum size.  In our file the oldest entry is the first line so the code just has to read the first line from one file, but not write it into the new file.  The following Python code snippet does just that:

See the Python code below; 

tmpName = LogFileName + '.bak'
with open(LogFileName, 'a') as readFrom, open(tmpName, 'w') as writeTo #open both files
        readFrom.readline() #read 1st line & throw it away. This moves the handle on by 1 line.
        # Now Read the rest of the lines from original file & write them to the dummy file
        for char in readFrom:
            writeTo.write(char)
 #now close all the handles and swap the file names around.
    readFrom.close()
    writeTo.close()
    os.remove(LogFileName)
    os.rename(tmpName,LogFileName)
    LED_FileWrite.value(0)

In this code we open the existing file with a handle called readFrom and a new file with a handle writeTo. The code then reads the first line out of the existing file. It does not do anything with the data that is read. The read is just there to move on the file handle by 1 line. It is also possible to position the file handle using a similar technique to the file size check.

Once the handle is positioned at 1 line into the old file the code runs a loop that just copies all the data from one file to the next. This is something to be aware of when writing rolling log files – although the maximum file size is limited to be able to run needs at least 2x the maximum file size of space – as at some points in the process there are 2 copies of the file. This applies no matter how the size of the file is kept track of.

In our example, the code will write incrementing numbers as the entries, and limit the file to 100 bytes. This is enough for about 20 lines of numbers.

The following code puts together what we have so far learnt. To assist in keeping the code easy to follow, the size, removal and writing are split into functions:

See the final Python code below; 

You can copy and paste the code from below, or download the code here.

import machine
import utime
import os

#we will use the inbuilt LED as a file activity light
LED_FileWrite = machine.Pin(25,machine.Pin.OUT)
#Define a file name and a maximum size in bytes
LogFileName = "log.txt"
Max_File_Size = 100
     
#This writes whatever is passed to it to the file     
def WriteFile(passed):
    LED_FileWrite.value(1) #indicate writing to file, so don't power off
    log = open(LogFileName,"a") #open in append - creates if not existing, will append if it exists
    log.write(passed)
    log.close()
    LED_FileWrite.value(0)

#This returns the size of the file, or 0 if the file does not exist
def CheckFileSize():
    # f is a file-like object.
    try:
        f = open(LogFileName,"r") # Open read - this throws an error if file does not exist - in that case the size is 0
        f.seek(0, 2)
        size = f.tell()
        f.close()
        return size
    except:
        # if we wanted to know we could print some diagnostics here:
        #print("Exception - File does not exist?")
        return 0 


#This removes one line from the file by copying the whole file except the first line to a new file 
# and then renaming it def RemoveOneLine(): LED_FileWrite.value(1) #indicate writing to file, so dont power off tmpName = LogFileName + '.bak' with open(LogFileName, 'r') as readFrom, open(tmpName, 'w') as writeTo: #open both files readFrom.readline() #read the first line and throw it away. This moves the file handle on by 1 line. # Now Read the rest of the lines from original file one by one and write them to the dummy file for char in readFrom: writeTo.write(char) #now close all the handles and swap the file names around. readFrom.close() writeTo.close() os.remove(LogFileName) os.rename(tmpName,LogFileName) LED_FileWrite.value(0) #We will just write an incrementing number to the file number = 0 while True: while(CheckFileSize() > Max_File_Size): RemoveOneLine() stringToWrite = str(number) + "\r\n" #add a line feed/carriage return to ensure each number is on its own line in the file WriteFile(stringToWrite) number+=1 utime.sleep_ms(100) #give a chance to break back in.

Run the code of a short period then stop it. The code blinks the onboard LED when file writes are occurring.  In Thonny open a file from the Pico, and there should be a file called log.txt that is about 100 bytes long, depending on if the code was stopped just before or after the removal of a line or the writing of a new one. It is possible to stop the code when both the log.txt and the log.txt.bak files are present. In that case open them both and compare. The oldest entry has been removed from the log.txt.bak file.

 

This tutorial has covered the basics of files – how to create them, write to them and find their sizes. There are many more advanced topics – such as binary files, removing data from the middle of a file,   – which we have not covered. File handling also increases the chances of an exception being created – for instance trying to open a file that is not there. The code here does not deal with the times when things go wrong completely.

In the next tutorial we will look at using threads to create a datalogger that writes to a file at the same time as also displaying the current temperature.

 

By
MEng (hons) DIS, CEng MIMarEST
Technical Director at Kitronik

 

 

Discovery Kit Extension Experiments:

The table below contains links to extension experiments for the Pico Discovery Kit. They have been listed in the order that they should be completed, and they should only be tackled once you have completed the 7 experiments supplied with the kit. Each extension experiment is thoroughly explained and code examples are provided. Although they cover quite challenging concepts, the information is provided in such a way as to walk you through the solution.

Exp No#. Experiment Name/Link.
1 Pico Thermometer.
2 Storing Data on the Pico.
3 Live Data Logging on the Pico.

 

 

If you enjoyed this guide, make sure you don't miss out on any other new free learning resources by signing up for our newsletter here

Project shopping list

For this project you will need the following items, you can adjust the quantities (including setting them to zero if you don't need the item), before pressing the "Buy all items" button.

Kitronik Discovery Kit for Raspberry Pi Pico (Pico not included)
£10.00
Qty
Kitronik Discovery Kit for Raspberry Pi Pico (Pico Included)
£15.50
Qty

Leave a comment

All comments are moderated before being published