PyGPSClient VMS usage rises over time

[pat-2010]

In the process of chasing down dropped NMEA messages I noticed that the PyGPSClient VMS memory usage grows. A few times it will take a step down, but the trend is upward. I wrote a python script to check the process ID for PyGPSClient every second and report any changes in VMS usage. Here is a sample:

Time, vms memory usage kB, delta memory usage kB
22:36:57 88232 88232 Initial startup
22:37:14 88368 136
22:37:29 88968 600
22:37:37 89100 132
22:37:49 162908 73808 Started rover
22:37:50 163316 408
22:37:52 163456 140
22:37:57 163596 140.
22:38:02 163732 136
22:38:08 163876 144
22:38:13 164008 132
22:38:19 164148 140
22:38:24 164280 132
22:38:30 164424 144
22:38:36 164568 144
22:38:41 164704 136
22:38:45 164840 136
22:38:47 165152 312
22:38:48 165308 156
22:40:04 165444 136
22:40:09 165576 132
22:40:15 165716 140
22:40:20 165848 132
22:40:26 165984 136
22:40:32 166132 148
22:40:38 166272 140
22:40:43 166416 144
22:40:49 166552 136
22:40:54 166684 132
22:40:59 166824 140
22:41:05 166960 136
22:41:11 167100 140
22:41:16 167232 132
22:41:22 167376 144
22:41:26 168400 1024
22:41:27 168536 136
22:41:33 168676 140
22:41:39 168820 144
22:41:44 168956 136

... to the next morning

10:56:52 313276 148
11:00:05 313412 136
11:03:28 313552 140
11:07:40 314576 1024
11:15:56 315600 1024
11:24:52 316624 1024

Here's a plot for the entire run:

I've had similar but not identical results using a Pi400, Pi4 and a Pi5 with a u-blox ZED-F9P and with Quectel LC29HDA and LC29HBS.

For the Pi400 and Pi4 I ran tests where I tried to eliminate other possible interference sources. I used a fresh install of the OS Trixie on the SD card followed by an install of PyGPSClient using the instructions at https://github.com/semuconsulting/pyubx2. I started PyGPSClient, then started the python script that tracked VM usage. I closed the map and satellite views and turned on the datalog using binary. After setting the data rate (9600 for ZED-F9P and 115200 for LC29) I connected to the GPS receiver. No other defaults were changed. The downlink rate was 1 Hz. I've checked and the CPU does not seem to be loaded at all (although I wouldn't likely see short bursts). Most of the time I run headless, but for the Pi400 case with fresh install I did not use VNC.

The memory usage was one consistent characteristic I could easily record. I've also seen:

Dropped NMEA messages that get worse over time. I had a run that lasted 46288 seconds. I scanned for time tags in GNRMC or GNGGA (either one or both) and found 19852 time tags. A plot showing the time between time tags for one case is:

In another case I had the scatter plots set with a 100 point average. In a 12 hour run I'd expect over 400 points, but only 33 were recorded. But in looking at log files I have never seen a message dropout rate that high so I think something else is going on.

Slow display updates (console display would update every 2 seconds or longer) and slow response to the mouse. I closed the map view and the console refresh rate went to normal. Reopened the map view and the console refresh rate stayed normal. Another time undocking the settings also made the console refresh rate go to normal. Looking at a run going now, the console is refreshing every 7 seconds. I opened the server config window. When I click on the close icon nothing happens until just before the console display refreshes.

Very infrequent warnings "Error parsing data stream Serial stream terminated unexpectedly. Line requested, xx bytes returned". Sometimes the error is on the checksum. I "think" this happens mostly or possibly only when the baud rate is 115200.

Also very infrequently the log looks like:

$GBGSV,2,1,07,13,59,159,22,27,39,311,19,28,79,249,09,33,75,031,19,17F
$GBGSV,2,2,07,38,33,151,21,41,50,236,13,42,19,047,23,145
$GQGSV,1,1,00,064
$GNGST,192253.00,57,27,13,98,5.3,11,1541
µb
1$��+))+)'()(+))-,+++,----3/12U5G95;225C85848?9@DABCGEHJLNRRXZ_fjnsx}ÅÇÑÑäñíçççåäãããåàâââàâÜáåÖÜÜÖÑÑÑÑÜàçëííäåååçåççéåçéããççèúîëèèèëèééèéçãäââäâàâàààäâââáâââçñåàáàáàäâääââäãåããäââàáÖÖÅ�~ywvtpmic]WVÄOD@=;<:897865344431622/303----,1.+,+*+++*)*)***(()()()( °� °�Ç≥a^-+,,/,.-.-00/0114436899>;;<?>ADGFEFHMKLNQOQRSTYWY\[]dlltjklqvÖ�yx{~~ÉâÖÖÑÖÜâåççñåääåíèéåçéèî§úîíëíîìòòîîîññòúõõõõúõû†õõòóòòùúîìííëìïìììììïïôóîóïñïóôóîîîììôóíëìêéèåçãäâáÜÖÜÇÄ}z|{~ê®é}vusuxqoijfdcb_[ZXVUVSNLIFFEDBA>===;;7764533132//0.//--.,.9,,-++,,,+,+),** °� °�j⁄ÙH'%ä$GNRMC,192254.00,A,3851.4242490,N,12116.8037205,W,0.227,,130726,,,A,V05
$GNGGA,192254.00,3851.4242490,N,12116.8037205,W,1,08,1.40,72.129,M,-26.730,M,,4A
$GNGSA,A,3,10,24,27,18,23,,,,,,,,3.01,1.40,2.66,10E
$GNGSA,A,3,81,,,,,,,,,,,,3.01,1.40,2.66,20E
$GNGSA,A,3,,,,,,,,,,,,,3.01,1.40,2.66,306
$

Desktop (please complete the following information):

Raspberry Pi Trixie OS, Bookworm on some of the earlier runs.
Raspberry Pi 400, Pi 4 and Pi 5. The Pi 400 and Pi 4 were running from SD card. The Pi5 has a NVMe installed in a hat.
Python 3.13.5
USB connection
u-blox ZED-F9P, Quectel LC29HDAN and LC29HBS
I don't have the firmware versions or protocols but I will work on getting them if it would help.

The challenge is that the problem is not consistent. Some times there are no (or very few) messages dropped. I'm working on compiling a set of tests that systematically tests:
1-GPS receiver (probably won't use the base station)
2-receiver baud rate
3-computer
4-PyGPSClient setup (minimal vs maximum windows open, no log, binary log, parsed log)
5- headless vs hard wired display.

I'm happy to run any test cases if that would help. The challenge is that it takes several hours for some of the characteristics to show up. I can log the memory usage and use the data log to detect dropped messages. But I haven't been able to figure out how to detect the refresh slowdown without looking at it. Or even if it is a useful symptom to measure.

I added just about everything that I saw. I hope that some of it will be useful. I expect that some of the symptoms are red herrings, I just don't know which.

[semuadmin]

Hi @pat-2010

Thanks for this information. For extended logging (> 2 hours or so) I would strongly recommend using an unattended (e.g. CLI) tool like GNSSStreamer rather than an attended (GUI) tool like PyGPSClient (especially if you're running 'headless' anyway), but this certainly warrants investigation. May for example be a memory leak issue in an underlying tkinter package?

Couple of quick questions:

1. Can you provide the python script you were using to monitor VM utilisation?
2. Why only 9600 baud for the F9P? That's extremely low.
3. DId you see any txbuffer alloc errors along the way?

[pat-2010]

Good morning.

I had started looking at GNSStreamer for my final setup. For now I'm learning the system and having the information on the GUI helps. Especially since I can get all three receivers up on my main computer using VNC.

1. Python script attached at the bottom. The script also checks the CPU usage but I didn't find it very interesting. The option I used computes the usage average since the last readout and I haven't yet researched what that really means. I only actually read out the CPU usage when the memory changes so I don't know if the average is since the last memory readout (1 per second) or the last time I read out the CPU usage. In any event I don't believe it would catch short bursts of high activity.
2. The P9P may have came set to 9600 baud or could have been set that way when I was using bluetooth to connect to my iPad. I'm using 1 per second data and that baud rate works so I just never changed it. I think I'll set it higher so the P9P and LC29 are consistent. There may still be some useful info at a the lower data rates depending on what goes into the data log. Would a message that failed parity check show up in the data log?
3. Where would I see the txbuffer alloc errors? I haven't seen any on the command window that I used to launch PyGPSClient. If they come up in the console I haven't seen them. But I do see errors pop up in the console. Mostly the "Serial stream terminated unexpectedly...". I have seen a few parity errors but not many. Is there a way to save the errors that show up on the console?

By the way I started a run at 9 last night. LC29DA on a Pi 4. This morning the console was refreshing every 30 seconds or longer.

I also think that the log that looked like the sample below was an operator error. I likely turned on NMEA and those are the binary RTCM messages.
$GNGST,192253.00,57,27,13,98,5.3,11,1541
µb
1$��+))+)'()(+))-,+++,----3/12U5G95;225C85848?9@DABCGEHJLNRRXZ_fjnsx}ÅÇÑÑäñíçççåäãããåàâââàâÜáåÖÜÜÖÑÑÑÑÜàçëííäåååçåççéåçéããçç

[semuadmin]

Hi,

I had started looking at GNSStreamer for my final setup. For now I'm learning the system and having the information on the GUI helps. Especially since I can get all three receivers up on my main computer using VNC.

Absolutely - PyGPSClient can be a very useful learning tool, but for extended data logging there are probably better options - it's not as though you're going to be eyeballing the data for hours on end?

Python script attached at the bottom.

Thanks for that - I'll take a look. Be aware that some memory stat tools and techniques can themselves impose something of a performance and memory overhead.

The P9P may have came set to 9600

Not necessarily an issue, but the u-blox default setting is normally 38400 and I'd recommend that as a bare minimum.

Where would I see the txbuffer alloc errors?

In the console, e.g.

Thanks again for taking the time to conduct these tests - they are helpful. There are a number of known potential memory leak issues with tkinter in particular - I thought I'd caught the worst of them, but there may be some still lurking.

[semuadmin]

That python attachment appears to have got garbled in the upload - could I trouble you to provide the script again as embedded text in the comment i.e. something like this:

print("this is an example")

Thanks

[pat-2010]

Here is the python script. It was adapted from one of the examples at the psutil readthedocs site

import psutil #https://psutil.readthedocs.io/stable/
import time
import datetime

#loop through processes to find the pid for PYGPSclient
for proc in psutil.process_iter(['pid', 'name', 'memory_info']):
    try:
        if proc.info['name']=="pygpsclient":
          print(f"PID: {proc.info['pid']}, Name: {proc.info['name']}, Memory Usage: {proc.info['memory_info'].vms} bytes")
          thispid=proc.info['pid']
    except (psutil.NoSuchProcess, psutil.AccessDenied):
        pass
    
print("Time, vms memory usage kB, delta memory usage kB, cpu percent")  #header line

oldvms=0    
process = psutil.Process(thispid)

while True:
    memory_info = process.memory_info()
    thisvms=int (memory_info.vms/1024) #convert to KB
    if thisvms!=oldvms:
        print("%27s %10d %8d %5.2f " % (datetime.datetime.now(),thisvms,thisvms-oldvms,process.cpu_percent(interval=None)))
        oldvms=thisvms
    
    time.sleep(1)

[pat-2010]

I looked through the datalog for the case that has the dropout plot above. There were no GNTXT messages. I assume those would come through on the datalog since they are NMEA. Is that correct? I don't remember seeing any fly by in the console but I will keep an eye out.

Well, I take that back. Found one:
$GNTXT,01,01,01,PQTM inv format*3A

Might be a pilot error. just before the NMEA string was:
µb
�‹�EXT CORE 1.00 (0fa0ae)��������00190000��ROM BASE 0x118B2060�����������FWVER=HPG 1.32����������������PROTVER=27.31�����������������MOD=ZED-F9P�������������������GPS;GLO;GAL;BDS���������������SBAS;QZSS���������������������O�$

So I may have been doing something that caused it. Later on in the datalog it looks like I turned on NMEA. I will do a more systematic check as soon as I figure out how to fix my Pi5 which is periodically shutting down. Sigh. I have looked through the logs for 6 cases and found no GNTXT messages.

[semuadmin]

Well, I take that back. Found one:
$GNTXT,01,01,01,PQTM inv format*3A

No you can ignore that one. Shortly after a connection is first started, PyGPSClient attempts to establish the receiver's hardware model and firmware version by sending out a series of poll messages using different manufacturers' protocols (u-blox UBX MON-VER, Quectel NMEA PQTMVERNO, Unicore VERSIONB and Septentrio ReceiverSetup, depending on which protocols are currently enabled). This NMEA message is just telling you that this particular u-blox receiver didn't recognise the PQTMVERNO poll, which is to be expected (other poll types may simply be ignored).

just before the NMEA string was:
µb
�‹�EXT CORE 1.00 (0fa0ae)��������00190000��ROM BASE 0x118B2060�����������FWVER=HPG

Again, this is the UBX response to the MON-VER poll, which gives information about u-blox hardware and firmware version.

[cturvey]

The 'µb' is a UBX packet
The subsequent line is the response to a UBX-MON-VER query, often sent by host to un-stall an I2C or SPI interface that's been neglected.
The $GxTXT "txbuf,alloc" is from the exhaustion of a common memory pool for queued messages, these get retired as each interface emits data, so a slow UART[1,2] with insufficient bandwidth will log-jam the txbuf resources, and data will then be dumped/discarded, I think one of the numbers indicates which interface is problematic, but you'd need to address the CFG-MSGOUT settings so data loss doesn't occur. The frequency will depend on the mismatch between fill and empty rates.

[semuadmin]

@pat-2010 ,

Seeing similar results, but interestingly only when the app is left unattended for an extended period. This was overnight on an RPi5 running stock Trixie with a ZED-X20P outputting standard NMEA cohort at 230400 baud. Will investigate further as and when time permits...

[pat-2010]

I'll stop looking at memory usage and work on using the CLI tools. I want to run a 24 hour test to see the position variability. I'm in a town so multi path may be a problem. Thanks for recommending the CLI tools. I'm just spoiled from the ease of use and system performance visibility that I get from PyGPSClient.

I did run two tests recently. Pi 400 running from a SD card and LC29N. 1 Hz sample rate. Datalog in binary. 115200 Baud. In one case I ran with the default window setup. In the second case I dropped the map and the two satellite related windows. The plot is below. I scaled it to match your plot. Your slope is much steeper (for what that is worth).

Let me know if there is anything I can do to help to collect data about the other 2 problems (dropped messages and lags in refreshing windows or responding to mouse clicks). I haven't yet checked the two Pi 400 tests for data drop outs. I'll do that today.

[pat-2010]

Looked at 2 runs where the only difference was one case with the default window set and a second case where the map window and the satellite window and the signals window were closed (lower load case). Found that in at least some (if not all) cases a block of dropped messages appears to be preceded by two copies of the same message.

The analysis was to look through the log file for GNMRC or GNGAA messages. In the low load case I always found both messages with the same time tag as expected. With a run of 43937 seconds I found no cases where there were missing GNRMC or GNGGA messages.

The default window test datalog showed dropped messages. With a run of 40949 seconds I only found 19233 time tags. It was not always both GNRMC and GNGGA. A plot of the time difference between successive time tags is:

When I was looking through the raw log files to crosscheck I noticed that the gaps appear to be preceded by duplicate messages. In the excerpt below the second and third messages are duplicates. Both have time tags of 9:18:18, while the 4th message with a time tag of 9:18:23 is the first one after a 5 second data gap.

$GNGGA,091818.000,3851.42464825,N,12116.91451424,W,1,08,43.53,41.686,M,-22.320,M,,4E
$GNRMC,091818.000,A,3851.42464825,N,12116.91451424,W,0.847,269.33,150726,,,A,V2F
$GNRMC,091818.000,A,3851.42464825,N,12116.91451424,W,0.847,269.33,150726,,,A,V*2F
$GNGGA,091823.000,3851.42443902,N,12116.91512337,W,1,08,43.51,41.683,M,-22.320,M,,43
$GNRMC,091823.000,A,3851.42443902,N,12116.91512337,W,0.660,269.33,150726,,,A,V2E

I scanned a handful of data gaps and the duplicate message was in all of the ones I checked. I don't plan on checking the other 20000 or so gaps ;-). I did spot check at the beginning of the run when the dropout rate was low and at the end of the run when the rate was high.

[pat-2010]

Well... not always duplicate messages.

I had a case with ZED-F9P running on a pi 400. Baud rate was 115000 and I had up the default windows. Messages were being dropped. But with a scan of 10 or so I did NOT find any duplicate messages.

Here's an example when the NMEA sentence from 14:35:04 was truncated and the stream restarted with the GNRMC at 14:35:37.

$GNRMC,143504.00,V,,,,,,,160726,,,N,V1A
$GNGGA,143504.00,,,,,0,05,25.06,,,,,,7B
$GNGSA,A,1,11,21,,,,,,,,,,,29.33,25.06,15.25,139
$GNGSA,A,1,82,68,67,,,,,,,,,,29.33,25.06,15.25,23C
$GNGSA,A,1,,,,,,,,,,,,,29.33,25.06,15.25,338
$GNGSA,A,1,,,,,,,,,,,,,29.33,25.06,15.25,43F
$GNGSA,A,1,,,,,,,,,,,,,29.33,25.06,15.25,53E
$GPGSV,2,1,05,05,63,084,19,11,12,060,23,20,12,117,22,21,33,050,21,161
$GPGSV,2,2,05,28,07,264,13,15E
$GNRMC,143537.00,V,,,,,,,160726,,,N,V1A
$GNGGA,143537.00,,,,,0,05,24.82,,,,,,*76

There were no partial messages or checksum errors in the datalog.

I also checked a case on the Pi 400 that was the same as above but the baud rate was 9600. I saw data dropouts at the end of NMEA sentences but no duplicate messages.

For reference here is a full NMEA sentence for that test:

$GNRMC,101851.00,A,3851.4266413,N,12116.7996908,W,1.087,,150726,,,A,V09
$GNGGA,101851.00,3851.4266413,N,12116.7996908,W,1,07,5.41,78.138,M,-26.729,M,,41
$GNGSA,A,3,06,14,17,22,,,,,,,,,6.80,5.41,4.13,10C
$GNGSA,A,3,87,77,,,,,,,,,,,6.80,5.41,4.13,205
$GNGSA,A,3,,,,,,,,,,,,,6.80,5.41,4.13,30B
$GNGSA,A,3,27,,,,,,,,,,,,6.80,5.41,4.13,409
$GNGSA,A,3,,,,,,,,,,,,,6.80,5.41,4.13,50D
$GPGSV,2,1,06,06,60,110,23,11,42,171,11,12,34,303,15,14,13,106,18,168
$GPGSV,2,2,06,17,26,054,25,22,30,097,20,169
$GPGSV,1,1,03,06,60,110,27,14,13,106,18,17,26,054,19,657
$GPGSV,2,1,05,03,03,040,,13,35,307,,19,60,040,,24,57,254,,06A
$GPGSV,2,2,05,25,00,296,,05A
$GLGSV,1,1,02,77,37,116,14,87,25,032,09,17D
$GLGSV,1,1,01,69,05,296,11,34C
$GLGSV,2,1,05,70,02,341,,78,69,004,,79,22,320,,81,27,204,,07D
$GLGSV,2,2,05,88,82,152,,040
$GAGSV,1,1,00,074
$GBGSV,1,1,03,27,17,046,14,29,49,217,13,39,39,312,13,143
$GBGSV,2,1,05,04,01,264,,11,72,199,,30,73,063,,36,87,140,,07D
$GBGSV,2,2,05,40,30,131,,046
$GQGSV,1,1,00,064
$GNGST,101851.00,332497,114,30,83,14,46,3259

[cturvey]

There were no partial messages or checksum errors in the datalog.

Well the pasted examples look wrong/inconsistent, and repeatedly missing the '*' delimiter for the NMEA checksum in many of them. Perhaps it's the mark-down formatting, as also seeing random lines in italics. Suggest pasting within triple '`' code blocks.

$GNGGA,143537.00,,,,,0,05,24.82,,,,,,*76

Large blocks missing, conveniently at sentence boundaries is highly suggestive of parsing / syncing anomalies, where something that persists in messages causes issues. Ideally data can be piped directly to a log file, that can subsequently replayed through the parser as test cases.

On UART side bit/byte level sync/framing can become an issue. I've seen this stress some MCU, or home-brew UART IP implementations. The data is very dense and continuous, perhaps more so than other implementations as uBlox DMA's data out of the UART, so there's next to no inter-symbol spacing, Each output epoch does provide an natural break in the data. Perhaps try enabling Two Stop Bits? A Start-Bit is a LOW, and a Stop-Bit is a HIGH, often used to resync the UART timing / bit-centering when oversampling at x8 or x16

[semuadmin]

@pat-2010

Appreciate the input but I would hold off on further testing for now. There's clearly a reproducible issue here affecting long term stability in unattended PyGPSClient instances which manifests itself in various ways. I suspect this may be a tkinter implementation issue on my part, but it may be a week or so before I can spare any serious time on investigation and remediation - I have a few potential strategies in mind. So please bear with me until then. Again, appreciate the information you've provided to date.

In the meantime, I did conduct an extended test of the CLI gnssstreamer tool (on a 'headless' RPi5 8MB running stock Trixie desktop) and can confirm that this is rock solid for extended data logging. PyGPSClient and gnssstreamer share the same I/O and parsing 'engine', so I'm reasonably confident this is not a UART or parsing issue per se (though, as @cturvey mentions above, such issues can arise in other circumstances)

(I've routinely run >24 hour RAW UBX logs for PPK purposes on an RPi using gnssstreamer without issue, but I thought I'd double check anyway).

I acquired a log of the standard NMEA output from a ZED-X20P over a roughly 14 hour period using the following command - 1,954,941 NMEA messages were captured in total:

gnssstreamer -P /dev/ttyACM0 --baudrate 115200 --timeout 3 --format 2 --clioutput 1 --output x20p_datalog.bin --verbosity 2

The ZED-X20P was in factory configuration other than the UART1 and UART2 baudrates, which I increased to 115200.

I checked the reported VMS memory footprint over time (as per your script) and the continuity of the logged GNRMC messages. VMS was static at 115,536 kB for the entire duration of the test:

 2026-07-16 08:10:22.004214     115536   115536  0.00 

The interval between successive GMRMC.time values was exactly 1 second for all 52,000 GNRMC messages captured - no dropouts whatsoever.