British Cave Monitoring Centre (Home)

Freshen the Data

This is an outline page - still to be formatted and expanded

View Uploaded Files & Technical: View Logs.

The logger data is uploaded automatically, on a scheule (details below). You can trigger a manual upload by clicking one of the buttons below.

Note: 15-Jun-2024 – Due to a slow broadband link, the uploads from Pooles cavern are taking a long time, and the BCRA server is timing out and not transferring all of the files. This is most likely an electrical fault on the phone line somewhere between Pooles Cavern and the BT cabinet. Some work-arounds have been implemented, as described below.

Both php and curl have had large limits placed on their execution time. In freshen.php I have set $time_limit = 5000; and in the cron job I have set --max-time 5000 ; 5000s being about 83 minutes.
The code in freshen.php has been altered so that only a quarter of the files are handled in any one operation. The cron job now executes every six hours, at 1,7,13 and 19 GMT and each execution handles a different set of files. Additionally, the freshen_log file is now a set of four files; and the system error output goes to a second set of log files.

Because the code was written in a hurry, and because debugging it was difficult, it uses what might be considered a rather convoluted procedure, as follows...

There are four cron entries. All are identical apart from the setting of a variable myChunk. That variable dictates the name of the log files as freshen_log_${myChunk}.html and freshen_log_${myChunk}_ERR.txp, but the variable is not passed to the php program.
The php program selects a quarter of the files to process. The decision is based on the time the program is executed. It looks at the hour modulo 4, and compares it with the file name index, modulo 4, and if they match it processes the file.

Update 23-Mar-2026: When the Freshen routine runs it now writes a file list to logs/mined_file_list.js. That list is in the form of a program that can be run in either PHP or JavaScript, to create an array of file paths. Further information is given within the file itself and there is also a demonstrator at test_data_mining.html, which checks that the file structure is valid.

An incremental upload of the CSV data files using TinyTag's 'restart' option.
A full upload of the 'raw' data files
For identification, the CSV files are named using the restart sequence number
A list of refreshes is available in the *.restart.txp files
See notes above, which describe how the files that are processed depend on th etime of day that the program is run

Note that the 'fractured' nature of the incremental CSV files may make them difficult to use.

Technical note: this runs freshen.php

A full upload of the CSV data files
A full upload of the 'raw' data files
See notes above, which describe how the files that are processed depend on th etime of day that the program is run

Note that *.csv files will be over-written, but the individual 'refresh' files already on the server will be left alone - although David Gibson might delete these manually as part of the ongoing development/debugging work.

Technical note: this runs freshen.php?restart=no

DEBUG: do not fetch any data files from the Gateway

Technical note: this runs freshen.php?debug=yes

Points to note...

When you ask to upload the logs, the process report appears on screen and is not stored in the log file
When you ask to upload the logs, the BCRA server will contact the TinyTag Gateway at Pooles Cavern and transfer the data to our storage here. This may take 30 minutes, during which time nothing much will happen, so please wait.
TinyTag provides the "freshen" option – which they refer to as a "restart" – because it is quicker, as only the changes to the CSV files are uploaded. However, the raw data files are also uploaded by my routine, and these do not have a 'freshen' option, so the upload is still going to take a while, whichever option you choose. Freshening the CSV files is useful for "real-time" access by humans but as we are running a CRON job, every hour, there is no useful purpose in freshening.
Technical note: you can add parameters to the query string.
- cron=yes — add a line to the output identifying this as a cron job
- debug=yes — do not fetch any data
- restart=no — do not fetch the restart info; download the full CSV files.
- fetchjf=no — do not fetch the *.jf files

At the moment, a CRON setting causes the "Replace the Stored Logs" function (see buttons above) to be executed as follows (E&OE). Please check with David Gibson to see if the CRON settings quoted below are still current. Prior to 15-June-2024 the setting was ...

CRON settings:
Times: 22 1,13 * * * (that is, those minutes, past those hours, every day)
Command: curl http://bcra.org.uk/data/freshen.php?restart=no\&cron=yes
   >/home/bcra/public_html/data/logs/freshen_log.html 2>&1

After June-15-2024 the hour settings were 1, 7, 13, 19 but I only discovered on 12-Mar-2026 that those were the wrong settings. The correct values are as below. The chunk of data that is fetched is governed by the time in hours, modulo 4. However, the name of the log file is constructed from the myChunk parameter. The reson for using two separate methods for fetching and logging the data is historic. Clearly we could control the data transfer by adding &chunk=${myChunk} to the query string, which would make the operation independent of the time of day.

13 1 * * *
myChunk=1; myPath=/home/bcra/public_html/data/logs/; \
curl https://bcra.org.uk/data/freshen.php?restart=no\&cron=yes --max-time 5000 \
>${myPath}freshen_log_${myChunk}.html 2>${myPath}freshen_ERR_${myChunk}.txp

13 6 * * *
myChunk=2; myPath=/home/bcra/public_html/data/logs/; \
curl https://bcra.org.uk/data/freshen.php?restart=no\&cron=yes --max-time 5000 \
>${myPath}freshen_log_${myChunk}.html 2>${myPath}freshen_ERR_${myChunk}.txp

13 11 * * *
myChunk=3; myPath=/home/bcra/public_html/data/logs/; \
curl https://bcra.org.uk/data/freshen.php?restart=no\&cron=yes --max-time 5000 \
>${myPath}freshen_log_${myChunk}.html 2>${myPath}freshen_ERR_${myChunk}.txp

13 16 * * *
myChunk=4; myPath=/home/bcra/public_html/data/logs/; \
curl https://bcra.org.uk/data/freshen.php?restart=no\&cron=yes --max-time 5000 \
>${myPath}freshen_log_${myChunk}.html 2>${myPath}freshen_ERR_${myChunk}.txp

This page, http://list.bcra.org.uk/data/freshen.html was last modified on Mon, 23 Mar 2026 10:39:29 +0000
Server: britiac4.british-caving.org.uk (31.25.186.126)