EE cache->save operation faulting during writing of large files / file cache in use

[jcogs-design]

Description of the problem
Have noticed that when saving a large array to the EE cache (associative array, about 5K rows, each row holding an array with 40 columns - generates a cache file of about 6Mbytes) the operation sometimes faults and the file written is truncated / corrupted. This faulting is observable in the cPanel resources plot ('faults' are recorded concurrent with when the file write operation fails).

Loss of this file is consequential for the add-on it is being written by.

How To Reproduce
Steps to reproduce the behavior:
Tricky to find a 100% reliable mechanism - currently behaviour is observed on a site with sufficiently large data requirements that triggers creation of the very large array to be cached, and has been seen with the site running on more than one servers - which suggests it is not a hardware issue.

Error Messages
None directly - EE does not appear to report that there was an issue writing the file. The issue is detected through the adverse consequences for site operations when the cache file is corrupted only, and via the cPanel Resources fault event log.

Screenshots / Videos / Template Code
The array is being saved to the cache file cache_log and reflects the total number of images stored by JCOGS Image on the site concerned - so typically grows to a size reflective of the total image estate for the site and stays there, but the file is periodically rewritten (data within the array being saved does change over time). Here you can see that the log file reduced from 1.3Mbytes to 327Kbytes over the course of a day or so. What has been observed is that when the file gets to a sufficiently large size, the probability of the write operation faulting (and all cached information being lost) goes up - subsequent attempts to re-save the file capture only changes made to image estate after the fault, hence why image cache file 'regrows' over time.

Concurrent with this the events that result in the loss of this file content appear to match exactly 'fault' reports for the server.

Apparently the date / time of this final 'fault' marker on this graph coincide with the time of creation of one instance of the cache_log file (a new file is created when existing one not found).

Environment Details:

• Version: 7.3.x and 7.4.x (issue has been under review for some time)
• PHP Version 8.1

Possible Solution
Looking at finding an alternative technical solution for that will avoid having to cache this array.