Large cache for zst and xz

[Suncatcher]

Something is obviously wrong with the architecture of the app. I got BSOD yesterday and started to investigate the reason. Having checked the Event viewer for errors during last 24h I found this sequence of events:

1. Space is low

   svchost (5892,T,97,0) SRUJet: The database engine is rejecting update operations due to low free disk space on the log disk.
   

2. Clonezilla-util crashes

   Application: clonezilla-util.exe
   CoreCLR Version: 8.0.1425.11118
   .NET Version: 8.0.14
   Description: The process was terminated due to an unhandled exception.
   Exception Info: exception code c0000005, exception address 00007FFF1A591926
   Stack:
   

Faulting application name: clonezilla-util.exe, version: 1.0.0.0, time stamp: 0x67ac2613
Faulting module name: dokan2.dll, version: 2.1.0.1000, time stamp: 0x6585e670
Exception code: 0xc0000005
Fault offset: 0x0000000000011926
Faulting process id: 0x2C98
Faulting application start time: 0x1DC5DBDABE5AA0B
Faulting application path: D:\clonezilla-util.v2.6.0.win-x64\clonezilla-util.exe
Faulting module path: C:\WINDOWS\SYSTEM32\dokan2.dll
Report Id: cc8dd554-0051-4c3d-830b-1833467bd9ac
Faulting package full name: 
Faulting package-relative application ID: 

3. Then all other apps start to crash due to low space

LogFileWriter:Write failed. System.IO.IOException: There is not enough space on the disk. : 'C:\Users\user\AppData\Local\Dell\Dell Display and Peripheral Manager\Log\DDPM.Subagent.User\S-1-5-21-1850704973-3694722576-301810095-1001\DDPMSubagentUser.01.log'
   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.Strategies.OSFileStreamStrategy.Write(ReadOnlySpan`1 buffer)
   at System.IO.Strategies.OSFileStreamStrategy.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.Strategies.BufferedFileStreamStrategy.Flush(Boolean flushToDisk)
   at Dell.Client.Framework.Common.FileStreamWindows.Flush()
   at Dell.Client.Framework.Common.LogFileWriter.Write(LogEntry logEntry)

The WerSvc service was unable to log on as NT AUTHORITY\SYSTEM with the currently configured password due to the following error: 
Not enough memory resources are available to process this command.

To ensure that the service is configured properly, use the Services snap-in in Microsoft Management Console (MMC).

The Windows Error Reporting Service service terminated with the following error: 
The paging file is too small for this operation to complete.

Then I investigated what caused so fast overflow of disc C space

And when I checked in WinDirStat what occupies most of the space I see this temp files

Which are exactly correspond to the file sizes I was trying to copy from the mounted via сlonezilla-util disk, and was copying via RoboCopy

Why on earth сlonezilla-util tries to cache each and every file in such a raw manner? What if the files are 500Gb each? It will try to copy anн file to disk C??? This is simply unwise. Folder with the clonezilla-util is one the disk D, so it makes no sense to copy all the files to C if the utility itself is located on D. I simply cannot understand that.

[fiddyschmitt]

Hi @Suncatcher,

Thanks for the detailed explanation. I'm sorry to hear your drive filled up & resulted in chaos.

There is a reason the cache is like that, and it comes down to the compression formats Clonezilla uses.

Very few compression formats support random access (meaning it allows a program to seek to any part of the decompressed content). clonezilla-util needs random access in order to extract nested files. To make the source files seekable, clonezilla-util creates a cache.

For some formats the cache is small. For example:

• To make a 21 GB gz archive seekable, the index is only 200 MB.
• To make a 21 GB bzip2 archive seekable, the index is only 2 MB.

But other formats (such as zst and xz) are harder to make seekable. So the program does the only thing it can to make them seekable - it extracts them in their entirety into the cache folder. I agree that that is not ideal. Ideally we would have a way to make each format seekable.

It's been a couple of years, so I'll research whether zst and xz can be made seekable.

Cheers,
Fidel

[Suncatcher]

OK, thanks for answer. That makes sense to make a cache, but my concern is about the location of this cache:

Folder with the clonezilla-util is one the disk D, so it makes no sense to copy all the files to C if the utility itself is located on D. I simply cannot understand that

If I have no space on disk C and the util dictates to put all the cache on C, which will cause crash of the system in case of too low space and will have fatal consequences for the system. I should be able to decide where to put cache! We should have a config key for this!