feat: save the storage file on a background thread by default

CHANGELOG.md

@@ -6,6 +6,18 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/)
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0/).
 
 ## [Unreleased]
+### Changed
+- Storage files are now written on a shared background thread. Changes are still serialized on the calling thread, but `Set` and auto-save no longer wait for the disk. `Save()` and `Dispose()` still block until the data has reached the disk.
+
+### Added
+- `SaveOnBackgroundThread(bool)` on the builder. Pass `false` to write the file before every change returns, as before.
+
+### Removed
+- `IsDirty`. With auto-save on it was almost always `false`, and with the background writer it meant "no change is waiting to be handed over", not "everything is on disk". Use `Save()` when you need the data on disk.
+
+### Fixed
+- Two storages opened through paths that differ only in form (`a/save.dat` and `./a/save.dat`) no longer publish the same file without serializing against each other.
+- A storage larger than 1 MB no longer re-grows its serialization buffer from 1 MB on every save.
 
 ## [1.0.5]
 - Baseline of the changelog. See the GitHub Releases page for earlier history.

README.md

@@ -15,7 +15,7 @@ Unity's standard `PlayerPrefs` has several limitations:
 
 All of these issues are addressed by BinaryPrefs: a configurable, strongly typed, binary key-value storage with support for Unity types, enums, collections, custom serializers, change events, and scoped sub-storages.
 
-And the feature I like most: **every change is persisted the moment it happens**. As soon as you set a value - or even mutate a stored list, set or dictionary - it is written to disk, with atomic, corruption-safe writes. You never have to remember to call `Save()`, and an unexpected crash or quit can't lose the last change. When you need it, many changes can still be batched into a single write.
+And the feature I like most: **every change is persisted the moment it happens**. As soon as you set a value - or even mutate a stored list, set or dictionary - it goes to disk on a background thread, with atomic, corruption-safe writes. You never have to remember to call `Save()`, and the calling thread never waits for the disk. When you need it, many changes can still be batched into a single write.
 
 <!-- omit from toc -->
 ## Table of content
@@ -96,6 +96,7 @@ using var storage = BinaryStorage.Construct(path)
     .SetMissingKeyBehaviour(MissingKeyBehavior.ReturnDefaultValueOnly)
     .SetTypeMismatchBehaviour(TypeMismatchBehaviour.OverrideValueAndType)
     .EnableAutoSaveOnChange()
+    .SaveOnBackgroundThread(true)                          // on by default, false writes the file before every change returns
     .Build(KeyLoadFailedBehaviour.IgnoreWithWarning);
 ```
 
@@ -164,18 +165,29 @@ using (storage.MultipleChangeScope())
 
 ## Saving
 
-By default (and with `BinaryStorage.Get`) auto-save is enabled, so **every change is written to disk immediately** - the moment you `Set` a value or mutate a stored collection. There is no "dirty" window where data lives only in memory: forget-to-save bugs and data lost to a crash simply don't happen.
+By default (and with `BinaryStorage.Get`) auto-save is enabled, so **every change goes to disk on its own** - the moment you `Set` a value or mutate a stored collection. Forget-to-save bugs simply don't happen.
 
-Writes are atomic: data is written to a temporary file first and only then swapped in, so an interrupted save can never corrupt your existing file.
+The change is serialized on your thread and handed to a shared background writer, so `Set` returns without waiting for the disk. A newer change replaces an older one that hasn't been written yet, because the file is always written whole and only the last state matters.
+
+Writes are atomic: data is written to a temporary file first and only then swapped in, so an interrupted save can never corrupt your existing file. A process killed in the window between the change and the write keeps the previous state intact and loses only the last change.
 
 Need to apply many changes as a single write? Wrap them in a [change scope](#batch-changes) - auto-save then fires once, when the scope ends.
 
-If you prefer full manual control, skip `EnableAutoSaveOnChange` and persist yourself:
+If you prefer full manual control, skip `EnableAutoSaveOnChange` and persist yourself. `Save()` blocks until the data has actually reached the disk - it writes the file on your thread rather than queueing behind the background writer:
 
 ```csharp
 storage.Save();
 ```
 
+Disposing a storage writes out whatever is still pending. To go back to writing the file before every change returns, turn the background writer off:
+
+```csharp
+using var storage = BinaryStorage.Construct(path)
+    .AddPrimitiveTypes()
+    .SaveOnBackgroundThread(false)
+    .Build();
+```
+
 ## Change events
 
 ```csharp

src/Runtime/BinaryStorage.Builder.cs

@@ -33,20 +33,11 @@ public static Builder Construct(string filePath)
         }
 
         /// <summary> Deletes the storage file at the specified path. </summary>
+        /// <remarks> Meant for a path no storage is open on. A storage still alive on this path may write it back from memory afterwards. </remarks>
         /// <param name="storagePath">The path to the storage file.</param>
         internal static void Delete(string storagePath)
         {
-            DeleteIfExists(storagePath);
-            DeleteIfExists(storagePath + BinaryStorageIO.TempFileExtension);
-            DeleteIfExists(storagePath + BinaryStorageIO.BackupFileExtension);
-        }
-
-        private static void DeleteIfExists(string filePath)
-        {
-            if (File.Exists(filePath))
-            {
-                File.Delete(filePath);
-            }
+            StorageFile.Of(storagePath).Remove();
         }
 
         /// <summary> Provides a fluent interface for configuring and building a <see cref="BinaryStorage"/> instance. </summary>
@@ -55,6 +46,7 @@ public class Builder
             private readonly string _filePath;
             private readonly List<BinarySection> _serializers = new();
             private bool _autoSave;
+            private bool _saveOnBackgroundThread = true;
             private bool _saveJsonForDebug;
             private MissingKeyBehavior _missingKeyBehavior = MissingKeyBehavior.InitializeWithDefaultValue;
             private TypeMismatchBehaviour _typeMismatchBehaviour = TypeMismatchBehaviour.ThrowException;
@@ -72,6 +64,20 @@ public Builder EnableAutoSaveOnChange()
                 return this;
             }
 
+            /// <summary> Controls whether saving happens on a background thread. </summary>
+            /// <remarks>
+            /// Enabled by default. Changes are serialized on the calling thread and the file is written on a shared background thread,
+            /// so a change no longer blocks the caller until the data is on disk. <see cref="Save"/> stays blocking in both modes.
+            /// Disable it to get the previous behaviour, where every change writes the file before returning.
+            /// </remarks>
+            /// <param name="enabled">Whether the storage file should be written on a background thread.</param>
+            /// <returns>The current <see cref="Builder"/> instance for method chaining.</returns>
+            public Builder SaveOnBackgroundThread(bool enabled = true)
+            {
+                _saveOnBackgroundThread = enabled;
+                return this;
+            }
+
             /// <summary> Enables writing a human-readable JSON copy of the storage next to the binary file for debugging. </summary>
             /// <remarks> The JSON copy is write-only and never loaded back; the binary file remains the only source of truth. Pass a build-type check (for example, a development-build flag) to toggle it without changing the rest of the configuration. </remarks>
             /// <param name="enabled">Whether the JSON debug copy should be written on each save.</param>
@@ -246,7 +252,7 @@ public Builder SupportDictionariesOf<TKey, TValue>()
             /// <exception cref="KeyLoadFailedException"> A key failed to load and <paramref name="keyLoadFailedBehaviour"/> is <see cref="KeyLoadFailedBehaviour.ThrowException"/>. </exception>
             public BinaryStorage Build(KeyLoadFailedBehaviour keyLoadFailedBehaviour = KeyLoadFailedBehaviour.IgnoreWithWarning)
             {
-                var storage = new BinaryStorage(_filePath, _serializers);
+                var storage = new BinaryStorage(_filePath, _serializers, _saveOnBackgroundThread);
                 storage.AutoSave = _autoSave;
                 storage.SaveJsonCopyForDebug = _saveJsonForDebug;
                 storage.MissingKeyBehavior = _missingKeyBehavior;

src/Runtime/BinaryStorage.cs

@@ -11,38 +11,41 @@ namespace Appegy.Storage
     public partial class BinaryStorage : IDisposable, IBinaryStorage
     {
         private readonly string _storageFilePath;
-        private readonly StorageFilePaths _storageFilePaths;
+        private readonly StoragePersistence _persistence;
         private readonly IReadOnlyList<BinarySection> _supportedTypes;
         private readonly Dictionary<string, Record> _data = new();
         private readonly Dictionary<IReactiveCollection, string> _collections = new();
         private int _changeScopeCounter;
+        private bool _hasUnsavedChanges;
 
         /// <summary> Gets or sets a value indicating whether data should be saved automatically. </summary>
         public bool AutoSave { get; set; }
 
         /// <summary> Gets or sets a value indicating whether a human-readable JSON copy is written next to the binary file on each save. The copy is write-only and never loaded back. </summary>
-        public bool SaveJsonCopyForDebug { get; set; }
+        public bool SaveJsonCopyForDebug
+        {
+            get => _persistence.SaveJsonCopyForDebug;
+            set => _persistence.SaveJsonCopyForDebug = value;
+        }
 
         /// <summary> Gets or sets the behavior when a requested key is not found in the storage. </summary>
         public MissingKeyBehavior MissingKeyBehavior { get; set; } = MissingKeyBehavior.ReturnDefaultValueOnly;
 
         /// <summary> Gets or sets the behavior when the type of value associated with a key does not match the expected type. </summary>
         public TypeMismatchBehaviour TypeMismatchBehaviour { get; set; } = TypeMismatchBehaviour.OverrideValueAndType;
 
-        /// <summary> Gets a value indicating whether there are unsaved changes. </summary>
-        public bool IsDirty { get; private set; }
-
         /// <summary> Gets a value indicating whether the storage has been disposed. </summary>
         public bool IsDisposed { get; private set; }
 
         /// <summary> Initializes a new instance of the <see cref="BinaryStorage"/> class. </summary>
         /// <param name="storageFilePath">The file path for storing data.</param>
         /// <param name="supportedTypes">The list of supported types for storage.</param>
-        internal BinaryStorage(string storageFilePath, IReadOnlyList<BinarySection> supportedTypes)
+        /// <param name="saveOnBackgroundThread">Whether the storage file is written on a background thread.</param>
+        internal BinaryStorage(string storageFilePath, IReadOnlyList<BinarySection> supportedTypes, bool saveOnBackgroundThread)
         {
             _storageFilePath = storageFilePath;
-            _storageFilePaths = new StorageFilePaths(storageFilePath);
             _supportedTypes = supportedTypes;
+            _persistence = new StoragePersistence(storageFilePath, supportedTypes, saveOnBackgroundThread);
         }
 
         #region Events
@@ -303,11 +306,11 @@ public virtual int RemoveAll()
             return count;
         }
 
-        /// <summary> Saves the current data to disk. </summary>
+        /// <summary> Saves the current data to disk and waits until it is there. </summary>
         /// <exception cref="ObjectDisposedException">Thrown if the storage is disposed.</exception>
         public virtual void Save()
         {
-            SaveDataFromDisk();
+            SaveDataOnDisk(true);
         }
 
         /// <summary> Begins a scope for making multiple changes. </summary>
@@ -595,9 +598,9 @@ private void DecreaseCounter()
             {
                 return;
             }
-            if (_changeScopeCounter == 0 && IsDirty && AutoSave)
+            if (_changeScopeCounter == 0 && _hasUnsavedChanges && AutoSave)
             {
-                SaveDataFromDisk();
+                SaveDataOnDisk(false);
             }
         }
 
@@ -618,10 +621,10 @@ private void MarkChanged()
         {
             if (!AutoSave || _changeScopeCounter > 0)
             {
-                IsDirty = true;
+                _hasUnsavedChanges = true;
                 return;
             }
-            SaveDataFromDisk();
+            SaveDataOnDisk(false);
         }
 
         /// <summary> Throws an exception if the storage has been disposed. </summary>
@@ -671,9 +674,16 @@ private void Dispose(bool disposing)
                 return;
             }
 
-            if (disposing && AutoSave && IsDirty)
+            if (disposing)
             {
-                SaveDataFromDisk();
+                if (AutoSave && _hasUnsavedChanges)
+                {
+                    SaveDataOnDisk(true);
+                }
+                else
+                {
+                    _persistence.Flush();
+                }
             }
 
             // Always dispose IReactiveCollection instances
@@ -718,7 +728,7 @@ private void Dispose(bool disposing)
         private void LoadDataFromDisk(KeyLoadFailedBehaviour keyLoadFailedBehaviour)
         {
             ThrowIfDisposed();
-            BinaryStorageIO.LoadDataFromDisk(_storageFilePaths, _supportedTypes, _data, keyLoadFailedBehaviour);
+            _persistence.Load(_data, keyLoadFailedBehaviour);
             foreach (var pair in _data)
             {
                 var rc = pair.Value.AsReactiveCollection();
@@ -731,24 +741,14 @@ private void LoadDataFromDisk(KeyLoadFailedBehaviour keyLoadFailedBehaviour)
         }
 
         /// <summary> Saves the data from memory to disk. </summary>
+        /// <param name="waitForDisk">Whether to block until the data has actually reached the disk.</param>
         /// <exception cref="ObjectDisposedException">Thrown if the storage is disposed.</exception>
         /// <exception cref="IOException"> An I/O error occurred </exception>
-        private void SaveDataFromDisk()
+        private void SaveDataOnDisk(bool waitForDisk)
         {
             ThrowIfDisposed();
-            BinaryStorageIO.SaveDataOnDisk(_storageFilePaths, _supportedTypes, _data);
-            IsDirty = false;
-            if (SaveJsonCopyForDebug)
-            {
-                try
-                {
-                    BinaryStorageIO.SaveJsonCopyOnDisk(_storageFilePath, _data);
-                }
-                catch (Exception exception)
-                {
-                    UnityEngine.Debug.LogWarning($"Failed to save JSON debug copy of '{_storageFilePath}'. Reason: {exception.Message}");
-                }
-            }
+            _persistence.Save(_data, waitForDisk);
+            _hasUnsavedChanges = false;
         }
 
         #endregion