Repository navigation
fix: release SV trial memmaps before temporary directory cleanup - #45
Open
xiangshengyun wants to merge 1 commit into
Open
xiangshengyun wants to merge 1 commit into
xiangshengyun wants to merge 1 commit into
Conversation
An open np.memmap keeps its file locked on Windows, so removing the SV pipeline temporary directory failed with PermissionError [WinError 32] and masked the original result or error. Close the trial mappings explicitly in the pipeline, release the embeddings/scores/labels mappings on every cosine_trial_scores error path, and add regression tests that assert the work directory is empty after both successful scoring and a failing metric.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When the SV pipeline finishes, it removes its temporary work directory. On
Windows an open
np.memmapkeeps the backing file locked, so cleanup raisedPermissionError [WinError 32]ontrial_labels.uint8.mmap/trial_scores.float32.mmap. That exception replaced the real evaluationresult or error, so every SV run failed on Windows regardless of the metric
outcome.
Fix
cosine_trial_scores/node.py: create thetrial_scores/trial_labelsmappings inside a
try/finallyand close both on every exit path, includingthe validation errors that previously leaked them (missing embedding keys,
trial-count mismatch, zero-norm embedding, label-count mismatch).
sv/pipeline.py: releasescores/labelsin afinallyaround metricscoring so a failing metric no longer leaves them open. Add
_release_trial_memmap(), which closesarray._mmap.cannot keep the mappings alive.
Scoring results are unchanged; only the lifetime of the mappings changes.
Tests
Two regressions added to
tests/test_sv_pipeline.py:test_sv_cleans_work_dir_after_scoringtest_sv_cleans_work_dir_when_metric_failsBoth assert
list(work_dir.iterdir()) == []afterevaluate_sv_files.pytest tests/test_sv_pipeline.py→ 9 passed on Windows.Note
The visible failure only reproduces on Windows, because POSIX allows unlinking
a mapped file; Linux CI passed before this change too. The leak itself
(fd/mapping retention, plus arrays captured by exception tracebacks) is
platform-independent, so it is worth fixing upstream rather than only in a
fork.