Repository navigation
Fix splitSepByOneOf on large Word64 values - #3265
Merged
Merged
Conversation
The Karp-Rabin hash called fromEnum, which fails on Word64 values above maxBound :: Int. Hash the bytes of the elements instead, reading an element of size 1, 2, 4 or 8 as one unsigned word; a hash match is already confirmed by comparing bytes. splitSepByOneOf no longer needs an Enum constraint. The hash helpers are in the new Streamly.Internal.Data.SeqMatch module. In the loop phase, read the falling-out element of the longest pattern before overwriting it with the new element and the others after, in a single zipWithM pass.
Move the word packing and decoding and the Karp-Rabin hash arithmetic of Fold.takeEndBySeq, Fold.takeEndBySeq_ and, in Stream, takeEndBySeq, takeEndBySeq_, splitSepBySeq_, splitOnSuffixSeq, dropMatches, replaceMatches and dropSuffix to Streamly.Internal.Data.SeqMatch. The code is unchanged, elements are still packed and hashed using fromEnum.
The tests are marked pending: packing a negative element into a word without masking it gives a false match, decoding a buffered element without sign extension fails for Int8, and the Karp-Rabin hash calls fromEnum, which fails on Word64 values above maxBound :: Int.
The haddock build of streamly-core on GHC 9.8.4 needs more than the 500M set by limit-build-mem (it peaked at 522M in use). Override it for packdiff only, via cabal.project.local which survives packdiff's git checkouts.
harendra-kumar
force-pushed
the
splitSepByOneOf-fixes
branch
from
October 20, 2026 10:25
cccfe28 to
389e188
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
splitSepByOneOfhashed elements withfromEnum, which fails onWord64values above
maxBound :: Int. It now hashes the bytes of each elementinstead, and no longer needs
Enum.The hash and word-packing helpers moved to a separate module, shared by the other sequence search functions.
Also adds pending tests for the same kinds of bugs in the other functions:
signed elements and large
Word64values. The fixes will come in a follow-up PR.