Fix string hashing for utf16 strings - #1376
Open
tobil4sk wants to merge 4 commits into
Open
Conversation
Outside the BMP, characters take two utf16 code units. The old hashing was incorrect as it would only ever read a single unit at a time, and therefore calculate the wrong hash for a string containing a non-BMP character. The Char16Advance call handles this, ensuring that surrogate pairs are read as a single code point.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1375.
String::hashwas broken for (non-literal) utf16 strings because it didn't handle theisUTF16Encoded()case.String::calcHashalso did not handle surrogate pairs correctly, and instead iterated on individual code units.During hashing, invalid utf16 code units are replaced with the replacement character (
Char16AdvancewiththrowOnErr = false). This avoids introducing exceptions to code that wasn't throwing before, however, it means two different invalid utf16 strings can have hashes that collide. This seems to be an ok compromise.