For the web, we allow a wobbly UTF-16 encoding (i.e. lonely surrogates
are permitted). Only in a few exceptional cases do we strictly require
valid UTF-16. As such, our `validate(AllowLonelySurrogates::Yes)` calls
will always succeed. It's a wasted effort to ever make such a check.
This patch eliminates such invocations. The validation methods will now
only check for strict UTF-16, and are only invoked when needed.
When we build a UTF-16 string, we currently always switch to the UTF-16
storage mode inside StringBuilder. Then when it comes time to create the
string, we switch the storage to ASCII if possible (by shifting the
underlying bytes up).
Instead, let's start out with ASCII storage and then switch to UTF-16
storage once we see a non-ASCII code point. For most strings, this will
avoid allocating 2x the memory, and avoids many ASCII validation calls.
We now define GenericLexer as a template to allow using it with UTF-16
strings. To keep existing users happy, the template is defined in the
Detail namespace. Then AK::GenericLexer is an alias for a char-based
view, and AK::Utf16GenericLexer is an alias for a char16-based view.
* Remove completely unused methods.
* Deduplicate methods that were overloaded with both StringView and
char const* parameters.
A future commit will templatize GenericLexer by char type. This patch
serves to make that a tiny bit easier.
We were handling the special cases of NaN and Infinity in basically the
same way across both functions - we can reduce code duplication by
moving this to before we branch.
This is also required as we will be moving the logic to encode in
scientific notation above the branch in a later commit and the
`convert_floating_point_to_decimal_exponential_form` method doesn't work
with non-finite values.
In the following synthetic benchmark, the simdutf version is 4x faster:
BENCHMARK_CASE(find)
{
auto string = u"😀Foo😀Bar"sv;
for (size_t i = 0; i < 100'000'000; ++i)
(void)string.find_code_unit_offset('a');
}
...for the first byte.
This function only really needs to read a single byte at that point, so
read_until_filled() is useless and read_value<u8> is functionally
equivalent to just a read.
This showed up hot in a wasm parse benchmark.
These are quite bottlenecky in wasm, the next commit will try to make
use of this by calling them directly instead of doing a vcall, and
having them inlineable helps the compiler a bit.
Before now, you could compare a Utf16View to a StringView, but it would
only be valid if the StringView were ASCII. When porting code to UTF-16,
it will be handy to have a code point-aware implementation for non-ASCII
StringViews.
