mirrors/ladybird
0
0
Fork 0
mirror of https://github.com/LadybirdBrowser/ladybird.git synced 2025-10-19 06:29:43 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 9

LibJS: Port the JS lexer and parser to UTF-16

Browse source 
This ports the lexer to UTF-16 and deals with the immediate fallout up
to the AST. The AST will be dealt with in upcoming commits.

The lexer will still accept UTF-8 strings as input, and will transcode
them to UTF-16 for lexing. This doesn't actually incur a new allocation,
as we were already converting the input StringView to a ByteString for
each lexer.

One immediate logical benefit here is that we do not need to know off-
hand how many UTF-8 bytes some special code points occupy. They all
happen to be a single UTF-16 code unit. So instead of advancing the
lexer by 3 positions in some cases, we can just always advance by 1.
This commit is contained in:
Timothy Flynn 2025-08-06 07:18:45 -04:00 • committed by Tim Flynn
commit 00182a2405
Notes: github-actions[bot] 2025-08-13 13:57:27 +00:00
14 changed files with 467 additions and 474 deletions
Download patch file Download diff file Expand all files Collapse all files

4
Libraries/LibTest/JavaScriptTestRunner.h
View file

@ -219,7 +219,7 @@ inline AK::Result<GC::Ref<JS::Script>, ParserError> parse_script(StringView path
if (script_or_errors.is_error()) {
auto errors = script_or_errors.release_error();
return ParserError { errors[0], errors[0].source_location_hint(contents) };
return ParserError { errors[0], errors[0].source_location_hint(Utf16String::from_utf8(contents)) };
}
return script_or_errors.release_value();
@ -232,7 +232,7 @@ inline AK::Result<GC::Ref<JS::SourceTextModule>, ParserError> parse_module(Strin
if (script_or_errors.is_error()) {
auto errors = script_or_errors.release_error();
return ParserError { errors[0], errors[0].source_location_hint(contents) };
return ParserError { errors[0], errors[0].source_location_hint(Utf16String::from_utf8(contents)) };
}
return script_or_errors.release_value();