mirrors/ladybird
0
0
Fork 0
mirror of https://github.com/LadybirdBrowser/ladybird.git synced 2025-10-16 13:09:41 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 9

LibJS: Port the JS lexer and parser to UTF-16

Browse source 
This ports the lexer to UTF-16 and deals with the immediate fallout up
to the AST. The AST will be dealt with in upcoming commits.

The lexer will still accept UTF-8 strings as input, and will transcode
them to UTF-16 for lexing. This doesn't actually incur a new allocation,
as we were already converting the input StringView to a ByteString for
each lexer.

One immediate logical benefit here is that we do not need to know off-
hand how many UTF-8 bytes some special code points occupy. They all
happen to be a single UTF-16 code unit. So instead of advancing the
lexer by 3 positions in some cases, we can just always advance by 1.
This commit is contained in:
Timothy Flynn 2025-08-06 07:18:45 -04:00 • committed by Tim Flynn
commit 00182a2405
Notes: github-actions[bot] 2025-08-13 13:57:27 +00:00
14 changed files with 467 additions and 474 deletions
Download patch file Download diff file Expand all files Collapse all files

2
Libraries/LibJS/ParserError.h
View file

@ -22,7 +22,7 @@ struct JS_API ParserError {
String to_string() const;
ByteString to_byte_string() const;
ByteString source_location_hint(StringView source, char const spacer = ' ', char const indicator = '^') const;
ByteString source_location_hint(Utf16View const& source, char spacer = ' ', char indicator = '^') const;
};
}