Previously, if the NumericCharacterReferenceEnd state was reached when
current_input_character was None, then the
DONT_CONSUME_NEXT_INPUT_CHARACTER macro would restore back before the
EOF, and allow the next state (after the SWITCH_TO_RETURN_STATE) to
proceed with the last digit of the numeric character reference.
For example, with something like `ї`, before this commit the
output would incorrectly be `<code point with the value 1111>1` instead
of just `<code point with the value 1111>`.
Instead of putting the `if (current_input_character.has_value())` check
inside NumericCharacterReferenceEnd directly, it was instead added to
DONT_CONSUME_NEXT_INPUT_CHARACTER, because all usages of the macro
benefit from this check, even if the other existing usage sites don't
exhibit any bugs without it:
- In MarkupDeclarationOpen, if the current_input_character is EOF, then
the previous character is always `!`, so restoring and then checking
forward for strings like `--`, `DOCTYPE`, etc won't match and the
BogusComment state will run one extra time (once for `!` and once
for EOF) with no practical consequences. With the `has_value()` check,
BogusComment will only run once with EOF.
- In AfterDOCTYPEName, ConsumeNextResult::RanOutOfCharacters can only
occur when stopping at the insertion point, and because of how
the code is structured, it is guaranteed that current_input_character
is either `P` or `S`, so the `has_value()` check is irrelevant.
In particular, input character lookahead now knows how to stop at the
insertion point marker if needed.
This makes it possible to do amazing things like having document.write()
insert doctypes one character at a time.
If we reach the insertion point at the same time as we switch to another
tokenizer state, we have to bail immediately without proceeding with the
next code point. Otherwise we'd fetch the next token, get an EOF marker,
and then proceed as if we're at the end of the input stream, even though
more data may be coming (with more calls to document.write()..)
This fixes 4 issues:
- RECONSUME_IN_RETURN_STATE was functionally equivalent to
SWITCH_TO_RETURN_STATE, which caused us to lose characters.
For example, &test= would lose the =
- & characters by themselves would be lost. For example, 1 & 2
would become 1 2. This is because we forgot to flush
characters in the the ANYTHING_ELSE path in CharacterReference
- Named character references didn't work at all in attributes.
This is because there was a path that was checking the entity
code points instead of the entity itself. Plus, the path that
was checking the entity itself wasn't quite spec compliant.
- If we fail to match a named character reference, the first
character is lost. For example &test would become &est.
However, this relies on a little hack since I can't wrap my
head around on how to change the code to do as the spec says.
The hack is to reconsume in AmbigiousAmpersand instead of
just switching to it.
Fixes#3957
