Class Token

  • Direct Known Subclasses:
    TokenCharacter, TokenCharacterSet, TokenEOF, TokenEPSILON, TokenRegex, TokenString, TokenUndefined

    public abstract class Token
    extends Decoratable
    An input token.

    This is an abstraction for input tokens. It allows the parser to be used, for example, for both sequences of characters and sequences of strings. (Sequences of anything you like, provided you define the tokens.)

    The only thing that's important about tokens is that we can tell when they match each other. This is not the same as equality becuase, for example, the same regular expression token might match many different input strings.

    • Constructor Detail

      • Token

        public Token​(java.util.Collection<ParserAttribute> attributes)
        A token with attributes.
        Parameters:
        attributes - the attributes
        Throws:
        GrammarException - if the attribute names are not unique
        AttributeException - if an attribute has an invalid value
    • Method Detail

      • matches

        public abstract boolean matches​(Token input)
        Does this token match the input?
        Parameters:
        input - The input.
        Returns:
        true if this token matches that input.
      • getValue

        public abstract java.lang.String getValue()
        What is this token?
        Returns:
        the token string