Package mars.venus.editors.jeditsyntax.tokenmarker

Class Token

java.lang.Object
mars.venus.editors.jeditsyntax.tokenmarker.Token
public class Token extends Object
A linked list of tokens. Each token has three fields - a token identifier, which is a byte value that can be looked up in the array returned by SyntaxDocument.getColors() to get a color value, a length value which is the length of the token in the text, and a pointer to the next token in the list.
Version:
$Id: Token.java,v 1.12 1999/12/13 03:40:30 sp Exp $
Author:
Slava Pestov

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static byte
    COMMENT1
    Comment 1 token id.
    static byte
    COMMENT2
    Comment 2 token id.
    static byte
    END
    The token type, that along with a length of 0 marks the end of the token list.
    byte
    id
    The id of this token.
    static byte
    ID_COUNT
    The total number of defined token ids.
    static byte
    INTERNAL_FIRST
    The first id that can be used for internal state in a token marker.
    static byte
    INTERNAL_LAST
    The last id that can be used for internal state in a token marker.
    static byte
    INVALID
    Invalid token id.
    static byte
    KEYWORD1
    Keyword 1 token id.
    static byte
    KEYWORD2
    Keyword 2 token id.
    static byte
    KEYWORD3
    Keyword 3 token id.
    static byte
    LABEL
    Label token id.
    int
    length
    The length of this token.
    static byte
    LITERAL1
    Literal 1 token id.
    static byte
    LITERAL2
    Literal 2 token id.
    static byte
    MACRO_ARG
    Macro parameter token.
    Token
    next
    The next token in the linked list.
    static byte
    NULL
    Normal text token id.
    static byte
    OPERATOR
    Operator token id.

  • Constructor Summary

    Constructors
    Constructor
    Description
    Token​(int length, byte id)
    Creates a new token.

  • Method Summary

    Modifier and Type
    Method
    Description
    String
    toString()
    Returns a string representation of this token.

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

  • Field Details

    • NULL

      public static final byte NULL
      Normal text token id. This should be used to mark normal text.
      See Also:
      Constant Field Values

    • COMMENT1

      public static final byte COMMENT1
      Comment 1 token id. This can be used to mark a comment.
      See Also:
      Constant Field Values

    • COMMENT2

      public static final byte COMMENT2
      Comment 2 token id. This can be used to mark a comment.
      See Also:
      Constant Field Values

    • LITERAL1

      public static final byte LITERAL1
      Literal 1 token id. This can be used to mark a string literal (eg, C mode uses this to mark "..." literals)
      See Also:
      Constant Field Values

    • LITERAL2

      public static final byte LITERAL2
      Literal 2 token id. This can be used to mark an object literal (eg, Java mode uses this to mark true, false, etc)
      See Also:
      Constant Field Values

    • LABEL

      public static final byte LABEL
      Label token id. This can be used to mark labels (eg, C mode uses this to mark ...: sequences)
      See Also:
      Constant Field Values

    • KEYWORD1

      public static final byte KEYWORD1
      Keyword 1 token id. This can be used to mark a keyword. This should be used for general language constructs.
      See Also:
      Constant Field Values

    • KEYWORD2

      public static final byte KEYWORD2
      Keyword 2 token id. This can be used to mark a keyword. This should be used for preprocessor commands, or variables.
      See Also:
      Constant Field Values

    • KEYWORD3

      public static final byte KEYWORD3
      Keyword 3 token id. This can be used to mark a keyword. This should be used for data types.
      See Also:
      Constant Field Values

    • OPERATOR

      public static final byte OPERATOR
      Operator token id. This can be used to mark an operator. (eg, SQL mode marks * +, -, etc with this token type)
      See Also:
      Constant Field Values

    • INVALID

      public static final byte INVALID
      Invalid token id. This can be used to mark invalid or incomplete tokens, so the user can easily spot syntax errors.
      See Also:
      Constant Field Values

    • MACRO_ARG

      public static final byte MACRO_ARG
      Macro parameter token. Added for MARS 4.3.
      See Also:
      Constant Field Values

    • ID_COUNT

      public static final byte ID_COUNT
      The total number of defined token ids.
      See Also:
      Constant Field Values

    • INTERNAL_FIRST

      public static final byte INTERNAL_FIRST
      The first id that can be used for internal state in a token marker.
      See Also:
      Constant Field Values

    • INTERNAL_LAST

      public static final byte INTERNAL_LAST
      The last id that can be used for internal state in a token marker.
      See Also:
      Constant Field Values

    • END

      public static final byte END
      The token type, that along with a length of 0 marks the end of the token list.
      See Also:
      Constant Field Values

    • length

      public int length
      The length of this token.

    • id

      public byte id
      The id of this token.

    • next

      public Token next
      The next token in the linked list.

  • Constructor Details

    • Token

      public Token​(int length, byte id)
      Creates a new token.
      Parameters:
      length - The length of the token
      id - The id of the token

  • Method Details

    • toString

      public String toString()
      Returns a string representation of this token.
      Overrides:
      toString in class Object