CrazyEngineers
CrazyEngineers
  • Please explain the below paragraph

    "Typically with a lexer/parser a token is a structure that holds not only the name of the token, but the characters/symbols that make up the token and the start and end position of the string of characters that make up the token, with the start and end position being used for error reporting, highlighting, etc. a token will more likely hold the start and end position of the characters/symbols that represent the token and the lexeme, sequence of characters/symbols can be derived from the start and end position as needed because the input is static."

    I am not able to get the understanding of the start and ending position which the token will hold ,plz clarify it.
    Replies
Howdy guest!
Dear guest, you must be logged-in to participate on CrazyEngineers. We would love to have you as a member of our community. Consider creating an account or login.
Replies

  • ManojKiran Eda

    Member • Mar 23, 2015

    radha gogia
    Please explain the below paragraph

    "Typically with a lexer/parser a token is a structure that holds not only the name of the token, but the characters/symbols that make up the token and the start and end position of the string of characters that make up the token, with the start and end position being used for error reporting, highlighting, etc. a token will more likely hold the start and end position of the characters/symbols that represent the token and the lexeme, sequence of characters/symbols can be derived from the start and end position as needed because the input is static."

    I am not able to get the understanding of the start and ending position which the token will hold ,plz clarify it.
    Actually I know that a token is a small part of a program that may be a variable or a constant....But after reading the above mentioned paragraph I am actually confused tooo...
    Are you sure? This action cannot be undone.
    Cancel

  • rahul69

    Member • Mar 28, 2015

    MANOJ KIRAN
    Actually I know that a token is a small part of a program that may be a variable or a constant....But after reading the above mentioned paragraph I am actually confused tooo...
    In the original post, Info given about token is in context to lexical analyzer.
    Simply put, when data is fed to lex, it is just strings of characters, eg:
    a=b+c
    Then it will divide it into tokens:
    {a,=,b,+,c}
    This is just an analogy for understanding,(not the original process), but hope it clear some doubts. 😀
    Are you sure? This action cannot be undone.
    Cancel
Home Channels Search Login Register