what does a token actually represent?

Replies

• 

  ManojKiran Eda

  radha gogia

  Please explain the below paragraph
  
  "Typically with a lexer/parser a token is a structure that holds not only the name of the token, but the characters/symbols that make up the token and the start and end position of the string of characters that make up the token, with the start and end position being used for error reporting, highlighting, etc. a token will more likely hold the start and end position of the characters/symbols that represent the token and the lexeme, sequence of characters/symbols can be derived from the start and end position as needed because the input is static."
  
  I am not able to get the understanding of the start and ending position which the token will hold ,plz clarify it.

  Actually I know that a token is a small part of a program that may be a variable or a constant....But after reading the above mentioned paragraph I am actually confused tooo...
• 

  rahul69

  MANOJ KIRAN

  Actually I know that a token is a small part of a program that may be a variable or a constant....But after reading the above mentioned paragraph I am actually confused tooo...

  In the original post, Info given about token is in context to lexical analyzer.
  Simply put, when data is fed to lex, it is just strings of characters, eg:
  a=b+c
  Then it will divide it into tokens:
  {a,=,b,+,c}
  This is just an analogy for understanding,(not the original process), but hope it clear some doubts. 😀