\begin{frame}{Lexical Analysis}
    \emph{Lexical analysis} converts a sequence of \emph{characters}
    into a sequence of \emph{tokens}.
  Programs that do lexical analysis are \emph{lexers} or \emph{tokenizers}.
    For example the expression
      \texttt{sum = 15 + 2}
    could be converted to the sequence of tokens
        token & token category \\
        \texttt{sum} & identifier \\
        \texttt{=} & assignment \\
        \texttt{15} & integer literal \\
        \texttt{+} & operator \\
        \texttt{2} & integer literal \\
  Allows to write parsers on the more abstract level of tokens.