1

StringTokenization

Overview

StringTokenizer class

Some StringTokenizer methods

StringTokenizer examples

2

StringTokenizer class

A token is a portion of a string that is separated from another portion of that string by one or more chosen characters (called delimiters).

Example: Assuming that a while space character (i.e., blank, ‘\n’ (new line ), ‘\t’ (tab), or ‘\r’ (carriage return)) is a delimiter, then the string: “I like KFUPM very much” has the tokens: “I”, “like”, “KFUPM”, “very”, and “much”

The StringTokenizer class contained in the java.util package can be used to break a string into separate tokens. This is particularly useful in those situations in which we want to read and process one token at a time; the BufferedReader class does not have a method to read one token at a time.

The StringTokenizer constructors are:StringTokenizer(String str) Uses white space characters

as a delimiters. The delimiters are not returned.

StringTokenizer(String str,

String delimiters)

delimiters is a string that specifies the delimiters. The delimiters are not returned.

StringTokenizer(String str,

String delimiters, boolean

delimAsToken)

If delimAsToken is true, then each delimiter is also returned as a token; otherwise delimiters are not returned.

3

Some StringTokenizer methods Some StringTokenizer methods are:

To break a string into tokens, a loop having one of the following forms may be used:

StringTokenizer tokenizer = new StringTokenizer(stringName);while(tokenizer.hasMoreTokens( )){ String token = tokenizer.nextToken( );

// process the token. . .

}

int countTokens( ) Using the current set of delimiters, the method returns the number of tokens left.

boolean hasMoreTokens( ) Returns true if one or more tokens remain in the string; otherwise it returns false.

String nextToken( ) throws

NoSuchElementException

Returns the next token as a string. Throws an exception if there are no more tokens

String nextToken(String

newDelimiters) throws

NoSuchElementException

Returns the next token as a string and sets the delimiters to newDelimiters. Throws an exception if there are no more tokens.

4

StringTokenizer examples

StringTokenizer tokenizer = new StringTokenizer (stringName);

int tokenCount = tokenizer.countTokens( );

for(int k = 1; k <= tokenCount; k++)

{

String token = tokenizer.nextToken( );

// process token

. . .

}

Example1:

import java.util.StringTokenizer;

public class Tokenizer1

{

public static void main(String[ ] args)

{

StringTokenizer wordFinder = new StringTokenizer

("We like KFUPM very much");

while( wordFinder.hasMoreTokens( ) )

System.out.println( wordFinder.nextToken( ) );

}

}

5

StringTokenizer examples (Cont’d) Example2: The following program reads grades from the keyboard and

finds their average. The grades are read in one line.

import java.io.*;import java.util.StringTokenizer;

public class Tokenizer5{

public static void main(String[ ] args) throws IOException{

BufferedReader stdin = new BufferedReader(new InputStreamReader(System.in));

System.out.println("Enter grades in one line:");String inputLine = stdin.readLine( );StringTokenizer tokenizer = new StringTokenizer(inputLine);int count = 0;float grade, sum = 0.0F;try { while( tokenizer.hasMoreTokens( ) ) { grade = Float.parseFloat( tokenizer.nextToken( ) ); if(grade >= 0 && grade <= 100) {

sum += grade;count++;

} }

if(count > 0) System.out.println("\nThe average = "+ sum / count); else System.out.println("No valid grades entered");}

6

StringTokenizer examples (Cont’d)

catch(NumberFormatException e)

{

System.err.println("Error - an invalid float value read");

}

}

}

Example3: Given that a text file grades.txt contains ids and quiz grades of students:

980000 50.0 30.0 40.0

975348 50.0 35.0

960035 80.0 70.0 60.0 75.0

950000 20.0 40.0

996245 65.0 70.0 80.0 60.0 45.0

987645 50.0 60.0

the program on the next slide will display the id, number of quizzes taken, and average of each student:

7

StringTokenizer examples (Cont’d)import java.io.*;import java.util.StringTokenizer;

public class Tokenizer6{

public static void main(String[ ] args) throws IOException{ BufferedReader inputStream = new BufferedReader(new

FileReader("grades.txt")); StringTokenizer tokenizer; String inputLine, id; int count; float sum; System.out.println("ID# Number of Quizzes Average\n");

while((inputLine = inputStream.readLine( )) != null) {

tokenizer = new StringTokenizer(inputLine);id = tokenizer.nextToken( );count = tokenizer.countTokens( );sum = 0.0F;while( tokenizer.hasMoreTokens( ) ) sum += Float.parseFloat( tokenizer.nextToken( ) );

System.out.println(id + " " + count + " ” + sum / count);

}}

}