Home » Java » Core Java » “Did you mean” feature with Apache Lucene Spell-Checker

About Ilias Tsagklis

Ilias Tsagklis
Ilias is a senior software engineer working in the telecom domain. He is an applications developer in a wide variety of applications/services, currently the technical lead in a in-house PCRF solution. Particularly interested in multi-tier architecture, middleware services and mobile development (contact). Ilias Tsagklis is co-founder and Executive Editor at Java Code Geeks.

“Did you mean” feature with Apache Lucene Spell-Checker

Google’s “Did you mean” feature
After making an introduction to Lucene in a previous post, now it is time to take it up a notch and create a more sophisticated application. You are most surely familiar with Google’s “Did you mean” feature (other search engines support this too). Here is an example of that:

Lucene SpellChecker Subproject
This feature can be implemented in Java using Lucene project and this tutorial will show you how. The implementation will be based on one of Lucene’s sub-projects, named SpellChecker (see spell checker API). The inspiration for this post was a recent article that I read at InfoQ named “Implementing Google’s “Did you mean” Feature In Java”. The author shortly describes the edit distance algorithms that are used and then provides a simple code example. I also found a similar article named “Did You Mean: Lucene?”, but this was written back in 2005, so I guess it is now quite outdated. It is a good idea to first take a look on those and then come back for some action.

Finding the base dictionary
For the purposes of this tutorial, an English dictionary text file will be used. You can perform a simple search for that or you can directly download this zipped file. The file is very simple and just contains the words separated by lines. The extracted file is named “fulldictionary00.txt”.

Using the Lucene SpellChecker API
Let’s get started by creating a new Eclipse project, perhaps under the name “LuceneSuggestionsProject”, and import the necessary JAR files. Except for the main JAR lucene-core-3.0.1.jar, we will also need the lucene-spellchecker-3.0.1.jar. This can be found in the extracted tarball in the folder “lucene-3.0.1\contrib\spellchecker”. In case you don’t know how to find the distribution and get started with the project, please check my previous post (An Introduction to Apache Lucene for Full-Text Search). Create a new class named “SimpleSuggestionService” and make sure a main method is included. The source code for this class is the following:

package com.javacodegeeks.lucene.spellcheck;

import java.io.File;

import org.apache.lucene.search.spell.PlainTextDictionary;
import org.apache.lucene.search.spell.SpellChecker;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.FSDirectory;

public class SimpleSuggestionService {
    
    public static void main(String[] args) throws Exception {
        
        File dir = new File("c:/spellchecker/");
        
        Directory directory = FSDirectory.open(dir);
        
        SpellChecker spellChecker = new SpellChecker(directory);
        
        spellChecker.indexDictionary(
             new PlainTextDictionary(new File("c:/fulldictionary00.txt")));
        
        String wordForSuggestions = "hwllo";
        
        int suggestionsNumber = 5;

        String[] suggestions = spellChecker.
            suggestSimilar(wordForSuggestions, suggestionsNumber);

        if (suggestions!=null && suggestions.length>0) {
            for (String word : suggestions) {
                System.out.println("Did you mean:" + word);
            }
        }
        else {
            System.out.println("No suggestions found for word:"+wordForSuggestions);
        }
            
    }

}

Yes, it is as simple as that. If you run the application (using “hwllo” as the input word), the following output will occur:

Explaining the SpellChecker API
The code is very straightforward, but I will elaborate on some of the classes. To begin with, you need a Directory, which is a flat list of files. This class is abstract so we use FSDirectory as the actual implementation. FSDirectory stores the index files in the file system (for faster access RAMDirectory, which is the memory resident implementation, should be evaluated). The next step is to create a SpellChecker instance, which is the heart of our application. Note that the default StringDistance is the LevensteinDistance. If you desire to use an other algorithm (JaroWinklerDistance or NGramDistance), you can use the setStringDistance method. Next, we instantiate a PlainTextDictionary, which points to the text file we downloaded. The dictionary is indexed and then the SpellChecker object is used to retrieve the suggested words for a misspelled word.

I guess you are now ready to create the next big search engine. You can find the relevant Eclipse project here.

Enjoy!

Related Articles :

Do you want to know how to develop your skillset to become a Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you our best selling eBooks for FREE!

1. JPA Mini Book

2. JVM Troubleshooting Guide

3. JUnit Tutorial for Unit Testing

4. Java Annotations Tutorial

5. Java Interview Questions

6. Spring Interview Questions

7. Android UI Design

and many more ....

4 comments

  1. Thanks for to the point solution!! :)
    I have simply copied your program. However, I am facing one problem;
    In my text file which is being indexed has one entry ‘nike’. Now, when my query is ‘nike’, it shows NO RESULTS. But when my query is ‘mike’, it returns ‘nike’. Strange!!
    Do you have any clue about this behavior?

  2. spellChecker.indexDictionary(new PlainTextDictionary(new File(“c:/fulldictionary00.txt”)));

    for the above code, the warning comes like this……

    method indexDictionary in class org.apache.lucene.search.spell.SpellChecker cannot be applied to given types
    required: org.apache.lucene.search.spell.Dictionary,org.apache.lucene.index.IndexWriterConfig,boolean
    found: org.apache.lucene.search.spell.PlainTextDictionary

  3. spellChecker.indexDictionary(new PlainTextDictionary(new File(

    “Y:/fulldictionary00/fulldictionary00.txt”)),

    new IndexWriterConfig(Version.LUCENE_CURRENT,

    new StandardAnalyzer(Version.LUCENE_CURRENT)), false);

    Modified like this for lucene 4.1

  4. Hey webzeest,

    I tried your syntax with Lucene 4.1, but still not able to run the code.
    Getting the error:
    Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/lucene/document/Fieldable
    at ir.assignments.four.SimpleSuggestionService.main(SimpleSuggestionService.java:42)

    I am making a search engine and for incorporating it with the GUI I am planning to add the auto-spell check and auto-suggestion feature to it.

    Did you make any other changes as well to run the code successfully?

    Thanks in advance.

Leave a Reply

Your email address will not be published. Required fields are marked *

*


× 4 = twenty eight

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Do you want to know how to develop your skillset and become a ...

Subscribe to our newsletter to start Rocking right now!

To get you started we give you our best selling eBooks for FREE!
Get ready to Rock!
To download the books, please verify your email address by following the instructions found on the email we just sent you.

THANK YOU!

Close