Ilias Tsagklis

About Ilias Tsagklis

Ilias Tsagklis is a senior software engineer working in the telecom domain. He is an applications developer in a wide variety of applications/services. Ilias is co-founder and Executive Editor at Java Code Geeks.

“Did you mean” feature with Apache Lucene Spell-Checker

Google’s “Did you mean” feature
After making an introduction to Lucene in a previous post, now it is time to take it up a notch and create a more sophisticated application. You are most surely familiar with Google’s “Did you mean” feature (other search engines support this too). Here is an example of that:

Lucene SpellChecker Subproject
This feature can be implemented in Java using Lucene project and this tutorial will show you how. The implementation will be based on one of Lucene’s sub-projects, named SpellChecker (see spell checker API). The inspiration for this post was a recent article that I read at InfoQ named “Implementing Google’s “Did you mean” Feature In Java”. The author shortly describes the edit distance algorithms that are used and then provides a simple code example. I also found a similar article named “Did You Mean: Lucene?”, but this was written back in 2005, so I guess it is now quite outdated. It is a good idea to first take a look on those and then come back for some action.

Finding the base dictionary
For the purposes of this tutorial, an English dictionary text file will be used. You can perform a simple search for that or you can directly download this zipped file. The file is very simple and just contains the words separated by lines. The extracted file is named “fulldictionary00.txt”.

Using the Lucene SpellChecker API
Let’s get started by creating a new Eclipse project, perhaps under the name “LuceneSuggestionsProject”, and import the necessary JAR files. Except for the main JAR lucene-core-3.0.1.jar, we will also need the lucene-spellchecker-3.0.1.jar. This can be found in the extracted tarball in the folder “lucene-3.0.1\contrib\spellchecker”. In case you don’t know how to find the distribution and get started with the project, please check my previous post (An Introduction to Apache Lucene for Full-Text Search). Create a new class named “SimpleSuggestionService” and make sure a main method is included. The source code for this class is the following:

package com.javacodegeeks.lucene.spellcheck;

import java.io.File;

import org.apache.lucene.search.spell.PlainTextDictionary;
import org.apache.lucene.search.spell.SpellChecker;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.FSDirectory;

public class SimpleSuggestionService {
    
    public static void main(String[] args) throws Exception {
        
        File dir = new File("c:/spellchecker/");
        
        Directory directory = FSDirectory.open(dir);
        
        SpellChecker spellChecker = new SpellChecker(directory);
        
        spellChecker.indexDictionary(
             new PlainTextDictionary(new File("c:/fulldictionary00.txt")));
        
        String wordForSuggestions = "hwllo";
        
        int suggestionsNumber = 5;

        String[] suggestions = spellChecker.
            suggestSimilar(wordForSuggestions, suggestionsNumber);

        if (suggestions!=null && suggestions.length>0) {
            for (String word : suggestions) {
                System.out.println("Did you mean:" + word);
            }
        }
        else {
            System.out.println("No suggestions found for word:"+wordForSuggestions);
        }
            
    }

}

Yes, it is as simple as that. If you run the application (using “hwllo” as the input word), the following output will occur:

Explaining the SpellChecker API
The code is very straightforward, but I will elaborate on some of the classes. To begin with, you need a Directory, which is a flat list of files. This class is abstract so we use FSDirectory as the actual implementation. FSDirectory stores the index files in the file system (for faster access RAMDirectory, which is the memory resident implementation, should be evaluated). The next step is to create a SpellChecker instance, which is the heart of our application. Note that the default StringDistance is the LevensteinDistance. If you desire to use an other algorithm (JaroWinklerDistance or NGramDistance), you can use the setStringDistance method. Next, we instantiate a PlainTextDictionary, which points to the text file we downloaded. The dictionary is indexed and then the SpellChecker object is used to retrieve the suggested words for a misspelled word.

I guess you are now ready to create the next big search engine. You can find the relevant Eclipse project here.

Enjoy!

Related Articles :
Related Whitepaper:

Bulletproof Java Code: A Practical Strategy for Developing Functional, Reliable, and Secure Java Code

Use Java? If you do, you know that Java software can be used to drive application logic of Web services or Web applications. Perhaps you use it for desktop applications? Or, embedded devices? Whatever your use of Java code, functional errors are the enemy!

To combat this enemy, your team might already perform functional testing. Even so, you're taking significant risks if you have not yet implemented a comprehensive team-wide quality management strategy. Such a strategy alleviates reliability, security, and performance problems to ensure that your code is free of functionality errors.Read this article to learn about this simple four-step strategy that is proven to make Java code more reliable, more secure, and easier to maintain.

Get it Now!  

4 Responses to "“Did you mean” feature with Apache Lucene Spell-Checker"

  1. gaurang patel says:

    Thanks for to the point solution!! :)
    I have simply copied your program. However, I am facing one problem;
    In my text file which is being indexed has one entry ‘nike’. Now, when my query is ‘nike’, it shows NO RESULTS. But when my query is ‘mike’, it returns ‘nike’. Strange!!
    Do you have any clue about this behavior?

  2. Vinod says:

    spellChecker.indexDictionary(new PlainTextDictionary(new File(“c:/fulldictionary00.txt”)));

    for the above code, the warning comes like this……

    method indexDictionary in class org.apache.lucene.search.spell.SpellChecker cannot be applied to given types
    required: org.apache.lucene.search.spell.Dictionary,org.apache.lucene.index.IndexWriterConfig,boolean
    found: org.apache.lucene.search.spell.PlainTextDictionary

  3. webzeest says:

    spellChecker.indexDictionary(new PlainTextDictionary(new File(

    “Y:/fulldictionary00/fulldictionary00.txt”)),

    new IndexWriterConfig(Version.LUCENE_CURRENT,

    new StandardAnalyzer(Version.LUCENE_CURRENT)), false);

    Modified like this for lucene 4.1

  4. Vibhor Mathur says:

    Hey webzeest,

    I tried your syntax with Lucene 4.1, but still not able to run the code.
    Getting the error:
    Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/lucene/document/Fieldable
    at ir.assignments.four.SimpleSuggestionService.main(SimpleSuggestionService.java:42)

    I am making a search engine and for incorporating it with the GUI I am planning to add the auto-spell check and auto-suggestion feature to it.

    Did you make any other changes as well to run the code successfully?

    Thanks in advance.

Leave a Reply


4 × = twenty



Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.

Sign up for our Newsletter

20,709 insiders are already enjoying weekly updates and complimentary whitepapers! Join them now to gain exclusive access to the latest news in the Java world, as well as insights about Android, Scala, Groovy and other related technologies.

As an extra bonus, by joining you will get our brand new e-books, published by Java Code Geeks and their JCG partners for your reading pleasure! Enter your info and stay on top of things,

  • Fresh trends
  • Cases and examples
  • Research and insights
  • Two complimentary e-books