Ilias Tsagklis

About Ilias Tsagklis

Ilias Tsagklis is a senior software engineer working in the telecom domain. He is an applications developer in a wide variety of applications/services. Ilias is co-founder and Executive Editor at Java Code Geeks.

“Did you mean” feature with Apache Lucene Spell-Checker

Google’s “Did you mean” feature
After making an introduction to Lucene in a previous post, now it is time to take it up a notch and create a more sophisticated application. You are most surely familiar with Google’s “Did you mean” feature (other search engines support this too). Here is an example of that:

Lucene SpellChecker Subproject
This feature can be implemented in Java using Lucene project and this tutorial will show you how. The implementation will be based on one of Lucene’s sub-projects, named SpellChecker (see spell checker API). The inspiration for this post was a recent article that I read at InfoQ named “Implementing Google’s “Did you mean” Feature In Java”. The author shortly describes the edit distance algorithms that are used and then provides a simple code example. I also found a similar article named “Did You Mean: Lucene?”, but this was written back in 2005, so I guess it is now quite outdated. It is a good idea to first take a look on those and then come back for some action.

Finding the base dictionary
For the purposes of this tutorial, an English dictionary text file will be used. You can perform a simple search for that or you can directly download this zipped file. The file is very simple and just contains the words separated by lines. The extracted file is named “fulldictionary00.txt”.

Using the Lucene SpellChecker API
Let’s get started by creating a new Eclipse project, perhaps under the name “LuceneSuggestionsProject”, and import the necessary JAR files. Except for the main JAR lucene-core-3.0.1.jar, we will also need the lucene-spellchecker-3.0.1.jar. This can be found in the extracted tarball in the folder “lucene-3.0.1\contrib\spellchecker”. In case you don’t know how to find the distribution and get started with the project, please check my previous post (An Introduction to Apache Lucene for Full-Text Search). Create a new class named “SimpleSuggestionService” and make sure a main method is included. The source code for this class is the following:

package com.javacodegeeks.lucene.spellcheck;

import java.io.File;

import org.apache.lucene.search.spell.PlainTextDictionary;
import org.apache.lucene.search.spell.SpellChecker;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.FSDirectory;

public class SimpleSuggestionService {
    
    public static void main(String[] args) throws Exception {
        
        File dir = new File("c:/spellchecker/");
        
        Directory directory = FSDirectory.open(dir);
        
        SpellChecker spellChecker = new SpellChecker(directory);
        
        spellChecker.indexDictionary(
             new PlainTextDictionary(new File("c:/fulldictionary00.txt")));
        
        String wordForSuggestions = "hwllo";
        
        int suggestionsNumber = 5;

        String[] suggestions = spellChecker.
            suggestSimilar(wordForSuggestions, suggestionsNumber);

        if (suggestions!=null && suggestions.length>0) {
            for (String word : suggestions) {
                System.out.println("Did you mean:" + word);
            }
        }
        else {
            System.out.println("No suggestions found for word:"+wordForSuggestions);
        }
            
    }

}

Yes, it is as simple as that. If you run the application (using “hwllo” as the input word), the following output will occur:

Explaining the SpellChecker API
The code is very straightforward, but I will elaborate on some of the classes. To begin with, you need a Directory, which is a flat list of files. This class is abstract so we use FSDirectory as the actual implementation. FSDirectory stores the index files in the file system (for faster access RAMDirectory, which is the memory resident implementation, should be evaluated). The next step is to create a SpellChecker instance, which is the heart of our application. Note that the default StringDistance is the LevensteinDistance. If you desire to use an other algorithm (JaroWinklerDistance or NGramDistance), you can use the setStringDistance method. Next, we instantiate a PlainTextDictionary, which points to the text file we downloaded. The dictionary is indexed and then the SpellChecker object is used to retrieve the suggested words for a misspelled word.

I guess you are now ready to create the next big search engine. You can find the relevant Eclipse project here.

Enjoy!

Related Articles :

Do you want to know how to develop your skillset to become a Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you two of our best selling eBooks for FREE!

JPA Mini Book

Learn how to leverage the power of JPA in order to create robust and flexible Java applications. With this Mini Book, you will get introduced to JPA and smoothly transition to more advanced concepts.

JVM Troubleshooting Guide

The Java virtual machine is really the foundation of any Java EE platform. Learn how to master it with this advanced guide!

Given email address is already subscribed, thank you!
Oops. Something went wrong. Please try again later.
Please provide a valid email address.
Thank you, your sign-up request was successful! Please check your e-mail inbox.
Please complete the CAPTCHA.
Please fill in the required fields.

4 Responses to "“Did you mean” feature with Apache Lucene Spell-Checker"

  1. gaurang patel says:

    Thanks for to the point solution!! :)
    I have simply copied your program. However, I am facing one problem;
    In my text file which is being indexed has one entry ‘nike’. Now, when my query is ‘nike’, it shows NO RESULTS. But when my query is ‘mike’, it returns ‘nike’. Strange!!
    Do you have any clue about this behavior?

  2. Vinod says:

    spellChecker.indexDictionary(new PlainTextDictionary(new File(“c:/fulldictionary00.txt”)));

    for the above code, the warning comes like this……

    method indexDictionary in class org.apache.lucene.search.spell.SpellChecker cannot be applied to given types
    required: org.apache.lucene.search.spell.Dictionary,org.apache.lucene.index.IndexWriterConfig,boolean
    found: org.apache.lucene.search.spell.PlainTextDictionary

  3. webzeest says:

    spellChecker.indexDictionary(new PlainTextDictionary(new File(

    “Y:/fulldictionary00/fulldictionary00.txt”)),

    new IndexWriterConfig(Version.LUCENE_CURRENT,

    new StandardAnalyzer(Version.LUCENE_CURRENT)), false);

    Modified like this for lucene 4.1

  4. Vibhor Mathur says:

    Hey webzeest,

    I tried your syntax with Lucene 4.1, but still not able to run the code.
    Getting the error:
    Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/lucene/document/Fieldable
    at ir.assignments.four.SimpleSuggestionService.main(SimpleSuggestionService.java:42)

    I am making a search engine and for incorporating it with the GUI I am planning to add the auto-spell check and auto-suggestion feature to it.

    Did you make any other changes as well to run the code successfully?

    Thanks in advance.

Leave a Reply


six + = 13



Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.
Do you want to know how to develop your skillset and become a ...
Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you two of our best selling eBooks for FREE!

Get ready to Rock!
You can download the complementary eBooks using the links below:
Close