Core Java

“Did you mean” feature with Apache Lucene Spell-Checker

Google’s “Did you mean” feature
After making an introduction to Lucene in a previous post, now it is time to take it up a notch and create a more sophisticated application. You are most surely familiar with Google’s “Did you mean” feature (other search engines support this too). Here is an example of that:

Lucene SpellChecker Subproject
This feature can be implemented in Java using Lucene project and this tutorial will show you how. The implementation will be based on one of Lucene’s sub-projects, named SpellChecker (see spell checker API). The inspiration for this post was a recent article that I read at InfoQ named “Implementing Google’s “Did you mean” Feature In Java”. The author shortly describes the edit distance algorithms that are used and then provides a simple code example. I also found a similar article named “Did You Mean: Lucene?”, but this was written back in 2005, so I guess it is now quite outdated. It is a good idea to first take a look on those and then come back for some action.

Finding the base dictionary
For the purposes of this tutorial, an English dictionary text file will be used. You can perform a simple search for that or you can directly download this zipped file. The file is very simple and just contains the words separated by lines. The extracted file is named “fulldictionary00.txt”.

Using the Lucene SpellChecker API
Let’s get started by creating a new Eclipse project, perhaps under the name “LuceneSuggestionsProject”, and import the necessary JAR files. Except for the main JAR lucene-core-3.0.1.jar, we will also need the lucene-spellchecker-3.0.1.jar. This can be found in the extracted tarball in the folder “lucene-3.0.1\contrib\spellchecker”. In case you don’t know how to find the distribution and get started with the project, please check my previous post (An Introduction to Apache Lucene for Full-Text Search). Create a new class named “SimpleSuggestionService” and make sure a main method is included. The source code for this class is the following:

package com.javacodegeeks.lucene.spellcheck;

import java.io.File;

import org.apache.lucene.search.spell.PlainTextDictionary;
import org.apache.lucene.search.spell.SpellChecker;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.FSDirectory;

public class SimpleSuggestionService {
    
    public static void main(String[] args) throws Exception {
        
        File dir = new File("c:/spellchecker/");
        
        Directory directory = FSDirectory.open(dir);
        
        SpellChecker spellChecker = new SpellChecker(directory);
        
        spellChecker.indexDictionary(
             new PlainTextDictionary(new File("c:/fulldictionary00.txt")));
        
        String wordForSuggestions = "hwllo";
        
        int suggestionsNumber = 5;

        String[] suggestions = spellChecker.
            suggestSimilar(wordForSuggestions, suggestionsNumber);

        if (suggestions!=null && suggestions.length>0) {
            for (String word : suggestions) {
                System.out.println("Did you mean:" + word);
            }
        }
        else {
            System.out.println("No suggestions found for word:"+wordForSuggestions);
        }
            
    }

}

Yes, it is as simple as that. If you run the application (using “hwllo” as the input word), the following output will occur:

Explaining the SpellChecker API
The code is very straightforward, but I will elaborate on some of the classes. To begin with, you need a Directory, which is a flat list of files. This class is abstract so we use FSDirectory as the actual implementation. FSDirectory stores the index files in the file system (for faster access RAMDirectory, which is the memory resident implementation, should be evaluated). The next step is to create a SpellChecker instance, which is the heart of our application. Note that the default StringDistance is the LevensteinDistance. If you desire to use an other algorithm (JaroWinklerDistance or NGramDistance), you can use the setStringDistance method. Next, we instantiate a PlainTextDictionary, which points to the text file we downloaded. The dictionary is indexed and then the SpellChecker object is used to retrieve the suggested words for a misspelled word.

I guess you are now ready to create the next big search engine. You can find the relevant Eclipse project here.

Enjoy!

Related Articles :

Ilias Tsagklis

Ilias is a software developer turned online entrepreneur. He is co-founder and Executive Editor at Java Code Geeks.
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

5 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
gaurang patel
gaurang patel
12 years ago

Thanks for to the point solution!! :)
I have simply copied your program. However, I am facing one problem;
In my text file which is being indexed has one entry ‘nike’. Now, when my query is ‘nike’, it shows NO RESULTS. But when my query is ‘mike’, it returns ‘nike’. Strange!!
Do you have any clue about this behavior?

Vinod
Vinod
11 years ago

spellChecker.indexDictionary(new PlainTextDictionary(new File(“c:/fulldictionary00.txt”)));

for the above code, the warning comes like this……

method indexDictionary in class org.apache.lucene.search.spell.SpellChecker cannot be applied to given types
required: org.apache.lucene.search.spell.Dictionary,org.apache.lucene.index.IndexWriterConfig,boolean
found: org.apache.lucene.search.spell.PlainTextDictionary

webzeest
11 years ago

spellChecker.indexDictionary(new PlainTextDictionary(new File(

“Y:/fulldictionary00/fulldictionary00.txt”)),

new IndexWriterConfig(Version.LUCENE_CURRENT,

new StandardAnalyzer(Version.LUCENE_CURRENT)), false);

Modified like this for lucene 4.1

Vibhor Mathur
Vibhor Mathur
11 years ago

Hey webzeest,

I tried your syntax with Lucene 4.1, but still not able to run the code.
Getting the error:
Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/lucene/document/Fieldable
at ir.assignments.four.SimpleSuggestionService.main(SimpleSuggestionService.java:42)

I am making a search engine and for incorporating it with the GUI I am planning to add the auto-spell check and auto-suggestion feature to it.

Did you make any other changes as well to run the code successfully?

Thanks in advance.

magan
magan
7 years ago
Reply to  Vibhor Mathur

Hi webzeest,

I tried your syntax with Lucene 6.4.0, but still not able to run the code.
Getting the error:
Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/lucene/document/Fieldable
at ir.assignments.four.SimpleSuggestionService.main(SimpleSuggestionService.java:18)

Error while creating obejct of SpellChecker
I am making a search engine and for incorporating it with the GUI I am planning to add the auto-spell check and auto-suggestion feature to it.

Did you make any other changes as well to run the code successfully?

Thanks in advance.

Back to top button