Tonight Curiosity touched down on Mars and my feed was a blur of cheers and telemetry. It reminded me why I love search. You stare into the dark, you ask a question, and if your tools are good, you get a signal back. When you wire up Apache Lucene the right way, your app stops guessing and starts finding. I have lost count of the projects that stalled on search and then took off once Lucene came in. Let us make that happen for you too.
Lucene is a Java library for full text search. No server, just a sharp toolbox you embed. At its core it builds an inverted index. Instead of documents pointing to words, words point to documents. During index time your text streams through an Analyzer that breaks it into tokens, lowercases, removes stop words, stems if you ask for it, and writes postings with positions and frequencies. Each token lives inside a field like title, body, tags. That detail matters because you can search per field and boost some over others. The result is a fast dictionary that jumps straight to the right docs without scanning everything.
// Minimal indexing with Lucene 3.6
Directory dir = FSDirectory.open(new File("index"));
Analyzer analyzer = new StandardAnalyzer(Version.LUCENE_36);
IndexWriterConfig cfg = new IndexWriterConfig(Version.LUCENE_36, analyzer);
IndexWriter writer = new IndexWriter(dir, cfg);
Document doc = new Document();
doc.add(new Field("id", "1", Field.Store.YES, Field.Index.NOT_ANALYZED));
doc.add(new Field("title", "Hello Lucene", Field.Store.YES, Field.Index.ANALYZED));
doc.add(new Field("body", "Search that actually works.", Field.Store.YES, Field.Index.ANALYZED));
writer.addDocument(doc);
writer.commit();
writer.close();Good indexes start with good choices. Ask yourself what to store and what to index. Store means you can retrieve the original value at search time. Index means Lucene makes it searchable. Many folks try to store everything and pay for it in disk and I O, when all they really need is id and title for display. For identifiers and exact tags use NOT_ANALYZED so values are kept as a single token. For content use an Analyzer like StandardAnalyzer or a language specific one. Avoid adding a thousand fields per document unless you truly need them. Keep commits under control and let Lucene batch writes so segment merges are healthy and fast.
// Simple search with a QueryParser
Directory dir = FSDirectory.open(new File("index"));
Analyzer analyzer = new StandardAnalyzer(Version.LUCENE_36);
IndexReader reader = IndexReader.open(dir);
IndexSearcher searcher = new IndexSearcher(reader);
QueryParser qp = new QueryParser(Version.LUCENE_36, "body", analyzer);
Query q = qp.parse("hello lucene");
TopDocs top = searcher.search(q, 10);
for (ScoreDoc sd : top.scoreDocs) {
Document d = searcher.doc(sd.doc);
System.out.println(d.get("id") + " " + d.get("title") + " score=" + sd.score);
}
reader.close();QueryParser is a friendly start, but it will not stop you from shooting your foot. It treats text as a user query, so special characters need escaping or they turn into operators. Quotes mean phrases, which rely on positions written at index time. Boost fields with a caret to favor important text like title. For programmatic control build queries by hand. Use BooleanQuery to combine parts. For date and numeric ranges use NumericRangeQuery so you get fast trie encoded ranges. Lucene scores with TF IDF by default through DefaultSimilarity, which rewards rare terms and multiple hits in the same doc. You can set boosts per field or per document to bend the score toward business goals without throwing away relevance.
Now the fun part, tuning for your content. Pick the right Analyzer for the language you serve. English works fine with StandardAnalyzer, Snowball or Porter stemmers trim words to roots, and custom stop word lists remove noise terms your users never mean. For autocompletion stick to prefix queries or a dedicated field built from edge n grams. When you need fresh results without full reopen, use near real time by asking the writer for a reader. That gives you low latency visibility of new docs. For sorting on fields like date or price, index them as numeric or as a keyword field and use Sort with a proper type. Watch heap use and open file handles when your index grows, and prefer a warmup search after deploy so caches are hot before traffic hits.
There is more in the Lucene family. If you want a server with HTTP and schema managed configs, Apache Solr sits on top of Lucene and is battle tested. If you like a JSON API that speaks cluster out of the box, ElasticSearch is getting a lot of attention too. Both share the same core ideas because the core is Lucene. That is why the lesson sticks. Learn how fields, analyzers, queries, and scoring fit together and you can move between libraries and keep your results sharp. The apps we ship today expect smart search the way users expect auto complete in their browser. Getting it right is a feature, not a checkbox.
Quick takeaway
Apache Lucene gives you fast full text search in a small Java jar. Index text with the right Analyzer, choose what to store, keep ids not analyzed, and use QueryParser for simple cases or build queries by hand when you need control. Shape relevance with boosts and lean on TF IDF. Add near real time readers for fresh results, sort with typed fields, and keep an eye on memory and segments. Whether you stay embedded or jump to Solr or ElasticSearch later, the foundations are the same. Ship search that finds what people mean, not just what they typed.