Repair Lucene index

Today I was browsing through a Lucene version 2.4 index with Luke 0.9.9 and I finished off my session by running optimize. This broke my web application, as it couldn’t read the database anymore. Glancing at the server logs, I quickly realized the data files had been converted into latest Lucene 2.9 format. Okay, no problems I thought, I might as well upgrade.
I stopped Tomcat, removed all lucene-*2.4*.jar files and copied the new lucene-2.9.jars into place. After a Tomcat restart everything went back to working as it should for a while.

Now… I must have made a grave mistake somewhere after that since all of a sudden my index was broken. Since Luke didn’t want to shut down properly and it did seem a bit sluggish on quit, I must have terminated the process without even making a mental notice of it and maybe Luke damaged the index?

The error I get in my Tomcat log is as follows:

Caused by: java.io.FileNotFoundException: /var/lib/databases/forum-search/_t8.fnm (No such file or directory)
	at java.io.RandomAccessFile.open(Native Method)
	at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
	at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.<init>(SimpleFSDirectory.java:78)
	at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleFSDirectory.java:108)
	at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:94)
	at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:70)
	at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:691)
	at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:68)
	at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
	at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
	at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:599)
	at org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:103)
	at org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:27)
	at org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:73)
	at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:704)
	at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:68)
	at org.apache.lucene.index.IndexReader.open(IndexReader.java:476)
	at org.apache.lucene.index.IndexReader.open(IndexReader.java:314)
	at com.popodeus.forum.search.ForumSearcher.reopenReader(ForumSearcher.java:218)
	<snip>

Just perfect… What have I done to deserve this treatment…

Luke won’t open it anymore either. What I am looking for is some low-level index manager that can remove any offending files or clear the reference to the missing segment.

So far I’ve been unable to find such a tool, maybe I need to make one myself?

I’ll try and write progress reports as I try coming up with a solution. I would hate to lose all of my 420k entries from the index. Rebuilding the database isn’t really an option since I don’t have access to the original data anymore, or re-indexing it all would take too much time.

To be continued…