In the process of updating to Lucene version 6.3.0 we switched the main build system to Gradle. The source code is now better organized, following the structure of Maven with all the source divided into main and test, and the resources being separated. It allows for the use of Maven repositories, automates most tasks out-of-the-box and is extended easily using the Groovy programming language.
Gradle also comes with a wrapper, so as long Java is installed, the build process is fully automated and does not need additional software installed.
Moreover, build.gradle files can be easily used to import projects into JetBrains IDEA. LireDemo and SimpleApplication are sub projects with their own build.gradle files. Checkout the new structure at https://github.com/dermotte/lire.
We all know that setting up a server application is a huge pain in the ass, but there’s hope: Docker is an open source project for packaging those things and making them runnable within containers. Only thing one needs to do is to create an image, which then can be used to create a container.
LireSolr now provides such a Docker image on Docker Hub. Basically you can install Docker, then use the docker command in the shell (Linux) or the PowerShell (MS Windows) to automatically download and run a container based on the image with a single command:
$> docker run -p 8983:8983 dermotte/liresolr:latest
What you need to do then is to index some images and take a look at the sample application. A detailed step-by-step howto is provided in the documentation.
Apache Solr is a search server application widely in use around the world and LireSolr is the integration project of Lire into this search server. As a last work item of the year, we have updated LireSolr to work with Solr 6.3.0. This also involves the move to Lucene 6.3.0 for LIRE and — for both — Java 8 as minimum SDK version. LireSolr also found a new home on Github: https://github.com/dermotte/liresolr.
There’s detailed documentation on how to set up Solr, how to add a core, how to configure it for the use with LireSolr and there are even indexing tools to get your photos inside Solr. Give it a try at Github!
In the current SVN version three global features have been re-visited in terms of serialization. This was necessary as the index of the web demo with 300k images already exceed 1.5 GB.
This significant reduction in space leads to (i) smaller indexes, (ii) reduced I/O time, and (iii) therefore, to faster search.
How was this done? Basically it’s clever organization of bytes. In the case of JCD the histogram has 168 entries, each in [0,127], so basically half a byte.Therefore, you can stuff 2 of these values into one byte, but you have to take care of the fact, that Java only supports bit-wise operations on ints and bytes are signed. So the trick is to create an integer in [0, 2^8-1] and then subtract 128 to get it into byte range. The inverse is done for reading. The rest is common bit shifting.
The code can be seen either in the JCD.java file in the SVN, or in the snippet at pastebin.com for your convenience.
The realization that setting up the project is not too trivial led to the video howto. It’s available on YouTube and shows all steps from (an already started) fresh IntelliJ IDEA to running a Junit test for LIRE. Make sure you watch the video in 1080p / full HD to be able to read all the text.
With the implementation of the PHOG descriptor I came around the situation that no well-performing Canny Edge Detector in pure Java was available. “Pure” in my case means, that it just takes a Java BufferedImage instance and computes the edges. Therefore, I had to implement my own 🙂
As a result there is now a “simple implementation” available as part of LIRE. It takes a BufferedImage and returns another BufferedImage, which contains all the edges as black pixels, while the non-edges are white. Thresholds can be changed and the blurring filter using for preprocessing can be changed in code. Usage is dead simple:
BufferedImage in = ImageIO.read(new File("testdata/wang-1000/128.jpg"));
CannyEdgeDetector ced = new CannyEdgeDetector(in, 40, 80);
ImageIO.write(ced.filter(), "png", new File("out.png"));
People lately asked whether LIRE can do more than linear search and I always answered: Yes, it should … but you know I never tried. But: Finally I came around to index the MIR-FLICKR data set and some of my Flickr-crawled photos and ended up with an index of 1,443,613 images. I used CEDD as main feature and a hashing algorithm to put multiple hashes per images into Lucene — to be interpreted as words. By tuning similarity, employing a Boolean query, and adding a re-rank step I ended up with a pretty decent approximate retrieval scheme, which is much faster and does not loose too many images on the way, which means the method has an acceptable recall. The image below shows the numbers along with a sample query. Linear search took more than a minute, while the hashing based approach did (nearly) the same thing in less than a second. Note that this is just a sequential, straight forward approach, so no optimization has been done to the performance. Also the hashing approach has not yet been investigated in detail, i.e. there are some parameters that still need some tuning … but let’s say it’s a step into the right direction.
I just uploaded Lire 0.9.3 to the all new Google Code page. This is the first version with full support for Lucene 4.0. Run time and memory performance are comparable to the version using Lucene 3.6. I’ve made several improvements in terms of speed and memory consumption along the way, mostly within the CEDD feature. Also I’ve added two new features:
JointHistogram - a 64 bit RGB color histogram joined with pixel rank in the 8-neighborhood, normalized with max-norm, quantized to [0,127], and JSD for a distance function
Opponent Histogram - a 64 bit histogram utilizing the opponent color space, normalized with max-norm, quantized to [0,127], and JSD for a distance function
Both features are fast in extraction (the second one naturally being faster as it does not investigate the neighborhood) and yield nice, visually very similar results in search. See also the image below showing 4 queries, each with the new features. The first one of a pair is always based on JointHistogram, the second is based on the OpponentHistogram (click ko see full size).
I also changed the Histogram interface to double as the double type is so much faster than float in 64 bit Oracle Java 7 VM. Major bug fix was in the JSD dissimilarity function. So many histograms now turned to use JSD instead of L1, depending on whether they performed better in the SIMPLIcity data set (see TestWang.java in the sources).
Final addition is the Lire-SimpleApplication, which provides two classes for indexing and search with CEDD, ready to compile with all libraries and an Ant build file. This may — hopefully — help those that still seek Java enlightenment 😀
Finally this just leaves to say to all of you: Merry Christmas and a Happy New Year!
In the course of finishing the book, I reviewed several aspects of the LIRE code and came across some bugs, including one with the Jensen-Shannon divergence. This dissimilarity measure has never been used actively in any features as it didn’t work out in retrieval evaluation the way it was meant to. After two hours staring at the code the realization finally came. In Java the short if statement, “x ? y : z” is overruled by almost any operator including ‘+’. Hence,
System.out.print(true ? 1: 0 + 1) prints '1',
System.out.print((true ? 1: 0) + 1) prints '2'
With this problem identified I was finally able to fix the implementation of the Jensen-Shannon divergence implementation and came to new retrieval evaluation results on the SIMPLIcity data set:
Color Histogram - JSD
Joint Histogram - JSD
Note that the color histogram in the first row now performs similarly to the “good” descriptors in terms of precision at ten and error rate. Also note that a new feature creeped in: Joint Histogram. This is a histogram combining pixel rank and RGB-64 color.
All the new stuff can be found in SVN and in the nightly builds (starting tomorrow 🙂
I just uploaded version 0.9.2 of Lire and LireDemo to Google Code. Yes, Google Code! I also migrated (more or less in a under cover action some month ago) the SVN trunk to Google Code and will move on with development there. Main reasons were that ads were getting more and more aggressive over at sf.net and the interface of a Google Code project is so much cleaner and easier to handle from a project manager point of view.
Lire 0.9.2 fixes two bugs in KMeans and GenericImageSearcher. Both were critical. The KMeans fix allows now for the use of the bag of visual words approach. The GenericImageSearcher fix makes search much faster.