The new LIRE web demo is based on Apache Solr and features and index of the MIRFLICKR data set. The new architecture allows for extremely fast retrieval. Moreover, there’s a new walk through video with some short peeks behind the screen. The source of the plugin will be released in the near future.
I’ve just uploaded LIRE 0.9.4 beta to the Google Code downloads page. This is an intermediate release that reflects several changes within the SVN trunk. Basically I put it online as there are many, many bugs solved in this one and it’s performing much, much faster than the 0.9.3 release. If you want to get the latest version I’d recommend to stick to the SVN. However, currently I’m changing a lot of feature serialization methods, so there’s no guarantee that an index created with 0.9.4 beta will work out with any newer version. Note also that the release does not work with older indexes 😉
Major changes include, but are not limited to:
- New features: PHOG, local binary patterns and binary patterns pyramid
- Parallel indexing: a producer-consumer based indexing application that makes heavy use of available CPU cores. On a current Intel Core i7 or considerably large Intel Xeon system it is able to reduce extraction to a marginal overhead to disk I/O.
- Intermediate byte based feature data files: a new way to extract features in a distributed way
- In-memory cached ImageSearcher: as long as there is enough memory all linear searching is done in memory without much disk I/O (cp. class GenericFastImageSearcher and set caching to true)
- Approximate indexing based on hashing: tests with 1.5 million led to search time < 300ms (cp. GenericDocumentBuilder with hashing set to true and BitSamplingImageSearcher)
- Footprint of many global descriptors has been significantly reduced. Examples: EdgeHistogram 40 bytes, ColorLayout 504 bytes, FCTH 96 bytes, …
- New unit test for benchmarking features on the UCID data set.
All changes can be found in the CHANGES.txt file.
While I know that the performance did not skyrocket with Lucene 4.0 I finally came around to find out why. Unfortunately the field compression technique applied in Lucene 4.x compresses each and every stored field … and decompresses it upon access. This makes up for a nice overhead when reading the index in a linear way, which is excactly one of the main methods of LIRE.
The image shows a screen shot of the CPU sampler in VisualVM. 58.7% of the CPU time go to the LZ4 decompression routine. That’s quite a lot and makes a huge difference for search. If anyone has a workaround of sort, I’d be happy
Update (2013-07-03): With the great help of the people from the lucene-user list I found at least a speed-up. In the current SVN version, there is a nove LireCustomCodec for stored fields, which speeds up decompression a lot. Moreover there is now an in-memory caching approach implemented in the GeneriecFastImageSearcher class, which is turned off by default, but speeds up search time (as a trade off for memory and init time) by holding image features in-memory. It has been tested with up to 1.5M images.
The current LireDemo 0.9.4 beta release features a new indexing routine, which is much faster than the old one. It’s based on the producer-consumer principle and makes — hopefully — optimal use of I/O and up to 8 cores of a system. Moreover, the new PHOG feature implementation is included and you can give it a try. Furthermore JCD, FCTH and CEDD got a more compact representation of their descriptors and use much less storage space now. Several small changes include parameter tuning on several descriptors and so on. All the changes have been documented in the CHANGES.txt file in the SVN.
In the current SVN version three global features have been re-visited in terms of serialization. This was necessary as the index of the web demo with 300k images already exceed 1.5 GB.
|EdgeHistogram||320 bytes||40 bytes|
|JCD||1344 bytes||84 bytes|
|ColorLayout||14336 bytes||504 bytes|
This significant reduction in space leads to (i) smaller indexes, (ii) reduced I/O time, and (iii) therefore, to faster search.
How was this done? Basically it’s clever organization of bytes. In the case of JCD the histogram has 168 entries, each in [0,127], so basically half a byte.Therefore, you can stuff 2 of these values into one byte, but you have to take care of the fact, that Java only supports bit-wise operations on ints and bytes are signed. So the trick is to create an integer in [0, 2^8-1] and then subtract 128 to get it into byte range. The inverse is done for reading. The rest is common bit shifting.
The LIRE web demo now includes an RGB color histogram as well as the MPEG-7 edge histogram implementation. The color histogram works well for instance for line art, such as this query.The edge histogram works fine for clear, gloabl edge distributions like queries such as this one. However, it’s performing different from PHOG. An example for the difference is this PHOG query compared to the according edge histogram query. The image below shows both queries.
A new web based LIRE demo is online. Within this demo you are able to search in an index of 300.000 images from the MIRFLICKR data set. Currently online queries from within the index are allowed, so no custom query images can be uploaded. The backend is plain LIRE, so there’s no search server and alike, and it’s the current SVN version. Search is done based on hashing, so the results are approximate, but they are immediately there. Also it’s just a selection of global features, but it’s enough to get the idea. The image below shows the result of two example searches.
The Kindle version of our book “Visual Information Retrieval using Java and LIRE” is now available on amazon.com (as well as Amazon in Germany, France, Italy, and Canada). It’s a good deal with 10$ (or something like 7.90 €) for the book, which is far cheaper than the PDF version and the paperback.
The realization that setting up the project is not too trivial led to the video howto. It’s available on YouTube and shows all steps from (an already started) fresh IntelliJ IDEA to running a Junit test for LIRE. Make sure you watch the video in 1080p / full HD to be able to read all the text.
- Video Tutorial (YouTube)
With the implementation of the PHOG descriptor I came around the situation that no well-performing Canny Edge Detector in pure Java was available. “Pure” in my case means, that it just takes a Java BufferedImage instance and computes the edges. Therefore, I had to implement my own
As a result there is now a “simple implementation” available as part of LIRE. It takes a BufferedImage and returns another BufferedImage, which contains all the edges as black pixels, while the non-edges are white. Thresholds can be changed and the blurring filter using for preprocessing can be changed in code. Usage is dead simple:
BufferedImage in = ImageIO.read(new File("testdata/wang-1000/128.jpg")); CannyEdgeDetector ced = new CannyEdgeDetector(in, 40, 80); ImageIO.write(ced.filter(), "png", new File("out.png"));
The result is the picture below: