Category Archives: Software

ICMR 2017 – Call for Open Source Software Papers

ICMR 2017 is calling for papers presenting significant and innovative open source contributions of researchers and software developers to advance the field of multimedia retrieval by providing the community with implementations of middleware, frameworks, toolkits, libraries, players, authoring tools, and other multimedia software. We strongly believe that making available open source resources significantly advances the field by providing a common set of tools for (i) building upon previous work, (ii) improving multimedia research prototypes and (iii) allowing others to replicate research results more easily.

Each open source software paper should not be longer than 4 pages.

Important Dates

Paper Submission: January 27, 2017
Notification of Acceptance: March 29, 2017
Camera-Ready Papers Due: April 26, 2017

More information …

LIRE 1.0b2 released

LireLogo_small

Today, the first official beta version of LIRE 1.0 has been released. After loads of internal tests we decided to pin it down to quasi-stable. There are loads of new features compared to 0.9.5 including metric spaces indexing, DocValues based searching, the SIMPLE approach for localizing global descriptors, new global ones, like CENTRIST, a lot of performance tweaks, and so on.

For those of you using the nightly build or the repository version not much changed, everyone else might check out the new APIs and possibilities staring from the SimpleApplication collection moving over to the LIRE documentation.

You will find the pre-compiled binaries on the download page, hosted by ITEC / Klagenfurt University.

 

 

LIRE 0.9.5 released

It’s been more than a year since we made a release, so there have been lots of changes, fixes and new features. Most important of all is the integration of the SIMPLE descriptor, a local feature that works extremely well for content based image retrieval. This has been done with a lot of help from Nektarios Anagnostopoulos and Savvas Chatzichristofis!

Besides that we switched to OpenCV for SURF and SIFT extraction, added numerous bug fixes, updated Lucene to 4.10.2 and much more. Best you give it a try.

>> Head over to the downloads

 

LIRE 0.9.4 beta released

I’ve just uploaded LIRE 0.9.4 beta to the Google Code downloads page. This is an intermediate release that reflects several changes within the SVN trunk. Basically I put it online as there are many, many bugs solved in this one and it’s performing much, much faster than the 0.9.3 release. If you want to get the latest version I’d recommend to stick to the SVN. However, currently I’m changing a lot of feature serialization methods, so there’s no guarantee that an index created with 0.9.4 beta will work out with any newer version. Note also that the release does not work with older indexes 😉

Major changes include, but are not limited to:

  • New features: PHOG, local binary patterns and binary patterns pyramid
  • Parallel indexing: a producer-consumer based indexing application that makes heavy use of available CPU cores. On a current Intel Core i7 or considerably large Intel Xeon system it is able to reduce extraction to a marginal overhead to disk I/O.
  • Intermediate byte[] based feature data files: a new way to extract features in a distributed way
  • In-memory cached ImageSearcher: as long as there is enough memory all linear searching is done in memory without much disk I/O (cp. class GenericFastImageSearcher and set caching to true)
  • Approximate indexing based on hashing: tests with 1.5 million led to search time < 300ms (cp. GenericDocumentBuilder with hashing set to true and BitSamplingImageSearcher)
  • Footprint of many global descriptors has been significantly reduced. Examples: EdgeHistogram 40 bytes, ColorLayout 504 bytes, FCTH 96 bytes, …
  • New unit test for benchmarking features on the UCID data set.

All changes can be found in the CHANGES.txt file.

LireDemo 0.9.4 beta released

The current LireDemo 0.9.4 beta release features a new indexing routine, which is much faster than the old one. It’s based on the producer-consumer principle and makes — hopefully — optimal use of I/O and up to 8 cores of a system. Moreover, the new PHOG feature implementation is included and you can give it a try. Furthermore JCD, FCTH and CEDD got a more compact representation of their descriptors and use much less storage space now. Several small changes include parameter tuning on several descriptors and so on. All the changes have been documented in the CHANGES.txt file in the SVN.

Links

Serialization of LIRE global features updated

In the current SVN version three global features have been re-visited in terms of serialization. This was necessary as the index of the web demo with 300k images already exceed 1.5 GB.

Feature prior now
EdgeHistogram 320 bytes 40 bytes
JCD 1344 bytes 84 bytes
ColorLayout 14336 bytes 504 bytes

This significant reduction in space leads to (i) smaller indexes, (ii) reduced I/O time, and (iii) therefore, to faster search.

How was this done? Basically it’s clever organization of bytes. In the case of JCD the histogram has 168 entries, each in [0,127], so basically half a byte.Therefore, you can stuff 2 of these values into one byte, but you have to take care of the fact, that Java only supports bit-wise operations on ints and bytes are signed. So the trick is to create an integer in [0, 2^8-1] and then subtract 128 to get it into byte range. The inverse is done for reading. The rest is common bit shifting.

The code can be seen either in the JCD.java file in the SVN, or in the snippet at pastebin.com for your convenience.

Web based LIRE demo now online

A new web based LIRE demo is online. Within this demo you are able to search in an index of 300.000 images from the MIRFLICKR data set. Currently online queries from within the index are allowed, so no custom query images can be uploaded. The backend is plain LIRE, so there’s no search server and alike, and it’s the current SVN version. Search is done based on hashing, so the results are approximate, but they are immediately there. Also it’s just a selection of global features, but it’s enough to get the idea. The image below shows the result of two example searches.

LIRE web demo screen shots

 

Links

Canny Edge Detector in Java

With the implementation of the PHOG descriptor I came around the situation that no well-performing Canny Edge Detector in pure Java was available. “Pure” in my case means, that it just takes a Java BufferedImage instance and computes the edges. Therefore, I had to implement my own 🙂

As a result there is now a “simple implementation” available as part of LIRE. It takes a BufferedImage and returns another BufferedImage, which contains all the edges as black pixels, while the non-edges are white. Thresholds can be changed and the blurring filter using for preprocessing can be changed in code. Usage is dead simple:

BufferedImage in = ImageIO.read(new File("testdata/wang-1000/128.jpg")); 
CannyEdgeDetector ced = new CannyEdgeDetector(in, 40, 80); 
ImageIO.write(ced.filter(), "png", new File("out.png"));

The result is the picture below:

cannyedgeLinks

 

Large image data sets with LIRE – some new numbers

People lately asked whether LIRE can do more than linear search and I always answered: Yes, it should … but you know I never tried. But: Finally I came around to index the MIR-FLICKR data set and some of my Flickr-crawled photos and ended up with an index of 1,443,613 images. I used CEDD as main feature and a hashing algorithm to put multiple hashes per images into Lucene — to be interpreted as words. By tuning similarity, employing a Boolean query, and adding a re-rank step I ended up with a pretty decent approximate retrieval scheme, which is much faster and does not loose too many images on the way, which means the method has an acceptable recall. The image below shows the numbers along with a sample query. Linear search took more than a minute, while the hashing based approach did (nearly) the same thing in less than a second. Note that this is just a sequential, straight forward approach, so no optimization has been done to the performance. Also the hashing approach has not yet been investigated in detail, i.e. there are some parameters that still need some tuning … but let’s say it’s a step into the right direction.

Results-CEDD-Hashing

Lire 0.9.3 released

I just uploaded Lire 0.9.3 to the all new Google Code page. This is the first version with full support for Lucene 4.0. Run time and memory performance are comparable to the version using Lucene 3.6. I’ve made several improvements in terms of speed and memory consumption along the way, mostly within the CEDD feature. Also I’ve added two new features:

  • JointHistogram – a 64 bit RGB color histogram joined with pixel rank in the 8-neighborhood, normalized with max-norm, quantized to [0,127], and JSD for a distance function
  • Opponent Histogram – a 64 bit histogram utilizing the opponent color space, normalized with max-norm, quantized to [0,127], and JSD for a distance function

Both features are fast in extraction (the second one naturally being faster as it does not investigate the neighborhood) and yield nice, visually very similar results in search. See also the image below showing 4 queries, each with the new features. The first one of a pair is always based on JointHistogram, the second is based on the OpponentHistogram (click ko see full size).

Samples-JointHistogram

 

I also changed the Histogram interface to double[] as the double type is so much faster than float in 64 bit Oracle Java 7 VM. Major bug fix was in the JSD dissimilarity function. So many histograms now turned to use JSD instead of L1, depending on whether they performed better in the SIMPLIcity data set (see TestWang.java in the sources).

Final addition is the Lire-SimpleApplication, which provides two classes for indexing and search with CEDD, ready to compile with all libraries and an Ant build file. This may — hopefully — help those that still seek Java enlightenment 😀

Finally this just leaves to say to all of you: Merry Christmas and a Happy New Year!