Caliph & Emir are MPEG-7 based Java prototypes for digital photo and image annotation and retrieval supporting graph like annotation for semantic metadata and content based image retrieval using MPEG-7 descriptors.
About Citation of Caliph & Emir
We kindly ask you to refer the following paper in any publication mentioning Caliph & Emir:
Lux, Mathias. “Caliph & Emir: MPEG-7 photo annotation and retrieval.” Proceedings of the 17th ACM international conference on Multimedia. ACM, 2009.
- Download Paper
- Download BibTeX
Features of Caliph
- Extraction of IPTC and EXIF metadata and conversion to MPEG-7
- Extraction of following content based image retrieval descriptors:
- ColorLayout (measures color distribution in an image)
- ScalableColor (basically a color histogram)
- EdgeHistogram (measures edginess)
- DominantColor (measure dominant colors)
- Annotation of images with
- Free text metadata
- Quality rating
- Structured text metadata
- Meta^2 data (metadata about metadata)
- Semantic Annotations
- Based on a editable catalogue for reusable semantic objects (like specific persons, places, …)
- Fully visual & graphical user interface
- Automatic placement and layout of visual descriptions
- Export of semantic descriptions to Structured Text Descriptions
- Annotation of shapes
- Load and save annotations as MPEG-7 XML file
Features of Emir
- Retrieval of annotations using the file system
- with XPath queries
- with keyword queries
- with quality constraints
- with example images for content based image retrieval
- Retrieval of Annotations using a Lucene index
- with inbuilt index creation
- using different fields for structured retrieval
- Retrieval of semantic graphs based on a Lucene powered index
- supporting wildcards for relations and nodes
- based on query expansion
- graphical query editor
- 2D data repository visualization based on FastMap and FDP algorithms
- for image content descriptors ColorLayout, ScalableColor and EdgeHistogram
- for graphs based on the minimum common sub graph metric
Caliph
Besides reading existing information, which already exists in digital photographs, and transformation of this information to MPEG-7, Caliph supports the creation of new metadata. Semantic information about the image is presented as directed graph, where the nodes reflect semantic objects, locations, agents, states, times or concepts and the edges define the relations between these semantic entities. To enhance retrieval efficiency content-based metadata is extracted and new instances of the image for faster visualization, like thumbnails, are created. The MPEG-7 description consists of the following parts: metadata description, creation information, media information, textual annotation, semantics, visual descriptors.
Emir
The experimental metadata based image retrieval tool Emir supports retrieval in file system based photo repositories created with Caliph. Different types of retrieval mechanisms are supported:
- Content based image retrieval using the MPEG-7 descriptors ColorLayout, EdgeHistogram and ScalableColor.
- Keyword based searching using the Lucene search engine.
- Graph based retrieval supporting wildcards for semantic relations and semantic objects.
- 2D data repository and result set visualization based on FastMap & FDP algorithms.
Furthermore clustering abilities are built in but not acitvated yet.
Documentation
Packaged with the installer version there is a Annotation and Search HowTo can also find here.
For questions regarding the usage or for submitting bugs please use the sourceforge project forum of Caliph & Emir. Your help request will normally be answered within one workdays as long as I (Mathias Lux) am online.
Additionally there are two Flash demos, which show the basic usage of Caliph & Emir, available:
Screenshots Caliph
This screenshot shows the startup screen of Caliph. Here you can annotate the photos with free text and you can rate the quality. Pre existing metadata like EXIF or IPTC tags inside images is loaded and converted to MPEG-7.
Here the second panel, the semantic description panel, is shown. It offers a tool for visual creation of MPEG-7 based semantic descriptions using a drawn directed graph.
On the feature extraction panel the low-level descriptors that are automatically extracted are shown. Extracted MPEG-7 descriptors are ColorLayout, ScalableColor and EdgeHistogram.
The shape panel allows the creation of simple shapes on the images. All shapes are stored within the MPEG-7 document.
Screenshots Emir
Lucene based keyword search of Emir, search in different fields is supported.
Searching for similar semantic graphs based on query expansion by drawing sample graphs with keywords
Anonymous (wildcard) nodes and relations are supported, a node can be selected from the nodes found in the index with a context menu after defining it with a query string.
2D visualization of the data repository using the FastMap algorithm for multidimensional scaling based on color information extracted from the images
2D visualization of the data repository using the FastMap algorithm and force directed placement for multidimensional scaling based on the semantic descriptions based on the maximum common subgraph metric
Search for similar images using an example image
Results of above search for the term “Mathias Lux”
Publications
[1] Mathias Lux, Jutta Becker and Harald Krottmaier, “Caliph & Emir: Semantic Annotation and Retrieval in Personal Digital Photo Libraries”, Proceedings of CAiSE03 Forum at 15th Conference on Advanced Information Systems Engineering, p. 85-89, June 16th-20th 2003, Velden, Austria.
[Download paper]
[2] Mathias Lux, Werner Klieber and Michael Granitzer, “Caliph & Emir: Semantics in Multimedia Retrieval and Annotation“, 19th International CODATA Conference 2004: The Information Society: New Horizons for Science, November 7th-10th 2004, Berlin, Germany.
[Download paper] [Download Slides]
[3] Mathias Lux and Michael Granitzer , “Retrieval of MPEG-7 based Semantic Descriptions“, BTW-Workshop “WebDB Meets IR” in context of the “11. GI-Fachtagung für Datenbanksysteme in Business, Technologie und Web”, March 1st 2005, University of Karlsruhe, Germany.
[Download paper ] [Download slides]
[4] Mathias Lux and Michael Granitzer, “A Fast and Simple Path Index Based Retrieval Approach for Graph Based Semantic Descriptions“, in Proceedings Workshop on Text-Based Information Retrieval TIR 05, in context of the 28th German Conference on Artificial Intelligence, Koblenz, Germany, September 2005.
[Download paper]
[5] Mathias Lux, “Retrieval und Annotation von digitalen Photos mit MPEG-7”, MPEG-7 Workshop, March 10th-11th 2005, University of Klagenfurt, Austria.
[Download slides in german]
History
The development started by Mathias Lux in a lecture of the IICM, an institute of the University of Technology Graz, Austria. Further development was done by adapting it to a project of the Know-Center, a competence center for knowledge based applications and systems in Graz, Austria, by Mathias Lux and Werner Klieber. Quite some time ago it was decided to license the sources, which were partly under copyright of the Know-Center, under the GPL. All private copyrighted sources from the above mentioned lecture were also opend. After registering the project at sourceforge.net the project administration and development was carried on by Mathias Lux. Later additions include code from the VizIR project, which was lead by Horst Eidenberger. Furthermore, Werner Klieber added parts of the results from his master thesis to Caliph to allow the definition of shapes in images. Margit Lang ( MrsMojo) added a search routine for Emir combining different visual descriptors.
Thanks to:
Jutta Becker, Harald Krottmaier, Michael Granitzer, Wolfgang Kienreich, Helmut Neuschmied, Werner Klieber, Vedran Sabol, Christian Gütl, the IICM and the TU Graz in general for supporting me by being my scientific environment. Further thanks go to Harald Kosch from the University of Klagenfurt for all the MPEG-7 support and to Horst Eidenberger from Vienna Technical University and his VizIR project team, for contributing edge histogram and other descriptors.
In general I want to thank the Know-Center, in special Klaus Tochtermann, my former boss, who agreed in opening Caliph & Emir to the public licensed under GPL.