Hybrid operations for content-based Vietnamese agricultural multimedia information retrieval

Abstract

Content-based multimedia information retrieval is never a trivial task even with state-of-the-art approaches. Its mandatory challenge, called “semantic gap,” requires much more understanding of the way human perceive things (i.e., visual and auditory information). Computer scientists have spent thousands of hours seeking optimal solutions, only ended up falling in the bound of this gap for both visual and spoken contexts. While an over-the-gap approach is unreachable, we insist on assembling current viable techniques from both contexts, aligned with a domain concept base (i.e., an ontology), to construct an info service for the retrieval of agricultural multimedia information. The development process spans over three packages: (1) building a Vietnamese agricultural thesaurus; (2) crafting a visual-auditory intertwined search engine; and (3) system deployment as an info service. We spring our the thesaurus in 2 sub-boughs: the aquaculture ontology consists of 3455 concepts and 5396 terms, with 28 relationships, covering about 2200 fish species and their related terms; and the plant production ontology comprises of 3437 concepts and 6874 terms, with 5 relationships, covering farming, plant production, pests, etc. These ontologies serve as a global linkage between keywords, visual, and spoken features, as well as providing the reinforcement for the system performances (e.g., through query expansion, knowledge indexing…). On the other hand, constructing a visual-auditory intertwined search engine is a bit trickier. Automatic transcriptions of audio channels are marked as the anchor points for the collection of visual features. These features, in turn, got clustered based on the referenced thesauri, and ultimately tracking out missing info induced by the speech recognizer’s word error rates. This compensation technique bought us back 14 % of loss recall and an increase of 9 % accuracy over the baseline system. Finally, wrapping the retrieval system as an info service guarantees its practical deployment, asour target audiences are the majority of farmers in developing countries who are unable to reach modern farming information and knowledge.