With the rapid growth of the technology of multimedia and network, the amounts of multimedia data (such as image and video) are increasing greatly. How to effectively organize, store and retrieve the huge data is challenging us and making the multimedia retrieval become one of the hottest research fields. Content-based video retrieval (CBVR) as a branch of the multimedia retrieval has attracted increasing interests, for it directly relies on video content in contrast to traditional retrieval techniques using manual annotations. In this thesis, we focus on the following four parts: video feature extraction, retrieval mechanism, relevance feedback and information fusion to propose an original CBVR system named “An Interactive Video Retrieval Framework using Semantic Matching and Information Fusion”. The main contributions of this thesis are five-fold. (1) We propose a novel feature extraction method based on the model matching and semantic matching strategy. A new mid-level sequence feature named “Model- Matching Correlogram” is extracted to accurately describe video’s spatio-temporal information, while a new high-level semantic feature called “Semantic-Matching Histogram” is defined in order to uncover videos’ basic semantic content. (2) We establish an unsupervised learning-based retrieval mechanism, which consists of indexing process and querying process, using the Dominant Set clustering for the sake of low on-line complexity and high retrieval efficiency. (3) We develop a new relevance feedback algorithm called Semantic-Based Relevance Feedback working together with SMHs to correct the inaccurate semantic keywords labeled by SMHs and improves the retrieval performances remarkably. (4) We set up a video retrieval method based on the fusion of color and motion information. Optical flow analysis is used to reflect the local motion between adjacent frames and embedded into our semantic-matching strategy, while Dempster-Shafer theory is imported to fusion the SMHs on Color and SMHs on Motion. (5) We design and implement the interactive video retrieval prototype system using semantic matching and information fusion —— “SMIF VideoSearch system”.