This paper describes an effective unsupervised method for query-by-example speaker retrieval. We suppose that only one speaker is in each audio file or in audio segment. The audio data are modeled using a common universal codebook. The codebook is based on bag-of-frames (BOF). The features corresponding to the audio frames are extracted from all audio files. These features are grouped into clusters using the K-means algorithm. The individual audio files are modeled by the normalized distribution of the numbers of cluster bins corresponding to this file. In the first level the knearest to the query files are retrieved using vector space representation. In the second level the second-order statistical measure is applied to obtained k-nearest files to find the final result of the retrieval. The described method is evaluated on the subset of Ester corpus of French broadcast news.