o p i.@sdZddlZddlZddlZddlZddlZddlmZgZ dZ dZ ddZ dd Z ed d d d dddZed d d d dddZed d d d dddZed d d d dddZed d d d dddZdS)a IMDB dataset. This module downloads IMDB dataset from http://ai.stanford.edu/%7Eamaas/data/sentiment/. This dataset contains a set of 25,000 highly polar movie reviews for training, and 25,000 for testing. Besides, this module also provides API for building dictionary. N) deprecatedz6https://dataset.bj.bcebos.com/imdb%2FaclImdb_v1.tar.gzZ 7c2ac02c03563afcf9b574c7e56c153accsttjjtdt@}|}|durEt | |j r5| | ddtjdV|}|dusWddSWddS1sPwYdS)zQ Read files that match the given pattern. Tokenize and yield each file. imdbNs zlatin-1)tarfileopenpaddledatasetcommondownloadURLMD5nextboolmatchname extractfilereadrstrip translatestring punctuationencodelowersplit)patternZtarftfrZ/home/app/PaddleOCR-VL/.venv_paddleocr/lib/python3.10/site-packages/paddle/dataset/imdb.pytokenize&s" "rc stt}t|D]}|D] }||d7<q q fdd|D}t|ddd}tt|\}}ttt|t t |}t ||d<|S)z Build a word dictionary from the corpus. Keys of the dictionary are words, and values are zero-based IDs of these words. csg|] }|dkr|qS)rr).0xcutoffrr Jszbuild_dict..cSs|d |dfS)Nrrr)r rrrLszbuild_dict..)key) collections defaultdictintritemssortedlistzipdictrangelen) rr"Z word_freqdocword dictionarywords_word_idxrr!r build_dict?s   r7z2.0.0zpaddle.text.datasets.Imdbrz>Please use new dataset API which supports paddle.io.DataLoader)ZsinceZ update_tolevelreasoncsBdgfdd}||d||dfdd}|S)Nr&cs0t|D]}|fdd|D|fqdS)Ncsg|]}|qSr)get)rwUNKr6rrr#_sz0reader_creator..load..)rappend)routlabelr1r<rrload]s  zreader_creator..loadrrc3sEdHdS)Nrr)INSrrreaderdszreader_creator..readerr)Z pos_patternZ neg_patternr6rArCr)rBr=r6rreader_creatorSs   rDcCttdtd|S)a IMDB training set creator. It returns a reader creator, each sample in the reader is an zero-based ID sequence and label in [0, 1]. :param word_idx: word dictionary :type word_idx: dict :return: Training reader creator :rtype: callable zaclImdb/train/pos/.*\.txt$zaclImdb/train/neg/.*\.txt$rDrecompiler6rrrtrainj rJcCrE)a IMDB test set creator. It returns a reader creator, each sample in the reader is an zero-based ID sequence and label in [0, 1]. :param word_idx: word dictionary :type word_idx: dict :return: Test reader creator :rtype: callable zaclImdb/test/pos/.*\.txt$zaclImdb/test/neg/.*\.txt$rFrIrrrtestrKrLcCsttddS)za Build a word dictionary from the corpus. :return: Word dictionary :rtype: dict z/aclImdb/((train)|(test))/((pos)|(neg))/.*\.txt$)r7rGrHrrrr word_dicts rNcCstjjtdtdS)Nr)rrrr r r rrrrfetchsrO)__doc__r'rGrrZpaddle.dataset.commonrZ paddle.utilsr__all__r r rr7rDrJrLrNrOrrrrs^