a ´iskã@sdZdZddlZddlZddlZddlZddlZddlZddlZddl m Z ddl m Z m Z mZddlmZddlmZmZdd lmZd!d d „ZGd d„deƒZdd„Zdd„Zdd„Zdd„Zdd„ZGdd„deƒZGdd„deƒZ Gdd„deƒZ!Gdd „d e ƒZ"dS)"zÛRefactoring framework. Used as a main program, this can refactor any number of files and/or recursively descend down directories. Imported as a module, this provides infrastructure to write your own refactoring tool. z#Guido van Rossum éN)Úchainé)ÚdriverÚtokenizeÚtoken)Ú find_root)ÚpytreeÚpygram)Ú btm_matcherTcCsTt|ggdgƒ}g}t |j¡D].\}}}| d¡r |rD|dd…}| |¡q |S)zEReturn a sorted list of all available fix names in the given package.Ú*Úfix_éN)Ú __import__ÚpkgutilÚ iter_modulesÚ__path__Ú startswithÚappend)Z fixer_pkgZ remove_prefixÚpkgZ fix_namesÚfinderÚnameÚispkg©rú(/usr/lib64/python3.9/lib2to3/refactor.pyÚget_all_fix_namess   rc@s eZdZdS)Ú _EveryNodeN©Ú__name__Ú __module__Ú __qualname__rrrrr+srcCs’t|tjtjfƒr(|jdur t‚|jhSt|tjƒrH|jrDt|jƒSt‚t|tj ƒr‚t ƒ}|jD]}|D]}|  t|ƒ¡qhq`|St d|ƒ‚dS)zf Accepts a pytree Pattern Node and returns a set of the pattern types which will match first. Nz$Oh no! I don't understand pattern %s) Ú isinstancerZ NodePatternZ LeafPatternÚtyperZNegatedPatternZcontentÚ_get_head_typesZWildcardPatternÚsetÚupdateÚ Exception)ÚpatÚrÚpÚxrrrr"/s     r"c Cs¼t t¡}g}|D]v}|jrbzt|jƒ}WntyF| |¡Yqˆ0|D]}|| |¡qLq|jdur~||j |¡q| |¡qtt j j   ¡t j j ƒD]}|| |¡q t|ƒS)z^ Accepts a list of fixers and returns a dictionary of head node type --> fixer list. N)Ú collectionsÚ defaultdictÚlistÚpatternr"rrZ _accept_typerr Úpython_grammarZ symbol2numberÚvaluesÚtokensÚextendÚdict)Z fixer_listZ head_nodesZeveryÚfixerZheadsZ node_typerrrÚ_get_headnode_dictKs$     ÿr4cs‡fdd„tˆdƒDƒS)zN Return the fully qualified names for fixers in the package pkg_name. csg|]}ˆd|‘qS©Ú.r)Ú.0Úfix_name©Úpkg_namerrÚ hsÿz+get_fixers_from_package..F)rr9rr9rÚget_fixers_from_packageds ÿr<cCs|S©Nr)ÚobjrrrÚ _identityksr?csVd}t t |¡j¡‰‡fdd„}ttjtjtj hƒ}t ƒ}zü|ƒ\}}||vrTq>q>|tj krl|rfq6d}q>|tj kr6|dkr6|ƒ\}}|tj ksž|dkr¢q6|ƒ\}}|tj ks¾|dkrÂq6|ƒ\}}|tj krè|dkrè|ƒ\}}|tj kr4| |¡|ƒ\}}|tj ks.|d kr"q4|ƒ\}}qèq>q6q>WntyLYn0t|ƒS) NFcstˆƒ}|d|dfS)Nrr)Únext)Útok©ÚgenrrÚadvancersz(_detect_future_features..advanceTÚfromZ __future__Úimportú(ú,)rÚgenerate_tokensÚioÚStringIOÚreadlineÚ frozensetrÚNEWLINEÚNLÚCOMMENTr#ÚSTRINGÚNAMEÚOPÚaddÚ StopIteration)ÚsourceZhave_docstringrDÚignoreÚfeaturesÚtpÚvaluerrBrÚ_detect_future_featuresosB           r[c@seZdZdZdS)Ú FixerErrorzA fixer could not be loaded.N)rrrÚ__doc__rrrrr\—sr\c@sæeZdZddddœZdZdZd4dd„Zdd „Zd d „Zd d „Z dd„Z dd„Z d5dd„Z d6dd„Z dd„Zd7dd„Zdd„Zd8dd„Zdd„Zd d!„Zd9d"d#„Zd:d$d%„Zd&Zd'Zd(d)„Zd*d+„Zd,d-„Zd.d/„Zd0d1„Zd2d3„ZdS);ÚRefactoringToolF)Úprint_functionÚ exec_functionÚwrite_unchanged_filesZFixr NcCsJ||_|p g|_|j ¡|_|dur0|j |¡tj ¡|_|jdrR|jj d=n|jdrf|jj d=|j  d¡|_ g|_ t  d¡|_g|_d|_tj|jtj|jd |_| ¡\|_|_g|_t ¡|_g|_g|_t|j|jƒD]H}|j rü|j !|¡qä||jvr|j "|¡qä||jvrä|j "|¡qät#|jƒ|_$t#|jƒ|_%dS) zÑInitializer. Args: fixer_names: a list of fixers to import options: a dict with configuration. explicit: a list of fixers to run even if they are explicit. Nr_Úprintr`Úexecrar^F)ÚconvertÚlogger)&ÚfixersÚexplicitÚ_default_optionsÚcopyÚoptionsr$r r.ÚgrammarÚkeywordsÚgetraÚerrorsÚloggingÚ getLoggerreÚ fixer_logÚwroterZDriverrrdÚ get_fixersÚ pre_orderÚ post_orderÚfilesÚbmZ BottomMatcherÚBMZ bmi_pre_orderZbmi_post_orderrZ BM_compatibleZ add_fixerrr4Úbmi_pre_order_headsÚbmi_post_order_heads)ÚselfZ fixer_namesrjrgr3rrrÚ__init__¤sB         þ    zRefactoringTool.__init__c CsVg}g}|jD]}t|iidgƒ}| dd¡d}| |j¡rR|t|jƒd…}| d¡}|jd dd „|Dƒ¡}zt ||ƒ}Wn$t y¨t d ||fƒd‚Yn0||j |j ƒ} | jrà|jd urà||jvrà| d |¡q| d |¡| jdkr| | ¡q| jdkr| | ¡qt d| jƒ‚qt d¡} |j| d|j| d||fS)aInspects the options to load the requested patterns and handlers. Returns: (pre_order, post_order), where pre_order is the list of fixers that want a pre-order AST traversal, and post_order is the list that want post-order traversal. r r6réÿÿÿÿNÚ_ÚcSsg|] }| ¡‘qSr)Útitle)r7r(rrrr;ëóz.RefactoringTool.get_fixers..zCan't find %s.%sTzSkipping optional fixer: %szAdding transformation: %sÚpreZpostzIllegal fixer order: %rZ run_order©Úkey)rfrÚrsplitrÚ FILE_PREFIXÚlenÚsplitÚ CLASS_PREFIXÚjoinÚgetattrÚAttributeErrorr\rjrqrgÚ log_messageÚ log_debugÚorderrÚoperatorÚ attrgetterÚsort) r{Zpre_order_fixersZpost_order_fixersZ fix_mod_pathÚmodr8ÚpartsÚ class_nameZ fix_classr3Zkey_funcrrrrsÛs:    ÿ         zRefactoringTool.get_fixerscOs‚dS)zCalled when an error occurs.Nr)r{ÚmsgÚargsÚkwdsrrrÚ log_errorszRefactoringTool.log_errorcGs|r ||}|j |¡dS)zHook to log a message.N)reÚinfo©r{r–r—rrrrszRefactoringTool.log_messagecGs|r ||}|j |¡dSr=)reÚdebugr›rrrrŽ szRefactoringTool.log_debugcCsdS)zTCalled with the old version, new version, and filename of a refactored file.Nr)r{Úold_textÚnew_textÚfilenameÚequalrrrÚ print_outputszRefactoringTool.print_outputcCs8|D].}tj |¡r$| |||¡q| |||¡qdS)z)Refactor a list of files and directories.N)ÚosÚpathÚisdirÚ refactor_dirÚ refactor_file)r{ÚitemsÚwriteÚ doctests_onlyZ dir_or_filerrrÚrefactors zRefactoringTool.refactorc Csštjd}t |¡D]€\}}}| d|¡| ¡| ¡|D]>}| d¡s>tj |¡d|kr>tj ||¡} |  | ||¡q>dd„|Dƒ|dd…<qdS)zÄDescends down a directory and refactor every Python file found. Python files are assumed to have a .py extension. Files and subdirectories starting with '.' are skipped. ÚpyzDescending into %sr6rcSsg|]}| d¡s|‘qSr5)r)r7Zdnrrrr;2rz0RefactoringTool.refactor_dir..N) r¢ÚextsepÚwalkrŽr’rr£ÚsplitextrŠr¦) r{Zdir_namer¨r©Zpy_extÚdirpathÚdirnamesÚ filenamesrÚfullnamerrrr¥ s   ÿzRefactoringTool.refactor_dirc Cs®zt|dƒ}Wn4tyB}z| d||¡WYd}~dSd}~00zt |j¡d}W| ¡n | ¡0tj|d|dd}| ¡|fWdƒS1s 0YdS) zG Do our best to decode a Python source file correctly. ÚrbzCan't open %s: %sN)NNrr'r©ÚencodingÚnewline) ÚopenÚOSErrorr™rÚdetect_encodingrLÚcloserJÚread)r{rŸÚfÚerrrµrrrÚ_read_python_source4sz#RefactoringTool._read_python_sourcecCsº| |¡\}}|durdS|d7}|rn| d|¡| ||¡}|jsL||kr`| |||||¡q¶| d|¡nH| ||¡}|jsŠ|rª|jrª|jt|ƒdd…|||dn | d|¡dS)zRefactors a file.NÚ zRefactoring doctests in %szNo doctest changes in %sr})r¨rµzNo changes in %s)r¾rŽÚrefactor_docstringraÚprocessed_fileÚrefactor_stringÚ was_changedÚstr)r{rŸr¨r©ÚinputrµÚoutputÚtreerrrr¦Ds    ÿzRefactoringTool.refactor_filec Cs°t|ƒ}d|vrtj|j_zfz|j |¡}WnFtyr}z.| d||jj |¡WYd}~W|j|j_dSd}~00W|j|j_n |j|j_0||_ |  d|¡|  ||¡|S)aFRefactor a given input string. Args: data: a string holding the code to be refactored. name: a human-readable name for use in error/log messages. Returns: An AST corresponding to the refactored input stream; None if there were errors during the parse. r_zCan't parse %s: %s: %sNzRefactoring %s) r[r Z!python_grammar_no_print_statementrrkZ parse_stringr%r™Ú __class__rÚfuture_featuresrŽÚ refactor_tree)r{ÚdatarrXrÇr½rrrrÂ[s"   ÿ  þ  zRefactoringTool.refactor_stringcCsŒtj ¡}|rN| d¡| |d¡}|js2||krB| |d|¡qˆ| d¡n:| |d¡}|jsj|r~|jr~| t |ƒd|¡n | d¡dS)NzRefactoring doctests in stdinzzNo doctest changes in stdinzNo changes in stdin) ÚsysÚstdinr»rŽrÀrarÁrÂrÃrÄ)r{r©rÅrÆrÇrrrÚrefactor_stdinvs     zRefactoringTool.refactor_stdinc CsÚt|j|jƒD]}| ||¡q| |j| ¡¡| |j| ¡¡|j |  ¡¡}t |  ¡ƒr²|jj D]B}||vrj||rj||j tjjdd|jr°||j tjjdt||ƒD]ð}|||vrÚ|| |¡z t|ƒWntyþYq¼Yn0|jr||jvrq¼| |¡}|r¼| ||¡}|dur¼| |¡| ¡D] }|js\g|_|j |¡qJ|j |  ¡¡}|D]*} | |vr–g|| <||  || ¡q€q¼qjqTt|j|jƒD]}| ||¡qÀ|jS)aÏRefactors a parse tree (modifying the tree in place). For compatible patterns the bottom matcher module is used. Otherwise the tree is traversed node-to-node for matches. Args: tree: a pytree.Node instance representing the root of the tree to be refactored. name: a human-readable name for this tree. Returns: True if the tree was modified, False otherwise. T)r„ÚreverserƒN)rrtruZ start_treeÚ traverse_byryrzrxÚrunZleavesÚanyr/rfr’rZBaseÚdepthZkeep_line_orderZ get_linenor,ÚremoverÚ ValueErrorZfixers_appliedÚmatchÚ transformÚreplacerr1Z finish_treerÃ) r{rÇrr3Z match_setÚnodeÚresultsÚnewZ new_matchesZfxrrrrrʆsJ        zRefactoringTool.refactor_treecCsV|sdS|D]D}||jD]4}| |¡}|r| ||¡}|dur| |¡|}qq dS)aTraverse an AST, applying a set of fixers to each node. This is a helper method for refactor_tree(). Args: fixers: a list of fixer instances. traversal: a generator that yields AST nodes. Returns: None N)r!rÖr×rØ)r{rfZ traversalrÙr3rÚrÛrrrrÐÕs    zRefactoringTool.traverse_bycCs†|j |¡|dur.| |¡d}|dur.dS||k}| ||||¡|r`| d|¡|js`dS|rv| ||||¡n | d|¡dS)zR Called when a file has been refactored and there may be changes. NrzNo changes to %szNot writing changes to %s)rvrr¾r¡rŽraÚ write_file)r{ržrŸrr¨rµr rrrrÁìs  zRefactoringTool.processed_filec CsÈztj|d|dd}Wn4tyJ}z| d||¡WYd}~dSd}~00|Rz| |¡Wn2ty’}z| d||¡WYd}~n d}~00Wdƒn1s¨0Y| d|¡d|_dS) zÑWrites a string to a file. It first shows a unified diff between the old text and the new text, and then rewrites the file; the latter is only done if the write option is set. Úwrr´zCan't create %s: %sNzCan't write %s: %szWrote changes to %sT)rJr·r¸r™r¨rŽrr)r{ržrŸrrµÚfpr½rrrrÜsB zRefactoringTool.write_filez>>> z... c Csg}d}d}d}d}|jddD]È}|d7}| ¡ |j¡r~|durZ| | ||||¡¡|}|g}| |j¡} |d| …}q |dur¸| ||j¡s¬|||j ¡dkr¸|  |¡q |durÖ| | ||||¡¡d}d}|  |¡q |dur | | ||||¡¡d  |¡S)aËRefactors a docstring, looking for doctests. This returns a modified version of the input string. It looks for doctests, which start with a ">>>" prompt, and may be continued with "..." prompts, as long as the "..." is indented the same as the ">>>". (Unfortunately we can't use the doctest module's parser, since, like most parsers, it is not geared towards preserving the original source.) NrT©Úkeependsrr¿r) Ú splitlinesÚlstriprÚPS1r1Úrefactor_doctestÚfindÚPS2ÚrstriprrŠ) r{rÅrŸÚresultÚblockZ block_linenoÚindentÚlinenoÚlineÚirrrrÀsD  ÿ ÿþ  ÿ   ÿz"RefactoringTool.refactor_docstringc s.zˆ ||ˆ¡}Wnhtyz}zPˆj tj¡rL|D]}ˆ d| d¡¡q4ˆ d|||j j |¡|WYd}~Sd}~00ˆ  ||¡r*t |ƒj dd}|d|d…||dd…} }| dg|dksÖJ| ƒ‚|d d¡sö|dd7<ˆˆj| d ¡g}|r*|‡‡fd d „|Dƒ7}|S) zÞRefactors one doctest. A doctest is given as a block of lines, the first of which starts with ">>>" (possibly indented), while the remaining lines start with "..." (identically indented). z Source: %sr¿z+Can't parse docstring in %s line %s: %s: %sNTrßrr}rcsg|]}ˆˆj|‘qSr)ræ)r7rì©rêr{rrr;^rz4RefactoringTool.refactor_doctest..)Ú parse_blockr%reÚ isEnabledForroÚDEBUGrŽrçr™rÈrrÊrÄráÚendswithrãÚpop) r{rérërêrŸrÇr½rìrÛZclippedrrîrräDs( ÿ"z RefactoringTool.refactor_doctestcCsÌ|jr d}nd}|js$| d|¡n"| d|¡|jD]}| |¡q6|jrl| d¡|jD]}| |¡q\|jrÈt|jƒdkrŒ| d¡n| dt|jƒ¡|jD]"\}}}|j|g|¢Ri|¤Žq¤dS) NÚwerez need to bezNo files %s modified.zFiles that %s modified:z$Warnings/messages while refactoring:rzThere was 1 error:zThere were %d errors:)rrrvrrqrnr‡)r{rôÚfileÚmessager–r—r˜rrrÚ summarizeas$       zRefactoringTool.summarizecCs"|j | |||¡¡}tƒ|_|S)z³Parses a block into a tree. This is necessary to get correct line number / offset information in the parser diagnostics and embedded into the parse tree. )rZ parse_tokensÚ wrap_toksrMrÉ)r{rérërêrÇrrrrïxszRefactoringTool.parse_blockc csdt | ||¡j¡}|D]F\}}\}}\} } } ||d7}| |d7} ||||f| | f| fVqdS)z;Wraps a tokenize stream to systematically modify start/end.rN)rrIÚ gen_linesÚ__next__) r{rérërêr0r!rZZline0Zcol0Zline1Zcol1Z line_textrrrrø‚s   zRefactoringTool.wrap_toksccsx||j}||j}|}|D]N}| |¡r>|t|ƒd…Vn(|| ¡dkrVdVntd||fƒ‚|}qdVqldS)z–Generates lines as expected by tokenize from a list of lines. This strips the first len(indent + self.PS1) characters off each line. Nr¿zline=%r, prefix=%rr)rãrærr‡rçÚAssertionError)r{rérêÚprefix1Zprefix2Úprefixrìrrrrùs   zRefactoringTool.gen_lines)NN)FF)FF)FF)F)NFN)N)rrrrhr‰r†r|rsr™rrŽr¡rªr¥r¾r¦rÂrÎrÊrÐrÁrÜrãrærÀrär÷rïrørùrrrrr^›s@þ 7(   Oÿ  + r^c@s eZdZdS)ÚMultiprocessingUnsupportedNrrrrrrþ¤srþcsBeZdZ‡fdd„Zd ‡fdd„ Z‡fdd„Z‡fd d „Z‡ZS) ÚMultiprocessRefactoringToolcs&tt|ƒj|i|¤Žd|_d|_dSr=)Úsuperrÿr|ÚqueueÚ output_lock©r{r—Úkwargs©rÈrrr|ªsz$MultiprocessRefactoringTool.__init__Frc s>|dkrttˆƒ |||¡Sz ddl‰Wnty>t‚Yn0ˆjdurRtdƒ‚ˆ ¡ˆ_ˆ  ¡ˆ_ ‡‡fdd„t |ƒDƒ}zn|D] }|  ¡q„ttˆƒ |||¡Wˆj  ¡t |ƒD]}ˆj d¡qº|D]}| ¡rÐ|  ¡qÐdˆ_nLˆj  ¡t |ƒD]}ˆj d¡q|D]}| ¡r|  ¡qdˆ_0dS)Nrrz already doing multiple processescsg|]}ˆjˆjd‘qS))Útarget)ÚProcessÚ_child)r7rí©Úmultiprocessingr{rrr;¼sÿz8MultiprocessRefactoringTool.refactor..)rrÿrªr Ú ImportErrorrþrÚ RuntimeErrorÚ JoinableQueueÚLockrÚrangeÚstartrŠÚputÚis_alive)r{r§r¨r©Z num_processesÚ processesr(rírr rrª¯sF ÿ       ÿ ÿ   ú    z$MultiprocessRefactoringTool.refactorcs\|j ¡}|durX|\}}z$tt|ƒj|i|¤ŽW|j ¡n |j ¡0|j ¡}q dSr=)rrmrrÿr¦Ú task_done)r{Ztaskr—rrrrrÌs  ÿÿz"MultiprocessRefactoringTool._childcs6|jdur|j ||f¡ntt|ƒj|i|¤ŽSdSr=)rrrrÿr¦rrrrr¦×s  ÿÿz)MultiprocessRefactoringTool.refactor_file)FFr)rrrr|rªrr¦Ú __classcell__rrrrrÿ¨s  ÿ rÿ)T)#r]Ú __author__rJr¢rrÌrorr*Ú itertoolsrZpgen2rrrZ fixer_utilrrrr r rwrr%rr"r4r<r?r[r\Úobjectr^rþrÿrrrrÚs8    (