o |q iF>@s|ddlZddlmZmZmZmZmZmZmZddl m Z ddl m Z ddl mZddlmZddlmZGdd d eZdS) N)Any AsyncIteratorIteratorListOptionalSetUnion)urlparse) BeautifulSoup)Document) BaseLoader) WebBaseLoaderc@seZdZdZ     d1ddddeded eed ed ed ed eedeeefddZdedefddZ d2de edededefddZ de ee efde fddZdedefddZdede efd d!Z d3ded"eed#ee de efd$d%Z d3ded ed"eed#ee de ef d&d'Zdeefd(d)Zdeefd*d+Z d3ded,eedeefd-d.Zdede efd/d0ZdS)4 GitbookLoadera Load `GitBook` data. 1. load from either a single page, or 2. load all (relative) paths in the sitemap, handling nested sitemap indexes. When `load_all_paths=True`, the loader parses XML sitemaps and requires the `lxml` package to be installed (`pip install lxml`). FNmainT) sitemap_urlallowed_domainsweb_pageload_all_pathsbase_urlcontent_selectorcontinue_on_failure show_progressrrc Cs|p||_|jdr|jdd|_||_||_||_||_||_||_|jdur5t|j } | r5| h|_|rA|p>|jd|_ n||_ | |j sVt d|j d|jdS)aTInitialize with web page and whether to load all paths. Args: web_page: The web page to load or the starting point from where relative paths are discovered. load_all_paths: If set to True, all relative paths in the navbar are loaded instead of only `web_page`. Requires `lxml` package. base_url: If `load_all_paths` is True, the relative paths are appended to this base url. Defaults to `web_page`. content_selector: The CSS selector for the content to load. Defaults to "main". continue_on_failure: whether to continue loading the sitemap if an error occurs loading a url, emitting a warning instead of raising an exception. Setting this to True makes the loader more robust, but also may result in missing data. Default: False show_progress: whether to show a progress bar while loading. Default: True sitemap_url: Custom sitemap URL to use when load_all_paths is True. Defaults to "{base_url}/sitemap.xml". allowed_domains: Optional set of allowed domains to fetch from. If None (default), the loader will restrict crawling to the domain of the `web_page` URL to prevent potential SSRF vulnerabilities. Provide an explicit set (e.g., {"example.com", "docs.example.com"}) to allow crawling across multiple domains. Use with caution in server environments where users might control the input URLs. /Nz /sitemap.xmlz Domain in z% is not in the allowed domains list: ) rendswithrrrrrrr netloc start_url_is_url_allowed ValueError) selfrrrrrrrrZinitial_domainr s/home/app/PaddleOCR-VL/.venv_paddleocr/lib/python3.10/site-packages/langchain_community/document_loaders/gitbook.py__init__s. %     zGitbookLoader.__init__urlreturncCsV|jdurdSzt|}|jdvrWdS|jsWdS|j|jvWSty*YdSw)z0Check if a URL has an allowed scheme and domain.NF)httphttps)rr schemer Exception)rr#parsedr r r!rYs   zGitbookLoader._is_url_allowedURLurl_listurl_typecCs2||r ||dStd|d|dS)aBSafely add a URL to a list if it's from an allowed domain. Args: url_list: The list to add the URL to url: The URL to add url_type: Type of URL for warning message (e.g., "sitemap", "content") Returns: bool: True if URL was added, False if skipped TzSkipping disallowed z URL: F)rappendwarningswarn)rr+r#r,r r r! _safe_add_urlps zGitbookLoader._safe_add_url url_or_urlscCst||j|jdS)zCreate a new WebBaseLoader instance for the given URL(s). This ensures each operation gets its own isolated WebBaseLoader. )Zweb_pathrr)r rr)rr1r r r!_create_web_loaders z GitbookLoader._create_web_loadersoupcCs|dduS)z+Check if the soup contains a sitemap index.Z sitemapindexN)find)rr3r r r!_is_sitemap_indexszGitbookLoader._is_sitemap_indexcCs@|d}g}|D]}|d}|r|jr|||jdq |S)z*Extract sitemap URLs from a sitemap index.sitemaploc)find_allr4textr0)rr3Z sitemap_tagsurlsr6r7r r r!_extract_sitemap_urlss   z#GitbookLoader._extract_sitemap_urlsprocessed_urls web_loaderc Cs|dur ||j}||ro||}g}|D]T}||vr'td|q||z|j}|g|_|jdd}||_| |||} | | Wqt yl} z|j ratd|d| nWYd} ~ qd} ~ ww|S| |S)aOProcess a sitemap, handling both direct content URLs and sitemap indexes. Args: soup: The BeautifulSoup object of the sitemap processed_urls: Set of already processed URLs to avoid cycles web_loader: WebBaseLoader instance to reuse for all requests, created if None Nz(Skipping already processed sitemap URL: lxml-xmlparserError processing sitemap : )r2rr5r;r.r/add web_pathsscrape_process_sitemapextendr(r _get_paths) rr3r<r= sitemap_urlsall_content_urlsroriginal_web_paths sitemap_soup content_urlser r r!rFs<       zGitbookLoader._process_sitemapc s|dur ||j}||r|||}g}fdd|D}|s$gS|j}||_|j|ddIdH} ||_t|| D]=\} } | z|| ||IdH} | | Wq<t yy} z|j rnt d| d| nWYd} ~ qsz3GitbookLoader._aprocess_sitemap..r>r?rArB)r2rr5r;rD ascrape_allziprC_aprocess_sitemaprGr(rr.r/rH)rr3rr<r=rIrJZnew_urlsrKsoupsrrLrMrNr rPr!rTs:       zGitbookLoader._aprocess_sitemapc cs|js||j}|}|||j}|r|VdSdS||j}|jdd}t}|||}|sA|jrAt d|jg}|D] }| ||dqE|sSdS||} | |} t | |D]\}}|||}|rq|VqbdS)zDFetch text from one single GitBook page or recursively from sitemap.r>r?$No content URLs found in sitemap at contentN)rr2rrE _get_documentrsetrFrr.r/r0Z scrape_allrS) r temp_loaderr3doc soup_infor<relative_pathsr:r#content_loader soup_infosr r r! lazy_loads6         zGitbookLoader.lazy_loadc Cs$|js(||j}||jgIdH}|d}|||j}|r&|VdSdS||j}|j|jgddIdH}|d}t}|||j|IdH}|sZ|j rZt d|jg}|D] }| ||dq^|sldS||} | |IdH} t | |D]\}}|||} | dur| Vq~dS)z/Asynchronously fetch text from GitBook page(s).Nrr>r?rVrW)rr2rrRrXrrYrTrrr.r/r0rS) rrZrUr\r[r<r]r:r#r^r_Z maybe_docr r r! alazy_load?s>      zGitbookLoader.alazy_load custom_urlcCsX||j}|s dS|jdd}|d}|r|jnd}|p"|j|d}t||dS)z,Fetch content from page and return Document.N ) separatorZh1)sourcetitle)Z page_contentmetadata)r4rZget_textstripr9rr )rr3rbZpage_content_rawrWZtitle_if_existsrgrhr r r!rXms   zGitbookLoader._get_documentcCs*g}|dD] }|jr||jq|S)zFetch all URLs in the sitemap.r7)r8r9r-)rr3r:r7r r r!rHzs  zGitbookLoader._get_paths)FNrFT)r*)N)__name__ __module__ __qualname____doc__strboolrrr"rrr0rr r2r r5r;rFrTrr r`rrarrXrHr r r r!r s    C   @ =*/  r)r.typingrrrrrrr urllib.parser Zbs4r Zlangchain_core.documentsr Z)langchain_community.document_loaders.baser Z-langchain_community.document_loaders.web_baser rr r r r!s$