---
phase: 01-poc
plan: "02"
subsystem: parser
tags:
  - parser
  - dedup
  - parsel
  - gne
  - trafilatura
  - simhash
dependency_graph:
  requires:
    - govcrawler.models (Plan 01-01)
  provides:
    - govcrawler.utils.url_norm.normalize_url
    - govcrawler.utils.url_norm.url_hash
    - govcrawler.utils.url_norm.sha256_bytes
    - govcrawler.utils.simhash_utils.content_simhash
    - govcrawler.parser.cleaner.html_to_text
    - govcrawler.parser.list_parser.parse_list
    - govcrawler.parser.list_parser.ListItem
    - govcrawler.parser.detail_parser.parse_detail
    - govcrawler.parser.detail_parser.DetailFields
    - govcrawler.parser.extractor.gne_extract
    - govcrawler.parser.extractor.trafilatura_extract
    - govcrawler.sites.gdqy.*
  affects: []
tech_stack:
  added:
    - GNE lazy-loaded via _get_gne() singleton to avoid slow pytest collection
    - trafilatura lazily imported inside function to avoid slow import
    - parsel CSS compound selector split workaround (join comma-separated sub-selectors)
  patterns:
    - 3-tier fallback: main XPath -> GNE -> trafilatura
    - dataclass for structured parser output (ListItem, DetailFields)
    - used_fallback + fallback_engine fields for observability
    - compound CSS selector split before ::attr(href) append
key_files:
  created:
    - govcrawler/utils/__init__.py
    - govcrawler/utils/url_norm.py
    - govcrawler/utils/simhash_utils.py
    - govcrawler/parser/__init__.py
    - govcrawler/parser/cleaner.py
    - govcrawler/parser/extractor.py
    - govcrawler/parser/list_parser.py
    - govcrawler/parser/detail_parser.py
    - govcrawler/sites/__init__.py
    - govcrawler/sites/gdqy.py
    - tests/__init__.py
    - tests/conftest.py
    - tests/fixtures/README.md
    - tests/test_url_norm.py
    - tests/test_simhash.py
    - tests/test_cleaner.py
    - tests/test_list_parser.py
    - tests/test_detail_parser.py
    - tests/test_gne_fallback.py
  modified:
    - pyproject.toml (added pytest to dev deps — was missing from venv)
    - uv.lock
decisions:
  - "pytest was listed as a dev dep in pyproject.toml but not installed in venv; added via uv add --dev pytest — resolved by pip resolving extras correctly"
  - "parsel cannot append ::attr(href) to a compound CSS selector (comma-separated list) — only applies to the last clause. Fixed by splitting on comma and querying each sub-selector individually in detail_parser.py"
  - "GNE lazy-loaded via module-level _GNE singleton and _get_gne() to avoid slow module-level import that would slow pytest collection"
  - "trafilatura lazily imported inside trafilatura_extract() for same reason"
  - "SZFWJ_SELECTORS alias added to gdqy.py as backwards compat for Plan 03 interface spec"
metrics:
  duration: "12 minutes 35 seconds"
  completed: "2026-04-22"
  tasks_completed: 2
  tasks_total: 2
  files_created: 19
---

# Phase 1 Plan 02: 纯函数层 Parser + Utils + sites/gdqy.py Summary

**One-liner:** parsel XPath + GNE/trafilatura 双兜底 detail_parser，64-bit 2-gram SimHash URL 归一化，gdqy.gov.cn 硬编码 selectors，32 个 pytest 全绿，全部纯函数无网络/DB 依赖。

## What Was Built

### Task 1: utils + cleaner 纯函数 (23 tests)

**`govcrawler/utils/url_norm.py`**
- `normalize_url(u)`: 强制 https、lower host、去 fragment、按 key 排序 query、去尾 `/`（根路径保留 `/`）。对非 `http/https` scheme 抛 `ValueError`（T-01-02-01 threat mitigated）。
- `url_hash(u)`: `sha256(normalize_url(u))` hex，64 字符
- `sha256_bytes(data)`: 供 Plan 03 附件 file_hash 使用

**`govcrawler/utils/simhash_utils.py`**
- `content_simhash(text)`: 64-bit SimHash via 2-gram（跨 CJK + ASCII 均适用），返回 16 个 hex 字符；空串返回 `"0000000000000000"`

**`govcrawler/parser/cleaner.py`**
- `html_to_text(html_str)`: 去除 script/style/noscript/iframe（T-01-02-02/T-01-02-04 threat mitigated），块级标签后插 `\n`，`text_content()` 拿纯文本，折叠空行

**Tests:** 10（url_norm）+ 4（simhash）+ 6（cleaner）+ 3（sha256_bytes & url_hash class） = **23 tests, all green**

### Task 2: list_parser + detail_parser + GNE/trafilatura + gdqy selectors (9 tests)

**`govcrawler/parser/extractor.py`**
- `gne_extract(html, base_url)`: GNE 主兜底；`GeneralNewsExtractor()` 懒加载（`_get_gne()` singleton）避免拖慢 pytest 收集
- `trafilatura_extract(html, base_url)`: trafilatura 次兜底；lazily `import trafilatura` 在函数内部

**`govcrawler/parser/list_parser.py`**
- `parse_list(html, base_url, selectors) -> list[ListItem]`: parsel CSS，selectors dict 含 row/href/title/date 四键

**`govcrawler/parser/detail_parser.py`**
- `parse_detail(html, base_url, selectors) -> DetailFields`: 三级路径：
  1. 主 XPath：若 `len(content_html) >= 100` 且 `title` 非空 → 直接走
  2. GNE 兜底：`gne_extract` 内容 >= 50 字 → `used_fallback=True, fallback_engine="gne"`
  3. trafilatura 兜底：GNE 也不够 50 字 → `used_fallback=True, fallback_engine="trafilatura"`
- `DetailFields.used_fallback` + `fallback_engine` 字段提供 Plan 03 可观测性

**`govcrawler/sites/gdqy.py`**
- 硬编码 `SITE_ID="gdqy"`, `COLUMN_ID="szfwj"`, `COLUMN_LIST_URL`, `TARGET_ARTICLE_URL`, `LIST_SELECTORS`, `DETAIL_SELECTORS`
- 多候选 CSS selector 应对政务站模板差异（`ul.news_list li, ul.list_news li, div.list li`）
- 附件 selector 覆盖后缀 + 含参数 URL（`a[href*='.pdf?']` 等，RESEARCH §Anti-Patterns 对齐）

**Tests:**
- `test_list_parser.py`: 3 cases（抽行、跳过无 anchor 行、空 HTML）
- `test_detail_parser.py`: 3 cases（主路径字段正确、附件绝对 URL、content_text 无 HTML 标签）
- `test_gne_fallback.py`: 3 cases（BROKEN_SELECTORS 触发 GNE/trafilatura，monkeypatch gne→空走 trafilatura，双空返回 used_fallback=False）

## Pytest Results

```
32 passed, 1 warning in 0.31s
```

| Test File | Cases | Result |
|-----------|-------|--------|
| test_url_norm.py | 13 | PASSED |
| test_simhash.py | 4 | PASSED |
| test_cleaner.py | 6 | PASSED |
| test_list_parser.py | 3 | PASSED |
| test_detail_parser.py | 3 | PASSED |
| test_gne_fallback.py | 3 | PASSED |
| **Total** | **32** | **ALL PASSED** |

Note: 32 > 29 (plan predicted 29). The extra 3 come from TestSha256Bytes having 1 test counted separately, plus TestUrlHash having 2 tests — plan counted "10 url_norm" but the actual breakdown is 10 TestNormalize + 2 TestUrlHash + 1 TestSha256Bytes = 13.

Warning: `PytestUnknownMarkWarning` for `@pytest.mark.network_free` — this is a documentation-only mark; harmless. Can be silenced by adding it to `[tool.pytest.ini_options] markers` in pyproject.toml in Plan 03 if desired.

## GNE Fallback Verification

`test_gne_fallback_triggered_when_xpath_fails` uses `BROKEN_SELECTORS` (all selectors point to nonexistent CSS classes). With `RICH_HTML` containing 150+ character Chinese government announcement text:

- GNE successfully extracted content (`fallback_engine="gne"`, `used_fallback=True`, `len(content_text) > 50`)
- `test_trafilatura_fallback_when_gne_fails`: monkeypatched `gne_extract` to return `{}` → trafilatura stepped in (`fallback_engine="trafilatura"`, `used_fallback=True`, `len(content_text) > 50`)
- `test_both_fallbacks_empty_returns_short_content`: both patched to `{}` → `used_fallback=False, fallback_engine=None` (neither succeeded, no corruption of output)

## gdqy Selectors Notes

Selectors in `govcrawler/sites/gdqy.py` are **synthesized** from known gdqy.gov.cn DOM conventions (per CONTEXT.md canonical refs — article at `post_2136593` with title class `article-title`, time `span.time`, content `div.article-content`). These selectors are **best-effort** for PoC:

- If actual gdqy DOM differs, Plan 03 smoke test will surface the mismatch and the executor should update `DETAIL_SELECTORS`/`LIST_SELECTORS` in `gdqy.py` at that point.
- GNE/trafilatura fallback ensures content is extracted even if selectors miss.

## Deviations from Plan

### Auto-fixed Issues

**1. [Rule 1 - Bug] parsel compound CSS selector + ::attr(href) appends only to last clause**

- **Found during:** Task 2 test_detail_parser::test_attachment_absolute
- **Issue:** `content_root.css(f"{attach_css}::attr(href)")` where `attach_css` is `"a[href$='.pdf'], a[href$='.doc'], ..."` — parsel/cssselect applies `::attr(href)` only to the final sub-selector after the last comma. The first selectors returned full element HTML instead of just the href attribute value.
- **Fix:** Split `attach_css` on commas in `detail_parser.py`, query each sub-selector with `::attr(href)` individually, deduplicate results.
- **Files modified:** `govcrawler/parser/detail_parser.py`
- **Commit:** bf6c741

**2. [Rule 3 - Environment] pytest missing from venv despite being listed as dev dep**

- **Found during:** Task 1 verification
- **Issue:** `uv run pytest` failed with "No such file or directory". Dev dependencies were not being installed by `uv sync --dev` into `.venv`.
- **Fix:** `uv add --dev pytest` to ensure proper resolution + installation. `uv run --with pytest pytest` also works as workaround.
- **Files modified:** `pyproject.toml`, `uv.lock`
- **Commit:** 5ec473a

## Known Stubs

None. All public functions return real computed values from inputs. Selectors in `gdqy.py` are best-effort approximations of the actual gdqy DOM and may need adjustment in Plan 03 smoke testing — but this is intentional PoC behavior, not a stub.

## Threat Flags

None beyond what the Plan 02 threat model already covers. No new network endpoints, auth paths, or file I/O introduced.

## Self-Check: PASSED

Files present:
- govcrawler/utils/url_norm.py ✓
- govcrawler/utils/simhash_utils.py ✓
- govcrawler/parser/cleaner.py ✓
- govcrawler/parser/extractor.py ✓
- govcrawler/parser/list_parser.py ✓
- govcrawler/parser/detail_parser.py ✓
- govcrawler/sites/gdqy.py ✓
- tests/test_url_norm.py ✓
- tests/test_simhash.py ✓
- tests/test_cleaner.py ✓
- tests/test_list_parser.py ✓
- tests/test_detail_parser.py ✓
- tests/test_gne_fallback.py ✓

Commits:
- 5ec473a: Task 1 ✓
- bf6c741: Task 2 ✓

pytest: 32 passed ✓
